== replay-dual test 26: dbench and tar with mds failover ========================================================== 07:28:22 (1740486502) Starting client oleg429-client.virtnet: -o user_xattr,flock 192.168.204.129@tcp:/lustre /mnt/lustre Started clients oleg429-client.virtnet: 192.168.204.129@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project) Started tar loop with pid 14947 Started dbench loop with 14948 looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Tue Feb 25 07:28:23 EST 2025 waiting for dbench pid 14978 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs failed to create barrier semaphore 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 1 298 10.96 MB/sec warmup 1 sec latency 23.301 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3456 2205184 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3688448 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 29696 3641344 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 32768 7329792 1% /mnt/lustre 1 500 8.73 MB/sec warmup 2 sec latency 29.535 ms 1 798 7.16 MB/sec warmup 3 sec latency 34.282 ms 1 1100 5.74 MB/sec warmup 4 sec latency 36.706 ms test_26 fail mds1 1 times Failing mds1 on oleg429-server Stopping /mnt/lustre-mds1 (opts:) on oleg429-server 1 1497 5.21 MB/sec warmup 5 sec latency 34.498 ms 07:28:29 (1740486509) shut down facet: mds1 facet_host: oleg429-server facet_failover_host: oleg429-server 1 1523 4.35 MB/sec warmup 6 sec latency 920.787 ms 1 1523 3.73 MB/sec warmup 7 sec latency 1921.006 ms 1 1523 3.26 MB/sec warmup 8 sec latency 2921.202 ms 1 1523 2.90 MB/sec warmup 9 sec latency 3921.365 ms 1 1523 2.61 MB/sec warmup 10 sec latency 4921.528 ms 1 1523 2.37 MB/sec warmup 11 sec latency 5921.718 ms 1 1523 2.17 MB/sec warmup 12 sec latency 6921.886 ms 1 1523 2.01 MB/sec warmup 13 sec latency 7922.050 ms 1 1523 1.86 MB/sec warmup 14 sec latency 8922.225 ms 1 1523 1.74 MB/sec warmup 15 sec latency 9922.325 ms Failover mds1 to oleg429-server mount facets: mds1 1 1523 1.63 MB/sec warmup 16 sec latency 10922.414 ms Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 1523 1.53 MB/sec warmup 17 sec latency 11922.486 ms oleg429-server: oleg429-server.virtnet: executing set_default_debug -1 all pdsh@oleg429-client: oleg429-server: ssh exited with exit code 1 Started lustre-MDT0000 07:28:41 (1740486521) targets are mounted 07:28:41 (1740486521) facet_failover done 1 1523 1.45 MB/sec warmup 18 sec latency 12922.623 ms 1 1523 1.37 MB/sec warmup 19 sec latency 13922.743 ms 1 1523 0.00 MB/sec execute 1 sec latency 15922.949 ms 1 1523 0.00 MB/sec execute 2 sec latency 16923.148 ms 1 1523 0.00 MB/sec execute 3 sec latency 17923.294 ms 1 1523 0.00 MB/sec execute 4 sec latency 18923.461 ms 1 1597 0.01 MB/sec execute 5 sec latency 19701.816 ms oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 1954 0.06 MB/sec execute 6 sec latency 37.979 ms 1 2306 0.24 MB/sec execute 7 sec latency 30.756 ms 1 2717 0.73 MB/sec execute 8 sec latency 35.366 ms 1 3341 1.03 MB/sec execute 9 sec latency 29.065 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 4992 2203520 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 13312 3587072 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 43008 3512320 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 56320 7099392 1% /mnt/lustre 1 3797 1.37 MB/sec execute 10 sec latency 18.259 ms 1 4059 1.27 MB/sec execute 11 sec latency 12.758 ms 1 4351 1.20 MB/sec execute 12 sec latency 51.342 ms test_26 fail mds1 2 times Failing mds1 on oleg429-server Stopping /mnt/lustre-mds1 (opts:) on oleg429-server 1 4560 1.15 MB/sec execute 13 sec latency 176.808 ms 07:28:57 (1740486537) shut down facet: mds1 facet_host: oleg429-server facet_failover_host: oleg429-server 1 4560 1.07 MB/sec execute 14 sec latency 1176.978 ms 1 4560 1.00 MB/sec execute 15 sec latency 2177.124 ms 1 4560 0.93 MB/sec execute 16 sec latency 3177.228 ms 1 4560 0.88 MB/sec execute 17 sec latency 4177.407 ms 1 4560 0.83 MB/sec execute 18 sec latency 5177.541 ms 1 4560 0.79 MB/sec execute 19 sec latency 6177.667 ms 1 4560 0.75 MB/sec execute 20 sec latency 7177.812 ms 1 4560 0.71 MB/sec execute 21 sec latency 8178.015 ms 1 4560 0.68 MB/sec execute 22 sec latency 9178.140 ms 1 4560 0.65 MB/sec execute 23 sec latency 10178.244 ms Failover mds1 to oleg429-server mount facets: mds1 1 4560 0.62 MB/sec execute 24 sec latency 11178.367 ms Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 4560 0.60 MB/sec execute 25 sec latency 12178.487 ms oleg429-server: oleg429-server.virtnet: executing set_default_debug -1 all pdsh@oleg429-client: oleg429-server: ssh exited with exit code 1 Started lustre-MDT0000 07:29:09 (1740486549) targets are mounted 07:29:09 (1740486549) facet_failover done 1 4560 0.57 MB/sec execute 26 sec latency 13178.637 ms 1 4560 0.55 MB/sec execute 27 sec latency 14178.791 ms 1 4560 0.53 MB/sec execute 28 sec latency 15178.948 ms 1 4560 0.51 MB/sec execute 29 sec latency 16179.066 ms 1 4631 0.53 MB/sec execute 30 sec latency 16997.189 ms oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 5025 0.61 MB/sec execute 31 sec latency 15.343 ms 1 5294 0.60 MB/sec execute 32 sec latency 45.036 ms 1 5624 0.60 MB/sec execute 33 sec latency 26.832 ms 1 5988 0.62 MB/sec execute 34 sec latency 37.068 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 6912 2201600 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 17408 3437568 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 46080 3416064 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 63488 6853632 1% /mnt/lustre 1 6525 0.76 MB/sec execute 35 sec latency 41.851 ms 1 7122 0.87 MB/sec execute 36 sec latency 24.513 ms 1 7349 0.89 MB/sec execute 37 sec latency 24.088 ms test_26 fail mds1 3 times Failing mds1 on oleg429-server Stopping /mnt/lustre-mds1 (opts:) on oleg429-server 1 7573 0.87 MB/sec execute 38 sec latency 20.869 ms 07:29:22 (1740486562) shut down facet: mds1 facet_host: oleg429-server facet_failover_host: oleg429-server 1 7585 0.85 MB/sec execute 39 sec latency 964.782 ms 1 7585 0.83 MB/sec execute 40 sec latency 1964.942 ms 1 7585 0.81 MB/sec execute 41 sec latency 2965.172 ms 1 7585 0.79 MB/sec execute 42 sec latency 3965.435 ms 1 7585 0.77 MB/sec execute 43 sec latency 4965.654 ms 1 7585 0.75 MB/sec execute 44 sec latency 5965.923 ms 1 7585 0.74 MB/sec execute 45 sec latency 6966.239 ms 1 7585 0.72 MB/sec execute 46 sec latency 7966.497 ms 1 7585 0.70 MB/sec execute 47 sec latency 8966.806 ms 1 7585 0.69 MB/sec execute 48 sec latency 9967.141 ms 1 7585 0.68 MB/sec execute 49 sec latency 10967.399 ms Failover mds1 to oleg429-server mount facets: mds1 1 7585 0.66 MB/sec execute 50 sec latency 11967.673 ms Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 7585 0.65 MB/sec execute 51 sec latency 12967.859 ms 1 7585 0.64 MB/sec execute 52 sec latency 13968.077 ms oleg429-server: oleg429-server.virtnet: executing set_default_debug -1 all pdsh@oleg429-client: oleg429-server: ssh exited with exit code 1 Started lustre-MDT0000 07:29:36 (1740486576) targets are mounted 07:29:36 (1740486576) facet_failover done 1 7585 0.62 MB/sec execute 53 sec latency 14968.282 ms 1 7585 0.61 MB/sec execute 54 sec latency 15968.501 ms 1 7585 0.60 MB/sec execute 55 sec latency 16968.679 ms 1 7585 0.59 MB/sec execute 56 sec latency 17968.813 ms oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 7754 0.58 MB/sec execute 57 sec latency 17996.053 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 7911 0.58 MB/sec execute 58 sec latency 58.468 ms 1 8051 0.57 MB/sec execute 59 sec latency 60.133 ms 1 8264 0.59 MB/sec execute 60 sec latency 47.406 ms 1 8504 0.63 MB/sec execute 61 sec latency 29.451 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 8064 2200448 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 19456 3497984 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 48128 3490816 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 67584 6988800 1% /mnt/lustre 1 8662 0.62 MB/sec execute 62 sec latency 70.588 ms 1 8844 0.61 MB/sec execute 63 sec latency 37.233 ms 1 8997 0.60 MB/sec execute 64 sec latency 65.742 ms test_26 fail mds1 4 times Failing mds1 on oleg429-server 1 9244 0.60 MB/sec execute 65 sec latency 22.328 ms Stopping /mnt/lustre-mds1 (opts:) on oleg429-server 1 9308 0.59 MB/sec execute 66 sec latency 541.540 ms 07:29:50 (1740486590) shut down facet: mds1 facet_host: oleg429-server facet_failover_host: oleg429-server 1 9308 0.58 MB/sec execute 67 sec latency 1541.739 ms 1 9308 0.57 MB/sec execute 68 sec latency 2541.990 ms 1 9308 0.56 MB/sec execute 69 sec latency 3542.297 ms 1 9308 0.56 MB/sec execute 70 sec latency 4542.619 ms 1 9308 0.55 MB/sec execute 71 sec latency 5542.953 ms 1 9308 0.54 MB/sec execute 72 sec latency 6543.257 ms 1 9308 0.53 MB/sec execute 73 sec latency 7543.454 ms 1 9308 0.53 MB/sec execute 74 sec latency 8543.756 ms 1 9308 0.52 MB/sec execute 75 sec latency 9544.028 ms 1 9308 0.51 MB/sec execute 76 sec latency 10544.230 ms Failover mds1 to oleg429-server mount facets: mds1 1 9308 0.51 MB/sec execute 77 sec latency 11544.633 ms Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 9308 0.50 MB/sec execute 78 sec latency 12546.203 ms 1 9308 0.49 MB/sec execute 79 sec latency 13546.465 ms oleg429-server: oleg429-server.virtnet: executing set_default_debug -1 all pdsh@oleg429-client: oleg429-server: ssh exited with exit code 1 1 9308 0.49 MB/sec execute 80 sec latency 14546.697 ms Started lustre-MDT0000 07:30:03 (1740486603) targets are mounted 07:30:03 (1740486603) facet_failover done 1 9308 0.48 MB/sec execute 81 sec latency 15546.908 ms 1 9308 0.47 MB/sec execute 82 sec latency 16547.101 ms 1 9308 0.47 MB/sec execute 83 sec latency 17547.448 ms 1 9308 0.46 MB/sec execute 84 sec latency 18547.608 ms 1 9308 0.46 MB/sec execute 85 sec latency 19547.947 ms 1 9458 0.47 MB/sec execute 86 sec latency 19919.058 ms oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 9665 0.47 MB/sec execute 87 sec latency 47.911 ms 1 10001 0.51 MB/sec execute 88 sec latency 59.718 ms 1 10289 0.52 MB/sec execute 89 sec latency 54.611 ms 1 10618 0.56 MB/sec execute 90 sec latency 60.435 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 9344 2199168 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 21504 3557376 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 49152 3273728 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 70656 6831104 2% /mnt/lustre 1 10786 0.56 MB/sec execute 91 sec latency 25.073 ms 1 10958 0.56 MB/sec execute 92 sec latency 29.290 ms 1 11090 0.56 MB/sec execute 93 sec latency 25.831 ms test_26 fail mds1 5 times 1 11255 0.56 MB/sec execute 94 sec latency 22.212 ms Failing mds1 on oleg429-server Stopping /mnt/lustre-mds1 (opts:) on oleg429-server 1 11391 0.55 MB/sec execute 95 sec latency 396.095 ms 07:30:19 (1740486619) shut down facet: mds1 facet_host: oleg429-server facet_failover_host: oleg429-server 1 11391 0.55 MB/sec execute 96 sec latency 1396.359 ms 1 11391 0.54 MB/sec execute 97 sec latency 2396.695 ms 1 11391 0.54 MB/sec execute 98 sec latency 3397.036 ms 1 11391 0.53 MB/sec execute 99 sec latency 4397.372 ms 1 cleanup 100 sec 1 cleanup 101 sec 1 cleanup 102 sec 1 cleanup 103 sec 1 cleanup 104 sec 1 cleanup 105 sec Failover mds1 to oleg429-server mount facets: mds1 1 cleanup 106 sec Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 cleanup 107 sec oleg429-server: oleg429-server.virtnet: executing set_default_debug -1 all 1 cleanup 108 sec pdsh@oleg429-client: oleg429-server: ssh exited with exit code 1 Started lustre-MDT0000 07:30:32 (1740486632) targets are mounted 07:30:32 (1740486632) facet_failover done 1 cleanup 109 sec 1 cleanup 110 sec 1 cleanup 111 sec 1 cleanup 112 sec 1 cleanup 113 sec 1 cleanup 114 sec 1 cleanup 115 sec oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 0 cleanup 116 sec Operation Count AvgLat MaxLat ---------------------------------------- NTCreateX 1709 20.757 17996.036 Close 1241 31.658 19919.015 Rename 69 16.437 31.184 Unlink 354 5.420 22.488 Qpathinfo 1585 15.439 19701.798 Qfileinfo 271 0.487 4.430 Qfsinfo 282 0.544 3.540 Sfileinfo 121 180.309 20657.145 Find 601 1.456 19.309 WriteX 829 2.093 10.030 ReadX 2680 0.112 29.587 LockX 6 1.835 2.765 UnlockX 6 1.880 3.244 Flush 115 24.364 70.561 Throughput 0.530946 MB/sec 1 clients 1 procs max_latency=19919.058 ms stopping dbench on /mnt/lustre at Tue Feb 25 07:30:39 EST 2025 with return code 0 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Tue Feb 25 07:30:41 EST 2025 waiting for dbench pid 20102 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 1 186 7.45 MB/sec warmup 1 sec latency 35.446 ms 1 331 6.00 MB/sec warmup 2 sec latency 31.009 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 10368 2198144 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 40960 3484672 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 54272 3505152 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 95232 6989824 2% /mnt/lustre 1 501 5.82 MB/sec warmup 3 sec latency 44.124 ms 1 744 5.33 MB/sec warmup 4 sec latency 66.892 ms 1 913 4.35 MB/sec warmup 5 sec latency 41.704 ms test_26 fail mds1 6 times 1 1148 3.83 MB/sec warmup 6 sec latency 42.013 ms Failing mds1 on oleg429-server Stopping /mnt/lustre-mds1 (opts:) on oleg429-server 1 1393 3.72 MB/sec warmup 7 sec latency 382.436 ms 07:30:48 (1740486648) shut down facet: mds1 facet_host: oleg429-server facet_failover_host: oleg429-server 1 1393 3.25 MB/sec warmup 8 sec latency 1382.671 ms 1 1393 2.89 MB/sec warmup 9 sec latency 2382.891 ms 1 1393 2.60 MB/sec warmup 10 sec latency 3383.119 ms 1 1393 2.36 MB/sec warmup 11 sec latency 4383.374 ms 1 1393 2.17 MB/sec warmup 12 sec latency 5383.592 ms 1 1393 2.00 MB/sec warmup 13 sec latency 6383.860 ms 1 1393 1.86 MB/sec warmup 14 sec latency 7384.074 ms 1 1393 1.73 MB/sec warmup 15 sec latency 8384.293 ms 1 1393 1.63 MB/sec warmup 16 sec latency 9384.510 ms 1 1393 1.53 MB/sec warmup 17 sec latency 10384.724 ms Failover mds1 to oleg429-server mount facets: mds1 1 1393 1.44 MB/sec warmup 18 sec latency 11385.013 ms Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 1393 1.37 MB/sec warmup 19 sec latency 12385.239 ms oleg429-server: oleg429-server.virtnet: executing set_default_debug -1 all pdsh@oleg429-client: oleg429-server: ssh exited with exit code 1 1 1393 0.00 MB/sec execute 1 sec latency 14385.632 ms Started lustre-MDT0000 07:31:02 (1740486662) targets are mounted 07:31:02 (1740486662) facet_failover done 1 1393 0.00 MB/sec execute 2 sec latency 15385.823 ms 1 1393 0.00 MB/sec execute 3 sec latency 16386.096 ms 1 1393 0.00 MB/sec execute 4 sec latency 17386.438 ms 1 1393 0.00 MB/sec execute 5 sec latency 18386.688 ms 1 1393 0.00 MB/sec execute 6 sec latency 19386.896 ms 1 1393 0.00 MB/sec execute 7 sec latency 20387.056 ms oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 1538 0.01 MB/sec execute 8 sec latency 20836.065 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 1857 0.04 MB/sec execute 9 sec latency 24.815 ms 1 2198 0.07 MB/sec execute 10 sec latency 34.865 ms 1 2412 0.18 MB/sec execute 11 sec latency 79.921 ms 1 2583 0.25 MB/sec execute 12 sec latency 58.861 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 11520 2196992 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 53248 3483648 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 20480 3522560 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 73728 7006208 2% /mnt/lustre 1 2993 0.58 MB/sec execute 13 sec latency 49.274 ms 1 3359 0.67 MB/sec execute 14 sec latency 63.443 ms 1 3693 0.85 MB/sec execute 15 sec latency 21.397 ms test_26 fail mds1 7 times Failing mds1 on oleg429-server 1 3884 0.86 MB/sec execute 16 sec latency 23.976 ms Stopping /mnt/lustre-mds1 (opts:) on oleg429-server 1 3948 0.83 MB/sec execute 17 sec latency 695.737 ms 07:31:18 (1740486678) shut down facet: mds1 facet_host: oleg429-server facet_failover_host: oleg429-server 1 3948 0.78 MB/sec execute 18 sec latency 1695.973 ms 1 3948 0.74 MB/sec execute 19 sec latency 2696.273 ms 1 3948 0.70 MB/sec execute 20 sec latency 3696.557 ms 1 3948 0.67 MB/sec execute 21 sec latency 4696.835 ms 1 3948 0.64 MB/sec execute 22 sec latency 5697.159 ms 1 3948 0.61 MB/sec execute 23 sec latency 6697.380 ms 1 3948 0.59 MB/sec execute 24 sec latency 7697.597 ms 1 3948 0.56 MB/sec execute 25 sec latency 8697.836 ms 1 3948 0.54 MB/sec execute 26 sec latency 9698.112 ms 1 3948 0.52 MB/sec execute 27 sec latency 10698.360 ms Failover mds1 to oleg429-server mount facets: mds1 1 3948 0.50 MB/sec execute 28 sec latency 11698.564 ms Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 3948 0.48 MB/sec execute 29 sec latency 12698.842 ms 1 3948 0.47 MB/sec execute 30 sec latency 13699.114 ms oleg429-server: oleg429-server.virtnet: executing set_default_debug -1 all pdsh@oleg429-client: oleg429-server: ssh exited with exit code 1 Started lustre-MDT0000 07:31:32 (1740486692) targets are mounted 07:31:32 (1740486692) facet_failover done 1 3948 0.45 MB/sec execute 31 sec latency 14699.427 ms 1 3948 0.44 MB/sec execute 32 sec latency 15699.684 ms 1 3948 0.43 MB/sec execute 33 sec latency 16699.917 ms 1 3948 0.41 MB/sec execute 34 sec latency 17700.144 ms 1 3948 0.40 MB/sec execute 35 sec latency 18700.435 ms 1 3948 0.39 MB/sec execute 36 sec latency 19700.720 ms 1 3948 0.38 MB/sec execute 37 sec latency 20700.863 ms oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 4105 0.37 MB/sec execute 38 sec latency 21046.232 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 4281 0.37 MB/sec execute 39 sec latency 122.174 ms 1 4393 0.36 MB/sec execute 40 sec latency 72.916 ms 1 4528 0.36 MB/sec execute 41 sec latency 67.862 ms 1 4701 0.38 MB/sec execute 42 sec latency 52.885 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 9728 2198784 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 53248 3667968 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 27648 3699712 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 80896 7367680 2% /mnt/lustre 1 4862 0.39 MB/sec execute 43 sec latency 27.799 ms 1 5082 0.44 MB/sec execute 44 sec latency 73.102 ms 1 5225 0.43 MB/sec execute 45 sec latency 79.085 ms test_26 fail mds1 8 times Failing mds1 on oleg429-server 1 5372 0.42 MB/sec execute 46 sec latency 29.328 ms Stopping /mnt/lustre-mds1 (opts:) on oleg429-server 1 5467 0.42 MB/sec execute 47 sec latency 384.433 ms 07:31:48 (1740486708) shut down facet: mds1 facet_host: oleg429-server facet_failover_host: oleg429-server 1 5467 0.41 MB/sec execute 48 sec latency 1384.649 ms 1 5467 0.40 MB/sec execute 49 sec latency 2384.851 ms 1 5467 0.39 MB/sec execute 50 sec latency 3385.072 ms 1 5467 0.38 MB/sec execute 51 sec latency 4385.269 ms 1 5467 0.38 MB/sec execute 52 sec latency 5385.478 ms 1 5467 0.37 MB/sec execute 53 sec latency 6385.672 ms 1 5467 0.36 MB/sec execute 54 sec latency 7385.873 ms 1 5467 0.36 MB/sec execute 55 sec latency 8386.125 ms 1 5467 0.35 MB/sec execute 56 sec latency 9386.441 ms 1 5467 0.34 MB/sec execute 57 sec latency 10386.638 ms Failover mds1 to oleg429-server mount facets: mds1 1 5467 0.34 MB/sec execute 58 sec latency 11386.841 ms Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 5467 0.33 MB/sec execute 59 sec latency 12387.076 ms 1 5467 0.33 MB/sec execute 60 sec latency 13387.351 ms oleg429-server: oleg429-server.virtnet: executing set_default_debug -1 all 1 5467 0.32 MB/sec execute 61 sec latency 14387.584 ms pdsh@oleg429-client: oleg429-server: ssh exited with exit code 1 Started lustre-MDT0000 07:32:02 (1740486722) targets are mounted 07:32:02 (1740486722) facet_failover done 1 5467 0.32 MB/sec execute 62 sec latency 15387.803 ms 1 5467 0.31 MB/sec execute 63 sec latency 16388.008 ms 1 5467 0.31 MB/sec execute 64 sec latency 17388.233 ms 1 5467 0.30 MB/sec execute 65 sec latency 18388.501 ms 1 5467 0.30 MB/sec execute 66 sec latency 19388.765 ms 1 5467 0.29 MB/sec execute 67 sec latency 20388.932 ms 1 5547 0.29 MB/sec execute 68 sec latency 21075.569 ms oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 5740 0.29 MB/sec execute 69 sec latency 26.551 ms 1 5943 0.30 MB/sec execute 70 sec latency 105.573 ms 1 6064 0.31 MB/sec execute 71 sec latency 76.654 ms 1 6436 0.35 MB/sec execute 72 sec latency 70.058 ms 1 6662 0.37 MB/sec execute 73 sec latency 68.547 ms 1 6934 0.39 MB/sec execute 74 sec latency 59.896 ms 1 7214 0.42 MB/sec execute 75 sec latency 22.298 ms 1 7464 0.44 MB/sec execute 76 sec latency 21.244 ms 1 7668 0.43 MB/sec execute 77 sec latency 23.791 ms 1 7823 0.43 MB/sec execute 78 sec latency 90.196 ms 1 7943 0.43 MB/sec execute 79 sec latency 74.280 ms 1 8078 0.43 MB/sec execute 80 sec latency 74.212 ms 1 8245 0.43 MB/sec execute 81 sec latency 57.355 ms 1 8427 0.44 MB/sec execute 82 sec latency 29.323 ms 1 8614 0.46 MB/sec execute 83 sec latency 101.105 ms 1 8804 0.46 MB/sec execute 84 sec latency 39.215 ms 1 8940 0.45 MB/sec execute 85 sec latency 24.856 ms 1 9125 0.45 MB/sec execute 86 sec latency 70.664 ms 1 9307 0.45 MB/sec execute 87 sec latency 22.629 ms 1 9511 0.46 MB/sec execute 88 sec latency 91.074 ms 1 9679 0.46 MB/sec execute 89 sec latency 62.912 ms 1 10017 0.51 MB/sec execute 90 sec latency 81.668 ms 1 10297 0.51 MB/sec execute 91 sec latency 71.422 ms 1 10621 0.55 MB/sec execute 92 sec latency 29.919 ms dbench killed by signal 15 stopping dbench on /mnt/lustre at Tue Feb 25 07:32:34 EST 2025 with return code 0 20102 pts/0 S+ 0:00 dbench -c client.txt 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100 killed dbench main pid 20102 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished