== replay-dual test 26: dbench and tar with mds failover ========================================================== 05:51:44 (1715334704) Starting client oleg147-client.virtnet: -o user_xattr,flock oleg147-server@tcp:/lustre /mnt/lustre Started clients oleg147-client.virtnet: 192.168.201.147@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project) Started tar loop with pid 17399 Started dbench loop with 17400 looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Fri May 10 05:51:45 EDT 2024 waiting for dbench pid 17430 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs failed to create barrier semaphore 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 1 263 9.92 MB/sec warmup 1 sec latency 44.965 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3712 2204928 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3695616 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 29696 3652608 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 32768 7348224 1% /mnt/lustre 1 501 8.73 MB/sec warmup 2 sec latency 23.998 ms 1 801 7.16 MB/sec warmup 3 sec latency 49.579 ms 1 1127 5.75 MB/sec warmup 4 sec latency 38.172 ms test_26 fail mds1 1 times Failing mds1 on oleg147-server Stopping /mnt/lustre-mds1 (opts:) on oleg147-server 1 1485 5.21 MB/sec warmup 5 sec latency 86.641 ms 05:51:51 (1715334711) shut down 1 1485 4.34 MB/sec warmup 6 sec latency 1086.878 ms 1 1485 3.72 MB/sec warmup 7 sec latency 2087.064 ms 1 1485 3.26 MB/sec warmup 8 sec latency 3087.253 ms 1 1485 2.89 MB/sec warmup 9 sec latency 4087.440 ms 1 1485 2.60 MB/sec warmup 10 sec latency 5087.704 ms 1 1485 2.37 MB/sec warmup 11 sec latency 6087.937 ms 1 1485 2.17 MB/sec warmup 12 sec latency 7088.099 ms 1 1485 2.00 MB/sec warmup 13 sec latency 8088.261 ms 1 1485 1.86 MB/sec warmup 14 sec latency 9088.451 ms 1 1485 1.74 MB/sec warmup 15 sec latency 10088.630 ms Failover mds1 to oleg147-server mount facets: mds1 1 1485 1.63 MB/sec warmup 16 sec latency 11088.868 ms Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 1485 1.53 MB/sec warmup 17 sec latency 12089.045 ms oleg147-server: oleg147-server.virtnet: executing set_default_debug -1 all pdsh@oleg147-client: oleg147-server: ssh exited with exit code 1 1 1485 1.45 MB/sec warmup 18 sec latency 13089.223 ms Started lustre-MDT0000 05:52:04 (1715334724) targets are mounted 05:52:04 (1715334724) facet_failover done 1 1485 1.37 MB/sec warmup 19 sec latency 14089.415 ms 1 1485 0.00 MB/sec execute 1 sec latency 16089.752 ms 1 1485 0.00 MB/sec execute 2 sec latency 17089.940 ms 1 1485 0.00 MB/sec execute 3 sec latency 18090.113 ms 1 1485 0.00 MB/sec execute 4 sec latency 19090.295 ms 1 1485 0.00 MB/sec execute 5 sec latency 20090.426 ms 1 1496 0.00 MB/sec execute 6 sec latency 21037.487 ms oleg147-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 1736 0.03 MB/sec execute 7 sec latency 55.291 ms 1 2031 0.07 MB/sec execute 8 sec latency 39.345 ms 1 2327 0.20 MB/sec execute 9 sec latency 47.519 ms 1 2584 0.29 MB/sec execute 10 sec latency 47.464 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 4864 2203648 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 16384 3639296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 37888 3529728 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 54272 7169024 1% /mnt/lustre 1 3019 0.68 MB/sec execute 11 sec latency 44.754 ms 1 3551 1.03 MB/sec execute 12 sec latency 41.179 ms 1 3820 1.06 MB/sec execute 13 sec latency 17.089 ms test_26 fail mds1 2 times 1 4006 1.00 MB/sec execute 14 sec latency 19.993 ms Failing mds1 on oleg147-server Stopping /mnt/lustre-mds1 (opts:) on oleg147-server 1 4123 0.94 MB/sec execute 15 sec latency 442.194 ms 05:52:21 (1715334741) shut down 1 4123 0.88 MB/sec execute 16 sec latency 1442.373 ms 1 4123 0.83 MB/sec execute 17 sec latency 2442.531 ms 1 4123 0.78 MB/sec execute 18 sec latency 3442.695 ms 1 4123 0.74 MB/sec execute 19 sec latency 4442.887 ms 1 4123 0.70 MB/sec execute 20 sec latency 5443.052 ms 1 4123 0.67 MB/sec execute 21 sec latency 6443.222 ms 1 4123 0.64 MB/sec execute 22 sec latency 7443.437 ms 1 4123 0.61 MB/sec execute 23 sec latency 8443.610 ms 1 4123 0.59 MB/sec execute 24 sec latency 9443.751 ms 1 4123 0.56 MB/sec execute 25 sec latency 10443.931 ms Failover mds1 to oleg147-server mount facets: mds1 1 4123 0.54 MB/sec execute 26 sec latency 11444.117 ms Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 4123 0.52 MB/sec execute 27 sec latency 12444.342 ms oleg147-server: oleg147-server.virtnet: executing set_default_debug -1 all pdsh@oleg147-client: oleg147-server: ssh exited with exit code 1 Started lustre-MDT0000 05:52:33 (1715334753) targets are mounted 05:52:33 (1715334753) facet_failover done 1 4123 0.50 MB/sec execute 28 sec latency 13444.536 ms 1 4123 0.48 MB/sec execute 29 sec latency 14444.750 ms 1 4123 0.47 MB/sec execute 30 sec latency 15444.980 ms 1 4123 0.45 MB/sec execute 31 sec latency 16445.178 ms 1 4123 0.44 MB/sec execute 32 sec latency 17445.341 ms 1 4123 0.43 MB/sec execute 33 sec latency 18445.646 ms 1 4123 0.41 MB/sec execute 34 sec latency 19446.237 ms 1 4123 0.40 MB/sec execute 35 sec latency 20447.316 ms oleg147-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 4230 0.39 MB/sec execute 36 sec latency 21018.641 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 4452 0.40 MB/sec execute 37 sec latency 54.114 ms 1 4780 0.42 MB/sec execute 38 sec latency 42.036 ms 1 5173 0.49 MB/sec execute 39 sec latency 47.803 ms 1 5492 0.49 MB/sec execute 40 sec latency 41.223 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 6528 2202112 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 18432 3493888 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 43008 3454976 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 61440 6948864 1% /mnt/lustre 1 5918 0.51 MB/sec execute 41 sec latency 32.732 ms 1 6449 0.62 MB/sec execute 42 sec latency 33.447 ms test_26 fail mds1 3 times 1 7060 0.73 MB/sec execute 43 sec latency 31.095 ms Failing mds1 on oleg147-server Stopping /mnt/lustre-mds1 (opts:) on oleg147-server 1 7305 0.75 MB/sec execute 44 sec latency 344.712 ms 05:52:50 (1715334770) shut down 1 7305 0.73 MB/sec execute 45 sec latency 1344.884 ms 1 7305 0.71 MB/sec execute 46 sec latency 2345.067 ms 1 7305 0.70 MB/sec execute 47 sec latency 3345.244 ms 1 7305 0.68 MB/sec execute 48 sec latency 4345.403 ms 1 7305 0.67 MB/sec execute 49 sec latency 5345.564 ms 1 7305 0.66 MB/sec execute 50 sec latency 6345.762 ms 1 7305 0.64 MB/sec execute 51 sec latency 7345.927 ms 1 7305 0.63 MB/sec execute 52 sec latency 8346.108 ms 1 7305 0.62 MB/sec execute 53 sec latency 9346.267 ms 1 7305 0.61 MB/sec execute 54 sec latency 10346.437 ms Failover mds1 to oleg147-server mount facets: mds1 1 7305 0.60 MB/sec execute 55 sec latency 11346.575 ms Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 7305 0.59 MB/sec execute 56 sec latency 12346.804 ms oleg147-server: oleg147-server.virtnet: executing set_default_debug -1 all pdsh@oleg147-client: oleg147-server: ssh exited with exit code 1 Started lustre-MDT0000 05:53:02 (1715334782) targets are mounted 05:53:02 (1715334782) facet_failover done 1 7305 0.58 MB/sec execute 57 sec latency 13346.993 ms 1 7305 0.57 MB/sec execute 58 sec latency 14347.168 ms 1 7305 0.56 MB/sec execute 59 sec latency 15347.324 ms 1 7305 0.55 MB/sec execute 60 sec latency 16347.441 ms 1 7305 0.54 MB/sec execute 61 sec latency 17347.578 ms oleg147-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 7521 0.53 MB/sec execute 62 sec latency 17443.555 ms 1 7778 0.53 MB/sec execute 63 sec latency 50.947 ms 1 8025 0.53 MB/sec execute 64 sec latency 83.782 ms 1 8368 0.54 MB/sec execute 65 sec latency 30.346 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 8320 2200320 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 20480 3357696 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 46080 3469312 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 66560 6827008 1% /mnt/lustre 1 8740 0.58 MB/sec execute 66 sec latency 46.135 ms 1 9098 0.58 MB/sec execute 67 sec latency 44.217 ms 1 9510 0.59 MB/sec execute 68 sec latency 45.531 ms test_26 fail mds1 4 times Failing mds1 on oleg147-server Stopping /mnt/lustre-mds1 (opts:) on oleg147-server 1 10031 0.66 MB/sec execute 69 sec latency 40.168 ms 05:53:15 (1715334795) shut down 1 10081 0.65 MB/sec execute 70 sec latency 925.069 ms 1 10081 0.64 MB/sec execute 71 sec latency 1925.274 ms 1 10081 0.63 MB/sec execute 72 sec latency 2925.445 ms 1 10081 0.63 MB/sec execute 73 sec latency 3925.627 ms 1 10081 0.62 MB/sec execute 74 sec latency 4925.827 ms 1 10081 0.61 MB/sec execute 75 sec latency 5926.018 ms 1 10081 0.60 MB/sec execute 76 sec latency 6926.247 ms 1 10081 0.59 MB/sec execute 77 sec latency 7926.428 ms 1 10081 0.59 MB/sec execute 78 sec latency 8926.633 ms 1 10081 0.58 MB/sec execute 79 sec latency 9926.821 ms Failover mds1 to oleg147-server mount facets: mds1 1 10081 0.57 MB/sec execute 80 sec latency 10927.004 ms Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 10081 0.56 MB/sec execute 81 sec latency 11927.374 ms oleg147-server: oleg147-server.virtnet: executing set_default_debug -1 all 1 10081 0.56 MB/sec execute 82 sec latency 12927.533 ms pdsh@oleg147-client: oleg147-server: ssh exited with exit code 1 Started lustre-MDT0000 05:53:28 (1715334808) targets are mounted 05:53:28 (1715334808) facet_failover done 1 10081 0.55 MB/sec execute 83 sec latency 13927.703 ms 1 10081 0.54 MB/sec execute 84 sec latency 14927.922 ms 1 10081 0.54 MB/sec execute 85 sec latency 15928.121 ms 1 10081 0.53 MB/sec execute 86 sec latency 16928.330 ms 1 10081 0.53 MB/sec execute 87 sec latency 17928.551 ms 1 10081 0.52 MB/sec execute 88 sec latency 18928.737 ms 1 10081 0.51 MB/sec execute 89 sec latency 19928.943 ms 1 10081 0.51 MB/sec execute 90 sec latency 20929.092 ms 1 10081 0.50 MB/sec execute 91 sec latency 21929.226 ms oleg147-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 10451 0.52 MB/sec execute 92 sec latency 22085.613 ms 1 10806 0.55 MB/sec execute 93 sec latency 14.492 ms 1 11034 0.56 MB/sec execute 94 sec latency 17.698 ms 1 11228 0.55 MB/sec execute 95 sec latency 29.139 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 9856 2198784 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 20480 3344384 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 47104 3304448 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 67584 6648832 2% /mnt/lustre 1 11425 0.55 MB/sec execute 96 sec latency 115.588 ms 1 11638 0.55 MB/sec execute 97 sec latency 39.031 ms 1 11924 0.56 MB/sec execute 98 sec latency 32.655 ms test_26 fail mds1 5 times 1 12260 0.58 MB/sec execute 99 sec latency 34.011 ms Failing mds1 on oleg147-server Stopping /mnt/lustre-mds1 (opts:) on oleg147-server 1 cleanup 100 sec 05:53:46 (1715334826) shut down 1 cleanup 101 sec 1 cleanup 102 sec 1 cleanup 103 sec 1 cleanup 104 sec 1 cleanup 105 sec 1 cleanup 106 sec 1 cleanup 107 sec 1 cleanup 108 sec 1 cleanup 109 sec 1 cleanup 110 sec Failover mds1 to oleg147-server mount facets: mds1 1 cleanup 111 sec Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 cleanup 112 sec oleg147-server: oleg147-server.virtnet: executing set_default_debug -1 all pdsh@oleg147-client: oleg147-server: ssh exited with exit code 1 1 cleanup 113 sec Started lustre-MDT0000 05:53:59 (1715334839) targets are mounted 05:53:59 (1715334839) facet_failover done 1 cleanup 114 sec 1 cleanup 115 sec 1 cleanup 116 sec 1 cleanup 117 sec 1 cleanup 118 sec 1 cleanup 119 sec 1 cleanup 120 sec oleg147-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 cleanup 121 sec 0 cleanup 121 sec Operation Count AvgLat MaxLat ---------------------------------------- NTCreateX 1917 19.362 20832.655 Close 1413 13.946 17443.530 Rename 79 11.151 16.984 Unlink 377 60.387 21018.622 Qpathinfo 1747 2.549 38.200 Qfileinfo 303 0.441 2.674 Qfsinfo 306 0.479 2.548 Sfileinfo 150 5.820 11.475 Find 659 33.226 21037.468 WriteX 932 1.821 6.759 ReadX 2932 0.134 38.733 LockX 6 1.367 1.692 UnlockX 6 1.441 1.989 Flush 131 194.572 22085.601 Throughput 0.580407 MB/sec 1 clients 1 procs max_latency=22085.613 ms stopping dbench on /mnt/lustre at Fri May 10 05:54:07 EDT 2024 with return code 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Fri May 10 05:54:08 EDT 2024 waiting for dbench pid 22504 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 1 289 10.70 MB/sec warmup 1 sec latency 18.872 ms 1 542 9.55 MB/sec warmup 2 sec latency 19.628 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 11008 2197632 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 24576 3430400 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 50176 3350528 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 74752 6780928 2% /mnt/lustre 1 786 7.15 MB/sec warmup 3 sec latency 87.671 ms 1 1006 5.53 MB/sec warmup 4 sec latency 46.028 ms 1 1421 5.20 MB/sec warmup 5 sec latency 14.275 ms test_26 fail mds1 6 times Failing mds1 on oleg147-server Stopping /mnt/lustre-mds1 (opts:) on oleg147-server 1 1711 4.37 MB/sec warmup 6 sec latency 66.637 ms 05:54:15 (1715334855) shut down 1 1804 3.76 MB/sec warmup 7 sec latency 788.758 ms 1 1804 3.29 MB/sec warmup 8 sec latency 1788.929 ms 1 1804 2.93 MB/sec warmup 9 sec latency 2789.108 ms 1 1804 2.63 MB/sec warmup 10 sec latency 3789.288 ms 1 1804 2.39 MB/sec warmup 11 sec latency 4789.438 ms 1 1804 2.19 MB/sec warmup 12 sec latency 5789.611 ms 1 1804 2.03 MB/sec warmup 13 sec latency 6789.826 ms 1 1804 1.88 MB/sec warmup 14 sec latency 7789.991 ms 1 1804 1.76 MB/sec warmup 15 sec latency 8790.174 ms 1 1804 1.65 MB/sec warmup 16 sec latency 9790.347 ms Failover mds1 to oleg147-server mount facets: mds1 1 1804 1.55 MB/sec warmup 17 sec latency 10790.473 ms Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 1804 1.46 MB/sec warmup 18 sec latency 11790.668 ms oleg147-server: oleg147-server.virtnet: executing set_default_debug -1 all 1 1804 1.39 MB/sec warmup 19 sec latency 12790.835 ms pdsh@oleg147-client: oleg147-server: ssh exited with exit code 1 Started lustre-MDT0000 05:54:28 (1715334868) targets are mounted 05:54:28 (1715334868) facet_failover done 1 1804 0.00 MB/sec execute 1 sec latency 14791.180 ms 1 1804 0.00 MB/sec execute 2 sec latency 15791.309 ms 1 1804 0.00 MB/sec execute 3 sec latency 16791.456 ms oleg147-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 2069 0.07 MB/sec execute 4 sec latency 16882.275 ms 1 2376 0.33 MB/sec execute 5 sec latency 97.764 ms 1 2759 0.94 MB/sec execute 6 sec latency 46.378 ms 1 3209 1.14 MB/sec execute 7 sec latency 61.010 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 10624 2198016 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 29696 3621888 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 45056 3566592 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 74752 7188480 2% /mnt/lustre 1 3664 1.55 MB/sec execute 8 sec latency 30.655 ms 1 3915 1.53 MB/sec execute 9 sec latency 29.046 ms 1 4162 1.38 MB/sec execute 10 sec latency 15.234 ms test_26 fail mds1 7 times Failing mds1 on oleg147-server 1 4362 1.30 MB/sec execute 11 sec latency 70.425 ms Stopping /mnt/lustre-mds1 (opts:) on oleg147-server 1 4473 1.21 MB/sec execute 12 sec latency 514.557 ms 05:54:41 (1715334881) shut down 1 4473 1.11 MB/sec execute 13 sec latency 1514.743 ms 1 4473 1.03 MB/sec execute 14 sec latency 2514.954 ms 1 4473 0.96 MB/sec execute 15 sec latency 3515.129 ms 1 4473 0.90 MB/sec execute 16 sec latency 4515.343 ms 1 4473 0.85 MB/sec execute 17 sec latency 5515.539 ms 1 4473 0.80 MB/sec execute 18 sec latency 6515.734 ms 1 4473 0.76 MB/sec execute 19 sec latency 7515.952 ms 1 4473 0.72 MB/sec execute 20 sec latency 8516.131 ms 1 4473 0.69 MB/sec execute 21 sec latency 9516.326 ms 1 4473 0.66 MB/sec execute 22 sec latency 10516.512 ms Failover mds1 to oleg147-server mount facets: mds1 1 4473 0.63 MB/sec execute 23 sec latency 11516.773 ms Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 4473 0.60 MB/sec execute 24 sec latency 12516.968 ms oleg147-server: oleg147-server.virtnet: executing set_default_debug -1 all pdsh@oleg147-client: oleg147-server: ssh exited with exit code 1 Started lustre-MDT0000 05:54:53 (1715334893) targets are mounted 05:54:53 (1715334893) facet_failover done 1 4473 0.58 MB/sec execute 25 sec latency 13517.167 ms 1 4473 0.56 MB/sec execute 26 sec latency 14517.380 ms 1 4473 0.54 MB/sec execute 27 sec latency 15517.576 ms 1 4473 0.52 MB/sec execute 28 sec latency 16517.774 ms 1 4473 0.50 MB/sec execute 29 sec latency 17517.933 ms 1 4473 0.48 MB/sec execute 30 sec latency 18518.101 ms 1 4473 0.47 MB/sec execute 31 sec latency 19518.230 ms 1 4473 0.45 MB/sec execute 32 sec latency 20518.398 ms 1 4473 0.44 MB/sec execute 33 sec latency 21518.543 ms oleg147-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 4739 0.46 MB/sec execute 34 sec latency 21548.159 ms 1 5070 0.54 MB/sec execute 35 sec latency 102.776 ms 1 5348 0.53 MB/sec execute 36 sec latency 48.306 ms 1 5642 0.52 MB/sec execute 37 sec latency 55.238 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 8192 2200448 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 20480 3743744 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 49152 3702784 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 69632 7446528 1% /mnt/lustre 1 5949 0.55 MB/sec execute 38 sec latency 70.811 ms 1 6324 0.63 MB/sec execute 39 sec latency 42.369 ms 1 6851 0.70 MB/sec execute 40 sec latency 60.692 ms test_26 fail mds1 8 times Failing mds1 on oleg147-server 1 7248 0.77 MB/sec execute 41 sec latency 17.375 ms Stopping /mnt/lustre-mds1 (opts:) on oleg147-server 1 7368 0.77 MB/sec execute 42 sec latency 563.168 ms 05:55:10 (1715334910) shut down 1 7368 0.76 MB/sec execute 43 sec latency 1563.352 ms 1 7368 0.74 MB/sec execute 44 sec latency 2563.553 ms 1 7368 0.72 MB/sec execute 45 sec latency 3563.744 ms 1 7368 0.71 MB/sec execute 46 sec latency 4563.959 ms 1 7368 0.69 MB/sec execute 47 sec latency 5564.162 ms 1 7368 0.68 MB/sec execute 48 sec latency 6564.337 ms 1 7368 0.66 MB/sec execute 49 sec latency 7564.539 ms 1 7368 0.65 MB/sec execute 50 sec latency 8564.747 ms 1 7368 0.64 MB/sec execute 51 sec latency 9564.914 ms 1 7368 0.63 MB/sec execute 52 sec latency 10565.146 ms Failover mds1 to oleg147-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 7368 0.61 MB/sec execute 53 sec latency 11565.290 ms 1 7368 0.60 MB/sec execute 54 sec latency 12565.453 ms oleg147-server: oleg147-server.virtnet: executing set_default_debug -1 all pdsh@oleg147-client: oleg147-server: ssh exited with exit code 1 Started lustre-MDT0000 05:55:23 (1715334923) targets are mounted 05:55:23 (1715334923) facet_failover done 1 7368 0.59 MB/sec execute 55 sec latency 13565.672 ms 1 7368 0.58 MB/sec execute 56 sec latency 14565.887 ms 1 7368 0.57 MB/sec execute 57 sec latency 15566.064 ms 1 7368 0.56 MB/sec execute 58 sec latency 16566.277 ms 1 7368 0.55 MB/sec execute 59 sec latency 17566.497 ms 1 7368 0.54 MB/sec execute 60 sec latency 18566.722 ms 1 7368 0.53 MB/sec execute 61 sec latency 19566.902 ms 1 7368 0.52 MB/sec execute 62 sec latency 20567.053 ms 1 7368 0.52 MB/sec execute 63 sec latency 21567.213 ms oleg147-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 7512 0.51 MB/sec execute 64 sec latency 21820.397 ms 1 7705 0.51 MB/sec execute 65 sec latency 25.144 ms 1 7902 0.51 MB/sec execute 66 sec latency 148.489 ms 1 8147 0.52 MB/sec execute 67 sec latency 63.387 ms 1 8560 0.56 MB/sec execute 68 sec latency 14.318 ms 1 8823 0.55 MB/sec execute 69 sec latency 103.911 ms 1 9186 0.55 MB/sec execute 70 sec latency 56.263 ms 1 9565 0.57 MB/sec execute 71 sec latency 46.667 ms 1 10086 0.63 MB/sec execute 72 sec latency 41.071 ms 1 10675 0.69 MB/sec execute 73 sec latency 53.132 ms dbench killed by signal 15 stopping dbench on /mnt/lustre at Fri May 10 05:55:42 EDT 2024 with return code 0 22504 pts/0 S+ 0:00 dbench -c client.txt 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100 killed dbench main pid 22504 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished