== replay-dual test 26: dbench and tar with mds failover ========================================================== 10:14:04 (1746627244) Starting client oleg436-client.virtnet: -o user_xattr,flock 192.168.204.136@tcp:/lustre /mnt/lustre Started clients oleg436-client.virtnet: 192.168.204.136@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project) Started tar loop with pid 14976 Started dbench loop with 14977 looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Wed May 7 10:14:06 EDT 2025 waiting for dbench pid 15008 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs failed to create barrier semaphore 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 1 243 9.21 MB/sec warmup 1 sec latency 46.576 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3679232 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 29696 3703808 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 32768 7383040 1% /mnt/lustre 1 494 8.60 MB/sec warmup 2 sec latency 18.829 ms 1 789 7.15 MB/sec warmup 3 sec latency 47.907 ms 1 1046 5.72 MB/sec warmup 4 sec latency 42.069 ms test_26 fail mds1 1 times Failing mds1 on oleg436-server Stopping /mnt/lustre-mds1 (opts:) on oleg436-server 1 1396 5.20 MB/sec warmup 5 sec latency 120.045 ms 10:14:11 (1746627251) shut down facet: mds1 facet_host: oleg436-server facet_failover_host: oleg436-server 1 1396 4.34 MB/sec warmup 6 sec latency 1120.256 ms 1 1396 3.72 MB/sec warmup 7 sec latency 2120.516 ms 1 1396 3.25 MB/sec warmup 8 sec latency 3120.743 ms 1 1396 2.89 MB/sec warmup 9 sec latency 4120.962 ms 1 1396 2.60 MB/sec warmup 10 sec latency 5121.173 ms 1 1396 2.36 MB/sec warmup 11 sec latency 6121.387 ms 1 1396 2.17 MB/sec warmup 12 sec latency 7121.640 ms 1 1396 2.00 MB/sec warmup 13 sec latency 8121.835 ms 1 1396 1.86 MB/sec warmup 14 sec latency 9122.039 ms 1 1396 1.73 MB/sec warmup 15 sec latency 10122.231 ms Failover mds1 to oleg436-server mount facets: mds1 1 1396 1.63 MB/sec warmup 16 sec latency 11122.397 ms Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 1396 1.53 MB/sec warmup 17 sec latency 12122.490 ms oleg436-server: oleg436-server.virtnet: executing set_default_debug -1 all 1 1396 1.45 MB/sec warmup 18 sec latency 13122.603 ms pdsh@oleg436-client: oleg436-server: ssh exited with exit code 1 Started lustre-MDT0000 10:14:24 (1746627264) targets are mounted 10:14:24 (1746627264) facet_failover done 1 1396 1.37 MB/sec warmup 19 sec latency 14122.798 ms 1 1396 0.00 MB/sec execute 1 sec latency 16123.159 ms 1 1396 0.00 MB/sec execute 2 sec latency 17123.292 ms 1 1396 0.00 MB/sec execute 3 sec latency 18123.443 ms oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 1578 0.03 MB/sec execute 4 sec latency 18445.718 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 1871 0.07 MB/sec execute 5 sec latency 36.630 ms 1 2200 0.12 MB/sec execute 6 sec latency 33.692 ms 1 2473 0.37 MB/sec execute 7 sec latency 40.459 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 4992 2203520 1% /mnt/lustre[MDT:0] 1 2978 0.94 MB/sec execute 8 sec latency 40.743 ms lustre-OST0000_UUID 3771392 14336 3606528 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 43008 3535872 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 57344 7142400 1% /mnt/lustre 1 3507 1.37 MB/sec execute 9 sec latency 39.105 ms 1 3826 1.37 MB/sec execute 10 sec latency 14.165 ms test_26 fail mds1 2 times 1 4059 1.28 MB/sec execute 11 sec latency 14.504 ms Failing mds1 on oleg436-server Stopping /mnt/lustre-mds1 (opts:) on oleg436-server 1 4266 1.19 MB/sec execute 12 sec latency 209.847 ms 10:14:38 (1746627278) shut down facet: mds1 facet_host: oleg436-server facet_failover_host: oleg436-server 1 4266 1.10 MB/sec execute 13 sec latency 1210.025 ms 1 4266 1.02 MB/sec execute 14 sec latency 2210.219 ms 1 4266 0.95 MB/sec execute 15 sec latency 3210.451 ms 1 4266 0.89 MB/sec execute 16 sec latency 4210.639 ms 1 4266 0.84 MB/sec execute 17 sec latency 5210.865 ms 1 4266 0.79 MB/sec execute 18 sec latency 6211.107 ms 1 4266 0.75 MB/sec execute 19 sec latency 7211.420 ms 1 4266 0.71 MB/sec execute 20 sec latency 8211.648 ms 1 4266 0.68 MB/sec execute 21 sec latency 9211.918 ms 1 4266 0.65 MB/sec execute 22 sec latency 10212.237 ms Failover mds1 to oleg436-server mount facets: mds1 1 4266 0.62 MB/sec execute 23 sec latency 11212.398 ms Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 4266 0.60 MB/sec execute 24 sec latency 12212.479 ms oleg436-server: oleg436-server.virtnet: executing set_default_debug -1 all pdsh@oleg436-client: oleg436-server: ssh exited with exit code 1 1 4266 0.57 MB/sec execute 25 sec latency 13212.629 ms Started lustre-MDT0000 10:14:51 (1746627291) targets are mounted 10:14:51 (1746627291) facet_failover done 1 4266 0.55 MB/sec execute 26 sec latency 14212.830 ms 1 4266 0.53 MB/sec execute 27 sec latency 15213.068 ms 1 4266 0.51 MB/sec execute 28 sec latency 16213.343 ms 1 4266 0.49 MB/sec execute 29 sec latency 17213.546 ms 1 4266 0.48 MB/sec execute 30 sec latency 18213.699 ms 1 4266 0.46 MB/sec execute 31 sec latency 19213.884 ms 1 4266 0.45 MB/sec execute 32 sec latency 20214.006 ms 1 4266 0.43 MB/sec execute 33 sec latency 21214.122 ms oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 4421 0.43 MB/sec execute 34 sec latency 21581.897 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 4680 0.46 MB/sec execute 35 sec latency 61.419 ms 1 5047 0.53 MB/sec execute 36 sec latency 15.825 ms 1 5308 0.52 MB/sec execute 37 sec latency 47.161 ms 1 5583 0.52 MB/sec execute 38 sec latency 35.791 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 6528 2201984 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 17408 3464192 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 46080 3452928 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 63488 6917120 1% /mnt/lustre 1 5936 0.54 MB/sec execute 39 sec latency 25.322 ms 1 6335 0.63 MB/sec execute 40 sec latency 37.222 ms test_26 fail mds1 3 times 1 6897 0.69 MB/sec execute 41 sec latency 36.873 ms Failing mds1 on oleg436-server Stopping /mnt/lustre-mds1 (opts:) on oleg436-server 1 7227 0.76 MB/sec execute 42 sec latency 193.367 ms 10:15:08 (1746627308) shut down facet: mds1 facet_host: oleg436-server facet_failover_host: oleg436-server 1 7227 0.74 MB/sec execute 43 sec latency 1193.563 ms 1 7227 0.72 MB/sec execute 44 sec latency 2193.780 ms 1 7227 0.71 MB/sec execute 45 sec latency 3193.974 ms 1 7227 0.69 MB/sec execute 46 sec latency 4194.185 ms 1 7227 0.68 MB/sec execute 47 sec latency 5194.383 ms 1 7227 0.66 MB/sec execute 48 sec latency 6194.609 ms 1 7227 0.65 MB/sec execute 49 sec latency 7194.780 ms 1 7227 0.64 MB/sec execute 50 sec latency 8196.873 ms 1 7227 0.62 MB/sec execute 51 sec latency 9197.416 ms 1 7227 0.61 MB/sec execute 52 sec latency 10197.672 ms Failover mds1 to oleg436-server mount facets: mds1 1 7227 0.60 MB/sec execute 53 sec latency 11199.710 ms Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 7227 0.59 MB/sec execute 54 sec latency 12199.916 ms 1 7227 0.58 MB/sec execute 55 sec latency 13200.507 ms oleg436-server: oleg436-server.virtnet: executing set_default_debug -1 all pdsh@oleg436-client: oleg436-server: ssh exited with exit code 1 Started lustre-MDT0000 10:15:21 (1746627321) targets are mounted 10:15:21 (1746627321) facet_failover done 1 7227 0.57 MB/sec execute 56 sec latency 14200.767 ms 1 7227 0.56 MB/sec execute 57 sec latency 15201.443 ms 1 7227 0.55 MB/sec execute 58 sec latency 16202.700 ms 1 7227 0.54 MB/sec execute 59 sec latency 17203.219 ms 1 7227 0.53 MB/sec execute 60 sec latency 18203.473 ms 1 7227 0.52 MB/sec execute 61 sec latency 19203.675 ms 1 7227 0.51 MB/sec execute 62 sec latency 20203.849 ms 1 7227 0.51 MB/sec execute 63 sec latency 21204.648 ms 1 7227 0.50 MB/sec execute 64 sec latency 22204.795 ms 1 7237 0.49 MB/sec execute 65 sec latency 23112.849 ms oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 7429 0.50 MB/sec execute 66 sec latency 23.203 ms 1 7655 0.50 MB/sec execute 67 sec latency 18.556 ms 1 7891 0.49 MB/sec execute 68 sec latency 78.418 ms 1 8094 0.49 MB/sec execute 69 sec latency 40.185 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 8192 2200320 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 19456 3548160 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 49152 3527680 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 68608 7075840 1% /mnt/lustre 1 8382 0.51 MB/sec execute 70 sec latency 25.141 ms 1 8708 0.54 MB/sec execute 71 sec latency 31.672 ms 1 8973 0.54 MB/sec execute 72 sec latency 28.721 ms test_26 fail mds1 4 times Failing mds1 on oleg436-server 1 9296 0.53 MB/sec execute 73 sec latency 33.464 ms Stopping /mnt/lustre-mds1 (opts:) on oleg436-server 1 9471 0.54 MB/sec execute 74 sec latency 542.540 ms 10:15:40 (1746627340) shut down facet: mds1 facet_host: oleg436-server facet_failover_host: oleg436-server 1 9471 0.53 MB/sec execute 75 sec latency 1542.742 ms 1 9471 0.53 MB/sec execute 76 sec latency 2542.900 ms 1 9471 0.52 MB/sec execute 77 sec latency 3543.052 ms 1 9471 0.51 MB/sec execute 78 sec latency 4543.242 ms 1 9471 0.51 MB/sec execute 79 sec latency 5543.442 ms 1 9471 0.50 MB/sec execute 80 sec latency 6543.649 ms 1 9471 0.50 MB/sec execute 81 sec latency 7543.882 ms 1 9471 0.49 MB/sec execute 82 sec latency 8544.062 ms 1 9471 0.48 MB/sec execute 83 sec latency 9544.281 ms 1 9471 0.48 MB/sec execute 84 sec latency 10544.435 ms Failover mds1 to oleg436-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 9471 0.47 MB/sec execute 85 sec latency 11544.634 ms 1 9471 0.47 MB/sec execute 86 sec latency 12544.802 ms oleg436-server: oleg436-server.virtnet: executing set_default_debug -1 all pdsh@oleg436-client: oleg436-server: ssh exited with exit code 1 Started lustre-MDT0000 10:15:52 (1746627352) targets are mounted 10:15:52 (1746627352) facet_failover done 1 9471 0.46 MB/sec execute 87 sec latency 13544.957 ms 1 9471 0.46 MB/sec execute 88 sec latency 14545.129 ms 1 9471 0.45 MB/sec execute 89 sec latency 15545.314 ms 1 9471 0.45 MB/sec execute 90 sec latency 16545.721 ms 1 9471 0.44 MB/sec execute 91 sec latency 17545.875 ms 1 9471 0.44 MB/sec execute 92 sec latency 18546.185 ms 1 9471 0.43 MB/sec execute 93 sec latency 19546.378 ms 1 9480 0.43 MB/sec execute 94 sec latency 20464.373 ms oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 9735 0.44 MB/sec execute 95 sec latency 37.938 ms 1 10296 0.48 MB/sec execute 96 sec latency 42.305 ms 1 10667 0.52 MB/sec execute 97 sec latency 39.310 ms 1 10917 0.53 MB/sec execute 98 sec latency 18.055 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 9472 2199040 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 20480 3429376 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 51200 3373056 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 71680 6802432 2% /mnt/lustre 1 11136 0.53 MB/sec execute 99 sec latency 16.257 ms 1 cleanup 100 sec 0 cleanup 101 sec Operation Count AvgLat MaxLat ---------------------------------------- NTCreateX 1743 32.054 21581.888 Close 1265 19.913 23112.827 Rename 69 14.408 27.863 Unlink 357 4.607 14.513 Qpathinfo 1606 15.322 20464.357 Qfileinfo 278 0.429 2.797 Qfsinfo 282 0.507 2.476 Sfileinfo 120 8.572 16.368 Find 607 1.199 14.073 WriteX 843 1.902 8.194 ReadX 2692 0.129 32.658 LockX 6 1.456 1.796 UnlockX 6 1.491 1.797 Flush 112 24.745 78.406 Throughput 0.528063 MB/sec 1 clients 1 procs max_latency=23112.849 ms stopping dbench on /mnt/lustre at Wed May 7 10:16:06 EDT 2025 with return code 0 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt test_26 fail mds1 5 times '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Wed May 7 10:16:08 EDT 2025 waiting for dbench pid 19371 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs 0 of 1 processes prepared for launch 0 sec Failing mds1 on oleg436-server 1 of 1 processes prepared for launch 0 sec releasing clients Stopping /mnt/lustre-mds1 (opts:) on oleg436-server 1 113 5.24 MB/sec warmup 1 sec latency 618.880 ms 10:16:09 (1746627369) shut down facet: mds1 facet_host: oleg436-server facet_failover_host: oleg436-server 1 113 2.62 MB/sec warmup 2 sec latency 1619.062 ms 1 113 1.75 MB/sec warmup 3 sec latency 2619.220 ms 1 113 1.31 MB/sec warmup 4 sec latency 3619.373 ms 1 113 1.05 MB/sec warmup 5 sec latency 4619.524 ms 1 113 0.87 MB/sec warmup 6 sec latency 5619.690 ms 1 113 0.75 MB/sec warmup 7 sec latency 6619.820 ms 1 113 0.66 MB/sec warmup 8 sec latency 7619.976 ms 1 113 0.58 MB/sec warmup 9 sec latency 8620.120 ms 1 113 0.52 MB/sec warmup 10 sec latency 9620.279 ms 1 113 0.48 MB/sec warmup 11 sec latency 10620.438 ms Failover mds1 to oleg436-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 113 0.44 MB/sec warmup 12 sec latency 11620.662 ms oleg436-server: oleg436-server.virtnet: executing set_default_debug -1 all 1 113 0.40 MB/sec warmup 13 sec latency 12620.848 ms pdsh@oleg436-client: oleg436-server: ssh exited with exit code 1 Started lustre-MDT0000 10:16:21 (1746627381) targets are mounted 10:16:21 (1746627381) facet_failover done 1 113 0.37 MB/sec warmup 14 sec latency 13621.037 ms 1 113 0.35 MB/sec warmup 15 sec latency 14621.214 ms 1 113 0.33 MB/sec warmup 16 sec latency 15621.402 ms 1 113 0.31 MB/sec warmup 17 sec latency 16621.579 ms 1 113 0.29 MB/sec warmup 18 sec latency 17621.797 ms 1 113 0.28 MB/sec warmup 19 sec latency 18621.988 ms 1 113 0.00 MB/sec execute 1 sec latency 20622.245 ms oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 341 3.51 MB/sec execute 2 sec latency 20680.874 ms 1 664 5.31 MB/sec execute 3 sec latency 32.022 ms 1 913 4.13 MB/sec execute 4 sec latency 38.118 ms 1 1224 3.57 MB/sec execute 5 sec latency 28.642 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 11008 2197504 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 46080 3386368 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 24576 3399680 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 70656 6786048 2% /mnt/lustre 1 1609 3.49 MB/sec execute 6 sec latency 66.132 ms 1 1942 3.03 MB/sec execute 7 sec latency 33.447 ms 1 2292 2.81 MB/sec execute 8 sec latency 14.109 ms test_26 fail mds1 6 times Failing mds1 on oleg436-server 1 2562 2.63 MB/sec execute 9 sec latency 58.327 ms Stopping /mnt/lustre-mds1 (opts:) on oleg436-server 1 2773 2.67 MB/sec execute 10 sec latency 598.160 ms 10:16:38 (1746627398) shut down facet: mds1 facet_host: oleg436-server facet_failover_host: oleg436-server 1 2773 2.43 MB/sec execute 11 sec latency 1598.355 ms 1 2773 2.23 MB/sec execute 12 sec latency 2598.500 ms 1 2773 2.06 MB/sec execute 13 sec latency 3598.788 ms 1 2773 1.91 MB/sec execute 14 sec latency 4598.955 ms 1 2773 1.78 MB/sec execute 15 sec latency 5599.125 ms 1 2773 1.67 MB/sec execute 16 sec latency 6599.270 ms 1 2773 1.57 MB/sec execute 17 sec latency 7599.447 ms 1 2773 1.48 MB/sec execute 18 sec latency 8599.588 ms 1 2773 1.41 MB/sec execute 19 sec latency 9599.793 ms 1 2773 1.34 MB/sec execute 20 sec latency 10600.030 ms Failover mds1 to oleg436-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 2773 1.27 MB/sec execute 21 sec latency 11600.397 ms 1 2773 1.21 MB/sec execute 22 sec latency 12600.641 ms oleg436-server: oleg436-server.virtnet: executing set_default_debug -1 all pdsh@oleg436-client: oleg436-server: ssh exited with exit code 1 Started lustre-MDT0000 10:16:50 (1746627410) targets are mounted 10:16:50 (1746627410) facet_failover done 1 2773 1.16 MB/sec execute 23 sec latency 13600.794 ms 1 2773 1.11 MB/sec execute 24 sec latency 14600.965 ms 1 2773 1.07 MB/sec execute 25 sec latency 15601.121 ms 1 2773 1.03 MB/sec execute 26 sec latency 16601.264 ms 1 2773 0.99 MB/sec execute 27 sec latency 17601.441 ms 1 2773 0.95 MB/sec execute 28 sec latency 18601.627 ms 1 2773 0.92 MB/sec execute 29 sec latency 19601.788 ms 1 2773 0.89 MB/sec execute 30 sec latency 20601.914 ms 1 2773 0.86 MB/sec execute 31 sec latency 21602.041 ms oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 2990 0.88 MB/sec execute 32 sec latency 22079.926 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 3529 1.00 MB/sec execute 33 sec latency 48.816 ms 1 3832 1.02 MB/sec execute 34 sec latency 17.692 ms 1 4078 1.00 MB/sec execute 35 sec latency 14.536 ms 1 4281 0.97 MB/sec execute 36 sec latency 87.990 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 10112 2198400 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 49152 3602432 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 26624 3586048 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 75776 7188480 2% /mnt/lustre 1 4473 0.96 MB/sec execute 37 sec latency 56.337 ms 1 4766 0.97 MB/sec execute 38 sec latency 43.331 ms 1 5139 1.03 MB/sec execute 39 sec latency 54.245 ms test_26 fail mds1 7 times Failing mds1 on oleg436-server Stopping /mnt/lustre-mds1 (opts:) on oleg436-server 1 5403 1.01 MB/sec execute 40 sec latency 34.328 ms 10:17:08 (1746627428) shut down facet: mds1 facet_host: oleg436-server facet_failover_host: oleg436-server 1 5407 0.98 MB/sec execute 41 sec latency 985.716 ms 1 5407 0.96 MB/sec execute 42 sec latency 1985.892 ms 1 5407 0.94 MB/sec execute 43 sec latency 2986.045 ms 1 5407 0.91 MB/sec execute 44 sec latency 3986.208 ms 1 5407 0.89 MB/sec execute 45 sec latency 4986.380 ms 1 5407 0.87 MB/sec execute 46 sec latency 5986.538 ms 1 5407 0.86 MB/sec execute 47 sec latency 6986.726 ms 1 5407 0.84 MB/sec execute 48 sec latency 7986.897 ms 1 5407 0.82 MB/sec execute 49 sec latency 8987.053 ms 1 5407 0.80 MB/sec execute 50 sec latency 9987.219 ms Failover mds1 to oleg436-server mount facets: mds1 1 5407 0.79 MB/sec execute 51 sec latency 10987.401 ms Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 5407 0.77 MB/sec execute 52 sec latency 11987.607 ms oleg436-server: oleg436-server.virtnet: executing set_default_debug -1 all pdsh@oleg436-client: oleg436-server: ssh exited with exit code 1 1 5407 0.76 MB/sec execute 53 sec latency 12987.765 ms Started lustre-MDT0000 10:17:21 (1746627441) targets are mounted 10:17:21 (1746627441) facet_failover done 1 5407 0.74 MB/sec execute 54 sec latency 13987.946 ms 1 5407 0.73 MB/sec execute 55 sec latency 14988.111 ms 1 5407 0.72 MB/sec execute 56 sec latency 15988.264 ms 1 5407 0.71 MB/sec execute 57 sec latency 16988.434 ms 1 5407 0.69 MB/sec execute 58 sec latency 17988.625 ms 1 5407 0.68 MB/sec execute 59 sec latency 18988.808 ms 1 5407 0.67 MB/sec execute 60 sec latency 19989.004 ms 1 5407 0.66 MB/sec execute 61 sec latency 20989.138 ms oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 5587 0.65 MB/sec execute 62 sec latency 21321.335 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 5936 0.66 MB/sec execute 63 sec latency 55.431 ms 1 6329 0.72 MB/sec execute 64 sec latency 48.756 ms 1 6851 0.76 MB/sec execute 65 sec latency 57.500 ms 1 7281 0.80 MB/sec execute 66 sec latency 18.250 ms 1 7531 0.81 MB/sec execute 67 sec latency 15.191 ms 1 7778 0.80 MB/sec execute 68 sec latency 100.704 ms 1 7988 0.79 MB/sec execute 69 sec latency 101.130 ms 1 8258 0.80 MB/sec execute 70 sec latency 55.026 ms 1 8611 0.83 MB/sec execute 71 sec latency 40.839 ms 1 8879 0.82 MB/sec execute 72 sec latency 47.320 ms 1 9166 0.82 MB/sec execute 73 sec latency 45.684 ms 1 9504 0.82 MB/sec execute 74 sec latency 61.300 ms 1 9981 0.87 MB/sec execute 75 sec latency 48.258 ms 1 10460 0.90 MB/sec execute 76 sec latency 38.270 ms dbench killed by signal 15 stopping dbench on /mnt/lustre at Wed May 7 10:17:44 EDT 2025 with return code 0 19371 pts/0 S+ 0:00 dbench -c client.txt 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100 killed dbench main pid 19371 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished