== replay-single test 70b: dbench 1mdts recovery; 1 clients ========================================================== 18:01:48 (1715205708) Starting client oleg408-client.virtnet: -o user_xattr,flock oleg408-server@tcp:/lustre /mnt/lustre Started clients oleg408-client.virtnet: 192.168.204.108@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt) + MISSING_DBENCH_OK= + PATH=/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin::/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests:/sbin:/usr/sbin:/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests/: + DBENCH_LIB= + TESTSUITE=replay-single + TESTNAME=test_70b + MOUNT=/mnt/lustre ++ hostname + DIR=/mnt/lustre/d70b.replay-single/oleg408-client.virtnet + LCTL=/home/green/git/lustre-release/lustre/utils/lctl + rundbench 1 -t 300 dbench: no process found Started rundbench load pid=11728 ... UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3712 2204928 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 31744 3689472 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 10240 3740672 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 41984 7430144 1% /mnt/lustre test_70b fail mds1 1 times Failing mds1 on oleg408-server Stopping /mnt/lustre-mds1 (opts:) on oleg408-server reboot facets: mds1 Failover mds1 to oleg408-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg408-server: oleg408-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3712 2204928 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 31744 3670016 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 13312 3738624 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 45056 7408640 1% /mnt/lustre test_70b fail mds1 2 times Failing mds1 on oleg408-server Stopping /mnt/lustre-mds1 (opts:) on oleg408-server oleg408-client.virtnet: looking for dbench program oleg408-client.virtnet: /usr/bin/dbench oleg408-client.virtnet: creating output directory /mnt/lustre/d70b.replay-single/oleg408-client.virtnet oleg408-client.virtnet: mkdir: created directory '/mnt/lustre/d70b.replay-single' oleg408-client.virtnet: mkdir: created directory '/mnt/lustre/d70b.replay-single/oleg408-client.virtnet' oleg408-client.virtnet: found dbench client file /usr/share/dbench/client.txt oleg408-client.virtnet: '/usr/share/dbench/client.txt' -> 'client.txt' oleg408-client.virtnet: running 'dbench 1 -t 300' on /mnt/lustre/d70b.replay-single/oleg408-client.virtnet at Wed May 8 18:01:49 EDT 2024 oleg408-client.virtnet: waiting for dbench pid 11753 oleg408-client.virtnet: dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 oleg408-client.virtnet: oleg408-client.virtnet: Running for 300 seconds with load 'client.txt' and minimum warmup 60 secs oleg408-client.virtnet: failed to create barrier semaphore oleg408-client.virtnet: 0 of 1 processes prepared for launch 0 sec oleg408-client.virtnet: 1 of 1 processes prepared for launch 0 sec oleg408-client.virtnet: releasing clients oleg408-client.virtnet: 1 466 16.30 MB/sec warmup 1 sec latency 16.778 ms oleg408-client.virtnet: 1 801 10.74 MB/sec warmup 2 sec latency 66.048 ms oleg408-client.virtnet: 1 1124 7.66 MB/sec warmup 3 sec latency 42.309 ms oleg408-client.virtnet: 1 1523 6.52 MB/sec warmup 4 sec latency 56.934 ms oleg408-client.virtnet: 1 1905 5.29 MB/sec warmup 5 sec latency 48.207 ms oleg408-client.virtnet: 1 1950 4.41 MB/sec warmup 6 sec latency 943.322 ms oleg408-client.virtnet: 1 1950 3.78 MB/sec warmup 7 sec latency 1943.525 ms oleg408-client.virtnet: 1 1950 3.31 MB/sec warmup 8 sec latency 2943.704 ms oleg408-client.virtnet: 1 1950 2.94 MB/sec warmup 9 sec latency 3943.861 ms oleg408-client.virtnet: 1 1950 2.64 MB/sec warmup 10 sec latency 4943.997 ms oleg408-client.virtnet: 1 1950 2.40 MB/sec warmup 11 sec latency 5944.150 ms oleg408-client.virtnet: 1 1950 2.20 MB/sec warmup 12 sec latency 6944.313 ms oleg408-client.virtnet: 1 1950 2.03 MB/sec warmup 13 sec latency 7944.538 ms oleg408-client.virtnet: 1 1950 1.89 MB/sec warmup 14 sec latency 8944.745 ms oleg408-client.virtnet: 1 1950 1.76 MB/sec warmup 15 sec latency 9944.911 ms oleg408-client.virtnet: 1 1950 1.65 MB/sec warmup 16 sec latency 10945.039 ms oleg408-client.virtnet: 1 1950 1.56 MB/sec warmup 17 sec latency 11945.192 ms oleg408-client.virtnet: 1 1950 1.47 MB/sec warmup 18 sec latency 12945.393 ms oleg408-client.virtnet: 1 1950 1.39 MB/sec warmup 19 sec latency 13945.581 ms oleg408-client.virtnet: 1 1950 1.32 MB/sec warmup 20 sec latency 14945.742 ms oleg408-client.virtnet: 1 1950 1.26 MB/sec warmup 21 sec latency 15945.908 ms oleg408-client.virtnet: 1 1950 1.20 MB/sec warmup 22 sec latency 16946.069 ms oleg408-client.virtnet: 1 1950 1.15 MB/sec warmup 23 sec latency 17946.218 ms oleg408-client.virtnet: 1 1950 1.10 MB/sec warmup 24 sec latency 18946.374 ms oleg408-client.virtnet: 1 2204 1.07 MB/sec warmup 25 sec latency 19497.513 ms oleg408-client.virtnet: 1 2903 1.26 MB/sec warmup 26 sec latency 40.094 ms oleg408-client.virtnet: 1 3650 1.43 MB/sec warmup 27 sec latency 47.305 ms oleg408-client.virtnet: 1 4007 1.43 MB/sec warmup 28 sec latency 17.123 ms oleg408-client.virtnet: 1 4362 1.40 MB/sec warmup 29 sec latency 55.453 ms oleg408-client.virtnet: 1 4823 1.41 MB/sec warmup 30 sec latency 43.866 ms oleg408-client.virtnet: 1 5449 1.47 MB/sec warmup 31 sec latency 28.014 ms oleg408-client.virtnet: 1 5936 1.47 MB/sec warmup 32 sec latency 332.610 mreboot facets: mds1 Failover mds1 to oleg408-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg408-server: oleg408-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Started lustre-MDT0000