== replay-dual test 26: dbench and tar with mds failover ========================================================== 10:13:42 (1761315222)
Starting client oleg337-client.virtnet:  -o user_xattr,flock oleg337-server@tcp:/lustre /mnt/lustre
Started clients oleg337-client.virtnet: 
192.168.203.137@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt)
Started tar loop with pid 65904
Started dbench loop with 65905
striped dir -i0 -c2 -H all_char /mnt/lustre/d26.replay-dual/run_tar
striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre2/d26.replay-dual/run_dbench
looking for dbench program
/usr/bin/dbench
found dbench client file /usr/share/dbench/client.txt
tar: etc/resolv.conf: Cannot open: No space left on device
'/usr/share/dbench/client.txt' -> 'client.txt'
cp: cannot create regular file 'client.txt': No space left on device
UUID                   1K-blocks        Used   Available Use% Mounted on
lustre-MDT0000_UUID      1414116        1964     1285724   1% /mnt/lustre[MDT:0]
lustre-MDT0001_UUID      1414116        1808     1285880   1% /mnt/lustre[MDT:1]
lustre-OST0000_UUID      3833116        1404     3605616   1% /mnt/lustre[OST:0]
stopping dbench on /mnt/lustre at Fri Oct 24 10:14:25 EDT 2025 with return code 1
clean dbench files on /mnt/lustre
/mnt/lustre /mnt/lustre
lustre-OST0001_UUID      3833116        1388     3605632   1% /mnt/lustre[OST:1]

filesystem_summary:      7666232        2792     7211248   1% /mnt/lustre

/mnt/lustre
dbench exit with error 1
 replay-dual test_26: @@@@@@ FAIL: dbench failed with rc 1 
  Trace dump:
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:6482:error()
  = /home/green/git/lustre-release/lustre/tests/replay-dual.sh:1033:test_26()
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:6840:run_one()
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:6887:run_one_logged()
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:6728:run_test()
  = /home/green/git/lustre-release/lustre/tests/replay-dual.sh:1072:main()
Dumping lctl log to /tmp/testlogs//replay-dual.test_26.*.1761315267.log
rsync: chown "/tmp/testlogs/.replay-dual.test_26.debug_log.oleg337-server.1761315267.log.PxUgjz" failed: Operation not permitted (1)
rsync: chown "/tmp/testlogs/.replay-dual.test_26.dmesg.oleg337-server.1761315267.log.WJqKbT" failed: Operation not permitted (1)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1668) [generator=3.1.3]
test_26 fail mds1 1 times
Failing mds1 on oleg337-server
Stopping /mnt/lustre-mds1 (opts:) on oleg337-server
reboot facets: mds1
Failover mds1 to oleg337-server
mount facets: mds1
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg337-server: oleg337-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg337-client: oleg337-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
/home/green/git/lustre-release/lustre/tests/replay-dual.sh: line 1051: kill: (65905) - No such process
 replay-dual test_26: @@@@@@ FAIL: dbench 65905 missing 
  Trace dump:
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:6482:error()
  = /home/green/git/lustre-release/lustre/tests/replay-dual.sh:1051:test_26()
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:6840:run_one()
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:6887:run_one_logged()
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:6728:run_test()
  = /home/green/git/lustre-release/lustre/tests/replay-dual.sh:1072:main()
Dumping lctl log to /tmp/testlogs//replay-dual.test_26.*.1761315310.log
rsync: chown "/tmp/testlogs/.replay-dual.test_26.debug_log.oleg337-server.1761315310.log.0DF2dN" failed: Operation not permitted (1)
rsync: chown "/tmp/testlogs/.replay-dual.test_26.dmesg.oleg337-server.1761315310.log.VLy3ad" failed: Operation not permitted (1)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1668) [generator=3.1.3]
tar: Unexpected EOF in archive
tar: Unexpected EOF in archive
tar: Error is not recoverable: exiting now
/home/green/git/lustre-release/lustre/tests/replay-dual.sh: line 945: kill: (65905) - No such process