== sanity-flr test 200: stress test ====================== 08:06:02 (1721217962)
Starting client: oleg427-client.virtnet:  -o user_xattr,flock oleg427-server@tcp:/lustre /mnt/lustre2
Starting client: oleg427-client.virtnet:  -o user_xattr,flock oleg427-server@tcp:/lustre /mnt/lustre3
fail_loc=0x1A03
fail_loc=0x1A03
resync file /mnt/lustre3/f200.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 5920000 ..
Extending file size to 8804480 ..
failed
Extending file size to 9057984 ..
resync file /mnt/lustre3/f200.sanity-flr with 'mirror_io resync -e resync_start' ..failed
resync file /mnt/lustre3/f200.sanity-flr with 'mirror_io resync -e resync_start' ..failed
lock to resync file /mnt/lustre3/f200.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done
resync file /mnt/lustre3/f200.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed
resync file /mnt/lustre3/f200.sanity-flr with 'mirror_io resync -e resync_start' ..failed
Extending file size to 9110784 ..
resync file /mnt/lustre3/f200.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed
resync file /mnt/lustre3/f200.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed
resync file /mnt/lustre3/f200.sanity-flr with 'mirror_io resync -e resync_start' ..failed
resync file /mnt/lustre3/f200.sanity-flr with 'mirror_io resync -e resync_start' ..failed
resync file /mnt/lustre3/f200.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed
resync file /mnt/lustre3/f200.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed
resync file /mnt/lustre3/f200.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed
lock to resync file /mnt/lustre3/f200.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done
resync file /mnt/lustre3/f200.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed
resync file /mnt/lustre3/f200.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed
resync file /mnt/lustre3/f200.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed
resync file /mnt/lustre3/f200.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done
lock to resync file /mnt/lustre3/f200.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done
Extending file size to 8986624 ..
resync file /mnt/lustre3/f200.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed
resync file /mnt/lustre3/f200.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed
resync file /mnt/lustre3/f200.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed
lock to resync file /mnt/lustre3/f200.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done
resync file /mnt/lustre3/f200.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed
lock to resync file /mnt/lustre3/f200.sanity-flr with 'mirror_io resync -e resync_start' ..failed
resync file /mnt/lustre3/f200.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed
resync file /mnt/lustre3/f200.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed
resync file /mnt/lustre3/f200.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed
resync file /mnt/lustre3/f200.sanity-flr with 'mirror_io resync -e resync_start' ..failed
resync file /mnt/lustre3/f200.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed
resync file /mnt/lustre3/f200.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done
lock to resync file /mnt/lustre3/f200.sanity-flr with 'mirror_io resync -e resync_start' ..failed
resync file /mnt/lustre3/f200.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed
resync file /mnt/lustre3/f200.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done
resync file /mnt/lustre3/f200.sanity-flr with 'mirror_io resync -e resync_start' ..failed
lock to resync file /mnt/lustre3/f200.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done
resync file /mnt/lustre3/f200.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed
resync file /mnt/lustre3/f200.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed
resync file /mnt/lustre3/f200.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed
lock to resync file /mnt/lustre3/f200.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done
resync file /mnt/lustre3/f200.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done
resync file /mnt/lustre3/f200.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed
resync file /mnt/lustre3/f200.sanity-flr with 'mirror_io resync -e resync_start' ..failed
lock to resync file /mnt/lustre3/f200.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done
resync file /mnt/lustre3/f200.sanity-flr with 'mirror_io resync -e resync_start' ..failed
resync file /mnt/lustre3/f200.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed
resync file /mnt/lustre3/f200.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed
resync file /mnt/lustre3/f200.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed
resync file /mnt/lustre3/f200.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed
Extending file size to 9180192 ..
resync file /mnt/lustre3/f200.sanity-flr with 'mirror_io resync -e resync_start' ..failed
Extending file size to 9345824 ..
resync file /mnt/lustre3/f200.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed
resync file /mnt/lustre3/f200.sanity-flr with 'mirror_io resync -e resync_start' ..failed
close: Input/output error
 sanity-flr test_200: @@@@@@ FAIL: failed writing to 5628672:1044224 
Waiting 24922 24923 24924 24926 24929
  Trace dump:
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:6414:error()
  = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:3077:write_file_200()
  = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:3153:test_200()
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:6749:run_one()
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:6796:run_one_logged()
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:6637:run_test()
  = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:3198:main()
Dumping lctl log to /tmp/testlogs//sanity-flr.test_200.*.1721217998.log
192.168.204.127@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0
rsync: chown "/tmp/testlogs/.sanity-flr.test_200.debug_log.oleg427-server.1721217998.log.faRUHM" failed: Operation not permitted (1)
Stopping client oleg427-client.virtnet /mnt/lustre2 (opts:)
rsync: chown "/tmp/testlogs/.sanity-flr.test_200.dmesg.oleg427-server.1721217998.log.F1fS0H" failed: Operation not permitted (1)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2]
192.168.204.127@tcp:/lustre /mnt/lustre3 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0
Stopping client oleg427-client.virtnet /mnt/lustre3 (opts:)
lfs mirror: '/mnt/lustre/f200.sanity-flr' llapi_mirror_resync_many: Input/output error.
3
 sanity-flr test_200: @@@@@@ FAIL: checksum error for mirror 2 
  Trace dump:
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:6414:error()
  = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:3193:test_200()
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:6749:run_one()
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:6796:run_one_logged()
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:6637:run_test()
  = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:3198:main()
Dumping lctl log to /tmp/testlogs//sanity-flr.test_200.*.1721218001.log
rsync: chown "/tmp/testlogs/.sanity-flr.test_200.debug_log.oleg427-server.1721218001.log.YQO5fc" failed: Operation not permitted (1)
rsync: chown "/tmp/testlogs/.sanity-flr.test_200.dmesg.oleg427-server.1721218001.log.ouZdBd" failed: Operation not permitted (1)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2]