== sanity-flr test 200: stress test ====================== 12:25:15 (1721406315)
Starting client: oleg135-client.virtnet:  -o user_xattr,flock oleg135-server@tcp:/lustre /mnt/lustre2
Starting client: oleg135-client.virtnet:  -o user_xattr,flock oleg135-server@tcp:/lustre /mnt/lustre3
fail_loc=0x1A03
fail_loc=0x1A03
resync file /mnt/lustre3/f200.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 1550400 ..
Extending file size to 5046208 ..
done
Extending file size to 3753216 ..
Extending file size to 6586912 ..
Extending file size to 4522048 ..
lock to resync file /mnt/lustre3/f200.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done
resync file /mnt/lustre3/f200.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8606336 ..
failed
Extending file size to 8125824 ..
resync file /mnt/lustre3/f200.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed
resync file /mnt/lustre3/f200.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed
resync file /mnt/lustre3/f200.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed
resync file /mnt/lustre3/f200.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed
Extending file size to 8428000 ..
resync file /mnt/lustre3/f200.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9059328 ..
failed
lock to resync file /mnt/lustre3/f200.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done
resync file /mnt/lustre3/f200.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed
lock to resync file /mnt/lustre3/f200.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done
lock to resync file /mnt/lustre3/f200.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done
resync file /mnt/lustre3/f200.sanity-flr with 'mirror_io resync -e resync_start' ..failed
Extending file size to 8635360 ..
resync file /mnt/lustre3/f200.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed
lock to resync file /mnt/lustre3/f200.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done
resync file /mnt/lustre3/f200.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed
resync file /mnt/lustre3/f200.sanity-flr with 'mirror_io resync -e resync_start' ..failed
lock to resync file /mnt/lustre3/f200.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done
resync file /mnt/lustre3/f200.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed
lock to resync file /mnt/lustre3/f200.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done
Extending file size to 9375456 ..
resync file /mnt/lustre3/f200.sanity-flr with 'mirror_io resync -e resync_start' ..failed
lock to resync file /mnt/lustre3/f200.sanity-flr with 'mirror_io resync -e resync_start' ..failed
lock to resync file /mnt/lustre3/f200.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done
lock to resync file /mnt/lustre3/f200.sanity-flr with 'mirror_io resync -e resync_start' ..failed
lock to resync file /mnt/lustre3/f200.sanity-flr with 'mirror_io resync -e resync_start' ..failed
lock to resync file /mnt/lustre3/f200.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done
lock to resync file /mnt/lustre3/f200.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done
resync file /mnt/lustre3/f200.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done
resync file /mnt/lustre3/f200.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed
lock to resync file /mnt/lustre3/f200.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done
resync file /mnt/lustre3/f200.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed
resync file /mnt/lustre3/f200.sanity-flr with 'mirror_io resync -e resync_start' ..failed
resync file /mnt/lustre3/f200.sanity-flr with 'mirror_io resync -e resync_start' ..failed
resync file /mnt/lustre3/f200.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed
resync file /mnt/lustre3/f200.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed
resync file /mnt/lustre3/f200.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed
resync file /mnt/lustre3/f200.sanity-flr with 'mirror_io resync -e resync_start' ..failed
resync file /mnt/lustre3/f200.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed
lock to resync file /mnt/lustre3/f200.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done
resync file /mnt/lustre3/f200.sanity-flr with 'mirror_io resync -e resync_start' ..failed
resync file /mnt/lustre3/f200.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done
resync file /mnt/lustre3/f200.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..close: Input/output error
 sanity-flr test_200: @@@@@@ FAIL: failed writing to 2691328:898976 
  Trace dump:
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:6414:error()
  = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:3077:write_file_200()
  = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:3159:test_200()
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:6749:run_one()
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:6796:run_one_logged()
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:6637:run_test()
  = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:3198:main()
Dumping lctl log to /tmp/testlogs//sanity-flr.test_200.*.1721406356.log
Waiting 22564 22565 22567 22568 22571
failed
 sanity-flr test_200: @@@@@@ FAIL: read failed 
  Trace dump:
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:6414:error()
  = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:3087:read_file_200()
  = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:3162:test_200()
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:6749:run_one()
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:6796:run_one_logged()
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:6637:run_test()
  = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:3198:main()
rsync: chown "/tmp/testlogs/.sanity-flr.test_200.debug_log.oleg135-server.1721406356.log.XL4ZRd" failed: Operation not permitted (1)
rsync: chown "/tmp/testlogs/.sanity-flr.test_200.dmesg.oleg135-server.1721406356.log.fcrrpM" failed: Operation not permitted (1)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2]
Dumping lctl log to /tmp/testlogs//sanity-flr.test_200.*.1721406357.log
rsync: chown "/tmp/testlogs/.sanity-flr.test_200.debug_log.oleg135-server.1721406357.log.CGJhId" failed: Operation not permitted (1)
rsync: chown "/tmp/testlogs/.sanity-flr.test_200.dmesg.oleg135-server.1721406357.log.oUQutO" failed: Operation not permitted (1)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2]
192.168.201.135@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0
Stopping client oleg135-client.virtnet /mnt/lustre2 (opts:)
192.168.201.135@tcp:/lustre /mnt/lustre3 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0
Stopping client oleg135-client.virtnet /mnt/lustre3 (opts:)
3