== sanity-flr test 70a: flr mode fsx test ================ 23:31:32 (1746156692) striped dir -i0 -c2 -H all_char /mnt/lustre/d70a.sanity-flr Seed set to 6693 fallocating to largest ever: 0x25bb6 000001 1746156693.456700 fallocate 0x0259e9 thru 0x025bb5 (0x1cd bytes) 000002 1746156693.470756 mapread 0x019302 thru 0x025bb5 (0xc8b4 bytes) 000003 1746156693.495321 trunc from 0x0058f3 to 0x0139f5 (0xe103 bytes) 000004 1746156693.500767 trunc from 0x025bb6 to 0x005ccc (0x1fee9 bytes) 000005 1746156693.523246 read 0x00072c thru 0x005ccc (0x55a1 bytes) 000006 1746156693.528824 write 0x039ef7 thru 0x03ffff (0x6109 bytes) 000007 1746156693.529834 fallocate 0x00a4ed thru 0x00e582 (0x4096 bytes) fallocating to largest ever: 0x3c45f 000008 1746156693.533919 fallocate 0x0373fb thru 0x03c45e (0x5064 bytes) 000009 1746156693.537698 write 0x02ab84 thru 0x02f26f (0x46ec bytes) 000010 1746156693.542857 write 0x013a3e thru 0x016526 (0x2ae9 bytes) 000011 1746156693.549080 write 0x01bc5e thru 0x02a1da (0xe57d bytes) 000012 1746156693.550166 mapwrite 0x0051b0 thru 0x00b164 (0x5fb5 bytes) 000013 1746156693.592036 read 0x03fae3 thru 0x03ffff (0x51d bytes) 000014 1746156693.592488 write 0x017a1d thru 0x0235ea (0xbbce bytes) 000015 1746156693.593572 punch from 0x019d78 to 0x0293ec (0xf675 bytes) 000016 1746156693.613033 write 0x0224be thru 0x028e4e (0x6991 bytes) 000017 1746156693.643339 mapwrite 0x03a78f thru 0x03b773 (0xfe5 bytes) 000018 1746156693.678666 read 0x0312f2 thru 0x03d183 (0xbe92 bytes) 000021 1746156693.681894 mapwrite 0x006363 thru 0x013e77 (0xdb15 bytes) 000022 1746156693.726297 mapwrite 0x038c67 thru 0x03f90c (0x6ca6 bytes) 000023 1746156693.755569 punch from 0x03d43b to 0x03ffff (0x2bc5 bytes) 000025 1746156693.764117 write 0x02fb03 thru 0x0316b7 (0x1bb5 bytes) 000026 1746156693.766281 mapwrite 0x03c50a thru 0x03ffff (0x3af6 bytes) 000027 1746156693.800641 mapread 0x032c04 thru 0x03b93a (0x8d37 bytes) 000028 1746156693.802067 write 0x003fb2 thru 0x0085f0 (0x463f bytes) 000029 1746156693.804673 fallocate 0x02575c thru 0x0260ff (0x9a4 bytes) 000030 1746156693.809430 mapread 0x00651d thru 0x009f4b (0x3a2f bytes) 000031 1746156693.810347 punch from 0x000e1e to 0x0060c0 (0x52a3 bytes) 000033 1746156693.827470 mapwrite 0x02c2ff thru 0x0334a1 (0x71a3 bytes) 000034 1746156693.858822 punch from 0x0001d7 to 0x001916 (0x1740 bytes) 000035 1746156693.864591 read 0x01435f thru 0x018898 (0x453a bytes) 000036 1746156693.867542 trunc from 0x040000 to 0x0071e6 (0x38e19 bytes) 000037 1746156693.881440 mapwrite 0x026f76 thru 0x02acb7 (0x3d42 bytes) 000038 1746156693.930952 punch from 0x00a326 to 0x00d75a (0x3435 bytes) 000039 1746156693.935771 trunc from 0x02acb8 to 0x01cd27 (0xdf90 bytes) 000040 1746156693.956167 trunc from 0x001fd6 to 0x00c507 (0xa532 bytes) 000042 1746156693.961271 write 0x02c108 thru 0x030fc8 (0x4ec1 bytes) 000043 1746156693.963480 trunc from 0x02ff3a to 0x030fc8 (0x108f bytes) 000044 1746156693.977228 read 0x0114c2 thru 0x020081 (0xebc0 bytes) 000045 1746156693.985985 punch from 0x0030af to 0x00497f (0x18d1 bytes) 000047 1746156693.991221 write 0x027258 thru 0x032320 (0xb0c9 bytes) fallocating to largest ever: 0x40000 000048 1746156693.992479 fallocate 0x03dd28 thru 0x03ffff (0x22d8 bytes) 000050 1746156693.996478 trunc from 0x040000 to 0x027608 (0x189f7 bytes) 000051 1746156694.012236 read 0x01da56 thru 0x023b14 (0x60bf bytes) 000052 1746156694.015922 fallocate 0x032bc6 thru 0x0373db (0x4816 bytes) 000053 1746156694.027927 punch from 0x0119e9 to 0x01f214 (0xd82c bytes) 000054 1746156694.038166 punch from 0x00ee3f to 0x0174c8 (0x868a bytes) 000055 1746156694.050501 write 0x0265ab thru 0x0301b3 (0x9c09 bytes) 000056 1746156694.052228 trunc from 0x0301b4 to 0x01554e (0x1ac65 bytes) 000057 1746156694.069988 mapread 0x0032b1 thru 0x012bee (0xf93e bytes) 000058 1746156694.086167 mapwrite 0x013d37 thru 0x01f57c (0xb846 bytes) 000059 1746156694.135449 trunc from 0x002670 to 0x010016 (0xd9a7 bytes) 000060 1746156694.141462 trunc from 0x015643 to 0x0161df (0xb9d bytes) 000061 1746156694.146887 mapwrite 0x0081f1 thru 0x00ec17 (0x6a27 bytes) 000062 1746156694.179745 trunc from 0x00c216 to 0x01303f (0x6e2a bytes) 000063 1746156694.186092 MIRROR_EXTEND to 2 mirrors lfs mirror mirror: fail to pread 0-4194304 of mirror 2: Invalid argument (22) lfs verify: failed to read data from mirror 2: Invalid argument. mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 22 lfs mirror mirror: fail to pread 0-4194304 of mirror 1: Invalid argument (22) lfs verify: failed to read data from mirror 1: Invalid argument. lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1f57d: skipped LOG DUMP (56 total operations): 1: 1746156693.456700 FALLOC 0x259e9 thru 0x25bb6 (0x001cd bytes)EXTENDING 2: 1746156693.470756 MAPREAD 0x19302 thru 0x25bb5 (0x0c8b4 bytes) 3: 1746156693.495321 ZERO 0x058f3 thru 0x139f5 (0x0e103 bytes) 4: 1746156693.500767 TRUNC_DN 0x25bb6 to 0x05ccd 5: 1746156693.523246 READ 0x0072c thru 0x05ccc (0x055a1 bytes) 6: 1746156693.528824 WRITE 0x39ef7 thru 0x3ffff (0x06109 bytes) HOLE 7: 1746156693.529834 FALLOC 0x0a4ed thru 0x0e583 (0x04096 bytes)INTERIOR 8: 1746156693.533919 FALLOC 0x373fb thru 0x3c45f (0x05064 bytes)INTERIOR 9: 1746156693.537698 WRITE 0x2ab84 thru 0x2f26f (0x046ec bytes) 10: 1746156693.542857 WRITE 0x13a3e thru 0x16526 (0x02ae9 bytes) 11: 1746156693.549080 WRITE 0x1bc5e thru 0x2a1da (0x0e57d bytes) 12: 1746156693.550166 MAPWRITE 0x051b0 thru 0x0b164 (0x05fb5 bytes) 13: 1746156693.592036 READ 0x3fae3 thru 0x3ffff (0x0051d bytes) 14: 1746156693.592488 WRITE 0x17a1d thru 0x235ea (0x0bbce bytes) 15: 1746156693.593572 PUNCH 0x19d78 thru 0x293ec (0x0f675 bytes) 16: 1746156693.613033 WRITE 0x224be thru 0x28e4e (0x06991 bytes) 17: 1746156693.643339 MAPWRITE 0x3a78f thru 0x3b773 (0x00fe5 bytes) 18: 1746156693.678666 READ 0x312f2 thru 0x3d183 (0x0be92 bytes) 19: 1746156693.681894 MAPWRITE 0x06363 thru 0x13e77 (0x0db15 bytes) 20: 1746156693.726297 MAPWRITE 0x38c67 thru 0x3f90c (0x06ca6 bytes) 21: 1746156693.755569 PUNCH 0x3d43b thru 0x3ffff (0x02bc5 bytes) 22: 1746156693.764117 WRITE 0x2fb03 thru 0x316b7 (0x01bb5 bytes) 23: 1746156693.766281 MAPWRITE 0x3c50a thru 0x3ffff (0x03af6 bytes) 24: 1746156693.800641 MAPREAD 0x32c04 thru 0x3b93a (0x08d37 bytes) 25: 1746156693.802067 WRITE 0x03fb2 thru 0x085f0 (0x0463f bytes) 26: 1746156693.804673 FALLOC 0x2575c thru 0x26100 (0x009a4 bytes)INTERIOR 27: 1746156693.809430 MAPREAD 0x0651d thru 0x09f4b (0x03a2f bytes) 28: 1746156693.810347 PUNCH 0x00e1e thru 0x060c0 (0x052a3 bytes) 29: 1746156693.827470 MAPWRITE 0x2c2ff thru 0x334a1 (0x071a3 bytes) 30: 1746156693.858822 PUNCH 0x001d7 thru 0x01916 (0x01740 bytes) 31: 1746156693.864591 READ 0x1435f thru 0x18898 (0x0453a bytes) 32: 1746156693.867542 TRUNC_DN 0x40000 to 0x071e7 33: 1746156693.881440 MAPWRITE 0x26f76 thru 0x2acb7 (0x03d42 bytes) 34: 1746156693.930952 PUNCH 0x0a326 thru 0x0d75a (0x03435 bytes) 35: 1746156693.935771 TRUNC_DN 0x2acb8 to 0x1cd28 36: 1746156693.956167 ZERO 0x01fd6 thru 0x0c507 (0x0a532 bytes) 37: 1746156693.961271 WRITE 0x2c108 thru 0x30fc8 (0x04ec1 bytes) HOLE 38: 1746156693.963480 ZERO 0x2ff3a thru 0x30fc8 (0x0108f bytes) 39: 1746156693.977228 READ 0x114c2 thru 0x20081 (0x0ebc0 bytes) 40: 1746156693.985985 PUNCH 0x030af thru 0x0497f (0x018d1 bytes) 41: 1746156693.991221 WRITE 0x27258 thru 0x32320 (0x0b0c9 bytes) EXTEND 42: 1746156693.992479 FALLOC 0x3dd28 thru 0x40000 (0x022d8 bytes)EXTENDING 43: 1746156693.996478 TRUNC_DN 0x40000 to 0x27609 44: 1746156694.012236 READ 0x1da56 thru 0x23b14 (0x060bf bytes) 45: 1746156694.015922 FALLOC 0x32bc6 thru 0x373dc (0x04816 bytes)PAST_EOF 46: 1746156694.027927 PUNCH 0x119e9 thru 0x1f214 (0x0d82c bytes) 47: 1746156694.038166 PUNCH 0x0ee3f thru 0x174c8 (0x0868a bytes) 48: 1746156694.050501 WRITE 0x265ab thru 0x301b3 (0x09c09 bytes) EXTEND 49: 1746156694.052228 TRUNC_DN 0x301b4 to 0x1554f 50: 1746156694.069988 MAPREAD 0x032b1 thru 0x12bee (0x0f93e bytes) 51: 1746156694.086167 MAPWRITE 0x13d37 thru 0x1f57c (0x0b846 bytes) 52: 1746156694.135449 ZERO 0x02670 thru 0x10016 (0x0d9a7 bytes) 53: 1746156694.141462 ZERO 0x15643 thru 0x161df (0x00b9d bytes) 54: 1746156694.146887 MAPWRITE 0x081f1 thru 0x0ec17 (0x06a27 bytes) 55: 1746156694.179745 ZERO 0x0c216 thru 0x1303f (0x06e2a bytes) 56: 1746156694.186092 MIRROR_EXTEND to 2 mirrors Using seed 6693 Correct content saved for comparison (maybe hexdump "/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr" vs "/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr.fsxgood") sanity-flr test_70a: @@@@@@ FAIL: fsx FLR file /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr failed Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7249:error() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:3351:test_70a() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7622:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7685:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7503:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:3353:main() Dumping lctl log to /tmp/testlogs//sanity-flr.test_70a.*.1746156695.log rsync: chown "/tmp/testlogs/.sanity-flr.test_70a.debug_log.oleg447-server.1746156695.log.7x0BVj" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-flr.test_70a.dmesg.oleg447-server.1746156695.log.mIJwPJ" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2]