== sanity-flr test 70a: flr mode fsx test ================ 18:01:05 (1746136865) striped dir -i0 -c2 -H all_char /mnt/lustre/d70a.sanity-flr Seed set to 6866 000001 1746136866.822623 mapwrite 0x02e8d7 thru 0x02f7da (0xf04 bytes) 000002 1746136866.859514 punch from 0x02487a to 0x02a5f8 (0x5d7f bytes) 000003 1746136866.863013 punch from 0x0160cc to 0x01d9b9 (0x78ee bytes) 000004 1746136866.865750 read 0x007ec1 thru 0x00b938 (0x3a78 bytes) 000006 1746136866.870676 mapread 0x00af08 thru 0x00fdb8 (0x4eb1 bytes) fallocating to largest ever: 0x139c3 000007 1746136866.876049 fallocate 0x00edd7 thru 0x0139c2 (0x4bec bytes) 000008 1746136866.881513 trunc from 0x02f7db to 0x000706 (0x2f0d4 bytes) 000009 1746136866.896667 write 0x012362 thru 0x018b76 (0x6815 bytes) 000010 1746136866.897716 write 0x00c79c thru 0x01463d (0x7ea2 bytes) truncating to largest ever: 0x161c1 000011 1746136866.908595 trunc from 0x018b77 to 0x0161c0 (0x29b6 bytes) 000012 1746136866.927758 write 0x015b49 thru 0x021d97 (0xc24f bytes) fallocating to largest ever: 0x24b52 000013 1746136866.933378 fallocate 0x02235f thru 0x024b51 (0x27f3 bytes) 000014 1746136866.938164 mapwrite 0x00cda8 thru 0x011d9e (0x4ff7 bytes) 000015 1746136866.955010 write 0x025725 thru 0x02cb18 (0x73f4 bytes) 000017 1746136866.961733 mapread 0x015857 thru 0x0175dc (0x1d86 bytes) 000018 1746136866.962363 punch from 0x01131b to 0x016743 (0x5429 bytes) 000019 1746136866.966126 mapwrite 0x01c221 thru 0x027c05 (0xb9e5 bytes) 000020 1746136866.992805 punch from 0x00e45b to 0x014fb5 (0x6b5b bytes) 000021 1746136867.002993 write 0x039ca1 thru 0x03b8de (0x1c3e bytes) 000022 1746136867.003763 mapwrite 0x01a318 thru 0x027d77 (0xda60 bytes) 000023 1746136867.019991 read 0x02bc43 thru 0x03a311 (0xe6cf bytes) 000024 1746136867.026168 trunc from 0x01234d to 0x0174d7 (0x518b bytes) 000025 1746136867.030224 mapwrite 0x037c1e thru 0x03ffff (0x83e2 bytes) 000026 1746136867.062733 trunc from 0x040000 to 0x00fcdb (0x30324 bytes) 000027 1746136867.077647 trunc from 0x00fcdc to 0x01fc78 (0xff9d bytes) 000028 1746136867.105851 mapread 0x009a39 thru 0x016b9a (0xd162 bytes) 000029 1746136867.131838 trunc from 0x01fc79 to 0x012ea4 (0xcdd4 bytes) 000030 1746136867.143079 punch from 0x00409a to 0x00dc8a (0x9bf1 bytes) truncating to largest ever: 0x2da67 000031 1746136867.147159 trunc from 0x012ea5 to 0x02da66 (0x1abc2 bytes) 000032 1746136867.160603 trunc from 0x02da67 to 0x022fe2 (0xaa84 bytes) 000033 1746136867.173369 write 0x03844e thru 0x03d625 (0x51d8 bytes) 000035 1746136867.174877 read 0x010bfb thru 0x0177e8 (0x6bee bytes) fallocating to largest ever: 0x3868f 000036 1746136867.184154 fallocate 0x02c02c thru 0x03868e (0xc663 bytes) 000037 1746136867.188924 mapwrite 0x032b60 thru 0x03df88 (0xb429 bytes) 000039 1746136867.210073 mapwrite 0x00cf23 thru 0x0142cf (0x73ad bytes) 000040 1746136867.223668 mapread 0x0158dc thru 0x0188a5 (0x2fca bytes) 000041 1746136867.230934 trunc from 0x01368a to 0x020f23 (0xd89a bytes) 000042 1746136867.237854 mapwrite 0x02805d thru 0x033c9c (0xbc40 bytes) 000043 1746136867.265405 punch from 0x02f6bc to 0x03985d (0xa1a2 bytes) 000045 1746136867.269401 read 0x011491 thru 0x0190aa (0x7c1a bytes) 000046 1746136867.275497 mapwrite 0x00483b thru 0x00a445 (0x5c0b bytes) 000050 1746136867.287996 read 0x01888a thru 0x02505f (0xc7d6 bytes) zero_range to largest ever: 0x3d1c6 000051 1746136867.288487 trunc from 0x039ba4 to 0x03d1c5 (0x3622 bytes) 000052 1746136867.292983 trunc from 0x03df89 to 0x024f97 (0x18ff1 bytes) 000053 1746136867.302381 write 0x018283 thru 0x01dec0 (0x5c3e bytes) 000054 1746136867.303258 fallocate 0x030625 thru 0x034f9c (0x4978 bytes) 000055 1746136867.308018 mapread 0x00c817 thru 0x013b14 (0x72fe bytes) 000056 1746136867.308997 punch from 0x00c7d7 to 0x01482a (0x8054 bytes) 000057 1746136867.313249 write 0x02ff58 thru 0x03c065 (0xc10e bytes) 000058 1746136867.314410 read 0x0331ab thru 0x03ab3b (0x7991 bytes) 000059 1746136867.315145 mapread 0x01319a thru 0x013374 (0x1db bytes) 000060 1746136867.327744 write 0x023259 thru 0x027876 (0x461e bytes) 000062 1746136867.329112 read 0x010318 thru 0x01f73d (0xf426 bytes) 000063 1746136867.329675 read 0x013a01 thru 0x01715d (0x375d bytes) 000064 1746136867.330042 trunc from 0x03c066 to 0x029e6d (0x121f8 bytes) 000065 1746136867.341087 mapwrite 0x03c18a thru 0x03f8d2 (0x3749 bytes) 000066 1746136867.397434 trunc from 0x03f8d3 to 0x012953 (0x2cf7f bytes) 000067 1746136867.412687 fallocate 0x00bbb6 thru 0x018d4e (0xd199 bytes) 000068 1746136867.416485 punch from 0x0126fb to 0x0130a3 (0x9a9 bytes) 000069 1746136867.421418 MIRROR_EXTEND to 2 mirrors lfs mirror mirror: fail to pread 0-4194304 of mirror 2: Invalid argument (22) lfs verify: failed to read data from mirror 2: Invalid argument. mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 22 lfs mirror mirror: fail to pread 0-4194304 of mirror 1: Invalid argument (22) lfs verify: failed to read data from mirror 1: Invalid argument. lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x18d4f: skipped LOG DUMP (60 total operations): 1: 1746136866.822623 MAPWRITE 0x2e8d7 thru 0x2f7da (0x00f04 bytes) 2: 1746136866.859514 PUNCH 0x2487a thru 0x2a5f8 (0x05d7f bytes) 3: 1746136866.863013 PUNCH 0x160cc thru 0x1d9b9 (0x078ee bytes) 4: 1746136866.865750 READ 0x07ec1 thru 0x0b938 (0x03a78 bytes) 5: 1746136866.870676 MAPREAD 0x0af08 thru 0x0fdb8 (0x04eb1 bytes) 6: 1746136866.876049 FALLOC 0x0edd7 thru 0x139c3 (0x04bec bytes)INTERIOR 7: 1746136866.881513 TRUNC_DN 0x2f7db to 0x00707 8: 1746136866.896667 WRITE 0x12362 thru 0x18b76 (0x06815 bytes) HOLE 9: 1746136866.897716 WRITE 0x0c79c thru 0x1463d (0x07ea2 bytes) 10: 1746136866.908595 TRUNC_DN 0x18b77 to 0x161c1 11: 1746136866.927758 WRITE 0x15b49 thru 0x21d97 (0x0c24f bytes) EXTEND 12: 1746136866.933378 FALLOC 0x2235f thru 0x24b52 (0x027f3 bytes)PAST_EOF 13: 1746136866.938164 MAPWRITE 0x0cda8 thru 0x11d9e (0x04ff7 bytes) 14: 1746136866.955010 WRITE 0x25725 thru 0x2cb18 (0x073f4 bytes) HOLE 15: 1746136866.961733 MAPREAD 0x15857 thru 0x175dc (0x01d86 bytes) 16: 1746136866.962363 PUNCH 0x1131b thru 0x16743 (0x05429 bytes) 17: 1746136866.966126 MAPWRITE 0x1c221 thru 0x27c05 (0x0b9e5 bytes) 18: 1746136866.992805 PUNCH 0x0e45b thru 0x14fb5 (0x06b5b bytes) 19: 1746136867.002993 WRITE 0x39ca1 thru 0x3b8de (0x01c3e bytes) HOLE 20: 1746136867.003763 MAPWRITE 0x1a318 thru 0x27d77 (0x0da60 bytes) 21: 1746136867.019991 READ 0x2bc43 thru 0x3a311 (0x0e6cf bytes) 22: 1746136867.026168 ZERO 0x1234d thru 0x174d7 (0x0518b bytes) 23: 1746136867.030224 MAPWRITE 0x37c1e thru 0x3ffff (0x083e2 bytes) 24: 1746136867.062733 TRUNC_DN 0x40000 to 0x0fcdc 25: 1746136867.077647 TRUNC_UP 0x0fcdc to 0x1fc79 26: 1746136867.105851 MAPREAD 0x09a39 thru 0x16b9a (0x0d162 bytes) 27: 1746136867.131838 TRUNC_DN 0x1fc79 to 0x12ea5 28: 1746136867.143079 PUNCH 0x0409a thru 0x0dc8a (0x09bf1 bytes) 29: 1746136867.147159 TRUNC_UP 0x12ea5 to 0x2da67 30: 1746136867.160603 TRUNC_DN 0x2da67 to 0x22fe3 31: 1746136867.173369 WRITE 0x3844e thru 0x3d625 (0x051d8 bytes) HOLE 32: 1746136867.174877 READ 0x10bfb thru 0x177e8 (0x06bee bytes) 33: 1746136867.184154 FALLOC 0x2c02c thru 0x3868f (0x0c663 bytes)INTERIOR 34: 1746136867.188924 MAPWRITE 0x32b60 thru 0x3df88 (0x0b429 bytes) 35: 1746136867.210073 MAPWRITE 0x0cf23 thru 0x142cf (0x073ad bytes) 36: 1746136867.223668 MAPREAD 0x158dc thru 0x188a5 (0x02fca bytes) 37: 1746136867.230934 ZERO 0x1368a thru 0x20f23 (0x0d89a bytes) 38: 1746136867.237854 MAPWRITE 0x2805d thru 0x33c9c (0x0bc40 bytes) 39: 1746136867.265405 PUNCH 0x2f6bc thru 0x3985d (0x0a1a2 bytes) 40: 1746136867.269401 READ 0x11491 thru 0x190aa (0x07c1a bytes) 41: 1746136867.275497 MAPWRITE 0x0483b thru 0x0a445 (0x05c0b bytes) 42: 1746136867.287996 READ 0x1888a thru 0x2505f (0x0c7d6 bytes) 43: 1746136867.288487 ZERO 0x39ba4 thru 0x3d1c5 (0x03622 bytes) 44: 1746136867.292983 TRUNC_DN 0x3df89 to 0x24f98 45: 1746136867.302381 WRITE 0x18283 thru 0x1dec0 (0x05c3e bytes) 46: 1746136867.303258 FALLOC 0x30625 thru 0x34f9d (0x04978 bytes)PAST_EOF 47: 1746136867.308018 MAPREAD 0x0c817 thru 0x13b14 (0x072fe bytes) 48: 1746136867.308997 PUNCH 0x0c7d7 thru 0x1482a (0x08054 bytes) 49: 1746136867.313249 WRITE 0x2ff58 thru 0x3c065 (0x0c10e bytes) HOLE 50: 1746136867.314410 READ 0x331ab thru 0x3ab3b (0x07991 bytes) 51: 1746136867.315145 MAPREAD 0x1319a thru 0x13374 (0x001db bytes) 52: 1746136867.327744 WRITE 0x23259 thru 0x27876 (0x0461e bytes) 53: 1746136867.329112 READ 0x10318 thru 0x1f73d (0x0f426 bytes) 54: 1746136867.329675 READ 0x13a01 thru 0x1715d (0x0375d bytes) 55: 1746136867.330042 TRUNC_DN 0x3c066 to 0x29e6e 56: 1746136867.341087 MAPWRITE 0x3c18a thru 0x3f8d2 (0x03749 bytes) 57: 1746136867.397434 TRUNC_DN 0x3f8d3 to 0x12954 58: 1746136867.412687 FALLOC 0x0bbb6 thru 0x18d4f (0x0d199 bytes)EXTENDING 59: 1746136867.416485 PUNCH 0x126fb thru 0x130a3 (0x009a9 bytes) 60: 1746136867.421418 MIRROR_EXTEND to 2 mirrors Using seed 6866 Correct content saved for comparison (maybe hexdump "/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr" vs "/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr.fsxgood") sanity-flr test_70a: @@@@@@ FAIL: fsx FLR file /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr failed Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7249:error() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:3351:test_70a() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7622:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7685:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7503:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:3353:main() Dumping lctl log to /tmp/testlogs//sanity-flr.test_70a.*.1746136868.log rsync: chown "/tmp/testlogs/.sanity-flr.test_70a.debug_log.oleg349-server.1746136868.log.8JXj0l" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-flr.test_70a.dmesg.oleg349-server.1746136868.log.XyvrG9" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2]