== sanity-flr test 70a: flr mode fsx test ================ 21:00:26 (1746147626) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70a.sanity-flr Seed set to 7627 fallocating to largest ever: 0x2043c 000001 1746147627.565257 fallocate 0x01a12e thru 0x02043b (0x630e bytes) 000002 1746147627.572379 trunc from 0x02043c to 0x019242 (0x71f9 bytes) 000003 1746147627.582368 punch from 0x00cff2 to 0x00d4dc (0x4eb bytes) 000004 1746147627.585086 read 0x019009 thru 0x019242 (0x23a bytes) 000005 1746147627.594111 mapread 0x015009 thru 0x019242 (0x423a bytes) 000006 1746147627.609412 trunc from 0x019243 to 0x00478d (0x14ab5 bytes) 000007 1746147627.624224 read 0x0010c8 thru 0x002770 (0x16a9 bytes) 000008 1746147627.630828 read 0x0031b3 thru 0x00478d (0x15db bytes) 000009 1746147627.631274 trunc from 0x003b70 to 0x00478d (0xc1e bytes) 000010 1746147627.634134 punch from 0x003dd3 to 0x00478d (0x9bb bytes) 000011 1746147627.637711 mapwrite 0x003334 thru 0x00dbb5 (0xa882 bytes) 000012 1746147627.669421 mapread 0x00c759 thru 0x00dbb5 (0x145d bytes) 000014 1746147627.670192 mapwrite 0x025fb6 thru 0x034180 (0xe1cb bytes) 000015 1746147627.703474 mapread 0x027842 thru 0x0294aa (0x1c69 bytes) fallocating to largest ever: 0x3cf24 000016 1746147627.704452 fallocate 0x02e783 thru 0x03cf23 (0xe7a1 bytes) 000017 1746147627.708216 write 0x02cf3b thru 0x033a01 (0x6ac7 bytes) 000018 1746147627.709024 punch from 0x02ec7f to 0x037b7f (0x8f01 bytes) 000019 1746147627.716212 trunc from 0x03cf24 to 0x00bbb0 (0x31373 bytes) 000020 1746147627.727868 trunc from 0x00bbb1 to 0x0397ff (0x2dc4f bytes) 000021 1746147627.735508 write 0x03bb14 thru 0x03ffff (0x44ec bytes) 000023 1746147627.736666 mapwrite 0x0145a5 thru 0x01ff88 (0xb9e4 bytes) fallocating to largest ever: 0x40000 000024 1746147627.783044 fallocate 0x03566b thru 0x03ffff (0xa995 bytes) 000025 1746147627.789473 MIRROR_EXTEND to 2 mirrors 000026 1746147628.064566 fallocate 0x03ddf9 thru 0x03ffff (0x2207 bytes) 000027 1746147628.096904 write 0x015725 thru 0x025480 (0xfd5c bytes) 000028 1746147628.111008 fallocate 0x009062 thru 0x012766 (0x9705 bytes) 000029 1746147628.116514 mapwrite 0x007488 thru 0x00e905 (0x747e bytes) 000030 1746147628.144219 mapwrite 0x0291a5 thru 0x0320ac (0x8f08 bytes) 000031 1746147628.169741 trunc from 0x00339e to 0x00381c (0x47f bytes) 000032 1746147628.173189 trunc from 0x03b9bc to 0x03ffff (0x4644 bytes) 000033 1746147628.176702 punch from 0x005714 to 0x0078dc (0x21c9 bytes) 000034 1746147628.184848 MIRROR_RESYNC 2 mirrors 000035 1746147628.544018 read 0x01e961 thru 0x022f8e (0x462e bytes) 000036 1746147628.549805 write 0x01ac88 thru 0x01c276 (0x15ef bytes) 000037 1746147628.586774 mapread 0x0397cb thru 0x03ffff (0x6835 bytes) 000038 1746147628.594822 fallocate 0x02d4ae thru 0x033696 (0x61e9 bytes) 000039 1746147628.600213 write 0x0177b6 thru 0x01940e (0x1c59 bytes) 000041 1746147628.613804 MIRROR_EXTEND to 3 mirrors 000042 1746147629.066687 mapread 0x01293b thru 0x01ab04 (0x81ca bytes) 000043 1746147629.091021 write 0x022854 thru 0x025a39 (0x31e6 bytes) 000044 1746147629.128247 write 0x016542 thru 0x016eb5 (0x974 bytes) 000045 1746147629.134345 mapwrite 0x0036b0 thru 0x006dbe (0x370f bytes) 000046 1746147629.154126 trunc from 0x040000 to 0x00c8e0 (0x3371f bytes) 000047 1746147629.180637 trunc from 0x00a975 to 0x00c8e0 (0x1f6c bytes) 000049 1746147629.190651 read 0x000d0d thru 0x009efe (0x91f2 bytes) 000050 1746147629.200226 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x554:0x0]: Invalid argument (22) 000051 1746147629.319976 trunc from 0x00968d to 0x00c8e0 (0x3254 bytes) 000052 1746147629.323599 write 0x023905 thru 0x031447 (0xdb43 bytes) 000053 1746147629.330813 mapread 0x030cd3 thru 0x031447 (0x775 bytes) 000054 1746147629.332280 write 0x02c0c0 thru 0x038c15 (0xcb56 bytes) 000055 1746147629.333520 trunc from 0x0231e3 to 0x025299 (0x20b7 bytes) 000056 1746147629.345354 trunc from 0x00bad1 to 0x01adf3 (0xf323 bytes) 000057 1746147629.351200 MIRROR_EXTEND to 3 mirrors lfs mirror mirror: fail to pread 0-4194304 of mirror 3: Invalid argument (22) lfs verify: failed to read data from mirror 3: Invalid argument. mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 22 lfs mirror mirror: fail to pread 0-4194304 of mirror 1: Invalid argument (22) lfs verify: failed to read data from mirror 1: Invalid argument. lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38c16: skipped LOG DUMP (53 total operations): 1: 1746147627.565257 FALLOC 0x1a12e thru 0x2043c (0x0630e bytes)EXTENDING 2: 1746147627.572379 TRUNC_DN 0x2043c to 0x19243 3: 1746147627.582368 PUNCH 0x0cff2 thru 0x0d4dc (0x004eb bytes) 4: 1746147627.585086 READ 0x19009 thru 0x19242 (0x0023a bytes) 5: 1746147627.594111 MAPREAD 0x15009 thru 0x19242 (0x0423a bytes) 6: 1746147627.609412 TRUNC_DN 0x19243 to 0x0478e 7: 1746147627.624224 READ 0x010c8 thru 0x02770 (0x016a9 bytes) 8: 1746147627.630828 READ 0x031b3 thru 0x0478d (0x015db bytes) 9: 1746147627.631274 ZERO 0x03b70 thru 0x0478d (0x00c1e bytes) 10: 1746147627.634134 PUNCH 0x03dd3 thru 0x0478d (0x009bb bytes) 11: 1746147627.637711 MAPWRITE 0x03334 thru 0x0dbb5 (0x0a882 bytes) 12: 1746147627.669421 MAPREAD 0x0c759 thru 0x0dbb5 (0x0145d bytes) 13: 1746147627.670192 MAPWRITE 0x25fb6 thru 0x34180 (0x0e1cb bytes) 14: 1746147627.703474 MAPREAD 0x27842 thru 0x294aa (0x01c69 bytes) 15: 1746147627.704452 FALLOC 0x2e783 thru 0x3cf24 (0x0e7a1 bytes)EXTENDING 16: 1746147627.708216 WRITE 0x2cf3b thru 0x33a01 (0x06ac7 bytes) 17: 1746147627.709024 PUNCH 0x2ec7f thru 0x37b7f (0x08f01 bytes) 18: 1746147627.716212 TRUNC_DN 0x3cf24 to 0x0bbb1 19: 1746147627.727868 TRUNC_UP 0x0bbb1 to 0x39800 20: 1746147627.735508 WRITE 0x3bb14 thru 0x3ffff (0x044ec bytes) HOLE 21: 1746147627.736666 MAPWRITE 0x145a5 thru 0x1ff88 (0x0b9e4 bytes) 22: 1746147627.783044 FALLOC 0x3566b thru 0x40000 (0x0a995 bytes)INTERIOR 23: 1746147627.789473 MIRROR_EXTEND to 2 mirrors 24: 1746147628.064566 FALLOC 0x3ddf9 thru 0x40000 (0x02207 bytes)INTERIOR 25: 1746147628.096904 WRITE 0x15725 thru 0x25480 (0x0fd5c bytes) 26: 1746147628.111008 FALLOC 0x09062 thru 0x12767 (0x09705 bytes)INTERIOR 27: 1746147628.116514 MAPWRITE 0x07488 thru 0x0e905 (0x0747e bytes) 28: 1746147628.144219 MAPWRITE 0x291a5 thru 0x320ac (0x08f08 bytes) 29: 1746147628.169741 ZERO 0x0339e thru 0x0381c (0x0047f bytes) 30: 1746147628.173189 ZERO 0x3b9bc thru 0x3ffff (0x04644 bytes) 31: 1746147628.176702 PUNCH 0x05714 thru 0x078dc (0x021c9 bytes) 32: 1746147628.184848 MIRROR_RESYNC 2 mirrors 33: 1746147628.544018 READ 0x1e961 thru 0x22f8e (0x0462e bytes) 34: 1746147628.549805 WRITE 0x1ac88 thru 0x1c276 (0x015ef bytes) 35: 1746147628.586774 MAPREAD 0x397cb thru 0x3ffff (0x06835 bytes) 36: 1746147628.594822 FALLOC 0x2d4ae thru 0x33697 (0x061e9 bytes)INTERIOR 37: 1746147628.600213 WRITE 0x177b6 thru 0x1940e (0x01c59 bytes) 38: 1746147628.613804 MIRROR_EXTEND to 3 mirrors 39: 1746147629.066687 MAPREAD 0x1293b thru 0x1ab04 (0x081ca bytes) 40: 1746147629.091021 WRITE 0x22854 thru 0x25a39 (0x031e6 bytes) 41: 1746147629.128247 WRITE 0x16542 thru 0x16eb5 (0x00974 bytes) 42: 1746147629.134345 MAPWRITE 0x036b0 thru 0x06dbe (0x0370f bytes) 43: 1746147629.154126 TRUNC_DN 0x40000 to 0x0c8e1 44: 1746147629.180637 ZERO 0x0a975 thru 0x0c8e0 (0x01f6c bytes) 45: 1746147629.190651 READ 0x00d0d thru 0x09efe (0x091f2 bytes) 46: 1746147629.200226 MIRROR_SPLIT mirror 2 to 2 mirrors 47: 1746147629.319976 ZERO 0x0968d thru 0x0c8e0 (0x03254 bytes) 48: 1746147629.323599 WRITE 0x23905 thru 0x31447 (0x0db43 bytes) HOLE 49: 1746147629.330813 MAPREAD 0x30cd3 thru 0x31447 (0x00775 bytes) 50: 1746147629.332280 WRITE 0x2c0c0 thru 0x38c15 (0x0cb56 bytes) EXTEND 51: 1746147629.333520 ZERO 0x231e3 thru 0x25299 (0x020b7 bytes) 52: 1746147629.345354 ZERO 0x0bad1 thru 0x1adf3 (0x0f323 bytes) 53: 1746147629.351200 MIRROR_EXTEND to 3 mirrors Using seed 7627 Correct content saved for comparison (maybe hexdump "/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr" vs "/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr.fsxgood") sanity-flr test_70a: @@@@@@ FAIL: fsx FLR file /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr failed Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7249:error() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:3351:test_70a() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7622:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7685:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7503:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:3353:main() Dumping lctl log to /tmp/testlogs//sanity-flr.test_70a.*.1746147630.log rsync: chown "/tmp/testlogs/.sanity-flr.test_70a.debug_log.oleg249-server.1746147630.log.kvTyj9" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-flr.test_70a.dmesg.oleg249-server.1746147630.log.JU6fBe" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2]