== sanity-flr test 70a: flr mode fsx test ================ 18:42:37 (1746139357) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70a.sanity-flr Seed set to 9358 skipping zero length punch hole truncating to largest ever: 0x19ce4 000002 1746139358.881508 trunc from 00000000 to 0x019ce3 (0x19ce4 bytes) truncating to largest ever: 0x33bef 000003 1746139358.893368 trunc from 0x019ce4 to 0x033bee (0x19f0b bytes) 000004 1746139358.903297 trunc from 0x002515 to 0x00ec67 (0xc753 bytes) 000005 1746139358.906754 trunc from 0x033bef to 0x002591 (0x3165d bytes) 000007 1746139358.920180 fallocate 0x01bc54 thru 0x02b5b1 (0xf95e bytes) 000008 1746139358.925037 mapwrite 0x00f67a thru 0x01e759 (0xf0e0 bytes) 000009 1746139359.025898 mapread 0x015c45 thru 0x019d52 (0x410e bytes) 000011 1746139359.027986 punch from 0x010699 to 0x016214 (0x5b7c bytes) 000013 1746139359.044434 mapwrite 0x00727d thru 0x01343e (0xc1c2 bytes) 000014 1746139359.075516 read 0x005ae5 thru 0x00d284 (0x77a0 bytes) 000015 1746139359.076819 trunc from 0x01251f to 0x0168a8 (0x438a bytes) 000017 1746139359.081809 trunc from 0x00be8b to 0x013228 (0x739e bytes) 000019 1746139359.085885 mapread 0x010508 thru 0x01c2be (0xbdb7 bytes) 000020 1746139359.093718 fallocate 0x013f13 thru 0x01578a (0x1878 bytes) 000021 1746139359.099799 read 0x0099b9 thru 0x00eef7 (0x553f bytes) 000022 1746139359.106604 mapwrite 0x016642 thru 0x01d0b0 (0x6a6f bytes) 000023 1746139359.128350 read 0x00f2d4 thru 0x0114ab (0x21d8 bytes) 000024 1746139359.129200 trunc from 0x001f2d to 0x00c992 (0xaa66 bytes) 000025 1746139359.134899 mapread 0x0053cd thru 0x00f63e (0xa272 bytes) 000027 1746139359.144720 read 0x019a7d thru 0x01d08c (0x3610 bytes) 000028 1746139359.145199 trunc from 0x0116a5 to 0x01e759 (0xd0b5 bytes) 000029 1746139359.156478 read 0x00bcf6 thru 0x016ce1 (0xafec bytes) 000030 1746139359.169856 mapwrite 0x02c763 thru 0x031d9f (0x563d bytes) 000032 1746139359.211690 write 0x022959 thru 0x02f4fd (0xcba5 bytes) 000033 1746139359.212834 trunc from 0x031da0 to 0x018faa (0x18df5 bytes) 000034 1746139359.230218 write 0x033e22 thru 0x03e7fc (0xa9db bytes) 000035 1746139359.231604 punch from 0x014897 to 0x019437 (0x4ba1 bytes) 000036 1746139359.238316 trunc from 0x03e7fd to 0x0144ed (0x2a30f bytes) 000037 1746139359.256433 trunc from 0x0144ee to 0x032f6b (0x1ea7e bytes) 000038 1746139359.276751 punch from 0x01a89d to 0x02961b (0xed7f bytes) 000039 1746139359.283089 trunc from 0x025eea to 0x03002c (0xa143 bytes) 000041 1746139359.290092 trunc from 0x032f6c to 0x029efb (0x9070 bytes) 000042 1746139359.306279 trunc from 0x017c03 to 0x01c709 (0x4b07 bytes) 000043 1746139359.313504 mapwrite 0x011c6c thru 0x01bdb5 (0xa14a bytes) 000044 1746139359.370909 trunc from 0x014e41 to 0x0153a1 (0x561 bytes) 000045 1746139359.375845 trunc from 0x029efc to 0x018e70 (0x1108b bytes) truncating to largest ever: 0x38733 000046 1746139359.391333 trunc from 0x018e71 to 0x038732 (0x1f8c2 bytes) 000047 1746139359.403580 mapread 0x002fb2 thru 0x00fe21 (0xce70 bytes) 000048 1746139359.405174 mapwrite 0x021df6 thru 0x0295bc (0x77c7 bytes) 000049 1746139359.453981 mapread 0x0170d7 thru 0x01cb6a (0x5a94 bytes) 000050 1746139359.468482 mapwrite 0x00e763 thru 0x0122e8 (0x3b86 bytes) 000051 1746139359.486556 trunc from 0x006203 to 0x00aa08 (0x4806 bytes) 000053 1746139359.493196 punch from 0x01db64 to 0x02a49f (0xc93c bytes) 000056 1746139359.498307 read 0x003f07 thru 0x00c7d3 (0x88cd bytes) 000057 1746139359.528241 read 0x002582 thru 0x00cce6 (0xa765 bytes) 000058 1746139359.529332 read 0x01a825 thru 0x01e5f1 (0x3dcd bytes) 000059 1746139359.529788 write 0x003b05 thru 0x01059b (0xca97 bytes) 000060 1746139359.530824 mapwrite 0x01cdd4 thru 0x01f8ff (0x2b2c bytes) 000062 1746139359.552580 punch from 0x011ba9 to 0x020a0c (0xee64 bytes) 000063 1746139359.561549 mapwrite 0x016cee thru 0x0187ec (0x1aff bytes) 000066 1746139359.577723 write 0x001198 thru 0x00755f (0x63c8 bytes) 000067 1746139359.578593 fallocate 0x00878e thru 0x00dbe7 (0x545a bytes) 000068 1746139359.584360 punch from 0x030420 to 0x038732 (0x8313 bytes) 000069 1746139359.589016 punch from 0x024596 to 0x02a482 (0x5eed bytes) 000070 1746139359.591914 mapwrite 0x02e60f thru 0x0381f2 (0x9be4 bytes) 000071 1746139359.630403 punch from 0x01e869 to 0x023a0f (0x51a7 bytes) fallocating to largest ever: 0x3a129 000072 1746139359.634375 fallocate 0x036304 thru 0x03a128 (0x3e25 bytes) 000074 1746139359.637497 trunc from 0x023c42 to 0x027892 (0x3c51 bytes) 000075 1746139359.642320 read 0x0234c7 thru 0x02a54a (0x7084 bytes) 000076 1746139359.648997 mapread 0x00ff0b thru 0x01f9ba (0xfab0 bytes) 000077 1746139359.650035 trunc from 0x03762b to 0x038732 (0x1108 bytes) 000078 1746139359.656859 write 0x0021d2 thru 0x010a86 (0xe8b5 bytes) 000079 1746139359.658954 punch from 0x002956 to 0x0082fc (0x59a7 bytes) 000081 1746139359.676089 punch from 0x0311c2 to 0x038732 (0x7571 bytes) 000082 1746139359.684301 read 0x02809a thru 0x034dc7 (0xcd2e bytes) 000083 1746139359.690879 write 0x037f9c thru 0x03f737 (0x779c bytes) 000085 1746139359.693761 trunc from 0x03f738 to 0x02cd50 (0x129e7 bytes) 000086 1746139359.719481 trunc from 0x02cd51 to 0x0288cc (0x4484 bytes) 000087 1746139359.737764 trunc from 0x0288cd to 0x00e086 (0x1a846 bytes) 000088 1746139359.762571 trunc from 0x00e087 to 0x01c0c8 (0xe042 bytes) 000090 1746139359.796030 trunc from 0x004625 to 0x010d50 (0xc72c bytes) 000092 1746139359.802643 mapread 0x018d91 thru 0x01c0c8 (0x3338 bytes) 000094 1746139359.834707 punch from 0x01b36b to 0x01c0c8 (0xd5e bytes) 000095 1746139359.847205 trunc from 0x0084aa to 0x010f7d (0x8ad4 bytes) 000096 1746139359.859588 MIRROR_EXTEND to 2 mirrors lfs mirror mirror: fail to pread 0-4194304 of mirror 2: Invalid argument (22) lfs verify: failed to read data from mirror 2: Invalid argument. mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 22 lfs mirror mirror: fail to pread 0-4194304 of mirror 1: Invalid argument (22) lfs verify: failed to read data from mirror 1: Invalid argument. lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1c0c9: skipped LOG DUMP (76 total operations): 1: 1746139358.877328 SKIPPED (no operation) 2: 1746139358.881508 TRUNC_UP 0x00000 to 0x19ce4 3: 1746139358.893368 TRUNC_UP 0x19ce4 to 0x33bef 4: 1746139358.903297 ZERO 0x02515 thru 0x0ec67 (0x0c753 bytes) 5: 1746139358.906754 TRUNC_DN 0x33bef to 0x02592 6: 1746139358.920180 FALLOC 0x1bc54 thru 0x2b5b2 (0x0f95e bytes)PAST_EOF 7: 1746139358.925037 MAPWRITE 0x0f67a thru 0x1e759 (0x0f0e0 bytes) 8: 1746139359.025898 MAPREAD 0x15c45 thru 0x19d52 (0x0410e bytes) 9: 1746139359.027986 PUNCH 0x10699 thru 0x16214 (0x05b7c bytes) 10: 1746139359.044434 MAPWRITE 0x0727d thru 0x1343e (0x0c1c2 bytes) 11: 1746139359.075516 READ 0x05ae5 thru 0x0d284 (0x077a0 bytes) 12: 1746139359.076819 ZERO 0x1251f thru 0x168a8 (0x0438a bytes) 13: 1746139359.081809 ZERO 0x0be8b thru 0x13228 (0x0739e bytes) 14: 1746139359.085885 MAPREAD 0x10508 thru 0x1c2be (0x0bdb7 bytes) 15: 1746139359.093718 FALLOC 0x13f13 thru 0x1578b (0x01878 bytes)INTERIOR 16: 1746139359.099799 READ 0x099b9 thru 0x0eef7 (0x0553f bytes) 17: 1746139359.106604 MAPWRITE 0x16642 thru 0x1d0b0 (0x06a6f bytes) 18: 1746139359.128350 READ 0x0f2d4 thru 0x114ab (0x021d8 bytes) 19: 1746139359.129200 ZERO 0x01f2d thru 0x0c992 (0x0aa66 bytes) 20: 1746139359.134899 MAPREAD 0x053cd thru 0x0f63e (0x0a272 bytes) 21: 1746139359.144720 READ 0x19a7d thru 0x1d08c (0x03610 bytes) 22: 1746139359.145199 ZERO 0x116a5 thru 0x1e759 (0x0d0b5 bytes) 23: 1746139359.156478 READ 0x0bcf6 thru 0x16ce1 (0x0afec bytes) 24: 1746139359.169856 MAPWRITE 0x2c763 thru 0x31d9f (0x0563d bytes) 25: 1746139359.211690 WRITE 0x22959 thru 0x2f4fd (0x0cba5 bytes) 26: 1746139359.212834 TRUNC_DN 0x31da0 to 0x18fab 27: 1746139359.230218 WRITE 0x33e22 thru 0x3e7fc (0x0a9db bytes) HOLE 28: 1746139359.231604 PUNCH 0x14897 thru 0x19437 (0x04ba1 bytes) 29: 1746139359.238316 TRUNC_DN 0x3e7fd to 0x144ee 30: 1746139359.256433 TRUNC_UP 0x144ee to 0x32f6c 31: 1746139359.276751 PUNCH 0x1a89d thru 0x2961b (0x0ed7f bytes) 32: 1746139359.283089 ZERO 0x25eea thru 0x3002c (0x0a143 bytes) 33: 1746139359.290092 TRUNC_DN 0x32f6c to 0x29efc 34: 1746139359.306279 ZERO 0x17c03 thru 0x1c709 (0x04b07 bytes) 35: 1746139359.313504 MAPWRITE 0x11c6c thru 0x1bdb5 (0x0a14a bytes) 36: 1746139359.370909 ZERO 0x14e41 thru 0x153a1 (0x00561 bytes) 37: 1746139359.375845 TRUNC_DN 0x29efc to 0x18e71 38: 1746139359.391333 TRUNC_UP 0x18e71 to 0x38733 39: 1746139359.403580 MAPREAD 0x02fb2 thru 0x0fe21 (0x0ce70 bytes) 40: 1746139359.405174 MAPWRITE 0x21df6 thru 0x295bc (0x077c7 bytes) 41: 1746139359.453981 MAPREAD 0x170d7 thru 0x1cb6a (0x05a94 bytes) 42: 1746139359.468482 MAPWRITE 0x0e763 thru 0x122e8 (0x03b86 bytes) 43: 1746139359.486556 ZERO 0x06203 thru 0x0aa08 (0x04806 bytes) 44: 1746139359.493196 PUNCH 0x1db64 thru 0x2a49f (0x0c93c bytes) 45: 1746139359.498307 READ 0x03f07 thru 0x0c7d3 (0x088cd bytes) 46: 1746139359.528241 READ 0x02582 thru 0x0cce6 (0x0a765 bytes) 47: 1746139359.529332 READ 0x1a825 thru 0x1e5f1 (0x03dcd bytes) 48: 1746139359.529788 WRITE 0x03b05 thru 0x1059b (0x0ca97 bytes) 49: 1746139359.530824 MAPWRITE 0x1cdd4 thru 0x1f8ff (0x02b2c bytes) 50: 1746139359.552580 PUNCH 0x11ba9 thru 0x20a0c (0x0ee64 bytes) 51: 1746139359.561549 MAPWRITE 0x16cee thru 0x187ec (0x01aff bytes) 52: 1746139359.577723 WRITE 0x01198 thru 0x0755f (0x063c8 bytes) 53: 1746139359.578593 FALLOC 0x0878e thru 0x0dbe8 (0x0545a bytes)INTERIOR 54: 1746139359.584360 PUNCH 0x30420 thru 0x38732 (0x08313 bytes) 55: 1746139359.589016 PUNCH 0x24596 thru 0x2a482 (0x05eed bytes) 56: 1746139359.591914 MAPWRITE 0x2e60f thru 0x381f2 (0x09be4 bytes) 57: 1746139359.630403 PUNCH 0x1e869 thru 0x23a0f (0x051a7 bytes) 58: 1746139359.634375 FALLOC 0x36304 thru 0x3a129 (0x03e25 bytes)PAST_EOF 59: 1746139359.637497 ZERO 0x23c42 thru 0x27892 (0x03c51 bytes) 60: 1746139359.642320 READ 0x234c7 thru 0x2a54a (0x07084 bytes) 61: 1746139359.648997 MAPREAD 0x0ff0b thru 0x1f9ba (0x0fab0 bytes) 62: 1746139359.650035 ZERO 0x3762b thru 0x38732 (0x01108 bytes) 63: 1746139359.656859 WRITE 0x021d2 thru 0x10a86 (0x0e8b5 bytes) 64: 1746139359.658954 PUNCH 0x02956 thru 0x082fc (0x059a7 bytes) 65: 1746139359.676089 PUNCH 0x311c2 thru 0x38732 (0x07571 bytes) 66: 1746139359.684301 READ 0x2809a thru 0x34dc7 (0x0cd2e bytes) 67: 1746139359.690879 WRITE 0x37f9c thru 0x3f737 (0x0779c bytes) EXTEND 68: 1746139359.693761 TRUNC_DN 0x3f738 to 0x2cd51 69: 1746139359.719481 TRUNC_DN 0x2cd51 to 0x288cd 70: 1746139359.737764 TRUNC_DN 0x288cd to 0x0e087 71: 1746139359.762571 TRUNC_UP 0x0e087 to 0x1c0c9 72: 1746139359.796030 ZERO 0x04625 thru 0x10d50 (0x0c72c bytes) 73: 1746139359.802643 MAPREAD 0x18d91 thru 0x1c0c8 (0x03338 bytes) 74: 1746139359.834707 PUNCH 0x1b36b thru 0x1c0c8 (0x00d5e bytes) 75: 1746139359.847205 ZERO 0x084aa thru 0x10f7d (0x08ad4 bytes) 76: 1746139359.859588 MIRROR_EXTEND to 2 mirrors Using seed 9358 Correct content saved for comparison (maybe hexdump "/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr" vs "/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr.fsxgood") sanity-flr test_70a: @@@@@@ FAIL: fsx FLR file /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr failed Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7249:error() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:3351:test_70a() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7622:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7685:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7503:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:3353:main() Dumping lctl log to /tmp/testlogs//sanity-flr.test_70a.*.1746139361.log rsync: chown "/tmp/testlogs/.sanity-flr.test_70a.debug_log.oleg422-server.1746139361.log.S4PjMb" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-flr.test_70a.dmesg.oleg422-server.1746139361.log.b4xBbj" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2]