== sanity-flr test 70a: flr mode fsx test ================ 18:37:41 (1763336261) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70a.sanity-flr Seed set to 6262 skipping zero size read skipping zero length zero range 000003 1763336262.944006 write 0x025253 thru 0x029338 (0x40e6 bytes) 000005 1763336262.946552 write 0x02b449 thru 0x0312dc (0x5e94 bytes) 000006 1763336262.948790 mapwrite 0x02ab8f thru 0x02ca56 (0x1ec8 bytes) truncating to largest ever: 0x506b 000007 1763336262.969633 trunc from 0x0312dd to 0x00506a (0x2c272 bytes) truncating to largest ever: 0x391f1 000008 1763336262.985598 trunc from 0x00506b to 0x0391f0 (0x34186 bytes) 000010 1763336263.008435 punch from 0x0002ef to 0x004209 (0x3f1b bytes) 000011 1763336263.014426 mapwrite 0x037cb5 thru 0x03ffff (0x834b bytes) 000012 1763336263.061174 trunc from 0x040000 to 0x00df4d (0x320b2 bytes) fallocating to largest ever: 0x3c267 000013 1763336263.082059 fallocate 0x02f941 thru 0x03c266 (0xc926 bytes) 000014 1763336263.087229 mapwrite 0x034726 thru 0x03c526 (0x7e01 bytes) 000015 1763336263.134175 trunc from 0x02af48 to 0x03a90d (0xf9c6 bytes) 000016 1763336263.152081 fallocate 0x0216c8 thru 0x0255d4 (0x3f0d bytes) 000017 1763336263.161841 mapread 0x03178e thru 0x033e0b (0x267e bytes) 000019 1763336263.178399 punch from 0x01698c to 0x02497b (0xdff0 bytes) 000020 1763336263.183472 mapread 0x034cd5 thru 0x03c526 (0x7852 bytes) zero_range to largest ever: 0x3c527 000021 1763336263.188034 trunc from 0x03c297 to 0x03c526 (0x290 bytes) 000022 1763336263.193995 mapwrite 0x012f45 thru 0x01d01b (0xa0d7 bytes) 000023 1763336263.232609 trunc from 0x01bdff to 0x020ddc (0x4fde bytes) 000024 1763336263.237367 trunc from 0x00477b to 0x01036b (0xbbf1 bytes) 000025 1763336263.247472 trunc from 0x00cbeb to 0x012ba7 (0x5fbd bytes) 000026 1763336263.254850 mapread 0x01a341 thru 0x02770c (0xd3cc bytes) 000027 1763336263.271553 mapread 0x03c4e8 thru 0x03c526 (0x3f bytes) 000028 1763336263.276278 write 0x0398b1 thru 0x03ffff (0x674f bytes) 000029 1763336263.277634 mapwrite 0x01f5a6 thru 0x024923 (0x537e bytes) 000030 1763336263.296819 trunc from 0x040000 to 0x030fba (0xf045 bytes) 000031 1763336263.322773 write 0x01562b thru 0x016e52 (0x1828 bytes) 000032 1763336263.323792 punch from 0x02393c to 0x027d54 (0x4419 bytes) 000033 1763336263.330946 mapwrite 0x02cfe2 thru 0x02d5b8 (0x5d7 bytes) 000034 1763336263.355523 fallocate 0x019905 thru 0x01f287 (0x5983 bytes) 000035 1763336263.362469 fallocate 0x006cb0 thru 0x011bb0 (0xaf01 bytes) 000036 1763336263.374747 MIRROR_EXTEND to 2 mirrors 000037 1763336263.943557 trunc from 0x01cfff to 0x022fb5 (0x5fb7 bytes) 000038 1763336263.974164 read 0x01f8a2 thru 0x02c629 (0xcd88 bytes) 000040 1763336263.987158 mapwrite 0x0387ea thru 0x03ffff (0x7816 bytes) 000041 1763336264.056873 trunc from 0x040000 to 0x020e9e (0x1f161 bytes) 000042 1763336264.087577 mapread 0x017202 thru 0x020e9e (0x9c9d bytes) 000043 1763336264.101780 trunc from 0x01b3da to 0x020e9e (0x5ac5 bytes) 000046 1763336264.108823 read 0x00167b thru 0x00d859 (0xc1df bytes) 000047 1763336264.129739 MIRROR_RESYNC 2 mirrors 000048 1763336264.619664 trunc from 0x0071ae to 0x008092 (0xee5 bytes) 000049 1763336264.645383 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x20e9f): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x20e9f): Mirror 2: 0xd57f5fd2 (stale) CRC-32 checksum value for chunk [0, 0x20e9f): Mirror 1: 0xd57f5fd2 Mirror 3: 0xd57f5fd2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x20e9f: skipped 000051 1763336265.201662 punch from 0x00ccaf to 0x00fd90 (0x30e2 bytes) 000053 1763336265.235448 fallocate 0x02cb2a thru 0x02ec19 (0x20f0 bytes) 000054 1763336265.244580 MIRROR_RESYNC 3 mirrors 000055 1763336265.864717 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x4b:0x0]: Invalid argument (22) 000057 1763336266.167113 write 0x03119c thru 0x031b63 (0x9c8 bytes) 000058 1763336266.206726 punch from 0x00eac8 to 0x01cade (0xe017 bytes) 000059 1763336266.210025 mapread 0x01dbaf thru 0x0276df (0x9b31 bytes) 000060 1763336266.233172 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x31b64): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31b64): Mirror 2: 0xc7f0b787 (stale) CRC-32 checksum value for chunk [0, 0x31b64): Mirror 1: 0xdc8247c4 Mirror 3: 0xdc8247c4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31b64: skipped 000062 1763336266.909267 read 0x001c34 thru 0x01144f (0xf81c bytes) 000063 1763336266.926347 read 0x019fd0 thru 0x028445 (0xe476 bytes) 000064 1763336266.936475 mapread 0x003466 thru 0x012d37 (0xf8d2 bytes) 000066 1763336266.937442 fallocate 0x01d7e3 thru 0x01eee7 (0x1705 bytes) 000067 1763336266.990432 trunc from 0x026cb1 to 0x02d861 (0x6bb1 bytes) 000068 1763336266.995860 read 0x016e74 thru 0x01e2e2 (0x746f bytes) 000069 1763336267.007221 punch from 0x00840f to 0x008691 (0x283 bytes) 000070 1763336267.024228 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x4b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x31b64): Mirror 3: 0xdc8247c4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31b64): Mirror 3: 0xdc8247c4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31b64: skipped 000071 1763336267.309948 mapwrite 0x01c908 thru 0x02529a (0x8993 bytes) 000072 1763336267.383972 trunc from 0x00ef22 to 0x019853 (0xa932 bytes) 000075 1763336267.390446 fallocate 0x012097 thru 0x021177 (0xf0e1 bytes) 000076 1763336267.397071 mapwrite 0x00ea28 thru 0x016ec7 (0x84a0 bytes) 000077 1763336267.439501 write 0x02db4b thru 0x03a103 (0xc5b9 bytes) 000078 1763336267.481916 MIRROR_RESYNC 2 mirrors 000079 1763336268.175264 fallocate 0x02ab3c thru 0x0362ef (0xb7b4 bytes) 000080 1763336268.244836 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3a104): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a104): Mirror 3: 0x1915dadb (stale) CRC-32 checksum value for chunk [0, 0x3a104): Mirror 1: 0x1915dadb Mirror 4: 0x1915dadb lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a104: skipped 000081 1763336269.071106 write 0x03310a thru 0x03f3ae (0xc2a5 bytes) 000083 1763336269.143451 read 0x025254 thru 0x0349ea (0xf797 bytes) 000084 1763336269.160437 trunc from 0x01fb22 to 0x021fae (0x248d bytes) 000086 1763336269.168482 mapwrite 0x03b435 thru 0x03ffff (0x4bcb bytes) 000087 1763336269.215931 write 0x02d515 thru 0x034606 (0x70f2 bytes) 000088 1763336269.217225 write 0x01b64d thru 0x02b5c1 (0xff75 bytes) 000089 1763336269.231703 write 0x01a52e thru 0x02373c (0x920f bytes) 000090 1763336269.242684 punch from 0x02802a to 0x028c2f (0xc06 bytes) 000091 1763336269.268278 read 0x005852 thru 0x010d51 (0xb500 bytes) 000092 1763336269.281482 punch from 0x003658 to 0x00cba5 (0x954e bytes) 000093 1763336269.288427 write 0x03b1b7 thru 0x03ffff (0x4e49 bytes) 000094 1763336269.290725 trunc from 0x040000 to 0x035459 (0xaba6 bytes) 000095 1763336269.327943 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3545a): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3545a): Mirror 3: 0x5cd5151a (stale) Mirror 4: 0x5cd5151a (stale) CRC-32 checksum value for chunk [0, 0x3545a): Mirror 1: 0x4678b5f8 Mirror 5: 0x4678b5f8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3545a: skipped 000096 1763336270.083094 write 0x02bd76 thru 0x0318fd (0x5b88 bytes) 000098 1763336270.181910 mapread 0x0254c7 thru 0x02776a (0x22a4 bytes) 000099 1763336270.194698 write 0x01c2db thru 0x027043 (0xad69 bytes) 000100 1763336270.205380 trunc from 0x019b12 to 0x01bd82 (0x2271 bytes) 000101 1763336270.209891 mapwrite 0x033d51 thru 0x039724 (0x59d4 bytes) 000102 1763336270.286790 mapwrite 0x03dc39 thru 0x03ffff (0x23c7 bytes) 000103 1763336270.329683 trunc from 0x040000 to 0x02be54 (0x141ab bytes) 000104 1763336270.345744 write 0x01eead thru 0x021ad7 (0x2c2b bytes) 000105 1763336270.346866 mapwrite 0x01d85d thru 0x02428f (0x6a33 bytes) 000106 1763336270.366428 trunc from 0x004cea to 0x008377 (0x368e bytes) 000107 1763336270.371452 trunc from 0x02be55 to 0x0183e3 (0x13a71 bytes) 000108 1763336270.389729 write 0x01d955 thru 0x01f8fa (0x1fa6 bytes) 000109 1763336270.395646 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x1f8fb): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1f8fb): Mirror 3: 0x68808e51 (stale) Mirror 4: 0x68808e51 (stale) Mirror 5: 0x47a576d2 (stale) CRC-32 checksum value for chunk [0, 0x1f8fb): Mirror 1: 0x265a78b6 Mirror 6: 0x265a78b6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1f8fb: skipped 000110 1763336271.263306 trunc from 0x01f8fb to 0x0385e6 (0x18cec bytes) 000111 1763336271.334752 write 0x0199aa thru 0x02663f (0xcc96 bytes) 000112 1763336271.356249 punch from 0x02f43e to 0x033bd7 (0x479a bytes) 000113 1763336271.363818 mapread 0x02b29d thru 0x02fed1 (0x4c35 bytes) 000114 1763336271.375000 trunc from 0x008874 to 0x00cc84 (0x4411 bytes) 000115 1763336271.386748 punch from 0x035695 to 0x0378bd (0x2229 bytes) 000117 1763336271.397468 mapread 0x017119 thru 0x01f9c7 (0x88af bytes) 000118 1763336271.410060 punch from 0x00f810 to 0x015d31 (0x6522 bytes) 000119 1763336271.416000 write 0x03c1ba thru 0x03ffff (0x3e46 bytes) 000120 1763336271.417267 mapread 0x01e1af thru 0x01f78f (0x15e1 bytes) 000122 1763336271.418700 mapread 0x005888 thru 0x01138e (0xbb07 bytes) 000123 1763336271.435395 write 0x02f04c thru 0x03b3e0 (0xc395 bytes) 000124 1763336271.444525 write 0x03b6cc thru 0x03f0cb (0x3a00 bytes) 000125 1763336271.448047 trunc from 0x0062f6 to 0x008ea1 (0x2bac bytes) 000126 1763336271.466734 mapread 0x033428 thru 0x03c20c (0x8de5 bytes) fallocating to largest ever: 0x3d766 000127 1763336271.469599 fallocate 0x038ce9 thru 0x03d765 (0x4a7d bytes) 000128 1763336271.473851 write 0x0176ba thru 0x0263d6 (0xed1d bytes) 000129 1763336271.475362 read 0x01699d thru 0x020565 (0x9bc9 bytes) 000131 1763336271.480559 MIRROR_RESYNC 5 mirrors 000133 1763336272.902190 trunc from 0x022628 to 0x02d4a4 (0xae7d bytes) 000134 1763336272.946568 write 0x00de6d thru 0x0168b5 (0x8a49 bytes) 000135 1763336272.959351 trunc from 0x03d498 to 0x03ffff (0x2b68 bytes) 000136 1763336272.970361 fallocate 0x02dc2d thru 0x033a76 (0x5e4a bytes) 000137 1763336272.983435 mapread 0x016d1d thru 0x02664f (0xf933 bytes) 000138 1763336272.997874 write 0x038a99 thru 0x03ffff (0x7567 bytes) 000139 1763336273.012154 trunc from 0x01ab2f to 0x029925 (0xedf7 bytes) 000140 1763336273.019517 punch from 0x036648 to 0x03ffff (0x99b8 bytes) 000141 1763336273.043049 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x4b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x14697343 (stale) Mirror 4: 0x14697343 (stale) Mirror 6: 0x14697343 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x14697343 (stale) Mirror 4: 0x14697343 (stale) Mirror 6: 0x14697343 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000142 1763336273.423704 punch from 0x015854 to 0x021622 (0xbdcf bytes) 000143 1763336273.430260 read 0x026969 thru 0x0334e3 (0xcb7b bytes) 000144 1763336273.447162 punch from 0x031442 to 0x034dae (0x396d bytes) 000145 1763336273.453863 write 0x0014e1 thru 0x005dff (0x491f bytes) 000146 1763336273.486032 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x4b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x14697343 (stale) Mirror 6: 0x14697343 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x14697343 (stale) Mirror 6: 0x14697343 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000148 1763336273.843712 mapwrite 0x01c889 thru 0x01fc64 (0x33dc bytes) 000149 1763336273.878078 mapread 0x017bd7 thru 0x01cc4f (0x5079 bytes) 000150 1763336273.879144 mapread 0x02cbca thru 0x0357de (0x8c15 bytes) 000151 1763336273.900275 write 0x022340 thru 0x024c83 (0x2944 bytes) 000153 1763336273.905980 trunc from 0x009e3f to 0x018771 (0xe933 bytes) 000154 1763336273.912948 write 0x00ce2c thru 0x01b67f (0xe854 bytes) 000155 1763336273.923951 trunc from 0x0096a8 to 0x01742f (0xdd88 bytes) 000156 1763336273.938916 read 0x03daf8 thru 0x03ffff (0x2508 bytes) 000158 1763336273.945087 mapread 0x02e9a7 thru 0x037cdd (0x9337 bytes) 000159 1763336273.947423 write 0x036790 thru 0x038042 (0x18b3 bytes) 000160 1763336273.948755 read 0x02880b thru 0x034751 (0xbf47 bytes) 000161 1763336273.950055 trunc from 0x028f33 to 0x035cda (0xcda8 bytes) 000162 1763336273.972202 trunc from 0x027700 to 0x030814 (0x9115 bytes) 000164 1763336273.984674 trunc from 0x020944 to 0x0256d4 (0x4d91 bytes) 000166 1763336273.993068 trunc from 0x02252a to 0x02bbc7 (0x969e bytes) 000167 1763336274.001770 mapwrite 0x03b31f thru 0x03e6ed (0x33cf bytes) 000168 1763336274.029666 trunc from 0x040000 to 0x037414 (0x8beb bytes) 000169 1763336274.045391 trunc from 0x024bed to 0x027ccc (0x30e0 bytes) 000170 1763336274.053767 punch from 0x00f286 to 0x015709 (0x6484 bytes) 000171 1763336274.061527 punch from 0x010e31 to 0x012b7c (0x1d4c bytes) 000172 1763336274.065991 punch from 0x01d877 to 0x021199 (0x3923 bytes) 000173 1763336274.071789 mapwrite 0x034fe9 thru 0x03ffff (0xb017 bytes) 000174 1763336274.142904 punch from 0x01e2a8 to 0x0278e4 (0x963d bytes) 000175 1763336274.148805 mapread 0x038fca thru 0x03b10b (0x2142 bytes) 000176 1763336274.149799 trunc from 0x040000 to 0x009af3 (0x3650c bytes) 000177 1763336274.171802 mapread 0x005a6f thru 0x009af3 (0x4085 bytes) 000178 1763336274.173573 read 0x0023a7 thru 0x009af3 (0x774d bytes) 000179 1763336274.174659 write 0x01ece3 thru 0x02d201 (0xe51f bytes) 000180 1763336274.178280 trunc from 0x02343b to 0x024af7 (0x16bd bytes) 000181 1763336274.196951 trunc from 0x02d202 to 0x039998 (0xc797 bytes) 000182 1763336274.216848 MIRROR_RESYNC 3 mirrors 000184 1763336275.196935 fallocate 0x01efd3 thru 0x0219d9 (0x2a07 bytes) 000185 1763336275.229285 trunc from 0x039999 to 0x01d958 (0x1c040 bytes) 000186 1763336275.261177 write 0x003e71 thru 0x00995e (0x5aee bytes) 000187 1763336275.293546 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x4b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1d959): Mirror 6: 0xcb01ca95 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1d959): Mirror 6: 0xcb01ca95 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1d959: skipped 000188 1763336275.629472 read 0x01b7b6 thru 0x01d958 (0x21a3 bytes) 000189 1763336275.636250 read 0x004344 thru 0x00bc91 (0x794e bytes) 000190 1763336275.646444 mapread 0x00d0e0 thru 0x013ee4 (0x6e05 bytes) 000192 1763336275.648090 write 0x02f9ad thru 0x03c7a9 (0xcdfd bytes) 000193 1763336275.658107 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x4b:0x0]: Invalid argument (22) 000194 1763336275.795420 mapwrite 0x01542c thru 0x01e2b6 (0x8e8b bytes) 000195 1763336275.833239 read 0x01bfaf thru 0x01dc26 (0x1c78 bytes) 000196 1763336275.837047 mapread 0x0197b0 thru 0x01dcd0 (0x4521 bytes) 000197 1763336275.848438 MIRROR_EXTEND to 2 mirrors 000198 1763336276.596722 write 0x00e918 thru 0x0189bd (0xa0a6 bytes) 000199 1763336276.663464 write 0x01a301 thru 0x020832 (0x6532 bytes) 000200 1763336276.675988 mapwrite 0x03840e thru 0x03f51c (0x710f bytes) 000201 1763336276.735219 write 0x00dccf thru 0x016f15 (0x9247 bytes) 000203 1763336276.748438 mapread 0x003c8b thru 0x00b1f5 (0x756b bytes) 000204 1763336276.765987 write 0x00e744 thru 0x01b8c3 (0xd180 bytes) 000205 1763336276.776214 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3f51d): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f51d): Mirror 2: 0xa032b08a (stale) CRC-32 checksum value for chunk [0, 0x3f51d): Mirror 1: 0xa677c021 Mirror 3: 0xa677c021 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f51d: skipped 000207 1763336277.573876 fallocate 0x010a40 thru 0x01e604 (0xdbc5 bytes) 000208 1763336277.609722 trunc from 0x03f51d to 0x00cad7 (0x32a45 bytes) 000209 1763336277.632433 trunc from 0x00ca11 to 0x00cad7 (0xc7 bytes) 000210 1763336277.646194 write 0x0254b6 thru 0x029ad0 (0x461b bytes) 000211 1763336277.647638 read 0x0240d3 thru 0x029ad0 (0x59fe bytes) 000212 1763336277.654975 read 0x0146a6 thru 0x01d555 (0x8eb0 bytes) 000213 1763336277.663698 trunc from 0x01b9ab to 0x02981e (0xde74 bytes) 000214 1763336277.676401 trunc from 0x013210 to 0x022a7e (0xf86f bytes) 000215 1763336277.680880 trunc from 0x018d17 to 0x0198ed (0xbd7 bytes) 000217 1763336277.693107 trunc from 0x004148 to 0x010bc2 (0xca7b bytes) 000218 1763336277.700084 mapread 0x01f969 thru 0x029ad0 (0xa168 bytes) 000219 1763336277.773874 write 0x01ca7d thru 0x02565e (0x8be2 bytes) 000220 1763336277.781469 punch from 0x00a9a0 to 0x019d81 (0xf3e2 bytes) 000221 1763336277.786734 punch from 0x0264e9 to 0x028b6e (0x2686 bytes) 000222 1763336277.790380 write 0x02c452 thru 0x0350b8 (0x8c67 bytes) 000223 1763336277.791937 mapread 0x026af1 thru 0x02dccc (0x71dc bytes) 000224 1763336277.813583 mapread 0x01ab2b thru 0x027c63 (0xd139 bytes) 000225 1763336277.831748 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x4b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x350b9): Mirror 3: 0x296952d8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x350b9): Mirror 3: 0x296952d8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x350b9: skipped 000226 1763336278.109879 trunc from 0x0350b9 to 0x008269 (0x2ce4f bytes) 000227 1763336278.132109 MIRROR_RESYNC 2 mirrors 000228 1763336278.679812 write 0x00f4c0 thru 0x01deb3 (0xe9f4 bytes) 000229 1763336278.726873 fallocate 0x035225 thru 0x03d48f (0x826b bytes) 000230 1763336278.733920 mapwrite 0x011f88 thru 0x01c69e (0xa717 bytes) 000231 1763336278.750671 read 0x0123f4 thru 0x01e396 (0xbfa3 bytes) 000233 1763336278.756146 mapwrite 0x026257 thru 0x02f3c6 (0x9170 bytes) 000234 1763336278.797111 read 0x018727 thru 0x01da1e (0x52f8 bytes) 000235 1763336278.803774 mapwrite 0x00ada8 thru 0x018000 (0xd259 bytes) 000236 1763336278.841272 write 0x01d819 thru 0x027963 (0xa14b bytes) 000237 1763336278.845792 write 0x019e34 thru 0x01fe83 (0x6050 bytes) 000238 1763336278.847092 mapwrite 0x02f643 thru 0x035482 (0x5e40 bytes) 000239 1763336278.865490 punch from 0x00a30b to 0x00fd32 (0x5a28 bytes) 000240 1763336278.872986 trunc from 0x03d490 to 0x00c3c7 (0x310c8 bytes) 000241 1763336278.915610 write 0x01a0cd thru 0x0299fc (0xf930 bytes) 000242 1763336278.917787 punch from 0x007988 to 0x00863f (0xcb8 bytes) 000244 1763336278.925221 mapwrite 0x0352c8 thru 0x03ffff (0xad38 bytes) 000245 1763336279.028469 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x49680e4d (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x66e29ec9 Mirror 4: 0x66e29ec9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000246 1763336279.718377 punch from 0x014e6a to 0x01b1b0 (0x6347 bytes) 000247 1763336279.766805 MIRROR_RESYNC 3 mirrors 000248 1763336280.618695 punch from 0x02c552 to 0x034d2e (0x87dd bytes) 000249 1763336280.657857 read 0x019816 thru 0x020b69 (0x7354 bytes) 000250 1763336280.670392 write 0x00aa57 thru 0x019b38 (0xf0e2 bytes) 000252 1763336280.679586 trunc from 0x040000 to 0x021bec (0x1e413 bytes) 000253 1763336280.695264 punch from 0x017285 to 0x01c64b (0x53c7 bytes) 000254 1763336280.710607 read 0x00cdc2 thru 0x014172 (0x73b1 bytes) 000255 1763336280.711192 mapwrite 0x035190 thru 0x03ffff (0xae70 bytes) 000256 1763336280.747893 fallocate 0x022c4a thru 0x0255cc (0x2983 bytes) 000257 1763336280.752328 fallocate 0x0239f0 thru 0x030df2 (0xd403 bytes) 000258 1763336280.755891 punch from 0x03101b to 0x038eee (0x7ed4 bytes) 000259 1763336280.760468 read 0x02bf76 thru 0x02e9ca (0x2a55 bytes) 000260 1763336280.761240 trunc from 0x03ed80 to 0x03ffff (0x1280 bytes) 000262 1763336280.767475 write 0x01b44d thru 0x01e290 (0x2e44 bytes) 000263 1763336280.768647 mapwrite 0x00a66c thru 0x012007 (0x799c bytes) 000264 1763336280.788236 trunc from 0x040000 to 0x02a10b (0x15ef4 bytes) 000265 1763336280.799227 read 0x0087d6 thru 0x009982 (0x11ad bytes) 000266 1763336280.799985 fallocate 0x004f66 thru 0x0076e2 (0x277d bytes) 000267 1763336280.805779 mapwrite 0x02fc31 thru 0x033383 (0x3753 bytes) 000268 1763336280.860968 read 0x031cbe thru 0x033383 (0x16c6 bytes) 000270 1763336280.863152 mapwrite 0x00205e thru 0x009fe7 (0x7f8a bytes) 000271 1763336280.883445 read 0x00ea57 thru 0x01d581 (0xeb2b bytes) 000272 1763336280.887824 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x33384): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33384): Mirror 3: 0xcaa75e58 (stale) Mirror 4: 0xcaa75e58 (stale) CRC-32 checksum value for chunk [0, 0x33384): Mirror 1: 0x7dd2917f Mirror 5: 0x7dd2917f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33384: skipped 000273 1763336281.731516 write 0x012928 thru 0x01de6f (0xb548 bytes) 000274 1763336281.806788 fallocate 0x02c637 thru 0x0350bd (0x8a87 bytes) 000275 1763336281.822726 MIRROR_RESYNC 4 mirrors 000276 1763336282.641845 read 0x032a83 thru 0x033383 (0x901 bytes) 000277 1763336282.660501 mapwrite 0x024f72 thru 0x02fe3d (0xaecc bytes) 000278 1763336282.757033 write 0x00f7c5 thru 0x01ed1e (0xf55a bytes) 000279 1763336282.771234 mapwrite 0x024387 thru 0x02e404 (0xa07e bytes) truncating to largest ever: 0x3ef86 000280 1763336282.798446 trunc from 0x033384 to 0x03ef85 (0xbc02 bytes) 000281 1763336282.818944 write 0x02dc05 thru 0x02f86a (0x1c66 bytes) 000282 1763336282.819996 mapread 0x01a0f6 thru 0x01b557 (0x1462 bytes) 000283 1763336282.820822 read 0x018c46 thru 0x01fa84 (0x6e3f bytes) 000284 1763336282.824082 write 0x0091e9 thru 0x00aee7 (0x1cff bytes) 000285 1763336282.828102 trunc from 0x03ef86 to 0x0095d4 (0x359b1 bytes) 000286 1763336282.849127 write 0x0187b8 thru 0x01a305 (0x1b4e bytes) 000287 1763336282.851461 read 0x0018bd thru 0x004abb (0x31ff bytes) 000288 1763336282.859916 punch from 0x0108d5 to 0x014b36 (0x4262 bytes) 000289 1763336282.865473 fallocate 0x015294 thru 0x018e58 (0x3bc5 bytes) 000290 1763336282.870932 mapread 0x01159a thru 0x0150b6 (0x3b1d bytes) 000291 1763336282.890035 trunc from 0x01a306 to 0x020153 (0x5e4e bytes) 000292 1763336282.909025 fallocate 0x036b87 thru 0x03ac7a (0x40f4 bytes) 000293 1763336282.914515 trunc from 0x01b2b5 to 0x01ee17 (0x3b63 bytes) 000294 1763336282.929663 fallocate 0x0084a8 thru 0x013e1a (0xb973 bytes) 000295 1763336282.941435 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3ac7b): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b000): Mirror 3: 0xb1351866 (stale) Mirror 4: 0xb1351866 (stale) Mirror 5: 0xb1351866 (stale) CRC-32 checksum value for chunk [0, 0x3b000): Mirror 1: 0x73da2f6d Mirror 6: 0x73da2f6d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b000: skipped Size error: expected 0x3ac7b stat 0x3ac7b seek 0x3b000 LOG DUMP (260 total operations): 1: 1763336262.939847 SKIPPED (no operation) 2: 1763336262.943571 SKIPPED (no operation) 3: 1763336262.944006 WRITE 0x25253 thru 0x29338 (0x040e6 bytes) HOLE 4: 1763336262.946552 WRITE 0x2b449 thru 0x312dc (0x05e94 bytes) HOLE 5: 1763336262.948790 MAPWRITE 0x2ab8f thru 0x2ca56 (0x01ec8 bytes) 6: 1763336262.969633 TRUNC_DN 0x312dd to 0x0506b 7: 1763336262.985598 TRUNC_UP 0x0506b to 0x391f1 8: 1763336263.008435 PUNCH 0x002ef thru 0x04209 (0x03f1b bytes) 9: 1763336263.014426 MAPWRITE 0x37cb5 thru 0x3ffff (0x0834b bytes) 10: 1763336263.061174 TRUNC_DN 0x40000 to 0x0df4e 11: 1763336263.082059 FALLOC 0x2f941 thru 0x3c267 (0x0c926 bytes)PAST_EOF 12: 1763336263.087229 MAPWRITE 0x34726 thru 0x3c526 (0x07e01 bytes) 13: 1763336263.134175 ZERO 0x2af48 thru 0x3a90d (0x0f9c6 bytes) 14: 1763336263.152081 FALLOC 0x216c8 thru 0x255d5 (0x03f0d bytes)INTERIOR 15: 1763336263.161841 MAPREAD 0x3178e thru 0x33e0b (0x0267e bytes) 16: 1763336263.178399 PUNCH 0x1698c thru 0x2497b (0x0dff0 bytes) 17: 1763336263.183472 MAPREAD 0x34cd5 thru 0x3c526 (0x07852 bytes) 18: 1763336263.188034 ZERO 0x3c297 thru 0x3c526 (0x00290 bytes) 19: 1763336263.193995 MAPWRITE 0x12f45 thru 0x1d01b (0x0a0d7 bytes) 20: 1763336263.232609 ZERO 0x1bdff thru 0x20ddc (0x04fde bytes) 21: 1763336263.237367 ZERO 0x0477b thru 0x1036b (0x0bbf1 bytes) 22: 1763336263.247472 ZERO 0x0cbeb thru 0x12ba7 (0x05fbd bytes) 23: 1763336263.254850 MAPREAD 0x1a341 thru 0x2770c (0x0d3cc bytes) 24: 1763336263.271553 MAPREAD 0x3c4e8 thru 0x3c526 (0x0003f bytes) 25: 1763336263.276278 WRITE 0x398b1 thru 0x3ffff (0x0674f bytes) EXTEND 26: 1763336263.277634 MAPWRITE 0x1f5a6 thru 0x24923 (0x0537e bytes) 27: 1763336263.296819 TRUNC_DN 0x40000 to 0x30fbb 28: 1763336263.322773 WRITE 0x1562b thru 0x16e52 (0x01828 bytes) 29: 1763336263.323792 PUNCH 0x2393c thru 0x27d54 (0x04419 bytes) 30: 1763336263.330946 MAPWRITE 0x2cfe2 thru 0x2d5b8 (0x005d7 bytes) 31: 1763336263.355523 FALLOC 0x19905 thru 0x1f288 (0x05983 bytes)INTERIOR 32: 1763336263.362469 FALLOC 0x06cb0 thru 0x11bb1 (0x0af01 bytes)INTERIOR 33: 1763336263.374747 MIRROR_EXTEND to 2 mirrors 34: 1763336263.943557 ZERO 0x1cfff thru 0x22fb5 (0x05fb7 bytes) 35: 1763336263.974164 READ 0x1f8a2 thru 0x2c629 (0x0cd88 bytes) 36: 1763336263.987158 MAPWRITE 0x387ea thru 0x3ffff (0x07816 bytes) 37: 1763336264.056873 TRUNC_DN 0x40000 to 0x20e9f 38: 1763336264.087577 MAPREAD 0x17202 thru 0x20e9e (0x09c9d bytes) 39: 1763336264.101780 ZERO 0x1b3da thru 0x20e9e (0x05ac5 bytes) 40: 1763336264.108823 READ 0x0167b thru 0x0d859 (0x0c1df bytes) 41: 1763336264.129739 MIRROR_RESYNC 2 mirrors 42: 1763336264.619664 ZERO 0x071ae thru 0x08092 (0x00ee5 bytes) 43: 1763336264.645383 MIRROR_EXTEND to 3 mirrors 44: 1763336265.201662 PUNCH 0x0ccaf thru 0x0fd90 (0x030e2 bytes) 45: 1763336265.235448 FALLOC 0x2cb2a thru 0x2ec1a (0x020f0 bytes)EXTENDING 46: 1763336265.244580 MIRROR_RESYNC 3 mirrors 47: 1763336265.864717 MIRROR_SPLIT mirror 2 to 2 mirrors 48: 1763336266.167113 WRITE 0x3119c thru 0x31b63 (0x009c8 bytes) HOLE 49: 1763336266.206726 PUNCH 0x0eac8 thru 0x1cade (0x0e017 bytes) 50: 1763336266.210025 MAPREAD 0x1dbaf thru 0x276df (0x09b31 bytes) 51: 1763336266.233172 MIRROR_EXTEND to 3 mirrors 52: 1763336266.909267 READ 0x01c34 thru 0x1144f (0x0f81c bytes) 53: 1763336266.926347 READ 0x19fd0 thru 0x28445 (0x0e476 bytes) 54: 1763336266.936475 MAPREAD 0x03466 thru 0x12d37 (0x0f8d2 bytes) 55: 1763336266.937442 FALLOC 0x1d7e3 thru 0x1eee8 (0x01705 bytes)INTERIOR 56: 1763336266.990432 ZERO 0x26cb1 thru 0x2d861 (0x06bb1 bytes) 57: 1763336266.995860 READ 0x16e74 thru 0x1e2e2 (0x0746f bytes) 58: 1763336267.007221 PUNCH 0x0840f thru 0x08691 (0x00283 bytes) 59: 1763336267.024228 MIRROR_SPLIT mirror 1 to 2 mirrors 60: 1763336267.309948 MAPWRITE 0x1c908 thru 0x2529a (0x08993 bytes) 61: 1763336267.383972 ZERO 0x0ef22 thru 0x19853 (0x0a932 bytes) 62: 1763336267.390446 FALLOC 0x12097 thru 0x21178 (0x0f0e1 bytes)INTERIOR 63: 1763336267.397071 MAPWRITE 0x0ea28 thru 0x16ec7 (0x084a0 bytes) 64: 1763336267.439501 WRITE 0x2db4b thru 0x3a103 (0x0c5b9 bytes) EXTEND 65: 1763336267.481916 MIRROR_RESYNC 2 mirrors 66: 1763336268.175264 FALLOC 0x2ab3c thru 0x362f0 (0x0b7b4 bytes)INTERIOR 67: 1763336268.244836 MIRROR_EXTEND to 3 mirrors 68: 1763336269.071106 WRITE 0x3310a thru 0x3f3ae (0x0c2a5 bytes) EXTEND 69: 1763336269.143451 READ 0x25254 thru 0x349ea (0x0f797 bytes) 70: 1763336269.160437 ZERO 0x1fb22 thru 0x21fae (0x0248d bytes) 71: 1763336269.168482 MAPWRITE 0x3b435 thru 0x3ffff (0x04bcb bytes) 72: 1763336269.215931 WRITE 0x2d515 thru 0x34606 (0x070f2 bytes) 73: 1763336269.217225 WRITE 0x1b64d thru 0x2b5c1 (0x0ff75 bytes) 74: 1763336269.231703 WRITE 0x1a52e thru 0x2373c (0x0920f bytes) 75: 1763336269.242684 PUNCH 0x2802a thru 0x28c2f (0x00c06 bytes) 76: 1763336269.268278 READ 0x05852 thru 0x10d51 (0x0b500 bytes) 77: 1763336269.281482 PUNCH 0x03658 thru 0x0cba5 (0x0954e bytes) 78: 1763336269.288427 WRITE 0x3b1b7 thru 0x3ffff (0x04e49 bytes) 79: 1763336269.290725 TRUNC_DN 0x40000 to 0x3545a 80: 1763336269.327943 MIRROR_EXTEND to 4 mirrors 81: 1763336270.083094 WRITE 0x2bd76 thru 0x318fd (0x05b88 bytes) 82: 1763336270.181910 MAPREAD 0x254c7 thru 0x2776a (0x022a4 bytes) 83: 1763336270.194698 WRITE 0x1c2db thru 0x27043 (0x0ad69 bytes) 84: 1763336270.205380 ZERO 0x19b12 thru 0x1bd82 (0x02271 bytes) 85: 1763336270.209891 MAPWRITE 0x33d51 thru 0x39724 (0x059d4 bytes) 86: 1763336270.286790 MAPWRITE 0x3dc39 thru 0x3ffff (0x023c7 bytes) 87: 1763336270.329683 TRUNC_DN 0x40000 to 0x2be55 88: 1763336270.345744 WRITE 0x1eead thru 0x21ad7 (0x02c2b bytes) 89: 1763336270.346866 MAPWRITE 0x1d85d thru 0x2428f (0x06a33 bytes) 90: 1763336270.366428 ZERO 0x04cea thru 0x08377 (0x0368e bytes) 91: 1763336270.371452 TRUNC_DN 0x2be55 to 0x183e4 92: 1763336270.389729 WRITE 0x1d955 thru 0x1f8fa (0x01fa6 bytes) HOLE 93: 1763336270.395646 MIRROR_EXTEND to 5 mirrors 94: 1763336271.263306 TRUNC_UP 0x1f8fb to 0x385e7 95: 1763336271.334752 WRITE 0x199aa thru 0x2663f (0x0cc96 bytes) 96: 1763336271.356249 PUNCH 0x2f43e thru 0x33bd7 (0x0479a bytes) 97: 1763336271.363818 MAPREAD 0x2b29d thru 0x2fed1 (0x04c35 bytes) 98: 1763336271.375000 ZERO 0x08874 thru 0x0cc84 (0x04411 bytes) 99: 1763336271.386748 PUNCH 0x35695 thru 0x378bd (0x02229 bytes) 100: 1763336271.397468 MAPREAD 0x17119 thru 0x1f9c7 (0x088af bytes) 101: 1763336271.410060 PUNCH 0x0f810 thru 0x15d31 (0x06522 bytes) 102: 1763336271.416000 WRITE 0x3c1ba thru 0x3ffff (0x03e46 bytes) HOLE 103: 1763336271.417267 MAPREAD 0x1e1af thru 0x1f78f (0x015e1 bytes) 104: 1763336271.418700 MAPREAD 0x05888 thru 0x1138e (0x0bb07 bytes) 105: 1763336271.435395 WRITE 0x2f04c thru 0x3b3e0 (0x0c395 bytes) 106: 1763336271.444525 WRITE 0x3b6cc thru 0x3f0cb (0x03a00 bytes) 107: 1763336271.448047 ZERO 0x062f6 thru 0x08ea1 (0x02bac bytes) 108: 1763336271.466734 MAPREAD 0x33428 thru 0x3c20c (0x08de5 bytes) 109: 1763336271.469599 FALLOC 0x38ce9 thru 0x3d766 (0x04a7d bytes)INTERIOR 110: 1763336271.473851 WRITE 0x176ba thru 0x263d6 (0x0ed1d bytes) 111: 1763336271.475362 READ 0x1699d thru 0x20565 (0x09bc9 bytes) 112: 1763336271.480559 MIRROR_RESYNC 5 mirrors 113: 1763336272.902190 ZERO 0x22628 thru 0x2d4a4 (0x0ae7d bytes) 114: 1763336272.946568 WRITE 0x0de6d thru 0x168b5 (0x08a49 bytes) 115: 1763336272.959351 ZERO 0x3d498 thru 0x3ffff (0x02b68 bytes) 116: 1763336272.970361 FALLOC 0x2dc2d thru 0x33a77 (0x05e4a bytes)INTERIOR 117: 1763336272.983435 MAPREAD 0x16d1d thru 0x2664f (0x0f933 bytes) 118: 1763336272.997874 WRITE 0x38a99 thru 0x3ffff (0x07567 bytes) 119: 1763336273.012154 ZERO 0x1ab2f thru 0x29925 (0x0edf7 bytes) 120: 1763336273.019517 PUNCH 0x36648 thru 0x3ffff (0x099b8 bytes) 121: 1763336273.043049 MIRROR_SPLIT mirror 3 to 4 mirrors 122: 1763336273.423704 PUNCH 0x15854 thru 0x21622 (0x0bdcf bytes) 123: 1763336273.430260 READ 0x26969 thru 0x334e3 (0x0cb7b bytes) 124: 1763336273.447162 PUNCH 0x31442 thru 0x34dae (0x0396d bytes) 125: 1763336273.453863 WRITE 0x014e1 thru 0x05dff (0x0491f bytes) 126: 1763336273.486032 MIRROR_SPLIT mirror 2 to 3 mirrors 127: 1763336273.843712 MAPWRITE 0x1c889 thru 0x1fc64 (0x033dc bytes) 128: 1763336273.878078 MAPREAD 0x17bd7 thru 0x1cc4f (0x05079 bytes) 129: 1763336273.879144 MAPREAD 0x2cbca thru 0x357de (0x08c15 bytes) 130: 1763336273.900275 WRITE 0x22340 thru 0x24c83 (0x02944 bytes) 131: 1763336273.905980 ZERO 0x09e3f thru 0x18771 (0x0e933 bytes) 132: 1763336273.912948 WRITE 0x0ce2c thru 0x1b67f (0x0e854 bytes) 133: 1763336273.923951 ZERO 0x096a8 thru 0x1742f (0x0dd88 bytes) 134: 1763336273.938916 READ 0x3daf8 thru 0x3ffff (0x02508 bytes) 135: 1763336273.945087 MAPREAD 0x2e9a7 thru 0x37cdd (0x09337 bytes) 136: 1763336273.947423 WRITE 0x36790 thru 0x38042 (0x018b3 bytes) 137: 1763336273.948755 READ 0x2880b thru 0x34751 (0x0bf47 bytes) 138: 1763336273.950055 ZERO 0x28f33 thru 0x35cda (0x0cda8 bytes) 139: 1763336273.972202 ZERO 0x27700 thru 0x30814 (0x09115 bytes) 140: 1763336273.984674 ZERO 0x20944 thru 0x256d4 (0x04d91 bytes) 141: 1763336273.993068 ZERO 0x2252a thru 0x2bbc7 (0x0969e bytes) 142: 1763336274.001770 MAPWRITE 0x3b31f thru 0x3e6ed (0x033cf bytes) 143: 1763336274.029666 TRUNC_DN 0x40000 to 0x37415 144: 1763336274.045391 ZERO 0x24bed thru 0x27ccc (0x030e0 bytes) 145: 1763336274.053767 PUNCH 0x0f286 thru 0x15709 (0x06484 bytes) 146: 1763336274.061527 PUNCH 0x10e31 thru 0x12b7c (0x01d4c bytes) 147: 1763336274.065991 PUNCH 0x1d877 thru 0x21199 (0x03923 bytes) 148: 1763336274.071789 MAPWRITE 0x34fe9 thru 0x3ffff (0x0b017 bytes) 149: 1763336274.142904 PUNCH 0x1e2a8 thru 0x278e4 (0x0963d bytes) 150: 1763336274.148805 MAPREAD 0x38fca thru 0x3b10b (0x02142 bytes) 151: 1763336274.149799 TRUNC_DN 0x40000 to 0x09af4 152: 1763336274.171802 MAPREAD 0x05a6f thru 0x09af3 (0x04085 bytes) 153: 1763336274.173573 READ 0x023a7 thru 0x09af3 (0x0774d bytes) 154: 1763336274.174659 WRITE 0x1ece3 thru 0x2d201 (0x0e51f bytes) HOLE 155: 1763336274.178280 ZERO 0x2343b thru 0x24af7 (0x016bd bytes) 156: 1763336274.196951 TRUNC_UP 0x2d202 to 0x39999 157: 1763336274.216848 MIRROR_RESYNC 3 mirrors 158: 1763336275.196935 FALLOC 0x1efd3 thru 0x219da (0x02a07 bytes)INTERIOR 159: 1763336275.229285 TRUNC_DN 0x39999 to 0x1d959 160: 1763336275.261177 WRITE 0x03e71 thru 0x0995e (0x05aee bytes) 161: 1763336275.293546 MIRROR_SPLIT mirror 1 to 2 mirrors 162: 1763336275.629472 READ 0x1b7b6 thru 0x1d958 (0x021a3 bytes) 163: 1763336275.636250 READ 0x04344 thru 0x0bc91 (0x0794e bytes) 164: 1763336275.646444 MAPREAD 0x0d0e0 thru 0x13ee4 (0x06e05 bytes) 165: 1763336275.648090 WRITE 0x2f9ad thru 0x3c7a9 (0x0cdfd bytes) HOLE 166: 1763336275.658107 MIRROR_SPLIT mirror 1 to 1 mirrors 167: 1763336275.795420 MAPWRITE 0x1542c thru 0x1e2b6 (0x08e8b bytes) 168: 1763336275.833239 READ 0x1bfaf thru 0x1dc26 (0x01c78 bytes) 169: 1763336275.837047 MAPREAD 0x197b0 thru 0x1dcd0 (0x04521 bytes) 170: 1763336275.848438 MIRROR_EXTEND to 2 mirrors 171: 1763336276.596722 WRITE 0x0e918 thru 0x189bd (0x0a0a6 bytes) 172: 1763336276.663464 WRITE 0x1a301 thru 0x20832 (0x06532 bytes) 173: 1763336276.675988 MAPWRITE 0x3840e thru 0x3f51c (0x0710f bytes) 174: 1763336276.735219 WRITE 0x0dccf thru 0x16f15 (0x09247 bytes) 175: 1763336276.748438 MAPREAD 0x03c8b thru 0x0b1f5 (0x0756b bytes) 176: 1763336276.765987 WRITE 0x0e744 thru 0x1b8c3 (0x0d180 bytes) 177: 1763336276.776214 MIRROR_EXTEND to 3 mirrors 178: 1763336277.573876 FALLOC 0x10a40 thru 0x1e605 (0x0dbc5 bytes)INTERIOR 179: 1763336277.609722 TRUNC_DN 0x3f51d to 0x0cad8 180: 1763336277.632433 ZERO 0x0ca11 thru 0x0cad7 (0x000c7 bytes) 181: 1763336277.646194 WRITE 0x254b6 thru 0x29ad0 (0x0461b bytes) HOLE 182: 1763336277.647638 READ 0x240d3 thru 0x29ad0 (0x059fe bytes) 183: 1763336277.654975 READ 0x146a6 thru 0x1d555 (0x08eb0 bytes) 184: 1763336277.663698 ZERO 0x1b9ab thru 0x2981e (0x0de74 bytes) 185: 1763336277.676401 ZERO 0x13210 thru 0x22a7e (0x0f86f bytes) 186: 1763336277.680880 ZERO 0x18d17 thru 0x198ed (0x00bd7 bytes) 187: 1763336277.693107 ZERO 0x04148 thru 0x10bc2 (0x0ca7b bytes) 188: 1763336277.700084 MAPREAD 0x1f969 thru 0x29ad0 (0x0a168 bytes) 189: 1763336277.773874 WRITE 0x1ca7d thru 0x2565e (0x08be2 bytes) 190: 1763336277.781469 PUNCH 0x0a9a0 thru 0x19d81 (0x0f3e2 bytes) 191: 1763336277.786734 PUNCH 0x264e9 thru 0x28b6e (0x02686 bytes) 192: 1763336277.790380 WRITE 0x2c452 thru 0x350b8 (0x08c67 bytes) HOLE 193: 1763336277.791937 MAPREAD 0x26af1 thru 0x2dccc (0x071dc bytes) 194: 1763336277.813583 MAPREAD 0x1ab2b thru 0x27c63 (0x0d139 bytes) 195: 1763336277.831748 MIRROR_SPLIT mirror 1 to 2 mirrors 196: 1763336278.109879 TRUNC_DN 0x350b9 to 0x0826a 197: 1763336278.132109 MIRROR_RESYNC 2 mirrors 198: 1763336278.679812 WRITE 0x0f4c0 thru 0x1deb3 (0x0e9f4 bytes) HOLE 199: 1763336278.726873 FALLOC 0x35225 thru 0x3d490 (0x0826b bytes)EXTENDING 200: 1763336278.733920 MAPWRITE 0x11f88 thru 0x1c69e (0x0a717 bytes) 201: 1763336278.750671 READ 0x123f4 thru 0x1e396 (0x0bfa3 bytes) 202: 1763336278.756146 MAPWRITE 0x26257 thru 0x2f3c6 (0x09170 bytes) 203: 1763336278.797111 READ 0x18727 thru 0x1da1e (0x052f8 bytes) 204: 1763336278.803774 MAPWRITE 0x0ada8 thru 0x18000 (0x0d259 bytes) 205: 1763336278.841272 WRITE 0x1d819 thru 0x27963 (0x0a14b bytes) 206: 1763336278.845792 WRITE 0x19e34 thru 0x1fe83 (0x06050 bytes) 207: 1763336278.847092 MAPWRITE 0x2f643 thru 0x35482 (0x05e40 bytes) 208: 1763336278.865490 PUNCH 0x0a30b thru 0x0fd32 (0x05a28 bytes) 209: 1763336278.872986 TRUNC_DN 0x3d490 to 0x0c3c8 210: 1763336278.915610 WRITE 0x1a0cd thru 0x299fc (0x0f930 bytes) HOLE 211: 1763336278.917787 PUNCH 0x07988 thru 0x0863f (0x00cb8 bytes) 212: 1763336278.925221 MAPWRITE 0x352c8 thru 0x3ffff (0x0ad38 bytes) 213: 1763336279.028469 MIRROR_EXTEND to 3 mirrors 214: 1763336279.718377 PUNCH 0x14e6a thru 0x1b1b0 (0x06347 bytes) 215: 1763336279.766805 MIRROR_RESYNC 3 mirrors 216: 1763336280.618695 PUNCH 0x2c552 thru 0x34d2e (0x087dd bytes) 217: 1763336280.657857 READ 0x19816 thru 0x20b69 (0x07354 bytes) 218: 1763336280.670392 WRITE 0x0aa57 thru 0x19b38 (0x0f0e2 bytes) 219: 1763336280.679586 TRUNC_DN 0x40000 to 0x21bed 220: 1763336280.695264 PUNCH 0x17285 thru 0x1c64b (0x053c7 bytes) 221: 1763336280.710607 READ 0x0cdc2 thru 0x14172 (0x073b1 bytes) 222: 1763336280.711192 MAPWRITE 0x35190 thru 0x3ffff (0x0ae70 bytes) 223: 1763336280.747893 FALLOC 0x22c4a thru 0x255cd (0x02983 bytes)INTERIOR 224: 1763336280.752328 FALLOC 0x239f0 thru 0x30df3 (0x0d403 bytes)INTERIOR 225: 1763336280.755891 PUNCH 0x3101b thru 0x38eee (0x07ed4 bytes) 226: 1763336280.760468 READ 0x2bf76 thru 0x2e9ca (0x02a55 bytes) 227: 1763336280.761240 ZERO 0x3ed80 thru 0x3ffff (0x01280 bytes) 228: 1763336280.767475 WRITE 0x1b44d thru 0x1e290 (0x02e44 bytes) 229: 1763336280.768647 MAPWRITE 0x0a66c thru 0x12007 (0x0799c bytes) 230: 1763336280.788236 TRUNC_DN 0x40000 to 0x2a10c 231: 1763336280.799227 READ 0x087d6 thru 0x09982 (0x011ad bytes) 232: 1763336280.799985 FALLOC 0x04f66 thru 0x076e3 (0x0277d bytes)INTERIOR 233: 1763336280.805779 MAPWRITE 0x2fc31 thru 0x33383 (0x03753 bytes) 234: 1763336280.860968 READ 0x31cbe thru 0x33383 (0x016c6 bytes) 235: 1763336280.863152 MAPWRITE 0x0205e thru 0x09fe7 (0x07f8a bytes) 236: 1763336280.883445 READ 0x0ea57 thru 0x1d581 (0x0eb2b bytes) 237: 1763336280.887824 MIRROR_EXTEND to 4 mirrors 238: 1763336281.731516 WRITE 0x12928 thru 0x1de6f (0x0b548 bytes) 239: 1763336281.806788 FALLOC 0x2c637 thru 0x350be (0x08a87 bytes)PAST_EOF 240: 1763336281.822726 MIRROR_RESYNC 4 mirrors 241: 1763336282.641845 READ 0x32a83 thru 0x33383 (0x00901 bytes) 242: 1763336282.660501 MAPWRITE 0x24f72 thru 0x2fe3d (0x0aecc bytes) 243: 1763336282.757033 WRITE 0x0f7c5 thru 0x1ed1e (0x0f55a bytes) 244: 1763336282.771234 MAPWRITE 0x24387 thru 0x2e404 (0x0a07e bytes) 245: 1763336282.798446 TRUNC_UP 0x33384 to 0x3ef86 246: 1763336282.818944 WRITE 0x2dc05 thru 0x2f86a (0x01c66 bytes) 247: 1763336282.819996 MAPREAD 0x1a0f6 thru 0x1b557 (0x01462 bytes) 248: 1763336282.820822 READ 0x18c46 thru 0x1fa84 (0x06e3f bytes) 249: 1763336282.824082 WRITE 0x091e9 thru 0x0aee7 (0x01cff bytes) 250: 1763336282.828102 TRUNC_DN 0x3ef86 to 0x095d5 251: 1763336282.849127 WRITE 0x187b8 thru 0x1a305 (0x01b4e bytes) HOLE 252: 1763336282.851461 READ 0x018bd thru 0x04abb (0x031ff bytes) 253: 1763336282.859916 PUNCH 0x108d5 thru 0x14b36 (0x04262 bytes) 254: 1763336282.865473 FALLOC 0x15294 thru 0x18e59 (0x03bc5 bytes)INTERIOR 255: 1763336282.870932 MAPREAD 0x1159a thru 0x150b6 (0x03b1d bytes) 256: 1763336282.890035 TRUNC_UP 0x1a306 to 0x20154 257: 1763336282.909025 FALLOC 0x36b87 thru 0x3ac7b (0x040f4 bytes)EXTENDING 258: 1763336282.914515 ZERO 0x1b2b5 thru 0x1ee17 (0x03b63 bytes) 259: 1763336282.929663 FALLOC 0x084a8 thru 0x13e1b (0x0b973 bytes)INTERIOR 260: 1763336282.941435 MIRROR_EXTEND to 5 mirrors Using seed 6262 Correct content saved for comparison (maybe hexdump "/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr" vs "/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr.fsxgood") sanity-flr test_70a: @@@@@@ FAIL: fsx FLR file /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr failed Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7314:error() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:3373:test_70a() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7690:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7753:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7568:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:3375:main() Dumping lctl log to /tmp/testlogs//sanity-flr.test_70a.*.1763336285.log rsync: chown "/tmp/testlogs/.sanity-flr.test_70a.debug_log.oleg316-server.1763336285.log.xNV9Nn" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-flr.test_70a.dmesg.oleg316-server.1763336285.log.1DnhQm" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1668) [generator=3.1.3]