== sanity-flr test 70a: flr mode fsx test ================ 20:06:59 (1785456419) Seed set to 6421 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read 000003 1785456421.479995 mapwrite 0x01425e thru 0x02079e (0xc541 bytes) 000004 1785456421.629033 read 0x00be97 thru 0x01ab22 (0xec8c bytes) 000008 1785456421.662788 mapwrite 0x01245c thru 0x01d5ee (0xb193 bytes) 000009 1785456421.782541 MIRROR_EXTEND to 2 mirrors 000011 1785456422.675033 mapread 0x00b3dc thru 0x00d615 (0x223a bytes) 000012 1785456422.728815 mapwrite 0x0361d8 thru 0x03ffff (0x9e28 bytes) 000015 1785456423.022660 read 0x01ceda thru 0x01e7cc (0x18f3 bytes) 000016 1785456423.033138 mapwrite 0x032d86 thru 0x035082 (0x22fd bytes) 000021 1785456423.145143 write 0x003b80 thru 0x00d479 (0x98fa bytes) 000022 1785456423.168780 mapwrite 0x012fe2 thru 0x01a2be (0x72dd bytes) 000023 1785456423.325706 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18c:0x0]: Invalid argument (22) 000024 1785456423.560413 mapread 0x025cc6 thru 0x028860 (0x2b9b bytes) 000025 1785456423.626348 mapwrite 0x03c43f thru 0x03ffff (0x3bc1 bytes) truncating to largest ever: 0x7101 000030 1785456423.746660 trunc from 0x040000 to 0x007100 (0x38eff bytes) truncating to largest ever: 0x1b11a 000032 1785456423.780739 trunc from 0x007101 to 0x01b119 (0x14019 bytes) 000034 1785456423.811675 read 0x01adc9 thru 0x01b119 (0x351 bytes) 000035 1785456423.826719 mapread 0x008eed thru 0x00df49 (0x505d bytes) 000036 1785456423.852340 mapwrite 0x0336e3 thru 0x03d4ca (0x9de8 bytes) 000037 1785456424.001198 mapwrite 0x00deb2 thru 0x00f20f (0x135e bytes) 000040 1785456424.065963 MIRROR_EXTEND to 2 mirrors 000041 1785456424.900759 mapread 0x0337f6 thru 0x03d4ca (0x9cd5 bytes) 000046 1785456424.993834 write 0x000802 thru 0x00ede1 (0xe5e0 bytes) 000047 1785456425.094202 read 0x027fae thru 0x02acda (0x2d2d bytes) 000049 1785456425.103509 mapread 0x029e55 thru 0x029f9f (0x14b bytes) 000052 1785456425.107600 mapread 0x008f7d thru 0x012adc (0x9b60 bytes) 000053 1785456425.122320 mapwrite 0x01044f thru 0x017e2e (0x79e0 bytes) 000054 1785456425.190074 trunc from 0x03d4cb to 0x017cd0 (0x257fa bytes) 000056 1785456425.250756 read 0x00eb11 thru 0x0137c6 (0x4cb6 bytes) 000057 1785456425.258986 MIRROR_RESYNC 2 mirrors 000059 1785456426.086426 mapwrite 0x011dcc thru 0x013ea5 (0x20da bytes) 000061 1785456426.280595 mapwrite 0x03a741 thru 0x03c1ad (0x1a6d bytes) truncating to largest ever: 0x1bcef 000062 1785456426.408919 trunc from 0x03c1ae to 0x01bcee (0x204bf bytes) 000063 1785456426.433383 read 0x0129ee thru 0x01bcee (0x9301 bytes) truncating to largest ever: 0x22dec 000065 1785456426.441578 trunc from 0x01bcef to 0x022deb (0x70fd bytes) 000067 1785456426.485556 mapread 0x01e929 thru 0x022deb (0x44c3 bytes) 000068 1785456426.501544 read 0x0183be thru 0x01f223 (0x6e66 bytes) 000069 1785456426.502340 mapwrite 0x03dfdb thru 0x03ffff (0x2025 bytes) 000070 1785456426.599881 trunc from 0x040000 to 0x00c0ee (0x33f11 bytes) 000071 1785456426.639610 mapwrite 0x01567f thru 0x0206aa (0xb02c bytes) 000073 1785456426.836563 read 0x005c31 thru 0x01577e (0xfb4e bytes) 000074 1785456426.863761 trunc from 0x0206ab to 0x01288d (0xde1d bytes) 000075 1785456426.904416 mapread 0x00ce35 thru 0x01288d (0x5a59 bytes) 000076 1785456426.920617 mapread 0x000b2f thru 0x00b3a5 (0xa877 bytes) 000077 1785456426.925671 write 0x012728 thru 0x013167 (0xa40 bytes) truncating to largest ever: 0x3c6a8 000079 1785456426.939303 trunc from 0x013168 to 0x03c6a7 (0x29540 bytes) 000080 1785456426.972468 mapread 0x030c93 thru 0x03c6a7 (0xba15 bytes) 000081 1785456427.021981 write 0x01c819 thru 0x02723a (0xaa22 bytes) 000082 1785456427.030229 read 0x013a49 thru 0x0229df (0xef97 bytes) 000084 1785456427.044355 mapread 0x01a8da thru 0x028edf (0xe606 bytes) 000085 1785456427.050803 MIRROR_RESYNC 2 mirrors 000086 1785456427.811046 read 0x02d236 thru 0x03aa9e (0xd869 bytes) 000087 1785456427.832277 mapread 0x018774 thru 0x0230c7 (0xa954 bytes) 000091 1785456427.923835 write 0x0383f4 thru 0x03ffff (0x7c0c bytes) 000092 1785456428.067989 MIRROR_RESYNC 2 mirrors 000093 1785456428.940941 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18c:0x0]: Invalid argument (22) 000095 1785456429.100080 MIRROR_EXTEND to 2 mirrors 000096 1785456429.998481 mapread 0x0246a2 thru 0x02b815 (0x7174 bytes) 000097 1785456430.078270 write 0x0224a4 thru 0x0224f8 (0x55 bytes) 000099 1785456430.190965 read 0x00fd30 thru 0x012eb7 (0x3188 bytes) 000102 1785456430.201508 trunc from 0x040000 to 0x0326ab (0xd954 bytes) 000105 1785456430.226362 write 0x0086eb thru 0x00deb8 (0x57ce bytes) 000106 1785456430.243490 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x326ac): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x326ac): Mirror 2: 0xa62bf443 (stale) CRC-32 checksum value for chunk [0, 0x326ac): Mirror 1: 0xd6756613 Mirror 3: 0xd6756613 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x326ac: skipped 000107 1785456431.343354 mapwrite 0x00a0c7 thru 0x00c20e (0x2148 bytes) 000109 1785456431.536560 read 0x00357d thru 0x0054ca (0x1f4e bytes) 000110 1785456431.548558 read 0x0221c8 thru 0x0271c4 (0x4ffd bytes) 000112 1785456431.564830 mapwrite 0x03ce94 thru 0x03ffff (0x316c bytes) 000113 1785456431.673793 write 0x02ea34 thru 0x0391c7 (0xa794 bytes) 000114 1785456431.676718 mapwrite 0x03fe8c thru 0x03ffff (0x174 bytes) 000115 1785456431.725502 write 0x039b55 thru 0x03efa5 (0x5451 bytes) 000116 1785456431.731765 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xce9ee413 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xce9ee413 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000117 1785456432.041494 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18c:0x0]: Invalid argument (22) 000118 1785456432.167146 read 0x03a740 thru 0x03ffff (0x58c0 bytes) 000119 1785456432.176356 mapwrite 0x0169b2 thru 0x01d50f (0x6b5e bytes) 000120 1785456432.236442 read 0x02ffe9 thru 0x0320a9 (0x20c1 bytes) 000121 1785456432.247230 write 0x00be6a thru 0x00c908 (0xa9f bytes) 000123 1785456432.248787 mapread 0x01458f thru 0x0193a6 (0x4e18 bytes) 000124 1785456432.250849 trunc from 0x040000 to 0x009840 (0x367bf bytes) 000126 1785456432.270050 trunc from 0x009841 to 0x0004c8 (0x9378 bytes) 000129 1785456432.288618 mapwrite 0x01073d thru 0x016ee5 (0x67a9 bytes) 000130 1785456432.383385 mapwrite 0x00f838 thru 0x0183ff (0x8bc8 bytes) 000131 1785456432.461595 read 0x010c65 thru 0x0183ff (0x779b bytes) 000133 1785456432.462874 trunc from 0x018400 to 0x019e3a (0x1a3b bytes) 000134 1785456432.484272 mapwrite 0x02ccf2 thru 0x02de13 (0x1122 bytes) truncating to largest ever: 0x3fd84 000135 1785456432.547019 trunc from 0x02de14 to 0x03fd83 (0x11f70 bytes) 000136 1785456432.562479 write 0x02bcff thru 0x0313e1 (0x56e3 bytes) 000137 1785456432.568197 mapwrite 0x03cafb thru 0x03ffff (0x3505 bytes) 000140 1785456432.669588 mapread 0x036810 thru 0x03c7d3 (0x5fc4 bytes) 000141 1785456432.670873 write 0x0330cb thru 0x03ffff (0xcf35 bytes) 000143 1785456432.675521 mapread 0x036076 thru 0x0383a8 (0x2333 bytes) 000144 1785456432.677091 write 0x0254a0 thru 0x033717 (0xe278 bytes) 000145 1785456432.679960 mapread 0x006290 thru 0x00731e (0x108f bytes) 000146 1785456432.681918 write 0x02f6b0 thru 0x03b3b6 (0xbd07 bytes) 000147 1785456432.683531 read 0x001a41 thru 0x004e62 (0x3422 bytes) 000157 1785456432.694349 mapwrite 0x03df37 thru 0x03ffff (0x20c9 bytes) 000158 1785456432.770804 mapread 0x017356 thru 0x01de6c (0x6b17 bytes) 000159 1785456432.773951 write 0x03fa17 thru 0x03ffff (0x5e9 bytes) 000161 1785456432.777584 mapread 0x0209cc thru 0x02a162 (0x9797 bytes) 000162 1785456432.781462 trunc from 0x040000 to 0x02f179 (0x10e86 bytes) 000164 1785456432.808607 mapwrite 0x032225 thru 0x0379c8 (0x57a4 bytes) 000168 1785456432.936717 trunc from 0x0379c9 to 0x00c7a6 (0x2b222 bytes) 000173 1785456432.957245 mapread 0x00ab50 thru 0x00c7a6 (0x1c57 bytes) 000174 1785456432.958409 trunc from 0x00c7a7 to 0x01ac93 (0xe4ed bytes) 000175 1785456432.997970 trunc from 0x01ac94 to 0x01f17c (0x44e9 bytes) 000179 1785456433.024816 mapread 0x0118b4 thru 0x01cb19 (0xb266 bytes) 000180 1785456433.087112 mapwrite 0x00e47a thru 0x01de4d (0xf9d4 bytes) 000183 1785456433.185680 mapwrite 0x015252 thru 0x01d69e (0x844d bytes) 000185 1785456433.227875 read 0x010241 thru 0x017e17 (0x7bd7 bytes) 000186 1785456433.233581 MIRROR_EXTEND to 2 mirrors 000187 1785456433.855102 mapwrite 0x030632 thru 0x03990f (0x92de bytes) 000189 1785456434.198567 mapread 0x0207da thru 0x029b82 (0x93a9 bytes) 000190 1785456434.227828 mapread 0x0243a0 thru 0x02525b (0xebc bytes) 000192 1785456434.230814 mapread 0x028993 thru 0x02ffd4 (0x7642 bytes) 000194 1785456434.235578 write 0x0320a8 thru 0x03acdf (0x8c38 bytes) 000196 1785456434.239594 write 0x01bdfc thru 0x01f331 (0x3536 bytes) 000197 1785456434.244434 write 0x01dbd0 thru 0x02709c (0x94cd bytes) 000199 1785456434.255868 MIRROR_RESYNC 2 mirrors 000200 1785456435.283924 mapread 0x0028f1 thru 0x0070fc (0x480c bytes) 000202 1785456435.305614 mapread 0x030c51 thru 0x037060 (0x6410 bytes) 000204 1785456435.324230 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18c:0x0]: Invalid argument (22) 000205 1785456435.452515 write 0x007fd9 thru 0x013de0 (0xbe08 bytes) 000206 1785456435.474604 mapwrite 0x016dcf thru 0x026dbc (0xffee bytes) 000207 1785456435.585356 MIRROR_EXTEND to 2 mirrors 000208 1785456436.351821 mapread 0x0293a3 thru 0x034c2e (0xb88c bytes) 000209 1785456436.390912 mapread 0x00dad4 thru 0x013b18 (0x6045 bytes) 000210 1785456436.415384 mapwrite 0x006af6 thru 0x00f9dd (0x8ee8 bytes) 000212 1785456436.591040 mapwrite 0x034687 thru 0x03ffff (0xb979 bytes) 000213 1785456436.699468 write 0x012fac thru 0x01d3bd (0xa412 bytes) 000219 1785456436.724480 mapread 0x03ad7d thru 0x03ffff (0x5283 bytes) 000221 1785456436.741047 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18c:0x0]: Invalid argument (22) 000223 1785456436.985833 read 0x0189b7 thru 0x01e587 (0x5bd1 bytes) 000224 1785456436.997599 read 0x0025f4 thru 0x005245 (0x2c52 bytes) 000225 1785456437.019692 mapread 0x02a793 thru 0x032297 (0x7b05 bytes) 000227 1785456437.025031 write 0x030a06 thru 0x034bf1 (0x41ec bytes) 000228 1785456437.026366 read 0x036ab6 thru 0x039fbe (0x3509 bytes) 000230 1785456437.030084 write 0x0008bc thru 0x00cab7 (0xc1fc bytes) 000231 1785456437.032519 trunc from 0x040000 to 0x02a923 (0x156dc bytes) 000233 1785456437.056192 mapread 0x003a8f thru 0x0093da (0x594c bytes) 000237 1785456437.061252 mapwrite 0x00b660 thru 0x011c1e (0x65bf bytes) 000238 1785456437.106485 MIRROR_EXTEND to 2 mirrors 000240 1785456437.752101 trunc from 0x02a924 to 0x004530 (0x263f3 bytes) 000241 1785456437.931006 mapwrite 0x02b5be thru 0x031cb2 (0x66f5 bytes) 000243 1785456438.137901 read 0x02d20e thru 0x02dd0d (0xb00 bytes) 000244 1785456438.149063 MIRROR_RESYNC 2 mirrors 000245 1785456438.984516 mapread 0x004315 thru 0x0115a5 (0xd291 bytes) 000247 1785456439.016092 mapread 0x00a796 thru 0x0127e5 (0x8050 bytes) 000248 1785456439.017269 read 0x01b2f0 thru 0x021161 (0x5e72 bytes) 000249 1785456439.023851 trunc from 0x031cb3 to 0x02aa11 (0x72a1 bytes) 000253 1785456439.141183 read 0x000840 thru 0x00c305 (0xbac6 bytes) 000254 1785456439.152430 mapwrite 0x03e44d thru 0x03ffff (0x1bb3 bytes) 000256 1785456439.282679 MIRROR_RESYNC 2 mirrors 000258 1785456440.222108 write 0x01ffcc thru 0x02f582 (0xf5b7 bytes) 000261 1785456440.298100 trunc from 0x040000 to 0x01641c (0x29be3 bytes) 000266 1785456440.366224 mapread 0x00c70c thru 0x01641c (0x9d11 bytes) 000268 1785456440.407557 mapread 0x007615 thru 0x0131e9 (0xbbd5 bytes) 000269 1785456440.412801 trunc from 0x01641d to 0x031f4c (0x1bb30 bytes) 000270 1785456440.433008 write 0x01a320 thru 0x028ff4 (0xecd5 bytes) 000271 1785456440.450074 read 0x02136f thru 0x02c43a (0xb0cc bytes) 000273 1785456440.458384 mapwrite 0x00b2ae thru 0x01b289 (0xffdc bytes) 000276 1785456440.525925 trunc from 0x031f4d to 0x00ff52 (0x21ffa bytes) 000278 1785456440.561233 read 0x008dc8 thru 0x00ff52 (0x718b bytes) 000280 1785456440.565310 mapwrite 0x00cf8d thru 0x019740 (0xc7b4 bytes) 000281 1785456440.655121 mapread 0x008dcc thru 0x00c79b (0x39d0 bytes) 000283 1785456440.658306 mapread 0x018fdb thru 0x019150 (0x176 bytes) 000284 1785456440.659905 mapwrite 0x023a6a thru 0x029268 (0x57ff bytes) 000286 1785456440.733703 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x29269): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29269): Mirror 2: 0x2fbc4393 (stale) CRC-32 checksum value for chunk [0, 0x29269): Mirror 1: 0xffecd645 Mirror 3: 0xffecd645 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29269: skipped 000287 1785456441.858701 write 0x00c8d2 thru 0x00d120 (0x84f bytes) 000288 1785456441.962586 write 0x01239d thru 0x01ff99 (0xdbfd bytes) 000289 1785456441.976128 mapread 0x000cc7 thru 0x004ab2 (0x3dec bytes) 000291 1785456442.007415 trunc from 0x029269 to 0x015a66 (0x13802 bytes) 000292 1785456442.038512 trunc from 0x015a67 to 0x0198e1 (0x3e7b bytes) 000293 1785456442.055929 trunc from 0x0198e2 to 0x001d60 (0x17b81 bytes) 000294 1785456442.079772 mapwrite 0x0220e4 thru 0x02d1c4 (0xb0e1 bytes) 000295 1785456442.158956 trunc from 0x02d1c5 to 0x0180b7 (0x1510d bytes) 000297 1785456442.178602 trunc from 0x0180b8 to 0x03f620 (0x27569 bytes) 000298 1785456442.200252 write 0x01fd57 thru 0x0206f8 (0x9a2 bytes) 000299 1785456442.216617 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f621): Mirror 2: 0xe82f7eae (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f621): Mirror 2: 0xe82f7eae (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f621: skipped 000301 1785456442.568234 write 0x01f7e8 thru 0x025a00 (0x6219 bytes) 000302 1785456442.578050 trunc from 0x03f621 to 0x029bb7 (0x15a69 bytes) 000303 1785456442.592222 mapread 0x00df1e thru 0x00e7f0 (0x8d3 bytes) 000306 1785456442.603784 trunc from 0x029bb8 to 0x02325e (0x6959 bytes) 000307 1785456442.622403 mapwrite 0x004cc0 thru 0x00e9df (0x9d20 bytes) 000308 1785456442.675558 write 0x02af7a thru 0x02ed66 (0x3ded bytes) 000309 1785456442.676931 write 0x0044cc thru 0x011f49 (0xda7e bytes) 000310 1785456442.683313 trunc from 0x02ed67 to 0x02dcd9 (0x108d bytes) 000311 1785456442.701097 trunc from 0x02dcda to 0x0377a8 (0x9acf bytes) 000312 1785456442.714152 trunc from 0x0377a9 to 0x01d346 (0x1a462 bytes) 000313 1785456442.731556 trunc from 0x01d347 to 0x03da69 (0x20723 bytes) 000316 1785456442.752343 write 0x02622f thru 0x02b454 (0x5226 bytes) 000317 1785456442.771144 trunc from 0x03da6a to 0x01f2d0 (0x1e799 bytes) 000318 1785456442.803676 mapwrite 0x037b10 thru 0x03a701 (0x2bf2 bytes) 000319 1785456442.921450 read 0x024d50 thru 0x0301e5 (0xb496 bytes) 000322 1785456442.960823 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18c:0x0]: Invalid argument (22) 000324 1785456443.118372 write 0x0361cb thru 0x03ffff (0x9e35 bytes) 000325 1785456443.127957 read 0x022f79 thru 0x02adef (0x7e77 bytes) 000326 1785456443.161268 MIRROR_EXTEND to 2 mirrors 000328 1785456444.274232 MIRROR_RESYNC 2 mirrors 000329 1785456444.662556 mapread 0x03ac34 thru 0x03ec91 (0x405e bytes) 000330 1785456444.726501 trunc from 0x040000 to 0x02e920 (0x116df bytes) 000331 1785456444.869835 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2e921): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e921): Mirror 2: 0x9699f7c (stale) CRC-32 checksum value for chunk [0, 0x2e921): Mirror 1: 0x9699f7c Mirror 3: 0x9699f7c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e921: skipped 000333 1785456445.928894 trunc from 0x02e921 to 0x03c243 (0xd923 bytes) 000335 1785456446.096280 mapwrite 0x0057ff thru 0x00ca62 (0x7264 bytes) 000337 1785456446.164514 mapwrite 0x03ead8 thru 0x03ffff (0x1528 bytes) 000338 1785456446.226516 trunc from 0x040000 to 0x02df7a (0x12085 bytes) 000340 1785456446.245017 write 0x0069f0 thru 0x00d45d (0x6a6e bytes) 000341 1785456446.251826 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2df7b): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2df7b): Mirror 2: 0xd9ff6867 (stale) Mirror 3: 0xd9ff6867 (stale) CRC-32 checksum value for chunk [0, 0x2df7b): Mirror 1: 0x24b9e190 Mirror 4: 0x24b9e190 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2df7b: skipped 000343 1785456447.481667 write 0x03be93 thru 0x03fb08 (0x3c76 bytes) 000344 1785456447.565159 write 0x02703e thru 0x036bd4 (0xfb97 bytes) 000346 1785456447.581253 mapwrite 0x0399b7 thru 0x03ffff (0x6649 bytes) 000349 1785456447.676721 mapwrite 0x0256ff thru 0x029600 (0x3f02 bytes) 000350 1785456447.726035 mapwrite 0x0199d9 thru 0x022f4f (0x9577 bytes) 000354 1785456447.775391 read 0x03a766 thru 0x03e0f9 (0x3994 bytes) 000355 1785456447.776171 mapread 0x03fbbb thru 0x03ffff (0x445 bytes) 000356 1785456447.778201 trunc from 0x040000 to 0x02df65 (0x1209a bytes) 000358 1785456447.808868 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2df66): Mirror 2: 0x1604e487 (stale) Mirror 4: 0x70e55a5b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2df66): Mirror 2: 0x1604e487 (stale) Mirror 4: 0x70e55a5b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2df66: skipped 000360 1785456448.199896 trunc from 0x02df66 to 0x01e802 (0xf763 bytes) 000363 1785456448.233830 trunc from 0x01e803 to 0x000149 (0x1e6b9 bytes) 000365 1785456448.275303 MIRROR_RESYNC 3 mirrors 000366 1785456449.133508 read 0x00002f thru 0x000149 (0x11b bytes) 000369 1785456449.140220 trunc from 0x00014a to 0x0090b2 (0x8f69 bytes) 000370 1785456449.208208 mapread 0x0078b7 thru 0x0090b2 (0x17fc bytes) 000372 1785456449.228050 mapwrite 0x0088bc thru 0x00e698 (0x5ddd bytes) 000373 1785456449.302923 MIRROR_RESYNC 3 mirrors 000374 1785456449.931281 read 0x00cb06 thru 0x00e698 (0x1b93 bytes) 000375 1785456449.938262 write 0x002d99 thru 0x00ddcb (0xb033 bytes) 000377 1785456450.038159 MIRROR_RESYNC 3 mirrors 000379 1785456451.211534 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18c:0x0]: Invalid argument (22) 000380 1785456451.521600 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18c:0x0]: Invalid argument (22) 000384 1785456451.719200 write 0x022ebb thru 0x0278c8 (0x4a0e bytes) 000385 1785456451.733810 mapread 0x00ec52 thru 0x01abd6 (0xbf85 bytes) 000386 1785456451.761023 write 0x01b3b2 thru 0x02659c (0xb1eb bytes) 000387 1785456451.762509 write 0x01cd92 thru 0x02247f (0x56ee bytes) 000389 1785456451.765372 read 0x013c4a thru 0x019dad (0x6164 bytes) 000391 1785456451.767288 write 0x007303 thru 0x01418a (0xce88 bytes) 000392 1785456451.772926 trunc from 0x0278c9 to 0x0250b5 (0x2813 bytes) 000394 1785456451.789478 mapwrite 0x02acca thru 0x02b4c4 (0x7fb bytes) 000395 1785456451.866247 mapwrite 0x031e3a thru 0x03ffff (0xe1c6 bytes) 000398 1785456451.941237 mapread 0x02499f thru 0x032038 (0xd69a bytes) 000399 1785456451.943728 trunc from 0x040000 to 0x0221b1 (0x1de4e bytes) 000400 1785456451.958744 mapwrite 0x0352f1 thru 0x03f4a9 (0xa1b9 bytes) 000401 1785456452.097016 read 0x02fba8 thru 0x030bec (0x1045 bytes) 000402 1785456452.099059 mapread 0x012631 thru 0x01ba00 (0x93d0 bytes) 000403 1785456452.102601 read 0x0094a0 thru 0x00ed65 (0x58c6 bytes) 000405 1785456452.106245 mapwrite 0x0362e0 thru 0x03ffff (0x9d20 bytes) 000406 1785456452.213611 trunc from 0x040000 to 0x023994 (0x1c66b bytes) 000407 1785456452.240597 read 0x01d216 thru 0x023994 (0x677f bytes) 000408 1785456452.266533 write 0x02910d thru 0x035023 (0xbf17 bytes) 000410 1785456452.272478 read 0x00d86b thru 0x01b4d2 (0xdc68 bytes) 000411 1785456452.275234 mapread 0x012f22 thru 0x0179ad (0x4a8c bytes) 000414 1785456452.282148 write 0x003ac0 thru 0x008a10 (0x4f51 bytes) 000415 1785456452.284878 read 0x01a851 thru 0x020dc7 (0x6577 bytes) 000417 1785456452.287907 write 0x028c66 thru 0x02fd8e (0x7129 bytes) 000418 1785456452.308573 mapwrite 0x000804 thru 0x008745 (0x7f42 bytes) 000419 1785456452.387580 MIRROR_EXTEND to 2 mirrors 000421 1785456453.008011 MIRROR_EXTEND to 3 mirrors 000423 1785456453.821542 trunc from 0x035024 to 0x03e2e6 (0x92c3 bytes) 000425 1785456453.960216 MIRROR_RESYNC 3 mirrors 000426 1785456454.803957 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18c:0x0]: Invalid argument (22) 000429 1785456455.179347 mapread 0x016871 thru 0x025c20 (0xf3b0 bytes) 000431 1785456455.213271 trunc from 0x03e2e7 to 0x010b8e (0x2d758 bytes) 000434 1785456455.316593 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18c:0x0]: Invalid argument (22) 000436 1785456455.458570 write 0x03f701 thru 0x03ffff (0x8ff bytes) 000437 1785456455.461829 write 0x0022b0 thru 0x008f81 (0x6cd2 bytes) 000438 1785456455.492145 mapread 0x007ec5 thru 0x0091e9 (0x1325 bytes) 000440 1785456455.514580 trunc from 0x040000 to 0x027a67 (0x18598 bytes) 000442 1785456455.559201 mapread 0x01aa8d thru 0x02182e (0x6da2 bytes) 000443 1785456455.593445 trunc from 0x027a68 to 0x012365 (0x15702 bytes) 000445 1785456455.617550 MIRROR_EXTEND to 2 mirrors 000448 1785456456.572072 mapwrite 0x013caf thru 0x01a323 (0x6675 bytes) 000449 1785456456.744423 trunc from 0x01a324 to 0x00f935 (0xa9ee bytes) 000450 1785456456.775205 write 0x022b2f thru 0x02f482 (0xc954 bytes) 000452 1785456456.780200 trunc from 0x02f483 to 0x01990c (0x15b76 bytes) 000453 1785456456.810506 trunc from 0x01990d to 0x016af2 (0x2e1a bytes) 000455 1785456456.865046 mapwrite 0x005f68 thru 0x01316a (0xd203 bytes) 000456 1785456456.989768 mapwrite 0x03ca10 thru 0x03f45d (0x2a4e bytes) 000458 1785456457.098319 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18c:0x0]: Invalid argument (22) 000459 1785456457.301985 write 0x025944 thru 0x030ff4 (0xb6b1 bytes) 000467 1785456457.343494 mapwrite 0x029640 thru 0x03352b (0x9eec bytes) 000469 1785456457.437337 write 0x02d473 thru 0x02e537 (0x10c5 bytes) 000470 1785456457.439202 write 0x00d535 thru 0x014889 (0x7355 bytes) 000473 1785456457.462223 write 0x039784 thru 0x03ffff (0x687c bytes) 000474 1785456457.464228 mapwrite 0x039925 thru 0x03ffff (0x66db bytes) 000475 1785456457.508191 read 0x02c3c9 thru 0x03bd58 (0xf990 bytes) 000477 1785456457.511570 trunc from 0x040000 to 0x03744a (0x8bb5 bytes) 000478 1785456457.536785 mapwrite 0x0079b1 thru 0x00f06b (0x76bb bytes) 000479 1785456457.626091 trunc from 0x03744b to 0x03f453 (0x8009 bytes) 000480 1785456457.653565 MIRROR_EXTEND to 2 mirrors 000482 1785456458.697068 mapread 0x01725a thru 0x0186ec (0x1493 bytes) 000483 1785456458.725307 MIRROR_RESYNC 2 mirrors 000484 1785456459.133196 read 0x018f2b thru 0x01d31b (0x43f1 bytes) 000485 1785456459.213031 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18c:0x0]: Invalid argument (22) 000486 1785456459.427598 mapread 0x0112a4 thru 0x015797 (0x44f4 bytes) 000489 1785456459.482190 mapread 0x039e50 thru 0x03f453 (0x5604 bytes) 000491 1785456459.496176 read 0x00b375 thru 0x00cff9 (0x1c85 bytes) 000492 1785456459.503682 write 0x038ad5 thru 0x03942f (0x95b bytes) 000493 1785456459.534796 mapwrite 0x03c3f2 thru 0x03ffff (0x3c0e bytes) 000495 1785456459.619353 mapread 0x006de8 thru 0x0081c4 (0x13dd bytes) 000496 1785456459.636569 mapread 0x010070 thru 0x01ecc4 (0xec55 bytes) 000499 1785456459.666800 trunc from 0x040000 to 0x010e2f (0x2f1d0 bytes) 000500 1785456459.687900 trunc from 0x010e30 to 0x0175c2 (0x6793 bytes) 000501 1785456459.711282 write 0x03710f thru 0x03cf93 (0x5e85 bytes) 000502 1785456459.716097 mapwrite 0x018a94 thru 0x028a7a (0xffe7 bytes) 000504 1785456459.837408 write 0x0297cf thru 0x02d4fa (0x3d2c bytes) 000508 1785456459.868082 MIRROR_EXTEND to 2 mirrors 000512 1785456460.708557 trunc from 0x03cf94 to 0x013bec (0x293a7 bytes) 000514 1785456460.856742 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18c:0x0]: Invalid argument (22) 000516 1785456460.994378 mapwrite 0x03574d thru 0x03653f (0xdf3 bytes) 000518 1785456461.099711 mapwrite 0x0359a3 thru 0x03851a (0x2b78 bytes) 000520 1785456461.169030 mapread 0x014ad8 thru 0x017ac0 (0x2fe9 bytes) 000521 1785456461.186033 write 0x002609 thru 0x00d398 (0xad90 bytes) 000523 1785456461.193555 read 0x0049c3 thru 0x005736 (0xd74 bytes) 000525 1785456461.195155 mapread 0x033290 thru 0x03851a (0x528b bytes) 000527 1785456461.198549 mapread 0x026f01 thru 0x027533 (0x633 bytes) 000528 1785456461.211191 mapwrite 0x00464b thru 0x00a8db (0x6291 bytes) 000530 1785456461.260113 mapread 0x02fb69 thru 0x034de9 (0x5281 bytes) 000532 1785456461.268116 mapread 0x033486 thru 0x03851a (0x5095 bytes) 000535 1785456461.275032 read 0x016c70 thru 0x021efc (0xb28d bytes) 000536 1785456461.285733 write 0x03ff61 thru 0x03ffff (0x9f bytes) 000540 1785456461.290206 mapread 0x0058ef thru 0x009b5a (0x426c bytes) 000541 1785456461.297543 MIRROR_EXTEND to 2 mirrors 000543 1785456462.247168 mapwrite 0x01a1f3 thru 0x01c847 (0x2655 bytes) 000547 1785456462.462484 trunc from 0x040000 to 0x012b29 (0x2d4d6 bytes) 000550 1785456462.521557 read 0x000bac thru 0x00baf5 (0xaf4a bytes) 000551 1785456462.534008 mapwrite 0x0388d2 thru 0x03ffff (0x772e bytes) 000554 1785456462.616410 mapwrite 0x021c2a thru 0x026f90 (0x5367 bytes) 000556 1785456462.666716 mapread 0x03b5dc thru 0x03ffff (0x4a24 bytes) 000557 1785456462.668040 trunc from 0x040000 to 0x00c681 (0x3397e bytes) 000560 1785456462.700104 MIRROR_RESYNC 2 mirrors 000561 1785456463.553076 trunc from 0x00c682 to 0x013d82 (0x7701 bytes) 000562 1785456463.626763 read 0x0125a7 thru 0x013d82 (0x17dc bytes) 000564 1785456463.639985 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18c:0x0]: Invalid argument (22) 000565 1785456463.803158 trunc from 0x013d83 to 0x02e269 (0x1a4e7 bytes) 000568 1785456463.828726 mapwrite 0x03a2c8 thru 0x03ffff (0x5d38 bytes) 000570 1785456463.983463 mapwrite 0x03f4f8 thru 0x03ffff (0xb08 bytes) 000572 1785456464.058861 trunc from 0x040000 to 0x0348d2 (0xb72d bytes) 000574 1785456464.100208 read 0x03237b thru 0x0348d2 (0x2558 bytes) 000575 1785456464.114675 read 0x024cc2 thru 0x033c3d (0xef7c bytes) 000577 1785456464.146164 trunc from 0x0348d3 to 0x0246ef (0x101e3 bytes) 000578 1785456464.162705 mapread 0x001a4a thru 0x0033a5 (0x195c bytes) 000580 1785456464.171284 read 0x00a164 thru 0x00d352 (0x31ef bytes) 000582 1785456464.173611 mapread 0x00dbc6 thru 0x016952 (0x8d8d bytes) 000583 1785456464.175753 read 0x003578 thru 0x004cad (0x1736 bytes) 000586 1785456464.179387 mapread 0x01df87 thru 0x0246ef (0x6769 bytes) 000587 1785456464.181237 mapread 0x02258b thru 0x02371e (0x1194 bytes) 000588 1785456464.185489 trunc from 0x0246f0 to 0x020c0b (0x3ae4 bytes) 000595 1785456464.244147 write 0x000164 thru 0x005434 (0x52d1 bytes) 000596 1785456464.258361 read 0x01d74e thru 0x0200dc (0x298f bytes) 000598 1785456464.269571 MIRROR_EXTEND to 2 mirrors 000601 1785456465.247201 mapwrite 0x02f6cb thru 0x03cd7b (0xd6b1 bytes) 000603 1785456465.497325 mapread 0x03c6aa thru 0x03cd7b (0x6d2 bytes) 000604 1785456465.505146 read 0x026f49 thru 0x032c7f (0xbd37 bytes) 000606 1785456465.526573 mapwrite 0x0372f4 thru 0x03eb4f (0x785c bytes) 000607 1785456465.615949 write 0x031c07 thru 0x03ffff (0xe3f9 bytes) 000609 1785456465.623664 trunc from 0x040000 to 0x012792 (0x2d86d bytes) 000610 1785456465.651563 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x12793): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x12793): Mirror 2: 0xbee2414f (stale) CRC-32 checksum value for chunk [0, 0x12793): Mirror 1: 0xbee2414f Mirror 3: 0xbee2414f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x12793: skipped truncating to largest ever: 0x3feed 000615 1785456466.946827 trunc from 0x012793 to 0x03feec (0x2d75a bytes) 000616 1785456467.043664 read 0x028f28 thru 0x02a0a0 (0x1179 bytes) 000618 1785456467.053096 trunc from 0x03feed to 0x032c70 (0xd27c bytes) 000619 1785456467.087413 trunc from 0x032c71 to 0x00ebba (0x240b6 bytes) 000620 1785456467.107514 read 0x00950b thru 0x00ebba (0x56b0 bytes) 000621 1785456467.120465 mapwrite 0x01f9f5 thru 0x02c3d9 (0xc9e5 bytes) 000622 1785456467.266224 read 0x025dae thru 0x02c3d9 (0x662c bytes) 000625 1785456467.268592 mapwrite 0x009f4e thru 0x016148 (0xc1fb bytes) 000627 1785456467.354527 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2c3da): Mirror 3: 0x25099f09 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c3da): Mirror 3: 0x25099f09 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c3da: skipped 000628 1785456467.709197 MIRROR_RESYNC 2 mirrors 000629 1785456468.558170 mapread 0x00d958 thru 0x0130dc (0x5785 bytes) 000631 1785456468.580143 write 0x00617c thru 0x00b558 (0x53dd bytes) 000632 1785456468.640959 trunc from 0x02c3da to 0x000a9e (0x2b93b bytes) 000634 1785456468.674569 mapread 0x0005a7 thru 0x000a9e (0x4f8 bytes) 000637 1785456468.688016 mapread 0x0006a4 thru 0x000a9e (0x3fb bytes) 000638 1785456468.693837 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18c:0x0]: Invalid argument (22) 000639 1785456468.895394 mapwrite 0x013659 thru 0x01beed (0x8895 bytes) 000640 1785456468.984978 write 0x024a43 thru 0x034278 (0xf836 bytes) 000642 1785456468.992993 mapwrite 0x01132b thru 0x01ebd9 (0xd8af bytes) 000644 1785456469.119685 MIRROR_EXTEND to 2 mirrors 000648 1785456470.020858 write 0x03b7fe thru 0x03ffff (0x4802 bytes) 000649 1785456470.099414 mapwrite 0x00c01b thru 0x00e6c2 (0x26a8 bytes) 000650 1785456470.215532 trunc from 0x040000 to 0x013f98 (0x2c067 bytes) 000653 1785456470.250066 trunc from 0x013f99 to 0x020f26 (0xcf8e bytes) 000654 1785456470.267174 read 0x016d78 thru 0x020f26 (0xa1af bytes) 000658 1785456470.285493 mapread 0x01a5e0 thru 0x020f26 (0x6947 bytes) 000659 1785456470.289300 trunc from 0x020f27 to 0x02d05b (0xc135 bytes) 000660 1785456470.311480 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18c:0x0]: Invalid argument (22) 000662 1785456470.467153 write 0x025eb3 thru 0x028cc1 (0x2e0f bytes) 000663 1785456470.492347 MIRROR_EXTEND to 2 mirrors 000664 1785456471.114251 read 0x003e4a thru 0x004d06 (0xebd bytes) 000665 1785456471.126133 mapread 0x00a4ca thru 0x011d9c (0x78d3 bytes) 000666 1785456471.154252 write 0x005d92 thru 0x00749d (0x170c bytes) 000667 1785456471.224793 read 0x00fce2 thru 0x013a90 (0x3daf bytes) 000668 1785456471.241135 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18c:0x0]: Invalid argument (22) 000669 1785456471.392716 trunc from 0x02d05c to 0x02776f (0x58ec bytes) 000670 1785456471.431378 write 0x01a073 thru 0x025f1d (0xbeab bytes) 000671 1785456471.444552 read 0x00dbe3 thru 0x01410d (0x652b bytes) 000672 1785456471.458360 write 0x03f514 thru 0x03ffff (0xaec bytes) 000673 1785456471.459552 read 0x00c8a6 thru 0x012f42 (0x669d bytes) 000676 1785456471.476137 write 0x018668 thru 0x027402 (0xed9b bytes) 000678 1785456471.481185 mapread 0x03a711 thru 0x03ffff (0x58ef bytes) 000681 1785456471.488463 trunc from 0x040000 to 0x038d81 (0x727e bytes) 000683 1785456471.514987 mapwrite 0x039d77 thru 0x03ffff (0x6289 bytes) 000685 1785456471.646317 mapread 0x02afb2 thru 0x02d86e (0x28bd bytes) 000689 1785456471.653854 mapread 0x03df26 thru 0x03ffff (0x20da bytes) 000690 1785456471.655265 trunc from 0x040000 to 0x03734e (0x8cb1 bytes) 000691 1785456471.680231 write 0x023417 thru 0x026847 (0x3431 bytes) 000692 1785456471.681354 write 0x00268a thru 0x0039f0 (0x1367 bytes) 000694 1785456471.682744 write 0x03396c thru 0x03d4a1 (0x9b36 bytes) 000695 1785456471.684109 mapwrite 0x02a61c thru 0x02c69f (0x2084 bytes) 000696 1785456471.731146 trunc from 0x03d4a2 to 0x02b674 (0x11e2d bytes) 000697 1785456471.750739 read 0x0036e8 thru 0x0132f3 (0xfc0c bytes) 000699 1785456471.754566 mapwrite 0x0053f6 thru 0x011152 (0xbd5d bytes) 000700 1785456471.822425 write 0x03383f thru 0x03ef75 (0xb737 bytes) 000702 1785456471.829461 mapread 0x03297c thru 0x0375b9 (0x4c3e bytes) 000704 1785456471.842958 write 0x015187 thru 0x023d0c (0xeb86 bytes) 000709 1785456471.850381 read 0x0073a6 thru 0x0130ba (0xbd15 bytes) 000710 1785456471.851177 trunc from 0x03ef76 to 0x00e003 (0x30f72 bytes) 000711 1785456471.879224 read 0x0007dc thru 0x00a5f1 (0x9e16 bytes) 000712 1785456471.881088 mapwrite 0x03eb6b thru 0x03ffff (0x1495 bytes) 000713 1785456471.991119 read 0x02bd45 thru 0x035b3d (0x9df9 bytes) 000716 1785456471.996144 mapread 0x037fa6 thru 0x03c835 (0x4890 bytes) 000719 1785456472.006452 write 0x009ee6 thru 0x0186e1 (0xe7fc bytes) 000720 1785456472.008925 mapread 0x000046 thru 0x00434a (0x4305 bytes) 000722 1785456472.011191 trunc from 0x040000 to 0x01979e (0x26861 bytes) 000724 1785456472.035896 write 0x03811a thru 0x03ab5a (0x2a41 bytes) 000725 1785456472.037112 mapwrite 0x03027c thru 0x03e74e (0xe4d3 bytes) 000727 1785456472.148803 trunc from 0x03e74f to 0x005a64 (0x38cea bytes) 000728 1785456472.166059 trunc from 0x005a65 to 0x0390e3 (0x3367f bytes) 000731 1785456472.189302 mapread 0x034ac7 thru 0x0390e3 (0x461d bytes) 000733 1785456472.226002 mapread 0x02e4f4 thru 0x02e867 (0x374 bytes) 000734 1785456472.234504 read 0x013824 thru 0x022f10 (0xf6ed bytes) 000736 1785456472.256320 MIRROR_EXTEND to 2 mirrors 000737 1785456473.090530 write 0x01b21c thru 0x01f9e8 (0x47cd bytes) 000738 1785456473.236150 read 0x019c49 thru 0x020417 (0x67cf bytes) 000739 1785456473.248934 write 0x0307be thru 0x03ffff (0xf842 bytes) 000741 1785456473.256295 mapread 0x031f3a thru 0x03d504 (0xb5cb bytes) 000743 1785456473.262962 read 0x000f77 thru 0x00ce11 (0xbe9b bytes) 000745 1785456473.275158 mapwrite 0x012c6f thru 0x014432 (0x17c4 bytes) 000748 1785456473.346949 MIRROR_RESYNC 2 mirrors 000749 1785456474.206915 read 0x01de04 thru 0x028f63 (0xb160 bytes) 000750 1785456474.218942 read 0x01ddf8 thru 0x01fd03 (0x1f0c bytes) 000752 1785456474.219882 read 0x02a90e thru 0x02cbde (0x22d1 bytes) 000754 1785456474.229852 read 0x0139ef thru 0x0157b3 (0x1dc5 bytes) 000755 1785456474.230499 read 0x034fd1 thru 0x03694d (0x197d bytes) 000757 1785456474.234723 MIRROR_EXTEND to 3 mirrors 000759 1785456475.067040 read 0x00d655 thru 0x0193a4 (0xbd50 bytes) 000764 1785456475.077389 mapwrite 0x012171 thru 0x0188cb (0x675b bytes) 000768 1785456475.206621 write 0x029923 thru 0x031a68 (0x8146 bytes) 000771 1785456475.226453 trunc from 0x040000 to 0x025047 (0x1afb8 bytes) 000772 1785456475.294894 mapread 0x0206c8 thru 0x025047 (0x4980 bytes) 000773 1785456475.310221 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x25048): Mirror 3: 0xc2787d61 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x25048): Mirror 3: 0xc2787d61 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x25048: skipped 000774 1785456475.611151 trunc from 0x025048 to 0x030bee (0xbba7 bytes) 000775 1785456475.630269 read 0x029a58 thru 0x030bee (0x7197 bytes) 000776 1785456475.638146 mapwrite 0x01d9c2 thru 0x028347 (0xa986 bytes) 000778 1785456475.723070 MIRROR_RESYNC 2 mirrors 000779 1785456476.474451 mapread 0x0171fc thru 0x01c5f7 (0x53fc bytes) 000782 1785456476.504620 mapread 0x004594 thru 0x00cebb (0x8928 bytes) 000785 1785456476.522154 trunc from 0x030bef to 0x0155f9 (0x1b5f5 bytes) 000787 1785456476.631772 trunc from 0x0155fa to 0x0133f6 (0x2203 bytes) 000789 1785456476.660351 mapread 0x004844 thru 0x00cb05 (0x82c2 bytes) 000791 1785456476.683732 mapread 0x012cb4 thru 0x0133f6 (0x743 bytes) 000792 1785456476.697422 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18c:0x0]: Invalid argument (22) 000793 1785456476.870312 MIRROR_EXTEND to 2 mirrors 000794 1785456477.615850 write 0x0204b4 thru 0x029c00 (0x974d bytes) 000795 1785456477.676236 mapwrite 0x00d10c thru 0x012174 (0x5069 bytes) 000797 1785456477.755517 mapread 0x019d3d thru 0x027d7d (0xe041 bytes) 000798 1785456477.770382 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18c:0x0]: Invalid argument (22) 000800 1785456477.935420 mapread 0x024015 thru 0x029c00 (0x5bec bytes) 000802 1785456477.970744 mapread 0x00c3be thru 0x01b0b6 (0xecf9 bytes) 000803 1785456478.010159 mapread 0x0036bc thru 0x012bd9 (0xf51e bytes) 000804 1785456478.019772 mapwrite 0x002f4a thru 0x007e8c (0x4f43 bytes) 000806 1785456478.102980 write 0x00f4e2 thru 0x01991f (0xa43e bytes) 000808 1785456478.112968 trunc from 0x029c01 to 0x01d092 (0xcb6e bytes) 000809 1785456478.149154 write 0x024677 thru 0x026e45 (0x27cf bytes) 000810 1785456478.154443 trunc from 0x026e46 to 0x026cac (0x199 bytes) 000811 1785456478.178412 mapwrite 0x027d98 thru 0x032b06 (0xad6f bytes) 000812 1785456478.264314 mapwrite 0x033b10 thru 0x034b7c (0x106d bytes) 000814 1785456478.340100 write 0x01ec0c thru 0x02768c (0x8a81 bytes) 000818 1785456478.356867 mapread 0x01709e thru 0x022603 (0xb566 bytes) 000819 1785456478.358111 write 0x011f61 thru 0x01c1da (0xa27a bytes) 000822 1785456478.366028 mapread 0x025009 thru 0x029204 (0x41fc bytes) 000824 1785456478.369160 mapwrite 0x018c55 thru 0x019243 (0x5ef bytes) 000825 1785456478.430109 mapwrite 0x02a144 thru 0x02dbe1 (0x3a9e bytes) 000826 1785456478.466666 mapread 0x02bc3f thru 0x02c314 (0x6d6 bytes) 000827 1785456478.468256 mapwrite 0x02e636 thru 0x03c1da (0xdba5 bytes) 000834 1785456478.560511 mapread 0x00f764 thru 0x01b675 (0xbf12 bytes) 000837 1785456478.565161 mapwrite 0x020dd6 thru 0x02ef2e (0xe159 bytes) 000838 1785456478.612778 write 0x02ce12 thru 0x03548a (0x8679 bytes) 000840 1785456478.616203 trunc from 0x03c1db to 0x0329ed (0x97ed bytes) 000841 1785456478.637509 mapwrite 0x01dee7 thru 0x0208eb (0x2a05 bytes) 000844 1785456478.694952 trunc from 0x0329ee to 0x03195c (0x1091 bytes) 000845 1785456478.720294 mapwrite 0x0200db thru 0x028d95 (0x8cbb bytes) 000846 1785456478.778327 read 0x01c1f3 thru 0x022c63 (0x6a71 bytes) 000849 1785456478.781306 read 0x02553b thru 0x028acf (0x3595 bytes) 000850 1785456478.788595 MIRROR_EXTEND to 2 mirrors 000852 1785456479.595111 MIRROR_RESYNC 2 mirrors 000853 1785456479.887511 mapwrite 0x00e732 thru 0x01ac17 (0xc4e6 bytes) 000855 1785456480.099154 trunc from 0x03195d to 0x02fc3f (0x1d1d bytes) 000856 1785456480.120708 trunc from 0x02fc40 to 0x02aaa5 (0x519a bytes) 000858 1785456480.135733 mapread 0x00a1c1 thru 0x01761f (0xd45f bytes) 000859 1785456480.137209 trunc from 0x02aaa6 to 0x028754 (0x2351 bytes) 000861 1785456480.156685 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x28755): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x28755): Mirror 2: 0x50eba571 (stale) CRC-32 checksum value for chunk [0, 0x28755): Mirror 1: 0x6ac1a322 Mirror 3: 0x6ac1a322 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x28755: skipped 000863 1785456481.071928 read 0x01955a thru 0x01e4ba (0x4f61 bytes) 000864 1785456481.079968 trunc from 0x028755 to 0x01ba06 (0xcd4e bytes) 000865 1785456481.134755 write 0x011bf1 thru 0x013cb9 (0x20c9 bytes) 000866 1785456481.150981 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1ba07): Mirror 3: 0x823a9a36 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1ba07): Mirror 3: 0x823a9a36 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1ba07: skipped 000867 1785456481.482810 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x1ba07): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1ba07): Mirror 3: 0x823a9a36 (stale) CRC-32 checksum value for chunk [0, 0x1ba07): Mirror 1: 0xbecc7b5e Mirror 4: 0xbecc7b5e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1ba07: skipped 000869 1785456482.437563 mapread 0x007ad5 thru 0x0165bc (0xeae8 bytes) 000870 1785456482.493510 write 0x0099f9 thru 0x00ef44 (0x554c bytes) 000871 1785456482.620594 trunc from 0x01ba07 to 0x001d56 (0x19cb0 bytes) 000873 1785456482.658987 read 0x0013df thru 0x001d56 (0x978 bytes) 000876 1785456482.675398 MIRROR_RESYNC 3 mirrors 000877 1785456483.396746 read 0x0018f3 thru 0x001d56 (0x464 bytes) 000878 1785456483.405834 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18c:0x0]: Invalid argument (22) 000879 1785456483.708846 mapwrite 0x019eac thru 0x025340 (0xb495 bytes) 000880 1785456483.908280 MIRROR_RESYNC 2 mirrors 000884 1785456484.517445 mapread 0x024511 thru 0x025340 (0xe30 bytes) 000887 1785456484.529411 write 0x003a32 thru 0x010970 (0xcf3f bytes) 000888 1785456484.612060 read 0x017be4 thru 0x019df3 (0x2210 bytes) 000889 1785456484.619224 mapread 0x005be3 thru 0x00ac16 (0x5034 bytes) 000890 1785456484.621190 mapwrite 0x00a411 thru 0x019b92 (0xf782 bytes) 000891 1785456484.696317 trunc from 0x025341 to 0x02ecf7 (0x99b7 bytes) 000892 1785456484.715005 trunc from 0x02ecf8 to 0x023702 (0xb5f5 bytes) 000893 1785456484.736561 mapwrite 0x019bc9 thru 0x0223a0 (0x87d8 bytes) 000895 1785456484.786578 mapread 0x017964 thru 0x01b6ae (0x3d4b bytes) 000896 1785456484.788954 trunc from 0x023703 to 0x01db05 (0x5bfd bytes) 000897 1785456484.803373 trunc from 0x01db06 to 0x0127a6 (0xb35f bytes) 000898 1785456484.827175 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18c:0x0]: Invalid argument (22) 000899 1785456484.965615 mapread 0x00ee6f thru 0x0127a6 (0x3938 bytes) 000900 1785456484.989955 write 0x037e85 thru 0x038865 (0x9e1 bytes) 000901 1785456484.991836 trunc from 0x038866 to 0x0163a0 (0x224c5 bytes) 000902 1785456485.010290 write 0x03135b thru 0x038b88 (0x782e bytes) 000904 1785456485.017595 MIRROR_EXTEND to 2 mirrors 000905 1785456485.827447 trunc from 0x038b89 to 0x01cebf (0x1bcc9 bytes) 000906 1785456485.941182 mapread 0x013374 thru 0x019d8c (0x6a19 bytes) 000907 1785456485.981167 write 0x03b112 thru 0x03ffff (0x4eee bytes) 000910 1785456485.987019 read 0x026ce8 thru 0x035810 (0xeb29 bytes) 000911 1785456485.999092 write 0x004bc0 thru 0x00d4e9 (0x892a bytes) 000913 1785456486.010941 mapread 0x00bb53 thru 0x013fb3 (0x8461 bytes) 000915 1785456486.014444 write 0x0216ca thru 0x02d74e (0xc085 bytes) 000916 1785456486.019985 mapread 0x00cc35 thru 0x0190c2 (0xc48e bytes) 000917 1785456486.021287 trunc from 0x040000 to 0x02329b (0x1cd64 bytes) 000918 1785456486.049172 MIRROR_RESYNC 2 mirrors 000919 1785456486.860337 MIRROR_EXTEND to 3 mirrors 000922 1785456487.577322 read 0x006e8f thru 0x012e90 (0xc002 bytes) 000923 1785456487.589261 mapread 0x00f256 thru 0x019e08 (0xabb3 bytes) 000924 1785456487.611477 read 0x018d44 thru 0x01df07 (0x51c4 bytes) 000927 1785456487.612663 mapwrite 0x03a76e thru 0x03ffff (0x5892 bytes) 000928 1785456487.813719 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x669d231a (stale) Mirror 3: 0x669d231a (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xf5cfecf1 Mirror 4: 0xf5cfecf1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000929 1785456488.838231 read 0x014069 thru 0x01e16a (0xa102 bytes) 000930 1785456488.865950 read 0x0382df thru 0x039acd (0x17ef bytes) 000931 1785456488.899756 MIRROR_RESYNC 4 mirrors 000933 1785456489.813944 mapread 0x03378e thru 0x03ffff (0xc872 bytes) 000936 1785456489.840195 read 0x0179b0 thru 0x01f203 (0x7854 bytes) 000937 1785456489.848076 mapread 0x02c4c2 thru 0x03c084 (0xfbc3 bytes) 000938 1785456489.858773 mapread 0x011ddb thru 0x01ab49 (0x8d6f bytes) 000940 1785456489.878970 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18c:0x0]: Invalid argument (22) 000942 1785456490.222724 trunc from 0x040000 to 0x002037 (0x3dfc8 bytes) 000946 1785456490.347658 write 0x0208cc thru 0x029dd0 (0x9505 bytes) 000949 1785456490.350622 mapread 0x015705 thru 0x0190ac (0x39a8 bytes) 000950 1785456490.383914 MIRROR_RESYNC 3 mirrors 000951 1785456491.247718 trunc from 0x029dd1 to 0x010d09 (0x190c7 bytes) 000952 1785456491.352951 MIRROR_RESYNC 3 mirrors 000953 1785456492.150594 trunc from 0x010d0a to 0x026310 (0x15607 bytes) 000954 1785456492.242668 trunc from 0x026311 to 0x03965a (0x1334a bytes) 000955 1785456492.290798 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3965b): Mirror 2: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3965b): Mirror 2: 0xa3056f67 (stale) Mirror 4: 0xa3056f67 (stale) CRC-32 checksum value for chunk [0, 0x3965b): Mirror 1: 0xa3056f67 Mirror 5: 0xa3056f67 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3965b: skipped 000962 1785456493.614054 write 0x026fc5 thru 0x031df4 (0xae30 bytes) 000964 1785456493.687623 mapwrite 0x013567 thru 0x01fb40 (0xc5da bytes) 000965 1785456493.796605 MIRROR_RESYNC 4 mirrors 000967 1785456494.917128 mapwrite 0x02c1ca thru 0x038630 (0xc467 bytes) 000969 1785456495.200122 mapread 0x013ba1 thru 0x020818 (0xcc78 bytes) 000970 1785456495.228035 mapread 0x02d952 thru 0x03815c (0xa80b bytes) 000971 1785456495.236192 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3965b): Mirror 2: 0x57490682 (stale) Mirror 4: 0x57490682 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3965b): Mirror 2: 0x57490682 (stale) Mirror 4: 0x57490682 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3965b: skipped 000973 1785456495.621344 mapread 0x00c328 thru 0x00dca9 (0x1982 bytes) 000975 1785456495.637368 trunc from 0x03965b to 0x00d5d0 (0x2c08a bytes) 000976 1785456495.663176 write 0x0261c1 thru 0x02d60c (0x744c bytes) 000980 1785456495.679675 MIRROR_RESYNC 3 mirrors 000981 1785456496.835990 write 0x007e9e thru 0x0100da (0x823d bytes) 000982 1785456496.954484 mapread 0x02ac49 thru 0x02d60c (0x29c4 bytes) 000986 1785456496.999793 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2d60d): Mirror 2: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d60d): Mirror 2: 0x93c51409 (stale) Mirror 4: 0x93c51409 (stale) CRC-32 checksum value for chunk [0, 0x2d60d): Mirror 1: 0xae76d7f Mirror 5: 0xae76d7f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d60d: skipped 000988 1785456498.000987 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2d60d): Mirror 2: 0x93c51409 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d60d): Mirror 2: 0x93c51409 (stale) CRC-32 checksum value for chunk [0, 0x2d60d): Mirror 1: 0xae76d7f Mirror 5: 0xae76d7f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d60d: skipped 000989 1785456498.591723 trunc from 0x02d60d to 0x03ce8a (0xf87e bytes) 000990 1785456498.687397 read 0x007a9c thru 0x00b415 (0x397a bytes) 000991 1785456498.696292 write 0x015efb thru 0x01dda2 (0x7ea8 bytes) 000992 1785456498.704279 read 0x011a32 thru 0x0147b0 (0x2d7f bytes) 000993 1785456498.714545 read 0x017305 thru 0x01afb8 (0x3cb4 bytes) 000994 1785456498.715444 mapwrite 0x0383c9 thru 0x03ffff (0x7c37 bytes) 001000 1785456498.821984 read 0x028510 thru 0x029690 (0x1181 bytes) All operations completed A-OK!