== sanity-flr test 70a: flr mode fsx test ================ 22:37:32 (1787625452) Seed set to 5454 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read 000008 1787625454.880606 write 0x017903 thru 0x0228ae (0xafac bytes) 000012 1787625454.886626 mapread 0x0087a9 thru 0x00b343 (0x2b9b bytes) truncating to largest ever: 0x4757 000013 1787625454.905348 trunc from 0x0228af to 0x004756 (0x1e158 bytes) 000014 1787625454.934559 MIRROR_EXTEND to 2 mirrors truncating to largest ever: 0x3c239 000015 1787625455.866193 trunc from 0x004757 to 0x03c238 (0x37ae2 bytes) 000018 1787625455.957724 trunc from 0x03c239 to 0x02cb81 (0xf6b7 bytes) 000019 1787625455.986916 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2cb82): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2cb82): Mirror 2: 0x37e44eaf (stale) CRC-32 checksum value for chunk [0, 0x2cb82): Mirror 1: 0x37e44eaf Mirror 3: 0x37e44eaf lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2cb82: skipped 000021 1787625457.090562 trunc from 0x02cb82 to 0x026036 (0x6b4b bytes) 000022 1787625457.245688 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x26037): Mirror 2: 0xac0eb9a4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x26037): Mirror 2: 0xac0eb9a4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x26037: skipped 000023 1787625457.808361 read 0x01bdb5 thru 0x0220c5 (0x6311 bytes) 000024 1787625457.839620 MIRROR_RESYNC 2 mirrors 000025 1787625458.633566 mapwrite 0x02d91f thru 0x03adb8 (0xd49a bytes) 000026 1787625459.055297 mapwrite 0x001a22 thru 0x011537 (0xfb16 bytes) 000028 1787625459.258395 mapread 0x02d531 thru 0x02e403 (0xed3 bytes) 000029 1787625459.275243 MIRROR_RESYNC 2 mirrors 000030 1787625460.053874 MIRROR_EXTEND to 3 mirrors 000031 1787625460.832260 write 0x036616 thru 0x03ffff (0x99ea bytes) 000032 1787625460.939488 MIRROR_RESYNC 3 mirrors 000033 1787625461.951548 read 0x01042c thru 0x01c40c (0xbfe1 bytes) 000034 1787625461.966980 mapwrite 0x02720e thru 0x028231 (0x1024 bytes) 000037 1787625462.182587 write 0x020850 thru 0x0294d7 (0x8c88 bytes) 000039 1787625462.201582 mapwrite 0x0016a2 thru 0x00b718 (0xa077 bytes) 000041 1787625462.320806 read 0x018a36 thru 0x01fb8e (0x7159 bytes) 000044 1787625462.334825 write 0x02317f thru 0x024ed0 (0x1d52 bytes) 000047 1787625462.348569 MIRROR_RESYNC 3 mirrors 000049 1787625463.241789 mapwrite 0x017ad9 thru 0x025632 (0xdb5a bytes) 000050 1787625463.434885 write 0x02c530 thru 0x0382c2 (0xbd93 bytes) 000054 1787625463.454816 write 0x033bac thru 0x03a36d (0x67c2 bytes) 000056 1787625463.461679 mapwrite 0x038e57 thru 0x03ffff (0x71a9 bytes) 000057 1787625463.519515 trunc from 0x040000 to 0x0219c0 (0x1e63f bytes) 000058 1787625463.568685 write 0x02b5d8 thru 0x02d030 (0x1a59 bytes) 000062 1787625463.574452 trunc from 0x02d031 to 0x0224cd (0xab63 bytes) 000064 1787625463.599559 trunc from 0x0224ce to 0x0372bf (0x14df2 bytes) 000069 1787625463.626802 mapread 0x0114ca thru 0x019a2c (0x8563 bytes) 000070 1787625463.639859 read 0x02036d thru 0x027239 (0x6ecd bytes) 000074 1787625463.654780 read 0x0026a3 thru 0x009e44 (0x77a2 bytes) 000075 1787625463.673838 MIRROR_RESYNC 3 mirrors 000076 1787625464.752419 write 0x03c54a thru 0x03ffff (0x3ab6 bytes) 000080 1787625464.887908 mapwrite 0x027540 thru 0x02d225 (0x5ce6 bytes) 000081 1787625464.979526 read 0x00379a thru 0x007faa (0x4811 bytes) 000082 1787625464.989262 write 0x02b118 thru 0x030413 (0x52fc bytes) 000083 1787625464.994351 MIRROR_RESYNC 3 mirrors 000085 1787625465.693554 trunc from 0x040000 to 0x002a15 (0x3d5ea bytes) 000086 1787625465.768622 MIRROR_RESYNC 3 mirrors 000088 1787625466.703844 trunc from 0x002a16 to 0x02f198 (0x2c783 bytes) 000089 1787625466.799193 write 0x0381c5 thru 0x03ffff (0x7e3b bytes) 000090 1787625466.801844 read 0x03243c thru 0x0383ea (0x5faf bytes) 000091 1787625466.809751 write 0x006176 thru 0x00c85e (0x66e9 bytes) 000092 1787625466.822753 mapread 0x025086 thru 0x033177 (0xe0f2 bytes) 000093 1787625466.852042 trunc from 0x040000 to 0x009552 (0x36aad bytes) 000094 1787625466.881987 trunc from 0x009553 to 0x015b65 (0xc613 bytes) 000095 1787625466.899139 mapwrite 0x03d17f thru 0x03ffff (0x2e81 bytes) 000096 1787625466.993349 read 0x0118df thru 0x01c035 (0xa757 bytes) 000097 1787625467.006187 mapwrite 0x01587b thru 0x018705 (0x2e8b bytes) 000100 1787625467.070366 mapread 0x03766a thru 0x03c6a6 (0x503d bytes) 000101 1787625467.071969 mapread 0x033c92 thru 0x036f74 (0x32e3 bytes) 000104 1787625467.074236 trunc from 0x040000 to 0x01c1d4 (0x23e2b bytes) 000106 1787625467.094630 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1c1d5): Mirror 3: 0xd92bc42e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1c1d5): Mirror 3: 0xd92bc42e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1c1d5: skipped 000107 1787625467.405486 trunc from 0x01c1d5 to 0x02e771 (0x1259d bytes) 000108 1787625467.440961 MIRROR_RESYNC 2 mirrors 000111 1787625468.685224 read 0x01fa09 thru 0x02802f (0x8627 bytes) 000112 1787625468.695703 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b7:0x0]: Invalid argument (22) 000113 1787625468.813604 trunc from 0x02e772 to 0x027312 (0x745f bytes) 000114 1787625468.851407 write 0x039505 thru 0x03ffff (0x6afb bytes) 000115 1787625468.860653 MIRROR_EXTEND to 2 mirrors 000116 1787625469.624718 trunc from 0x040000 to 0x00733e (0x38cc1 bytes) 000119 1787625469.693431 trunc from 0x00733f to 0x01e399 (0x1705b bytes) 000120 1787625469.713892 write 0x027661 thru 0x030ffa (0x999a bytes) 000121 1787625469.727040 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x30ffb): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30ffb): Mirror 2: 0xfdd5b70d (stale) CRC-32 checksum value for chunk [0, 0x30ffb): Mirror 1: 0xfe8a4f95 Mirror 3: 0xfe8a4f95 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30ffb: skipped 000122 1787625470.381988 read 0x00e76c thru 0x01cacd (0xe362 bytes) 000124 1787625470.426478 read 0x00de83 thru 0x01038b (0x2509 bytes) 000126 1787625470.464630 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x30ffb): Mirror 2: 0xfdd5b70d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30ffb): Mirror 2: 0xfdd5b70d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30ffb: skipped 000127 1787625470.832256 trunc from 0x030ffb to 0x02840b (0x8bef bytes) 000128 1787625470.884048 mapwrite 0x004a61 thru 0x00a685 (0x5c25 bytes) 000129 1787625470.955724 write 0x014beb thru 0x01a85c (0x5c72 bytes) 000132 1787625470.964805 write 0x035edc thru 0x038815 (0x293a bytes) 000134 1787625470.966719 write 0x0179f3 thru 0x018935 (0xf43 bytes) 000136 1787625470.967876 mapwrite 0x016cd7 thru 0x01b04b (0x4375 bytes) 000137 1787625471.048774 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x38816): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38816): Mirror 2: 0xcb588be0 (stale) CRC-32 checksum value for chunk [0, 0x38816): Mirror 1: 0x30144c8a Mirror 3: 0x30144c8a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38816: skipped 000139 1787625471.879237 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b7:0x0]: Invalid argument (22) 000141 1787625472.311051 read 0x00ce67 thru 0x00d960 (0xafa bytes) 000144 1787625472.365989 mapread 0x038282 thru 0x038815 (0x594 bytes) 000145 1787625472.413339 trunc from 0x038816 to 0x039132 (0x91d bytes) 000147 1787625472.579318 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x39133): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39133): Mirror 3: 0x78b3d654 (stale) CRC-32 checksum value for chunk [0, 0x39133): Mirror 1: 0x78b3d654 Mirror 4: 0x78b3d654 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39133: skipped 000148 1787625473.575618 read 0x02e6f0 thru 0x038c6f (0xa580 bytes) 000149 1787625473.588650 mapread 0x030136 thru 0x039132 (0x8ffd bytes) 000150 1787625473.615853 trunc from 0x039133 to 0x004099 (0x35099 bytes) 000151 1787625473.719186 read 0x001748 thru 0x004099 (0x2952 bytes) 000152 1787625473.732108 mapread 0x0001e9 thru 0x004099 (0x3eb1 bytes) 000153 1787625473.733211 trunc from 0x00409a to 0x01a046 (0x15fad bytes) 000157 1787625473.768270 write 0x039f9c thru 0x03ffff (0x6064 bytes) 000159 1787625473.770970 write 0x02767c thru 0x02fb18 (0x849d bytes) 000160 1787625473.780703 mapwrite 0x02df9f thru 0x036bf6 (0x8c58 bytes) 000161 1787625473.856241 trunc from 0x040000 to 0x02c179 (0x13e86 bytes) 000165 1787625473.886004 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2c17a): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c17a): Mirror 3: 0x9baa851c (stale) Mirror 4: 0x9baa851c (stale) CRC-32 checksum value for chunk [0, 0x2c17a): Mirror 1: 0x525d213e Mirror 5: 0x525d213e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c17a: skipped 000167 1787625475.368041 trunc from 0x02c17a to 0x00bc92 (0x204e7 bytes) 000168 1787625475.572327 MIRROR_RESYNC 4 mirrors 000172 1787625476.852465 mapread 0x00481f thru 0x00bc92 (0x7474 bytes) 000174 1787625476.876174 read 0x0018d7 thru 0x004356 (0x2a80 bytes) 000175 1787625476.876929 trunc from 0x00bc93 to 0x03095c (0x24cca bytes) 000176 1787625476.982559 write 0x027c22 thru 0x0294ea (0x18c9 bytes) 000177 1787625477.003127 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3095d): Mirror 4: 0x8055b7ee (stale) Mirror 5: 0x8055b7ee (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3095d): Mirror 4: 0x8055b7ee (stale) Mirror 5: 0x8055b7ee (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3095d: skipped 000179 1787625477.426750 MIRROR_RESYNC 3 mirrors 000181 1787625479.315412 read 0x012665 thru 0x020a46 (0xe3e2 bytes) 000184 1787625479.337781 trunc from 0x03095d to 0x032ea4 (0x2548 bytes) 000186 1787625479.624935 trunc from 0x032ea5 to 0x01dc4f (0x15255 bytes) 000187 1787625479.666930 mapwrite 0x02310e thru 0x031351 (0xe244 bytes) 000188 1787625479.782350 mapwrite 0x03d112 thru 0x03ffff (0x2eee bytes) 000189 1787625479.869640 read 0x014ba2 thru 0x0225f5 (0xda54 bytes) 000190 1787625479.882719 mapread 0x027b22 thru 0x030a13 (0x8ef2 bytes) 000192 1787625479.883801 mapwrite 0x009cde thru 0x00dddc (0x40ff bytes) 000194 1787625479.997742 write 0x03b448 thru 0x03ffff (0x4bb8 bytes) 000195 1787625479.999185 mapwrite 0x03233e thru 0x03a9a0 (0x8663 bytes) 000196 1787625480.088617 mapwrite 0x006854 thru 0x00b4d6 (0x4c83 bytes) 000197 1787625480.132643 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x6b654ac1 (stale) Mirror 5: 0x6b654ac1 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xb412acee Mirror 6: 0xb412acee lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000198 1787625481.122941 mapwrite 0x00cf29 thru 0x01a83e (0xd916 bytes) 000199 1787625481.521072 read 0x01337b thru 0x01427d (0xf03 bytes) 000201 1787625481.523107 mapread 0x00a261 thru 0x00f50b (0x52ab bytes) 000202 1787625481.525128 mapwrite 0x009cc3 thru 0x017522 (0xd860 bytes) 000203 1787625481.681114 MIRROR_RESYNC 4 mirrors 000205 1787625482.863254 mapread 0x02302d thru 0x02d3dd (0xa3b1 bytes) 000209 1787625482.882998 mapwrite 0x017be3 thru 0x01f7c9 (0x7be7 bytes) 000210 1787625483.122186 write 0x03307f thru 0x03987b (0x67fd bytes) 000213 1787625483.139412 MIRROR_RESYNC 4 mirrors 000214 1787625484.595978 mapwrite 0x019ce5 thru 0x021560 (0x787c bytes) 000216 1787625484.847888 write 0x03afe0 thru 0x03ffff (0x5020 bytes) 000218 1787625484.855635 read 0x029bb7 thru 0x02f9d0 (0x5e1a bytes) 000220 1787625484.868035 write 0x0102b2 thru 0x015555 (0x52a4 bytes) 000221 1787625484.878841 mapwrite 0x030f8f thru 0x0356a4 (0x4716 bytes) 000223 1787625484.965372 mapread 0x034cff thru 0x035437 (0x739 bytes) 000224 1787625484.966140 trunc from 0x040000 to 0x00f8e9 (0x30716 bytes) 000230 1787625484.995919 trunc from 0x00f8ea to 0x02f53e (0x1fc55 bytes) 000231 1787625485.025443 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2f53f): Mirror 5: 0x469ca34f (stale) Mirror 6: 0x469ca34f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f53f): Mirror 5: 0x469ca34f (stale) Mirror 6: 0x469ca34f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f53f: skipped 000234 1787625485.395936 trunc from 0x02f53f to 0x00e670 (0x20ece bytes) 000235 1787625485.414078 mapwrite 0x03ba69 thru 0x03de4e (0x23e6 bytes) 000236 1787625485.534929 mapread 0x01532f thru 0x019bd3 (0x48a5 bytes) 000237 1787625485.554808 MIRROR_RESYNC 3 mirrors 000238 1787625486.571723 trunc from 0x03de4f to 0x026c12 (0x1723c bytes) 000241 1787625486.648118 trunc from 0x026c13 to 0x0336ed (0xcadb bytes) 000242 1787625486.670676 write 0x02eb93 thru 0x039d4f (0xb1bd bytes) 000245 1787625486.679396 trunc from 0x039d50 to 0x02f43c (0xa913 bytes) 000246 1787625486.700563 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2f43d): Mirror 6: 0x76d9215e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f43d): Mirror 6: 0x76d9215e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f43d: skipped 000249 1787625487.165820 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2f43d): Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f43d): Mirror 6: 0x76d9215e (stale) CRC-32 checksum value for chunk [0, 0x2f43d): Mirror 1: 0xc4b8572c Mirror 7: 0xc4b8572c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f43d: skipped 000251 1787625488.498930 trunc from 0x02f43d to 0x00c587 (0x22eb5 bytes) 000252 1787625488.573069 trunc from 0x00c588 to 0x0255e5 (0x1905e bytes) 000253 1787625488.595829 mapwrite 0x02832c thru 0x033a53 (0xb728 bytes) 000255 1787625488.690106 mapread 0x016ce5 thru 0x023d42 (0xd05e bytes) 000257 1787625488.706067 mapread 0x01263b thru 0x01760a (0x4fd0 bytes) 000258 1787625488.707112 mapread 0x00c0d5 thru 0x00d327 (0x1253 bytes) 000259 1787625488.714673 write 0x02c67d thru 0x035327 (0x8cab bytes) 000263 1787625488.718061 mapread 0x032b93 thru 0x035327 (0x2795 bytes) 000264 1787625488.719195 mapwrite 0x0176c6 thru 0x02362a (0xbf65 bytes) 000265 1787625488.826653 mapwrite 0x0223a4 thru 0x0284d7 (0x6134 bytes) 000266 1787625488.974807 trunc from 0x035328 to 0x02f7a9 (0x5b7e bytes) 000268 1787625488.995148 read 0x02e234 thru 0x02f7a9 (0x1576 bytes) 000271 1787625488.997134 write 0x01cc2a thru 0x021574 (0x494b bytes) 000272 1787625488.998825 write 0x01b9a7 thru 0x01c9d8 (0x1032 bytes) 000274 1787625489.000535 mapread 0x011f9c thru 0x01ac9e (0x8d03 bytes) 000279 1787625489.004956 write 0x03ac89 thru 0x03ffff (0x5377 bytes) 000281 1787625489.014225 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x2685fd8b (stale) Mirror 7: 0x52794e70 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x5688ddc2 Mirror 8: 0x5688ddc2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000282 1787625489.850278 mapwrite 0x0257cb thru 0x0325f6 (0xce2c bytes) 000283 1787625490.141509 mapread 0x029756 thru 0x02f06b (0x5916 bytes) 000285 1787625490.144404 mapread 0x026ea5 thru 0x02fa59 (0x8bb5 bytes) 000287 1787625490.150261 write 0x00846e thru 0x00ef7f (0x6b12 bytes) 000290 1787625490.184752 mapread 0x027542 thru 0x029f3d (0x29fc bytes) 000294 1787625490.187334 trunc from 0x040000 to 0x01cccc (0x23333 bytes) 000296 1787625490.240713 read 0x00f179 thru 0x017a3d (0x88c5 bytes) 000297 1787625490.256248 trunc from 0x01cccd to 0x003135 (0x19b97 bytes) 000298 1787625490.268735 read 0x002cd3 thru 0x003135 (0x463 bytes) 000299 1787625490.277369 mapread 0x000d21 thru 0x003135 (0x2415 bytes) 000302 1787625490.279242 mapwrite 0x020748 thru 0x02478b (0x4044 bytes) 000305 1787625490.335665 read 0x0074c6 thru 0x011242 (0x9d7d bytes) 000306 1787625490.336489 trunc from 0x02478c to 0x0209df (0x3dac bytes) 000309 1787625490.357635 trunc from 0x0209e0 to 0x024688 (0x3ca9 bytes) 000310 1787625490.372352 trunc from 0x024689 to 0x02f39b (0xad13 bytes) 000315 1787625490.392340 trunc from 0x02f39c to 0x02fd13 (0x978 bytes) 000317 1787625490.412569 write 0x026059 thru 0x02f63f (0x95e7 bytes) 000318 1787625490.425480 MIRROR_RESYNC 4 mirrors 000319 1787625491.621041 trunc from 0x02fd14 to 0x0155f8 (0x1a71b bytes) 000320 1787625491.721332 mapwrite 0x025127 thru 0x02578b (0x665 bytes) 000325 1787625491.805860 MIRROR_RESYNC 4 mirrors 000326 1787625492.807688 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b7:0x0]: Invalid argument (22) 000327 1787625493.207211 MIRROR_RESYNC 3 mirrors 000328 1787625493.556815 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b7:0x0]: Invalid argument (22) 000330 1787625493.808017 mapwrite 0x007029 thru 0x014849 (0xd821 bytes) 000331 1787625494.066521 write 0x0007b0 thru 0x002fef (0x2840 bytes) 000332 1787625494.078011 write 0x02aa2d thru 0x0353f1 (0xa9c5 bytes) 000333 1787625494.081790 mapread 0x01b68b thru 0x01c706 (0x107c bytes) 000336 1787625494.109141 trunc from 0x0353f2 to 0x018eba (0x1c537 bytes) 000337 1787625494.144937 read 0x00343a thru 0x00b615 (0x81dc bytes) 000339 1787625494.156387 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b7:0x0]: Invalid argument (22) 000341 1787625494.295073 write 0x009541 thru 0x00cc77 (0x3737 bytes) 000342 1787625494.305387 write 0x00194f thru 0x00f1f7 (0xd8a9 bytes) 000343 1787625494.315592 mapread 0x002530 thru 0x00c440 (0x9f11 bytes) 000344 1787625494.320679 MIRROR_EXTEND to 2 mirrors 000345 1787625494.909773 write 0x038fcd thru 0x03ffff (0x7033 bytes) 000349 1787625494.974087 mapread 0x014ac1 thru 0x021971 (0xceb1 bytes) 000350 1787625494.995893 mapread 0x03a684 thru 0x03ffff (0x597c bytes) 000351 1787625495.000740 MIRROR_RESYNC 2 mirrors 000353 1787625496.172890 mapread 0x007bc7 thru 0x00a810 (0x2c4a bytes) 000355 1787625496.240534 write 0x00de1b thru 0x0100ea (0x22d0 bytes) 000357 1787625496.421983 mapwrite 0x03e6e6 thru 0x03ffff (0x191a bytes) 000359 1787625496.592531 write 0x02421b thru 0x033898 (0xf67e bytes) 000360 1787625496.618627 mapread 0x02e903 thru 0x030f47 (0x2645 bytes) 000361 1787625496.622701 trunc from 0x040000 to 0x01bf79 (0x24086 bytes) 000362 1787625496.702901 read 0x005b0d thru 0x0140a0 (0xe594 bytes) 000363 1787625496.720436 read 0x019675 thru 0x01bf79 (0x2905 bytes) 000364 1787625496.751407 MIRROR_RESYNC 2 mirrors 000369 1787625497.850296 read 0x00ddad thru 0x01a27a (0xc4ce bytes) 000373 1787625497.888458 mapread 0x00169e thru 0x00df53 (0xc8b6 bytes) 000374 1787625497.937027 mapread 0x00c1f5 thru 0x00ef02 (0x2d0e bytes) 000375 1787625497.939689 write 0x023314 thru 0x02cac1 (0x97ae bytes) 000377 1787625498.137361 read 0x02440b thru 0x02a146 (0x5d3c bytes) 000378 1787625498.153387 MIRROR_RESYNC 2 mirrors 000379 1787625498.918406 read 0x014135 thru 0x018b80 (0x4a4c bytes) 000381 1787625498.938602 read 0x026322 thru 0x02cac1 (0x67a0 bytes) 000383 1787625498.959903 mapwrite 0x00669b thru 0x00ef8d (0x88f3 bytes) 000384 1787625499.215789 mapread 0x00dfbd thru 0x01a5a0 (0xc5e4 bytes) 000386 1787625499.286505 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b7:0x0]: Invalid argument (22) 000387 1787625499.424726 mapwrite 0x00b540 thru 0x00bad3 (0x594 bytes) 000388 1787625499.466958 mapwrite 0x03ea6a thru 0x03ffff (0x1596 bytes) 000389 1787625499.591958 mapread 0x00be52 thru 0x0192ac (0xd45b bytes) 000393 1787625499.670762 mapread 0x029256 thru 0x037e9d (0xec48 bytes) 000398 1787625499.697658 write 0x03a5cb thru 0x03ffff (0x5a35 bytes) 000401 1787625499.701310 mapread 0x01457a thru 0x0205aa (0xc031 bytes) 000405 1787625499.717026 read 0x038759 thru 0x03ffff (0x78a7 bytes) 000408 1787625499.721614 mapwrite 0x002221 thru 0x004599 (0x2379 bytes) 000411 1787625499.817679 mapwrite 0x02e000 thru 0x038ea4 (0xaea5 bytes) 000412 1787625499.869793 mapwrite 0x037e08 thru 0x03f3f2 (0x75eb bytes) 000413 1787625499.937283 write 0x013c2a thru 0x01b963 (0x7d3a bytes) 000416 1787625499.939503 read 0x01b664 thru 0x02acbb (0xf658 bytes) 000417 1787625499.943741 trunc from 0x040000 to 0x026960 (0x1969f bytes) 000418 1787625499.970486 read 0x0022cb thru 0x0103c6 (0xe0fc bytes) 000419 1787625499.971472 write 0x0138e7 thru 0x0173e2 (0x3afc bytes) 000420 1787625499.973797 mapread 0x0052de thru 0x00900c (0x3d2f bytes) 000422 1787625499.984449 mapwrite 0x031cce thru 0x037460 (0x5793 bytes) 000424 1787625500.112943 mapread 0x0300fe thru 0x037460 (0x7363 bytes) 000427 1787625500.115788 mapread 0x0345e8 thru 0x03580a (0x1223 bytes) 000428 1787625500.124860 MIRROR_EXTEND to 2 mirrors 000429 1787625501.079067 mapread 0x033920 thru 0x03592b (0x200c bytes) 000430 1787625501.150954 read 0x01836f thru 0x026bb2 (0xe844 bytes) 000432 1787625501.216660 trunc from 0x037461 to 0x031efe (0x5562 bytes) 000433 1787625501.455492 mapread 0x01f7ce thru 0x02c449 (0xcc7c bytes) 000435 1787625501.486053 mapwrite 0x006b90 thru 0x012b2a (0xbf9b bytes) 000437 1787625501.652795 read 0x00226a thru 0x004566 (0x22fd bytes) 000439 1787625501.653924 mapwrite 0x01b66a thru 0x01b8d5 (0x26c bytes) 000441 1787625501.784208 write 0x035d4f thru 0x0388e1 (0x2b93 bytes) 000442 1787625501.787588 write 0x0318af thru 0x039fed (0x873f bytes) 000443 1787625501.803617 trunc from 0x039fee to 0x0154df (0x24b0e bytes) 000446 1787625501.838964 read 0x00c257 thru 0x00dd9d (0x1b47 bytes) 000448 1787625501.841433 trunc from 0x0154e0 to 0x01436a (0x1175 bytes) 000451 1787625501.892307 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b7:0x0]: Invalid argument (22) 000452 1787625502.187579 mapread 0x007245 thru 0x01436a (0xd126 bytes) 000453 1787625502.270746 trunc from 0x01436b to 0x038abd (0x24753 bytes) 000454 1787625502.289852 mapwrite 0x009f68 thru 0x0133ef (0x9488 bytes) 000456 1787625502.458213 write 0x01714b thru 0x024d14 (0xdbca bytes) 000460 1787625502.478015 mapwrite 0x03466c thru 0x039ff4 (0x5989 bytes) 000462 1787625502.633344 trunc from 0x039ff5 to 0x013f24 (0x260d0 bytes) 000464 1787625502.673076 read 0x007b72 thru 0x00b680 (0x3b0f bytes) 000465 1787625502.683108 write 0x02f5e9 thru 0x031ed4 (0x28ec bytes) 000470 1787625502.695617 mapread 0x000622 thru 0x00289f (0x227e bytes) 000471 1787625502.706010 mapread 0x00e7e0 thru 0x012101 (0x3922 bytes) 000473 1787625502.708491 mapread 0x00f398 thru 0x01ad84 (0xb9ed bytes) 000480 1787625502.724510 read 0x031e17 thru 0x031ed4 (0xbe bytes) 000481 1787625502.725089 mapread 0x011d0a thru 0x02178d (0xfa84 bytes) 000482 1787625502.743720 write 0x01aa33 thru 0x028709 (0xdcd7 bytes) 000484 1787625502.745528 trunc from 0x031ed5 to 0x0333b5 (0x14e1 bytes) 000487 1787625502.793459 read 0x005c46 thru 0x005ce3 (0x9e bytes) 000488 1787625502.796427 read 0x010739 thru 0x01a660 (0x9f28 bytes) 000490 1787625502.800690 mapread 0x009ac6 thru 0x011507 (0x7a42 bytes) 000491 1787625502.801627 read 0x018510 thru 0x018610 (0x101 bytes) 000494 1787625502.805395 write 0x027f27 thru 0x02fb01 (0x7bdb bytes) 000495 1787625502.807667 mapwrite 0x034a51 thru 0x03ffff (0xb5af bytes) 000499 1787625502.896858 write 0x035aab thru 0x036727 (0xc7d bytes) 000500 1787625502.897775 write 0x00b2c0 thru 0x014112 (0x8e53 bytes) 000501 1787625502.905010 trunc from 0x040000 to 0x031dbc (0xe243 bytes) 000503 1787625502.924875 trunc from 0x031dbd to 0x014002 (0x1ddba bytes) 000507 1787625502.945332 mapread 0x006017 thru 0x013b31 (0xdb1b bytes) 000508 1787625502.946241 read 0x00913f thru 0x00b1af (0x2071 bytes) 000511 1787625502.952758 mapwrite 0x02d1c2 thru 0x036260 (0x909f bytes) 000514 1787625503.226331 MIRROR_EXTEND to 2 mirrors 000516 1787625504.144346 read 0x023db1 thru 0x028da6 (0x4ff6 bytes) 000517 1787625504.176787 trunc from 0x036261 to 0x032874 (0x39ec bytes) 000518 1787625504.327036 mapwrite 0x00aed1 thru 0x00d994 (0x2ac4 bytes) 000519 1787625504.424934 MIRROR_RESYNC 2 mirrors 000520 1787625505.378842 mapwrite 0x000256 thru 0x00358e (0x3339 bytes) 000523 1787625505.599501 mapread 0x01dad2 thru 0x01ff5d (0x248c bytes) 000524 1787625505.611878 mapread 0x01cfcb thru 0x02b0b5 (0xe0eb bytes) 000525 1787625505.623790 mapread 0x014b12 thru 0x0178ca (0x2db9 bytes) 000526 1787625505.632410 mapread 0x02f770 thru 0x032874 (0x3105 bytes) 000527 1787625505.638007 trunc from 0x032875 to 0x01d01a (0x1585a bytes) 000528 1787625505.688352 trunc from 0x01d01b to 0x006177 (0x16ea3 bytes) 000529 1787625505.725348 trunc from 0x006178 to 0x008517 (0x23a0 bytes) 000530 1787625505.763651 mapread 0x0081da thru 0x008517 (0x33e bytes) 000531 1787625505.779420 mapread 0x002e76 thru 0x008517 (0x56a2 bytes) 000532 1787625505.780363 write 0x001f95 thru 0x011122 (0xf18e bytes) 000534 1787625505.793809 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b7:0x0]: Invalid argument (22) 000535 1787625506.079076 trunc from 0x011123 to 0x00722e (0x9ef4 bytes) 000537 1787625506.106655 trunc from 0x00722f to 0x03a0af (0x32e81 bytes) 000538 1787625506.131401 mapwrite 0x0002e2 thru 0x001002 (0xd21 bytes) 000539 1787625506.182600 write 0x01abb9 thru 0x01e165 (0x35ad bytes) 000541 1787625506.193400 mapwrite 0x0217d1 thru 0x028b44 (0x7374 bytes) 000542 1787625506.345910 trunc from 0x03a0b0 to 0x019495 (0x20c1a bytes) 000545 1787625506.387409 read 0x017016 thru 0x019495 (0x2480 bytes) 000548 1787625506.411051 write 0x036290 thru 0x03d69d (0x740e bytes) 000550 1787625506.415958 mapwrite 0x029732 thru 0x031cfb (0x85ca bytes) 000551 1787625506.596185 read 0x00a3f0 thru 0x018fbe (0xebcf bytes) 000556 1787625506.618636 trunc from 0x03d69e to 0x038b98 (0x4b05 bytes) 000557 1787625506.652017 MIRROR_EXTEND to 2 mirrors 000559 1787625507.405485 mapwrite 0x002a4d thru 0x003500 (0xab4 bytes) 000560 1787625507.663520 read 0x00cbaa thru 0x017b0c (0xaf63 bytes) 000561 1787625507.684908 trunc from 0x038b99 to 0x03854c (0x64c bytes) 000562 1787625507.720260 write 0x039d65 thru 0x03a55c (0x7f8 bytes) 000563 1787625507.722652 mapread 0x0270a7 thru 0x02b409 (0x4363 bytes) 000564 1787625507.736133 mapread 0x00c98f thru 0x010284 (0x38f6 bytes) 000565 1787625507.737505 trunc from 0x03a55d to 0x027bab (0x129b1 bytes) 000569 1787625507.763784 read 0x001422 thru 0x008089 (0x6c68 bytes) 000570 1787625507.764491 write 0x00423d thru 0x005827 (0x15eb bytes) 000572 1787625507.765893 mapread 0x022b4d thru 0x027bab (0x505f bytes) 000573 1787625507.773002 mapread 0x015306 thru 0x018a36 (0x3731 bytes) 000574 1787625507.795064 mapread 0x01fb30 thru 0x027bab (0x807c bytes) 000575 1787625507.805985 mapwrite 0x0203a0 thru 0x0250c8 (0x4d29 bytes) 000576 1787625507.949887 trunc from 0x027bac to 0x036a11 (0xee66 bytes) 000577 1787625507.976010 mapread 0x023faa thru 0x02ee2b (0xae82 bytes) 000578 1787625507.999893 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x36a12): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36a12): Mirror 2: 0xffc849de (stale) CRC-32 checksum value for chunk [0, 0x36a12): Mirror 1: 0x6637363e Mirror 3: 0x6637363e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36a12: skipped 000579 1787625509.372756 write 0x03f465 thru 0x03ffff (0xb9b bytes) 000580 1787625509.437194 write 0x008096 thru 0x00ef11 (0x6e7c bytes) 000583 1787625509.451127 trunc from 0x040000 to 0x0048cf (0x3b730 bytes) 000587 1787625509.475891 mapread 0x003074 thru 0x0048cf (0x185c bytes) 000588 1787625509.486897 read 0x001848 thru 0x0047e6 (0x2f9f bytes) 000589 1787625509.491920 mapread 0x002f00 thru 0x0048cf (0x19d0 bytes) 000590 1787625509.497843 MIRROR_RESYNC 3 mirrors 000593 1787625510.174132 write 0x0289c7 thru 0x02c854 (0x3e8e bytes) 000594 1787625510.268239 mapwrite 0x006a39 thru 0x00c3be (0x5986 bytes) 000596 1787625510.336418 MIRROR_RESYNC 3 mirrors 000599 1787625511.271455 MIRROR_RESYNC 3 mirrors 000600 1787625511.676140 mapread 0x00228f thru 0x007a9a (0x580c bytes) 000601 1787625511.697856 mapread 0x026375 thru 0x02c854 (0x64e0 bytes) 000602 1787625511.706917 write 0x009927 thru 0x0169dd (0xd0b7 bytes) 000603 1787625511.771197 mapwrite 0x003a05 thru 0x003d89 (0x385 bytes) 000604 1787625511.833490 mapwrite 0x0076f3 thru 0x00c049 (0x4957 bytes) 000609 1787625511.969760 mapread 0x02680d thru 0x029e9c (0x3690 bytes) 000610 1787625512.005551 read 0x014e23 thru 0x01e729 (0x9907 bytes) 000611 1787625512.017910 read 0x00a5d9 thru 0x00b1b2 (0xbda bytes) 000612 1787625512.029963 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2c855): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c855): Mirror 2: 0x5435ed73 (stale) Mirror 3: 0x5435ed73 (stale) CRC-32 checksum value for chunk [0, 0x2c855): Mirror 1: 0xeefbf290 Mirror 4: 0xeefbf290 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c855: skipped 000613 1787625512.977271 write 0x02d616 thru 0x031602 (0x3fed bytes) 000614 1787625513.098203 mapwrite 0x00611a thru 0x0135a1 (0xd488 bytes) 000615 1787625513.251571 write 0x01b5ce thru 0x029a36 (0xe469 bytes) 000616 1787625513.287930 mapread 0x016b81 thru 0x0190ea (0x256a bytes) 000618 1787625513.291527 mapwrite 0x00125e thru 0x00478d (0x3530 bytes) 000620 1787625513.453266 write 0x010ed8 thru 0x02030f (0xf438 bytes) 000621 1787625513.474440 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x31603): Mirror 2: 0x82f26842 (stale) Mirror 4: 0xf6d6465b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31603): Mirror 2: 0x82f26842 (stale) Mirror 4: 0xf6d6465b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31603: skipped 000623 1787625513.930181 mapread 0x00b0bb thru 0x017ce6 (0xcc2c bytes) 000626 1787625513.967338 trunc from 0x031603 to 0x023fad (0xd655 bytes) 000628 1787625513.992410 read 0x010d92 thru 0x0147cc (0x3a3b bytes) 000630 1787625513.998735 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x23fae): Mirror 4: 0x3c5a35cd (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x23fae): Mirror 4: 0x3c5a35cd (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x23fae: skipped 000633 1787625514.237116 trunc from 0x023fae to 0x019ac2 (0xa4eb bytes) 000635 1787625514.256140 write 0x027eb9 thru 0x02b64a (0x3792 bytes) 000636 1787625514.257797 write 0x02396d thru 0x02b010 (0x76a4 bytes) 000637 1787625514.263008 trunc from 0x02b64b to 0x003992 (0x27cb8 bytes) 000638 1787625514.288451 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b7:0x0]: Invalid argument (22) 000639 1787625514.458070 read 0x0004c1 thru 0x003992 (0x34d2 bytes) 000640 1787625514.462982 mapwrite 0x006466 thru 0x010e86 (0xaa21 bytes) 000641 1787625514.637684 trunc from 0x010e87 to 0x01e8ef (0xda69 bytes) 000644 1787625514.670886 trunc from 0x01e8f0 to 0x03b67f (0x1cd90 bytes) 000645 1787625514.738836 read 0x00557f thru 0x01077a (0xb1fc bytes) 000646 1787625514.742098 read 0x0008ef thru 0x007bba (0x72cc bytes) 000649 1787625514.763438 MIRROR_EXTEND to 2 mirrors 000650 1787625515.754552 read 0x024c7a thru 0x0278b9 (0x2c40 bytes) 000651 1787625515.770780 write 0x02438b thru 0x02630b (0x1f81 bytes) 000652 1787625515.866926 read 0x005b47 thru 0x013301 (0xd7bb bytes) 000653 1787625515.885433 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3b680): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b680): Mirror 2: 0xa6cfdc92 (stale) CRC-32 checksum value for chunk [0, 0x3b680): Mirror 1: 0xaca03894 Mirror 3: 0xaca03894 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b680: skipped 000655 1787625517.149092 mapread 0x02528b thru 0x02f178 (0x9eee bytes) 000657 1787625517.183605 read 0x012efd thru 0x01f2bc (0xc3c0 bytes) 000659 1787625517.203121 mapread 0x02b9ea thru 0x038989 (0xcfa0 bytes) 000661 1787625517.234706 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3b680): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b680): Mirror 2: 0xa6cfdc92 (stale) CRC-32 checksum value for chunk [0, 0x3b680): Mirror 1: 0xaca03894 Mirror 3: 0xaca03894 Mirror 4: 0xaca03894 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b680: skipped 000662 1787625517.997124 read 0x033df0 thru 0x03b67f (0x7890 bytes) 000664 1787625518.021847 mapwrite 0x02cf55 thru 0x033b2f (0x6bdb bytes) 000665 1787625518.278620 mapread 0x033e83 thru 0x035fe9 (0x2167 bytes) 000666 1787625518.279537 write 0x03bf6e thru 0x03ffff (0x4092 bytes) 000667 1787625518.290337 trunc from 0x040000 to 0x01df35 (0x220ca bytes) 000668 1787625518.331402 write 0x0306a4 thru 0x03ffff (0xf95c bytes) 000669 1787625518.340774 mapread 0x023ccc thru 0x02843f (0x4774 bytes) 000671 1787625518.365489 MIRROR_RESYNC 4 mirrors 000673 1787625520.280072 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b7:0x0]: Invalid argument (22) 000674 1787625520.680347 trunc from 0x040000 to 0x0371c6 (0x8e39 bytes) 000681 1787625520.813533 mapwrite 0x026043 thru 0x03168b (0xb649 bytes) 000686 1787625520.890230 read 0x02b568 thru 0x02fc98 (0x4731 bytes) 000687 1787625520.891556 mapwrite 0x00c7e3 thru 0x013899 (0x70b7 bytes) truncating to largest ever: 0x3cf5a 000691 1787625520.984372 trunc from 0x0371c7 to 0x03cf59 (0x5d93 bytes) 000695 1787625521.044919 mapwrite 0x015786 thru 0x01cae9 (0x7364 bytes) 000697 1787625521.193063 mapread 0x00badf thru 0x01a483 (0xe9a5 bytes) 000698 1787625521.195702 read 0x039465 thru 0x03a68f (0x122b bytes) 000699 1787625521.219074 write 0x030fbf thru 0x033c73 (0x2cb5 bytes) 000704 1787625521.229984 read 0x037827 thru 0x03883c (0x1016 bytes) 000705 1787625521.231914 read 0x020338 thru 0x023529 (0x31f2 bytes) 000707 1787625521.236298 read 0x01e363 thru 0x029085 (0xad23 bytes) 000708 1787625521.237969 read 0x034ba1 thru 0x03c10c (0x756c bytes) 000711 1787625521.245644 trunc from 0x03cf5a to 0x016b72 (0x263e7 bytes) 000712 1787625521.303759 write 0x026180 thru 0x0335c4 (0xd445 bytes) 000713 1787625521.307395 mapwrite 0x03e9c3 thru 0x03ffff (0x163d bytes) 000714 1787625521.456031 write 0x013aed thru 0x01eff6 (0xb50a bytes) 000716 1787625521.460039 write 0x029f5b thru 0x039b1f (0xfbc5 bytes) 000718 1787625521.463772 read 0x02ef58 thru 0x03e848 (0xf8f1 bytes) 000722 1787625521.465559 read 0x038e1c thru 0x03ffff (0x71e4 bytes) 000723 1787625521.471946 mapwrite 0x0351fb thru 0x03ffff (0xae05 bytes) 000727 1787625521.556106 write 0x013948 thru 0x01a05a (0x6713 bytes) 000728 1787625521.559253 trunc from 0x040000 to 0x039494 (0x6b6b bytes) 000730 1787625521.611701 MIRROR_RESYNC 3 mirrors 000731 1787625522.747572 read 0x01161d thru 0x018bbb (0x759f bytes) 000732 1787625522.762790 read 0x012c16 thru 0x01ee87 (0xc272 bytes) 000735 1787625522.785197 mapread 0x00c4eb thru 0x00d9d0 (0x14e6 bytes) 000736 1787625522.786609 write 0x02abea thru 0x02d381 (0x2798 bytes) 000737 1787625522.985085 MIRROR_RESYNC 3 mirrors 000740 1787625524.075189 trunc from 0x039495 to 0x011f5e (0x27536 bytes) 000744 1787625524.228213 mapread 0x0023cd thru 0x0096d7 (0x730b bytes) 000749 1787625524.268220 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x11f5f): Mirror 4: 0xcc197566 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x11f5f): Mirror 4: 0xcc197566 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x11f5f: skipped 000752 1787625524.844064 mapwrite 0x035773 thru 0x03ffff (0xa88d bytes) 000754 1787625524.922433 trunc from 0x040000 to 0x02083d (0x1f7c2 bytes) 000755 1787625524.947467 mapwrite 0x032984 thru 0x039768 (0x6de5 bytes) 000757 1787625525.025332 MIRROR_RESYNC 2 mirrors 000759 1787625526.364472 trunc from 0x039769 to 0x022f74 (0x167f4 bytes) 000760 1787625526.555385 mapwrite 0x0116cb thru 0x01f081 (0xd9b7 bytes) 000764 1787625526.718551 mapwrite 0x003da8 thru 0x007b48 (0x3da1 bytes) 000766 1787625526.836247 read 0x0171bb thru 0x0216c0 (0xa506 bytes) 000767 1787625526.837469 read 0x00d4e8 thru 0x010f33 (0x3a4c bytes) 000768 1787625526.842831 MIRROR_RESYNC 2 mirrors 000769 1787625527.561584 write 0x01b280 thru 0x022cc9 (0x7a4a bytes) 000770 1787625527.667751 mapwrite 0x0310c2 thru 0x03f932 (0xe871 bytes) 000772 1787625527.794927 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b7:0x0]: Invalid argument (22) 000773 1787625527.994843 read 0x0351eb thru 0x03cb1e (0x7934 bytes) 000776 1787625528.021241 read 0x02cdef thru 0x02e474 (0x1686 bytes) 000780 1787625528.053520 write 0x023e49 thru 0x02dfc2 (0xa17a bytes) 000783 1787625528.083752 mapread 0x02b880 thru 0x03adf9 (0xf57a bytes) 000784 1787625528.084494 trunc from 0x03f933 to 0x0196e6 (0x2624c bytes) 000785 1787625528.098962 read 0x0180c2 thru 0x0196e6 (0x1625 bytes) 000786 1787625528.099757 write 0x0079b7 thru 0x008520 (0xb6a bytes) 000787 1787625528.100455 mapread 0x00fcd5 thru 0x0193fa (0x9726 bytes) 000788 1787625528.102497 write 0x007483 thru 0x007517 (0x95 bytes) 000789 1787625528.103560 trunc from 0x0196e7 to 0x0239dd (0xa2f7 bytes) 000792 1787625528.117642 read 0x015624 thru 0x01e603 (0x8fe0 bytes) 000793 1787625528.123985 write 0x021fd4 thru 0x023c54 (0x1c81 bytes) 000794 1787625528.124819 write 0x02aa32 thru 0x039060 (0xe62f bytes) 000796 1787625528.129520 mapread 0x0225a7 thru 0x0295b6 (0x7010 bytes) 000797 1787625528.141148 read 0x03813e thru 0x039060 (0xf23 bytes) 000800 1787625528.143495 read 0x025901 thru 0x026164 (0x864 bytes) 000801 1787625528.145098 write 0x023124 thru 0x0270e3 (0x3fc0 bytes) 000802 1787625528.147886 write 0x03764f thru 0x03ffff (0x89b1 bytes) 000803 1787625528.150617 mapread 0x01af2a thru 0x025a71 (0xab48 bytes) 000804 1787625528.152630 write 0x026f87 thru 0x02fb05 (0x8b7f bytes) 000805 1787625528.156528 read 0x023626 thru 0x0245b1 (0xf8c bytes) 000806 1787625528.158099 read 0x03a4c7 thru 0x03ffff (0x5b39 bytes) 000808 1787625528.160108 mapread 0x02ed55 thru 0x03a649 (0xb8f5 bytes) 000810 1787625528.167025 mapread 0x005b87 thru 0x00dd6e (0x81e8 bytes) 000811 1787625528.170960 mapwrite 0x02ee0d thru 0x03bf0b (0xd0ff bytes) 000814 1787625528.229807 trunc from 0x040000 to 0x02dc2d (0x123d2 bytes) 000815 1787625528.271703 trunc from 0x02dc2e to 0x014ec2 (0x18d6b bytes) 000816 1787625528.313920 mapwrite 0x02226d thru 0x026226 (0x3fba bytes) 000819 1787625528.476806 write 0x006e09 thru 0x0165d2 (0xf7ca bytes) 000820 1787625528.482482 trunc from 0x026227 to 0x01d748 (0x8ade bytes) 000822 1787625528.517246 mapread 0x008d27 thru 0x010d24 (0x7ffe bytes) 000824 1787625528.525487 read 0x01b477 thru 0x01d748 (0x22d2 bytes) 000825 1787625528.526745 write 0x02ab6c thru 0x035eb6 (0xb34b bytes) 000828 1787625528.536609 trunc from 0x035eb7 to 0x006257 (0x2fc5f bytes) 000830 1787625528.565435 trunc from 0x006258 to 0x020940 (0x1a6e9 bytes) 000834 1787625528.593818 mapread 0x00b13f thru 0x013e20 (0x8ce2 bytes) 000836 1787625528.607542 mapwrite 0x020747 thru 0x025289 (0x4b43 bytes) 000838 1787625528.705771 write 0x002fb0 thru 0x011b57 (0xeba8 bytes) 000839 1787625528.711757 write 0x03b814 thru 0x03ffff (0x47ec bytes) 000840 1787625528.714974 mapwrite 0x010cd4 thru 0x017a36 (0x6d63 bytes) 000841 1787625528.782460 write 0x035c3d thru 0x037d42 (0x2106 bytes) 000848 1787625528.813433 read 0x03e1a5 thru 0x03ffff (0x1e5b bytes) 000849 1787625528.814646 write 0x02e2d1 thru 0x034be0 (0x6910 bytes) 000850 1787625528.816967 mapwrite 0x005217 thru 0x0071e0 (0x1fca bytes) 000851 1787625528.909404 read 0x03ba8d thru 0x03ffff (0x4573 bytes) 000852 1787625528.911025 mapread 0x011854 thru 0x0145b2 (0x2d5f bytes) 000856 1787625528.915461 trunc from 0x040000 to 0x011a15 (0x2e5ea bytes) 000857 1787625528.950591 mapwrite 0x00ae29 thru 0x01a9b2 (0xfb8a bytes) 000861 1787625529.071690 mapread 0x006031 thru 0x00e9e7 (0x89b7 bytes) 000864 1787625529.078664 trunc from 0x01a9b3 to 0x037476 (0x1cac4 bytes) 000865 1787625529.107525 mapwrite 0x013b96 thru 0x020ef5 (0xd360 bytes) 000868 1787625529.203630 read 0x02ef5b thru 0x02fcd5 (0xd7b bytes) 000870 1787625529.208084 read 0x025d1e thru 0x03215a (0xc43d bytes) 000872 1787625529.210605 read 0x006074 thru 0x0157f0 (0xf77d bytes) 000873 1787625529.212285 read 0x0250f5 thru 0x02945b (0x4367 bytes) 000875 1787625529.215059 mapread 0x02b0f9 thru 0x02bd77 (0xc7f bytes) 000876 1787625529.217024 mapread 0x0292a0 thru 0x02acd1 (0x1a32 bytes) 000877 1787625529.219029 mapread 0x01b594 thru 0x024f39 (0x99a6 bytes) 000880 1787625529.229026 MIRROR_EXTEND to 2 mirrors 000881 1787625530.088109 MIRROR_RESYNC 2 mirrors 000882 1787625530.348039 mapread 0x02bfd3 thru 0x02e207 (0x2235 bytes) 000884 1787625530.391123 MIRROR_EXTEND to 3 mirrors 000885 1787625530.949460 trunc from 0x037477 to 0x0128ae (0x24bc8 bytes) 000887 1787625531.082563 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x128af): Mirror 3: 0x27812d8c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x128af): Mirror 3: 0x27812d8c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x128af: skipped 000889 1787625531.335094 mapwrite 0x02ef8d thru 0x03ba13 (0xca87 bytes) 000892 1787625531.483926 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b7:0x0]: Invalid argument (22) 000893 1787625531.638932 mapread 0x013464 thru 0x01f5ba (0xc157 bytes) 000895 1787625531.686738 read 0x024cd0 thru 0x03066c (0xb99d bytes) 000896 1787625531.703730 MIRROR_EXTEND to 2 mirrors 000897 1787625532.622873 mapread 0x01c408 thru 0x01c6eb (0x2e4 bytes) 000898 1787625532.648671 read 0x008958 thru 0x00f5ea (0x6c93 bytes) 000899 1787625532.675016 write 0x02048f thru 0x02d402 (0xcf74 bytes) 000901 1787625532.765395 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b7:0x0]: Invalid argument (22) 000902 1787625532.876107 write 0x016d10 thru 0x01caa8 (0x5d99 bytes) 000905 1787625532.888783 mapwrite 0x0194da thru 0x019979 (0x4a0 bytes) 000909 1787625532.937497 trunc from 0x03ba14 to 0x0255fb (0x16418 bytes) 000912 1787625532.956730 trunc from 0x0255fc to 0x00486f (0x20d8c bytes) 000916 1787625532.982706 read 0x003278 thru 0x00486f (0x15f8 bytes) 000917 1787625532.988537 read 0x003a69 thru 0x00486f (0xe07 bytes) 000919 1787625532.990517 read 0x0028cb thru 0x00486f (0x1fa5 bytes) 000922 1787625532.999206 mapread 0x000ad8 thru 0x002a4e (0x1f77 bytes) 000923 1787625533.000623 trunc from 0x004870 to 0x033080 (0x2e811 bytes) 000925 1787625533.023983 write 0x0214b9 thru 0x02190c (0x454 bytes) 000926 1787625533.036501 read 0x028d8b thru 0x02f212 (0x6488 bytes) 000927 1787625533.037782 read 0x02b2e7 thru 0x033080 (0x7d9a bytes) 000928 1787625533.038679 mapwrite 0x0292ea thru 0x02efe6 (0x5cfd bytes) 000931 1787625533.093651 read 0x026ee9 thru 0x033044 (0xc15c bytes) 000933 1787625533.096636 mapwrite 0x026a39 thru 0x02df65 (0x752d bytes) 000934 1787625533.149353 trunc from 0x033081 to 0x032513 (0xb6d bytes) 000935 1787625533.169350 mapwrite 0x001d61 thru 0x011858 (0xfaf8 bytes) 000937 1787625533.242450 mapread 0x01a905 thru 0x029f8c (0xf688 bytes) 000938 1787625533.243734 mapwrite 0x00cd38 thru 0x01b516 (0xe7df bytes) 000939 1787625533.296435 mapwrite 0x00a622 thru 0x0192e0 (0xecbf bytes) 000941 1787625533.351519 mapwrite 0x00364f thru 0x00e263 (0xac15 bytes) 000942 1787625533.444537 MIRROR_EXTEND to 2 mirrors 000944 1787625534.176421 mapwrite 0x0052c7 thru 0x00f956 (0xa690 bytes) 000946 1787625534.348293 write 0x037de7 thru 0x03ffff (0x8219 bytes) 000947 1787625534.350425 write 0x02f8ad thru 0x03452f (0x4c83 bytes) 000948 1787625534.360390 MIRROR_RESYNC 2 mirrors 000950 1787625535.056880 mapread 0x009cfd thru 0x0170a2 (0xd3a6 bytes) 000951 1787625535.102036 write 0x00c5f3 thru 0x014e00 (0x880e bytes) 000955 1787625535.238019 trunc from 0x040000 to 0x0054e2 (0x3ab1d bytes) 000956 1787625535.282970 trunc from 0x0054e3 to 0x021d02 (0x1c820 bytes) 000961 1787625535.319978 read 0x006cb5 thru 0x0123c4 (0xb710 bytes) 000962 1787625535.337948 trunc from 0x021d03 to 0x004d3e (0x1cfc4 bytes) 000963 1787625535.390651 mapread 0x003d2b thru 0x004d3e (0x1014 bytes) 000966 1787625535.403661 trunc from 0x004d3f to 0x035ac8 (0x30d8a bytes) 000967 1787625535.435646 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b7:0x0]: Invalid argument (22) 000971 1787625535.669727 trunc from 0x035ac9 to 0x037b50 (0x2088 bytes) 000972 1787625535.683631 write 0x027546 thru 0x03143c (0x9ef7 bytes) 000973 1787625535.691364 mapwrite 0x020b84 thru 0x02d81c (0xcc99 bytes) 000974 1787625535.777762 read 0x01b242 thru 0x0249e2 (0x97a1 bytes) 000976 1787625535.780081 write 0x03c651 thru 0x03ffff (0x39af bytes) 000977 1787625535.782565 mapread 0x00a4cd thru 0x00f0e0 (0x4c14 bytes) 000978 1787625535.797674 read 0x0252a2 thru 0x02f497 (0xa1f6 bytes) 000979 1787625535.798569 mapread 0x02778e thru 0x03642e (0xeca1 bytes) 000983 1787625535.813496 write 0x03343d thru 0x03f0a7 (0xbc6b bytes) 000985 1787625535.819673 trunc from 0x040000 to 0x02d2fd (0x12d02 bytes) 000987 1787625535.843750 mapread 0x014f97 thru 0x01e199 (0x9203 bytes) 000992 1787625535.860439 MIRROR_EXTEND to 2 mirrors 000996 1787625537.508655 MIRROR_RESYNC 2 mirrors 000997 1787625538.083878 trunc from 0x02d2fe to 0x0277fe (0x5aff bytes) 000999 1787625538.280241 MIRROR_RESYNC 2 mirrors All operations completed A-OK!