== sanity-flr test 70a: flr mode fsx test ================ 16:54:23 (1746132863) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70a.sanity-flr Seed set to 2864 000001 1746132864.801465 MIRROR_EXTEND to 2 mirrors skipping zero length punch hole skipping zero size read skipping zero length punch hole 000005 1746132865.061525 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x2ac:0x0]: Invalid argument (22) skipping zero length zero range skipping zero size read fallocating to largest ever: 0x25afb 000009 1746132865.305104 fallocate 0x01b219 thru 0x025afa (0xa8e2 bytes) 000010 1746132865.324611 punch from 0x013514 to 0x020b44 (0xd631 bytes) 000011 1746132865.331822 write 0x022b6c thru 0x03160b (0xeaa0 bytes) 000013 1746132865.342460 mapread 0x02fbf9 thru 0x03160b (0x1a13 bytes) 000014 1746132865.343159 write 0x02d74b thru 0x030cbe (0x3574 bytes) 000015 1746132865.343737 punch from 0x01505f to 0x01a2ae (0x5250 bytes) fallocating to largest ever: 0x40000 000016 1746132865.352953 fallocate 0x03ce5a thru 0x03ffff (0x31a6 bytes) 000017 1746132865.361649 fallocate 0x0021c1 thru 0x005bab (0x39eb bytes) 000018 1746132865.368037 mapwrite 0x0147e6 thru 0x01d16f (0x898a bytes) 000020 1746132865.429797 read 0x03038c thru 0x03dd4b (0xd9c0 bytes) 000021 1746132865.440638 mapwrite 0x01799b thru 0x01c304 (0x496a bytes) 000022 1746132865.463557 punch from 0x0392ae to 0x03ffff (0x6d52 bytes) 000023 1746132865.469080 mapread 0x02e931 thru 0x035754 (0x6e24 bytes) 000024 1746132865.470042 read 0x0032a2 thru 0x00aa28 (0x7787 bytes) 000025 1746132865.481038 read 0x01642e thru 0x0231bc (0xcd8f bytes) 000026 1746132865.481985 read 0x031b45 thru 0x033177 (0x1633 bytes) 000028 1746132865.483066 trunc from 0x02daf2 to 0x0367a8 (0x8cb7 bytes) 000029 1746132865.491753 punch from 0x01da72 to 0x01f585 (0x1b14 bytes) 000030 1746132865.496309 fallocate 0x03d8c1 thru 0x03ffff (0x273f bytes) 000031 1746132865.509445 punch from 0x03ff5b to 0x03ffff (0xa5 bytes) 000032 1746132865.514117 mapread 0x00cb7b thru 0x00efc5 (0x244b bytes) 000033 1746132865.518873 mapread 0x018e90 thru 0x01e681 (0x57f2 bytes) 000034 1746132865.533748 write 0x00a9b3 thru 0x00f28b (0x48d9 bytes) 000035 1746132865.539236 punch from 0x035a07 to 0x03ffff (0xa5f9 bytes) 000036 1746132865.546258 mapread 0x0195be thru 0x0277c9 (0xe20c bytes) 000037 1746132865.547770 mapread 0x03c9ca thru 0x03ffff (0x3636 bytes) 000040 1746132865.568974 fallocate 0x02a8ed thru 0x02ed3c (0x4450 bytes) 000041 1746132865.573460 mapwrite 0x01d386 thru 0x02c8bc (0xf537 bytes) 000042 1746132865.588537 mapread 0x01bc6b thru 0x021005 (0x539b bytes) 000043 1746132865.595713 MIRROR_EXTEND to 2 mirrors 000044 1746132866.033904 punch from 0x015233 to 0x01abf0 (0x59be bytes) 000045 1746132866.076354 MIRROR_RESYNC 2 mirrors 000046 1746132866.530909 write 0x036bcb thru 0x03eaaa (0x7ee0 bytes) 000048 1746132866.570843 mapwrite 0x019d2d thru 0x01a89a (0xb6e bytes) 000049 1746132866.613560 fallocate 0x016504 thru 0x0249dd (0xe4da bytes) 000050 1746132866.637855 fallocate 0x00344b thru 0x00f8de (0xc494 bytes) 000051 1746132866.652813 mapwrite 0x003797 thru 0x0121d9 (0xea43 bytes) 000052 1746132866.717790 punch from 0x032537 to 0x03636d (0x3e37 bytes) 000053 1746132866.733963 mapread 0x027609 thru 0x028b62 (0x155a bytes) 000054 1746132866.765373 read 0x009a9f thru 0x018d79 (0xf2db bytes) 000055 1746132866.766169 trunc from 0x040000 to 0x037d2b (0x82d4 bytes) 000056 1746132866.821408 fallocate 0x02eeff thru 0x033e7f (0x4f81 bytes) 000057 1746132866.835236 fallocate 0x0121ed thru 0x01ece0 (0xcaf4 bytes) 000059 1746132866.867691 MIRROR_RESYNC 2 mirrors lfs mirror mirror: fail to pread 221184-8192 of mirror 1: Invalid argument (22) lfs mirror mirror: error reading bytes 221184-8192 of mirror 1: Invalid argument (22) lfs mirror mirror: fail to mirror resync '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr': Invalid argument (22) lfs mirror resync /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 22 LOG DUMP (51 total operations): 1: 1746132864.801465 MIRROR_EXTEND to 2 mirrors 2: 1746132865.054659 SKIPPED (no operation) 3: 1746132865.054872 SKIPPED (no operation) 4: 1746132865.055206 SKIPPED (no operation) 5: 1746132865.061525 MIRROR_SPLIT mirror 1 to 1 mirrors 6: 1746132865.304198 SKIPPED (no operation) 7: 1746132865.304725 SKIPPED (no operation) 8: 1746132865.305104 FALLOC 0x1b219 thru 0x25afb (0x0a8e2 bytes)EXTENDING 9: 1746132865.324611 PUNCH 0x13514 thru 0x20b44 (0x0d631 bytes) 10: 1746132865.331822 WRITE 0x22b6c thru 0x3160b (0x0eaa0 bytes) EXTEND 11: 1746132865.342460 MAPREAD 0x2fbf9 thru 0x3160b (0x01a13 bytes) 12: 1746132865.343159 WRITE 0x2d74b thru 0x30cbe (0x03574 bytes) 13: 1746132865.343737 PUNCH 0x1505f thru 0x1a2ae (0x05250 bytes) 14: 1746132865.352953 FALLOC 0x3ce5a thru 0x40000 (0x031a6 bytes)EXTENDING 15: 1746132865.361649 FALLOC 0x021c1 thru 0x05bac (0x039eb bytes)INTERIOR 16: 1746132865.368037 MAPWRITE 0x147e6 thru 0x1d16f (0x0898a bytes) 17: 1746132865.429797 READ 0x3038c thru 0x3dd4b (0x0d9c0 bytes) 18: 1746132865.440638 MAPWRITE 0x1799b thru 0x1c304 (0x0496a bytes) 19: 1746132865.463557 PUNCH 0x392ae thru 0x3ffff (0x06d52 bytes) 20: 1746132865.469080 MAPREAD 0x2e931 thru 0x35754 (0x06e24 bytes) 21: 1746132865.470042 READ 0x032a2 thru 0x0aa28 (0x07787 bytes) 22: 1746132865.481038 READ 0x1642e thru 0x231bc (0x0cd8f bytes) 23: 1746132865.481985 READ 0x31b45 thru 0x33177 (0x01633 bytes) 24: 1746132865.483066 ZERO 0x2daf2 thru 0x367a8 (0x08cb7 bytes) 25: 1746132865.491753 PUNCH 0x1da72 thru 0x1f585 (0x01b14 bytes) 26: 1746132865.496309 FALLOC 0x3d8c1 thru 0x40000 (0x0273f bytes)INTERIOR 27: 1746132865.509445 PUNCH 0x3ff5b thru 0x3ffff (0x000a5 bytes) 28: 1746132865.514117 MAPREAD 0x0cb7b thru 0x0efc5 (0x0244b bytes) 29: 1746132865.518873 MAPREAD 0x18e90 thru 0x1e681 (0x057f2 bytes) 30: 1746132865.533748 WRITE 0x0a9b3 thru 0x0f28b (0x048d9 bytes) 31: 1746132865.539236 PUNCH 0x35a07 thru 0x3ffff (0x0a5f9 bytes) 32: 1746132865.546258 MAPREAD 0x195be thru 0x277c9 (0x0e20c bytes) 33: 1746132865.547770 MAPREAD 0x3c9ca thru 0x3ffff (0x03636 bytes) 34: 1746132865.568974 FALLOC 0x2a8ed thru 0x2ed3d (0x04450 bytes)INTERIOR 35: 1746132865.573460 MAPWRITE 0x1d386 thru 0x2c8bc (0x0f537 bytes) 36: 1746132865.588537 MAPREAD 0x1bc6b thru 0x21005 (0x0539b bytes) 37: 1746132865.595713 MIRROR_EXTEND to 2 mirrors 38: 1746132866.033904 PUNCH 0x15233 thru 0x1abf0 (0x059be bytes) 39: 1746132866.076354 MIRROR_RESYNC 2 mirrors 40: 1746132866.530909 WRITE 0x36bcb thru 0x3eaaa (0x07ee0 bytes) 41: 1746132866.570843 MAPWRITE 0x19d2d thru 0x1a89a (0x00b6e bytes) 42: 1746132866.613560 FALLOC 0x16504 thru 0x249de (0x0e4da bytes)INTERIOR 43: 1746132866.637855 FALLOC 0x0344b thru 0x0f8df (0x0c494 bytes)INTERIOR 44: 1746132866.652813 MAPWRITE 0x03797 thru 0x121d9 (0x0ea43 bytes) 45: 1746132866.717790 PUNCH 0x32537 thru 0x3636d (0x03e37 bytes) 46: 1746132866.733963 MAPREAD 0x27609 thru 0x28b62 (0x0155a bytes) 47: 1746132866.765373 READ 0x09a9f thru 0x18d79 (0x0f2db bytes) 48: 1746132866.766169 TRUNC_DN 0x40000 to 0x37d2c 49: 1746132866.821408 FALLOC 0x2eeff thru 0x33e80 (0x04f81 bytes)INTERIOR 50: 1746132866.835236 FALLOC 0x121ed thru 0x1ece1 (0x0caf4 bytes)INTERIOR 51: 1746132866.867691 MIRROR_RESYNC 2 mirrors Using seed 2864 Correct content saved for comparison (maybe hexdump "/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr" vs "/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr.fsxgood") sanity-flr test_70a: @@@@@@ FAIL: fsx FLR file /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr failed Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7249:error() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:3351:test_70a() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7622:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7685:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7503:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:3353:main() Dumping lctl log to /tmp/testlogs//sanity-flr.test_70a.*.1746132868.log rsync: chown "/tmp/testlogs/.sanity-flr.test_70a.debug_log.oleg158-server.1746132868.log.h7nrDk" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-flr.test_70a.dmesg.oleg158-server.1746132868.log.nFsi2M" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2]