== sanity-flr test 70a: flr mode fsx test ================ 18:57:50 (1763337470) Seed set to 7472 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000003 1763337472.293174 mapwrite 0x0228d0 thru 0x02c3b6 (0x9ae7 bytes) 000006 1763337472.406164 write 0x03a373 thru 0x03b561 (0x11ef bytes) 000009 1763337472.408000 mapwrite 0x0100fe thru 0x01f8fa (0xf7fd bytes) 000011 1763337472.506446 MIRROR_EXTEND to 2 mirrors 000012 1763337473.051855 mapwrite 0x003f65 thru 0x00590c (0x19a8 bytes) 000013 1763337473.247559 write 0x0244fe thru 0x030510 (0xc013 bytes) truncating to largest ever: 0xab47 000014 1763337473.269544 trunc from 0x03b562 to 0x00ab46 (0x30a1b bytes) 000015 1763337473.329863 write 0x0382e8 thru 0x03c468 (0x4181 bytes) 000016 1763337473.330978 mapwrite 0x029479 thru 0x038179 (0xed01 bytes) 000018 1763337473.425598 read 0x021457 thru 0x027bc8 (0x6772 bytes) 000022 1763337473.433951 mapwrite 0x02dcf2 thru 0x0399dc (0xbceb bytes) 000024 1763337473.488999 write 0x01c1ae thru 0x01d6bb (0x150e bytes) 000025 1763337473.501431 MIRROR_RESYNC 2 mirrors 000027 1763337474.189527 read 0x00a959 thru 0x00eb13 (0x41bb bytes) 000031 1763337474.198484 read 0x02bc88 thru 0x02dad7 (0x1e50 bytes) 000033 1763337474.216377 MIRROR_RESYNC 2 mirrors 000035 1763337474.552373 read 0x02a3d6 thru 0x02ecc9 (0x48f4 bytes) 000036 1763337474.552830 mapwrite 0x03a103 thru 0x03b54c (0x144a bytes) 000037 1763337474.696765 write 0x01b888 thru 0x027acd (0xc246 bytes) 000042 1763337474.726393 write 0x022a52 thru 0x031651 (0xec00 bytes) 000043 1763337474.743754 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3c469): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c469): Mirror 2: 0x80b34f4e (stale) CRC-32 checksum value for chunk [0, 0x3c469): Mirror 1: 0x9bc730a4 Mirror 3: 0x9bc730a4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c469: skipped truncating to largest ever: 0x12354 000045 1763337475.686328 trunc from 0x03c469 to 0x012353 (0x2a115 bytes) truncating to largest ever: 0x20539 000046 1763337475.760372 trunc from 0x012354 to 0x020538 (0xe1e5 bytes) 000048 1763337475.799993 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x20539): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x20539): Mirror 2: 0x32737d29 (stale) Mirror 3: 0x7f21bfac (stale) CRC-32 checksum value for chunk [0, 0x20539): Mirror 1: 0xe0367ca Mirror 4: 0xe0367ca lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x20539: skipped 000049 1763337476.592751 write 0x0337bb thru 0x03c769 (0x8faf bytes) 000053 1763337476.679094 write 0x003845 thru 0x006403 (0x2bbf bytes) truncating to largest ever: 0x2cfbe 000056 1763337476.692265 trunc from 0x03c76a to 0x02cfbd (0xf7ac bytes) 000057 1763337476.726961 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ab:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2cfbe): Mirror 3: 0x9db2e9fe (stale) Mirror 4: 0xab0a28b4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2cfbe): Mirror 3: 0x9db2e9fe (stale) Mirror 4: 0xab0a28b4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2cfbe: skipped 000058 1763337477.093487 mapwrite 0x022724 thru 0x02867f (0x5f5c bytes) 000060 1763337477.163251 trunc from 0x02cfbe to 0x00bdc2 (0x211fb bytes) 000061 1763337477.185377 trunc from 0x00bdc3 to 0x000ab0 (0xb312 bytes) 000063 1763337477.208924 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0xab1): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1000): Mirror 3: 0xc71c0011 (stale) Mirror 4: 0xc71c0011 (stale) CRC-32 checksum value for chunk [0, 0x1000): Mirror 1: 0xc71c0011 Mirror 5: 0xc71c0011 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1000: skipped Size error: expected 0xab1 stat 0xab1 seek 0x1000 LOG DUMP (57 total operations): 1: 1763337472.284702 SKIPPED (no operation) 2: 1763337472.292202 SKIPPED (no operation) 3: 1763337472.293174 MAPWRITE 0x228d0 thru 0x2c3b6 (0x09ae7 bytes) 4: 1763337472.405202 SKIPPED (no operation) 5: 1763337472.405669 SKIPPED (no operation) 6: 1763337472.406164 WRITE 0x3a373 thru 0x3b561 (0x011ef bytes) HOLE 7: 1763337472.407583 SKIPPED (no operation) 8: 1763337472.408000 MAPWRITE 0x100fe thru 0x1f8fa (0x0f7fd bytes) 9: 1763337472.501799 SKIPPED (no operation) 10: 1763337472.506446 MIRROR_EXTEND to 2 mirrors 11: 1763337473.051855 MAPWRITE 0x03f65 thru 0x0590c (0x019a8 bytes) 12: 1763337473.247559 WRITE 0x244fe thru 0x30510 (0x0c013 bytes) 13: 1763337473.269544 TRUNC_DN 0x3b562 to 0x0ab47 14: 1763337473.329863 WRITE 0x382e8 thru 0x3c468 (0x04181 bytes) HOLE 15: 1763337473.330978 MAPWRITE 0x29479 thru 0x38179 (0x0ed01 bytes) 16: 1763337473.425172 SKIPPED (no operation) 17: 1763337473.425598 READ 0x21457 thru 0x27bc8 (0x06772 bytes) 18: 1763337473.432646 SKIPPED (no operation) 19: 1763337473.433519 SKIPPED (no operation) 20: 1763337473.433951 MAPWRITE 0x2dcf2 thru 0x399dc (0x0bceb bytes) 21: 1763337473.488585 SKIPPED (no operation) 22: 1763337473.488999 WRITE 0x1c1ae thru 0x1d6bb (0x0150e bytes) 23: 1763337473.501431 MIRROR_RESYNC 2 mirrors 24: 1763337474.189527 READ 0x0a959 thru 0x0eb13 (0x041bb bytes) 25: 1763337474.197775 SKIPPED (no operation) 26: 1763337474.197990 SKIPPED (no operation) 27: 1763337474.198259 SKIPPED (no operation) 28: 1763337474.198484 READ 0x2bc88 thru 0x2dad7 (0x01e50 bytes) 29: 1763337474.216377 MIRROR_RESYNC 2 mirrors 30: 1763337474.551915 SKIPPED (no operation) 31: 1763337474.552373 READ 0x2a3d6 thru 0x2ecc9 (0x048f4 bytes) 32: 1763337474.552830 MAPWRITE 0x3a103 thru 0x3b54c (0x0144a bytes) 33: 1763337474.696765 WRITE 0x1b888 thru 0x27acd (0x0c246 bytes) 34: 1763337474.724250 SKIPPED (no operation) 35: 1763337474.724864 SKIPPED (no operation) 36: 1763337474.725429 SKIPPED (no operation) 37: 1763337474.726393 WRITE 0x22a52 thru 0x31651 (0x0ec00 bytes) 38: 1763337474.743754 MIRROR_EXTEND to 3 mirrors 39: 1763337475.677891 SKIPPED (no operation) 40: 1763337475.686328 TRUNC_DN 0x3c469 to 0x12354 41: 1763337475.760372 TRUNC_UP 0x12354 to 0x20539 42: 1763337475.794650 SKIPPED (no operation) 43: 1763337475.799993 MIRROR_EXTEND to 4 mirrors 44: 1763337476.592751 WRITE 0x337bb thru 0x3c769 (0x08faf bytes) HOLE 45: 1763337476.677244 SKIPPED (no operation) 46: 1763337476.677676 SKIPPED (no operation) 47: 1763337476.678116 SKIPPED (no operation) 48: 1763337476.679094 WRITE 0x03845 thru 0x06403 (0x02bbf bytes) 49: 1763337476.691814 SKIPPED (no operation) 50: 1763337476.692265 TRUNC_DN 0x3c76a to 0x2cfbe 51: 1763337476.726961 MIRROR_SPLIT mirror 1 to 3 mirrors 52: 1763337477.093487 MAPWRITE 0x22724 thru 0x2867f (0x05f5c bytes) 53: 1763337477.162768 SKIPPED (no operation) 54: 1763337477.163251 TRUNC_DN 0x2cfbe to 0x0bdc3 55: 1763337477.185377 TRUNC_DN 0x0bdc3 to 0x00ab1 56: 1763337477.202701 SKIPPED (no operation) 57: 1763337477.208924 MIRROR_EXTEND to 4 mirrors Using seed 7472 Correct content saved for comparison (maybe hexdump "/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr" vs "/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr.fsxgood") sanity-flr test_70a: @@@@@@ FAIL: fsx FLR file /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr failed Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7314:error() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:3373:test_70a() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7690:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7753:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7568:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:3375:main() Dumping lctl log to /tmp/testlogs//sanity-flr.test_70a.*.1763337479.log rsync: chown "/tmp/testlogs/.sanity-flr.test_70a.debug_log.oleg418-server.1763337479.log.GQgeeX" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-flr.test_70a.dmesg.oleg418-server.1763337479.log.xez6aH" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1668) [generator=3.1.3]