== sanity-flr test 70a: flr mode fsx test ================ 16:33:55 (1746131635) Seed set to 1636 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000001 1746131636.641153 write 0x036fc5 thru 0x03ffff (0x903b bytes) 000002 1746131636.663361 MIRROR_EXTEND to 2 mirrors 000006 1746131637.165252 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000403:0x2f5:0x0]: Invalid argument (22) 000007 1746131637.259287 mapwrite 0x01f4a8 thru 0x026854 (0x73ad bytes) 000008 1746131637.366833 mapwrite 0x022bf3 thru 0x02bfe6 (0x93f4 bytes) truncating to largest ever: 0x2c4b2 000010 1746131637.457094 trunc from 0x040000 to 0x02c4b1 (0x13b4e bytes) truncating to largest ever: 0x3ca1d 000011 1746131637.488113 trunc from 0x02c4b2 to 0x03ca1c (0x1056b bytes) 000016 1746131637.501382 mapread 0x026c21 thru 0x034e7a (0xe25a bytes) 000020 1746131637.509222 trunc from 0x03ca1d to 0x01a683 (0x22399 bytes) 000021 1746131637.520507 write 0x007eb7 thru 0x01650e (0xe658 bytes) 000022 1746131637.531843 mapread 0x008bae thru 0x0153f0 (0xc843 bytes) 000023 1746131637.533225 mapwrite 0x02adbc thru 0x0302f1 (0x5536 bytes) 000024 1746131637.640958 mapwrite 0x017ce8 thru 0x01eaf2 (0x6e0b bytes) 000025 1746131637.694796 write 0x03d0cf thru 0x03ffff (0x2f31 bytes) 000026 1746131637.695987 trunc from 0x040000 to 0x00a91f (0x356e0 bytes) 000027 1746131637.711033 mapwrite 0x01b471 thru 0x0263f9 (0xaf89 bytes) 000028 1746131637.797508 mapwrite 0x00f955 thru 0x011f27 (0x25d3 bytes) 000029 1746131637.851281 write 0x006c69 thru 0x00c75e (0x5af6 bytes) 000030 1746131637.852083 read 0x000cae thru 0x005425 (0x4778 bytes) 000032 1746131637.857829 write 0x03fc61 thru 0x03ffff (0x39f bytes) 000033 1746131637.858714 mapread 0x03ff52 thru 0x03ffff (0xae bytes) 000034 1746131637.859252 mapwrite 0x02ea50 thru 0x0381f5 (0x97a6 bytes) 000038 1746131637.966409 trunc from 0x040000 to 0x01ffa2 (0x2005d bytes) 000039 1746131637.985431 mapwrite 0x03e3f4 thru 0x03ffff (0x1c0c bytes) 000044 1746131638.058281 read 0x02d2d4 thru 0x030088 (0x2db5 bytes) 000045 1746131638.066106 mapread 0x0330b4 thru 0x03ffff (0xcf4c bytes) 000047 1746131638.068209 trunc from 0x040000 to 0x0359ac (0xa653 bytes) 000051 1746131638.084191 write 0x033afe thru 0x0345b6 (0xab9 bytes) 000054 1746131638.086163 read 0x034670 thru 0x0359ac (0x133d bytes) 000056 1746131638.087474 trunc from 0x0359ad to 0x00615f (0x2f84d bytes) 000059 1746131638.107654 MIRROR_EXTEND to 2 mirrors lfs mirror mirror: fail to pread 0-4194304 of mirror 2: Invalid argument (22) lfs verify: failed to read data from mirror 2: Invalid argument. mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 22 lfs mirror mirror: fail to pread 0-4194304 of mirror 1: Invalid argument (22) lfs verify: failed to read data from mirror 1: Invalid argument. lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x6160: skipped LOG DUMP (46 total operations): 1: 1746131636.641153 WRITE 0x36fc5 thru 0x3ffff (0x0903b bytes) HOLE 2: 1746131636.663361 MIRROR_EXTEND to 2 mirrors 3: 1746131637.156056 SKIPPED (no operation) 4: 1746131637.159885 SKIPPED (no operation) 5: 1746131637.165252 MIRROR_SPLIT mirror 1 to 1 mirrors 6: 1746131637.259287 MAPWRITE 0x1f4a8 thru 0x26854 (0x073ad bytes) 7: 1746131637.366833 MAPWRITE 0x22bf3 thru 0x2bfe6 (0x093f4 bytes) 8: 1746131637.457094 TRUNC_DN 0x40000 to 0x2c4b2 9: 1746131637.488113 TRUNC_UP 0x2c4b2 to 0x3ca1d 10: 1746131637.499447 SKIPPED (no operation) 11: 1746131637.500882 SKIPPED (no operation) 12: 1746131637.501382 MAPREAD 0x26c21 thru 0x34e7a (0x0e25a bytes) 13: 1746131637.508920 SKIPPED (no operation) 14: 1746131637.509222 TRUNC_DN 0x3ca1d to 0x1a684 15: 1746131637.520507 WRITE 0x07eb7 thru 0x1650e (0x0e658 bytes) 16: 1746131637.531843 MAPREAD 0x08bae thru 0x153f0 (0x0c843 bytes) 17: 1746131637.533225 MAPWRITE 0x2adbc thru 0x302f1 (0x05536 bytes) 18: 1746131637.640958 MAPWRITE 0x17ce8 thru 0x1eaf2 (0x06e0b bytes) 19: 1746131637.694796 WRITE 0x3d0cf thru 0x3ffff (0x02f31 bytes) HOLE 20: 1746131637.695987 TRUNC_DN 0x40000 to 0x0a920 21: 1746131637.711033 MAPWRITE 0x1b471 thru 0x263f9 (0x0af89 bytes) 22: 1746131637.797508 MAPWRITE 0x0f955 thru 0x11f27 (0x025d3 bytes) 23: 1746131637.851281 WRITE 0x06c69 thru 0x0c75e (0x05af6 bytes) 24: 1746131637.852083 READ 0x00cae thru 0x05425 (0x04778 bytes) 25: 1746131637.857829 WRITE 0x3fc61 thru 0x3ffff (0x0039f bytes) HOLE 26: 1746131637.858714 MAPREAD 0x3ff52 thru 0x3ffff (0x000ae bytes) 27: 1746131637.859252 MAPWRITE 0x2ea50 thru 0x381f5 (0x097a6 bytes) 28: 1746131637.965487 SKIPPED (no operation) 29: 1746131637.965811 SKIPPED (no operation) 30: 1746131637.966105 SKIPPED (no operation) 31: 1746131637.966409 TRUNC_DN 0x40000 to 0x1ffa3 32: 1746131637.985431 MAPWRITE 0x3e3f4 thru 0x3ffff (0x01c0c bytes) 33: 1746131638.056537 SKIPPED (no operation) 34: 1746131638.058281 READ 0x2d2d4 thru 0x30088 (0x02db5 bytes) 35: 1746131638.066106 MAPREAD 0x330b4 thru 0x3ffff (0x0cf4c bytes) 36: 1746131638.067899 SKIPPED (no operation) 37: 1746131638.068209 TRUNC_DN 0x40000 to 0x359ad 38: 1746131638.083882 SKIPPED (no operation) 39: 1746131638.084191 WRITE 0x33afe thru 0x345b6 (0x00ab9 bytes) 40: 1746131638.085014 SKIPPED (no operation) 41: 1746131638.085833 SKIPPED (no operation) 42: 1746131638.086163 READ 0x34670 thru 0x359ac (0x0133d bytes) 43: 1746131638.087140 SKIPPED (no operation) 44: 1746131638.087474 TRUNC_DN 0x359ad to 0x06160 45: 1746131638.103719 SKIPPED (no operation) 46: 1746131638.107654 MIRROR_EXTEND to 2 mirrors Using seed 1636 Correct content saved for comparison (maybe hexdump "/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr" vs "/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr.fsxgood") sanity-flr test_70a: @@@@@@ FAIL: fsx FLR file /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr failed Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7249:error() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:3351:test_70a() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7622:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7685:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7503:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:3353:main() Dumping lctl log to /tmp/testlogs//sanity-flr.test_70a.*.1746131639.log rsync: chown "/tmp/testlogs/.sanity-flr.test_70a.debug_log.oleg251-server.1746131639.log.6tfR1B" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-flr.test_70a.dmesg.oleg251-server.1746131639.log.TvynTt" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2]