== sanity-flr test 70a: flr mode fsx test ================ 23:01:59 (1746154919) striped dir -i0 -c2 -H crush /mnt/lustre/d70a.sanity-flr Seed set to 4920 truncating to largest ever: 0x174c6 000001 1746154921.021928 trunc from 00000000 to 0x0174c5 (0x174c6 bytes) 000002 1746154921.037275 mapread 0x00851c thru 0x0172dc (0xedc1 bytes) 000003 1746154921.063437 read 0x0161fd thru 0x0174c5 (0x12c9 bytes) 000004 1746154921.063832 mapread 0x01350a thru 0x0174c5 (0x3fbc bytes) 000005 1746154921.064568 trunc from 0x0058e1 to 0x007641 (0x1d61 bytes) 000006 1746154921.067145 mapwrite 0x023722 thru 0x031814 (0xe0f3 bytes) 000007 1746154921.126178 mapwrite 0x036f63 thru 0x03ffff (0x909d bytes) truncating to largest ever: 0x37090 000008 1746154921.173792 trunc from 0x040000 to 0x03708f (0x8f70 bytes) 000009 1746154921.193768 write 0x008f79 thru 0x01749e (0xe526 bytes) 000010 1746154921.195331 mapwrite 0x00806e thru 0x015443 (0xd3d6 bytes) 000011 1746154921.233348 mapwrite 0x03c76e thru 0x03ffff (0x3892 bytes) 000012 1746154921.293298 MIRROR_EXTEND to 2 mirrors 000013 1746154921.657883 read 0x028090 thru 0x03519a (0xd10b bytes) 000014 1746154921.665688 trunc from 0x040000 to 0x024eb8 (0x1b147 bytes) 000015 1746154921.734872 read 0x022cda thru 0x024eb8 (0x21df bytes) 000016 1746154921.746564 mapread 0x011710 thru 0x0158a5 (0x4196 bytes) 000017 1746154921.747726 mapwrite 0x031a0a thru 0x03abcc (0x91c3 bytes) fallocating to largest ever: 0x39119 000018 1746154921.792128 fallocate 0x029441 thru 0x039118 (0xfcd8 bytes) 000019 1746154921.798374 trunc from 0x01c22c to 0x02176e (0x5543 bytes) 000020 1746154921.807791 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x32a:0x0]: Invalid argument (22) 000021 1746154921.933725 read 0x01825b thru 0x01d184 (0x4f2a bytes) 000022 1746154921.939374 trunc from 0x03abcd to 0x02c87c (0xe350 bytes) 000023 1746154921.953020 trunc from 0x0066d4 to 0x00a551 (0x3e7e bytes) 000024 1746154921.959604 mapread 0x0217a7 thru 0x025da7 (0x4601 bytes) 000025 1746154921.976159 read 0x0265f9 thru 0x02c87c (0x6284 bytes) 000026 1746154921.985549 read 0x00b476 thru 0x01501a (0x9ba5 bytes) 000027 1746154921.987218 trunc from 0x00bf29 to 0x01aabf (0xeb97 bytes) 000028 1746154921.995432 MIRROR_EXTEND to 2 mirrors lfs mirror mirror: fail to pread 0-4194304 of mirror 2: Invalid argument (22) lfs verify: failed to read data from mirror 2: Invalid argument. mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 22 lfs mirror mirror: fail to pread 0-4194304 of mirror 1: Invalid argument (22) lfs verify: failed to read data from mirror 1: Invalid argument. lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c87d: skipped LOG DUMP (28 total operations): 1: 1746154921.021928 TRUNC_UP 0x00000 to 0x174c6 2: 1746154921.037275 MAPREAD 0x0851c thru 0x172dc (0x0edc1 bytes) 3: 1746154921.063437 READ 0x161fd thru 0x174c5 (0x012c9 bytes) 4: 1746154921.063832 MAPREAD 0x1350a thru 0x174c5 (0x03fbc bytes) 5: 1746154921.064568 ZERO 0x058e1 thru 0x07641 (0x01d61 bytes) 6: 1746154921.067145 MAPWRITE 0x23722 thru 0x31814 (0x0e0f3 bytes) 7: 1746154921.126178 MAPWRITE 0x36f63 thru 0x3ffff (0x0909d bytes) 8: 1746154921.173792 TRUNC_DN 0x40000 to 0x37090 9: 1746154921.193768 WRITE 0x08f79 thru 0x1749e (0x0e526 bytes) 10: 1746154921.195331 MAPWRITE 0x0806e thru 0x15443 (0x0d3d6 bytes) 11: 1746154921.233348 MAPWRITE 0x3c76e thru 0x3ffff (0x03892 bytes) 12: 1746154921.293298 MIRROR_EXTEND to 2 mirrors 13: 1746154921.657883 READ 0x28090 thru 0x3519a (0x0d10b bytes) 14: 1746154921.665688 TRUNC_DN 0x40000 to 0x24eb9 15: 1746154921.734872 READ 0x22cda thru 0x24eb8 (0x021df bytes) 16: 1746154921.746564 MAPREAD 0x11710 thru 0x158a5 (0x04196 bytes) 17: 1746154921.747726 MAPWRITE 0x31a0a thru 0x3abcc (0x091c3 bytes) 18: 1746154921.792128 FALLOC 0x29441 thru 0x39119 (0x0fcd8 bytes)INTERIOR 19: 1746154921.798374 ZERO 0x1c22c thru 0x2176e (0x05543 bytes) 20: 1746154921.807791 MIRROR_SPLIT mirror 1 to 1 mirrors 21: 1746154921.933725 READ 0x1825b thru 0x1d184 (0x04f2a bytes) 22: 1746154921.939374 TRUNC_DN 0x3abcd to 0x2c87d 23: 1746154921.953020 ZERO 0x066d4 thru 0x0a551 (0x03e7e bytes) 24: 1746154921.959604 MAPREAD 0x217a7 thru 0x25da7 (0x04601 bytes) 25: 1746154921.976159 READ 0x265f9 thru 0x2c87c (0x06284 bytes) 26: 1746154921.985549 READ 0x0b476 thru 0x1501a (0x09ba5 bytes) 27: 1746154921.987218 ZERO 0x0bf29 thru 0x1aabf (0x0eb97 bytes) 28: 1746154921.995432 MIRROR_EXTEND to 2 mirrors Using seed 4920 Correct content saved for comparison (maybe hexdump "/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr" vs "/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr.fsxgood") sanity-flr test_70a: @@@@@@ FAIL: fsx FLR file /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr failed Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7249:error() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:3351:test_70a() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7622:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7685:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7503:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:3353:main() Dumping lctl log to /tmp/testlogs//sanity-flr.test_70a.*.1746154923.log rsync: chown "/tmp/testlogs/.sanity-flr.test_70a.debug_log.oleg142-server.1746154923.log.F4lXmt" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-flr.test_70a.dmesg.oleg142-server.1746154923.log.CoaG8W" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2]