== sanity-flr test 70a: flr mode fsx test ================ 17:20:16 (1746134416) striped dir -i0 -c2 -H crush /mnt/lustre/d70a.sanity-flr Seed set to 4417 skipping zero length punch hole skipping zero size read skipping zero size read skipping zero size read truncating to largest ever: 0x35595 000007 1746134417.198093 trunc from 00000000 to 0x035594 (0x35595 bytes) 000010 1746134417.208186 mapwrite 0x02c569 thru 0x031ad0 (0x5568 bytes) 000012 1746134417.249295 trunc from 0x035595 to 0x010003 (0x25591 bytes) 000013 1746134417.261113 trunc from 0x006026 to 0x00c017 (0x5ff2 bytes) 000014 1746134417.267004 mapwrite 0x0013b7 thru 0x009751 (0x839b bytes) 000015 1746134417.287310 trunc from 0x010004 to 0x030590 (0x2058d bytes) 000016 1746134417.304339 mapread 0x0071b8 thru 0x00e789 (0x75d2 bytes) 000017 1746134417.305298 write 0x02996c thru 0x030724 (0x6db9 bytes) 000018 1746134417.316719 fallocate 0x021d66 thru 0x02423b (0x24d6 bytes) 000019 1746134417.323280 mapread 0x001782 thru 0x00626d (0x4aec bytes) 000020 1746134417.324280 read 0x019966 thru 0x022017 (0x86b2 bytes) 000021 1746134417.329021 punch from 0x026250 to 0x02de39 (0x7bea bytes) 000022 1746134417.337609 mapwrite 0x00737a thru 0x00a8b0 (0x3537 bytes) 000023 1746134417.359487 trunc from 0x020b52 to 0x02c52e (0xb9dd bytes) 000024 1746134417.364557 write 0x006f77 thru 0x011bc9 (0xac53 bytes) 000025 1746134417.367561 write 0x00264b thru 0x00d9e5 (0xb39b bytes) 000026 1746134417.368407 mapwrite 0x03538c thru 0x03abed (0x5862 bytes) 000028 1746134417.395155 trunc from 0x03664d to 0x03abed (0x45a1 bytes) 000030 1746134417.403129 fallocate 0x010803 thru 0x012270 (0x1a6e bytes) 000032 1746134417.406482 trunc from 0x01a825 to 0x01b7f6 (0xfd2 bytes) 000033 1746134417.408907 trunc from 0x03abee to 0x014b89 (0x26064 bytes) 000034 1746134417.415358 mapwrite 0x024e90 thru 0x033420 (0xe591 bytes) 000035 1746134417.462272 punch from 0x008bbc to 0x00cdcf (0x4214 bytes) 000036 1746134417.467813 read 0x02512e thru 0x0284b6 (0x3389 bytes) 000038 1746134417.470084 write 0x027d67 thru 0x03643f (0xe6d9 bytes) 000039 1746134417.471252 mapwrite 0x03ab36 thru 0x03b0d6 (0x5a1 bytes) 000040 1746134417.504395 mapwrite 0x02f366 thru 0x031327 (0x1fc2 bytes) 000041 1746134417.517040 write 0x0320d9 thru 0x03ef36 (0xce5e bytes) 000043 1746134417.518764 mapwrite 0x003e19 thru 0x00d133 (0x931b bytes) 000044 1746134417.535393 trunc from 0x02fa22 to 0x030b0f (0x10ee bytes) 000045 1746134417.543540 punch from 0x01897d to 0x0250f0 (0xc774 bytes) 000046 1746134417.547316 trunc from 0x0075e2 to 0x012417 (0xae36 bytes) 000048 1746134417.553365 read 0x03d3e4 thru 0x03df80 (0xb9d bytes) 000049 1746134417.553957 trunc from 0x03ef37 to 0x028b70 (0x163c6 bytes) 000050 1746134417.567226 MIRROR_EXTEND to 2 mirrors lfs mirror mirror: fail to pread 0-4194304 of mirror 2: Invalid argument (22) lfs verify: failed to read data from mirror 2: Invalid argument. mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 22 lfs mirror mirror: fail to pread 0-4194304 of mirror 1: Invalid argument (22) lfs verify: failed to read data from mirror 1: Invalid argument. lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x28b71: skipped LOG DUMP (39 total operations): 1: 1746134417.196003 SKIPPED (no operation) 2: 1746134417.197068 SKIPPED (no operation) 3: 1746134417.197377 SKIPPED (no operation) 4: 1746134417.197783 SKIPPED (no operation) 5: 1746134417.198093 TRUNC_UP 0x00000 to 0x35595 6: 1746134417.208186 MAPWRITE 0x2c569 thru 0x31ad0 (0x05568 bytes) 7: 1746134417.249295 TRUNC_DN 0x35595 to 0x10004 8: 1746134417.261113 ZERO 0x06026 thru 0x0c017 (0x05ff2 bytes) 9: 1746134417.267004 MAPWRITE 0x013b7 thru 0x09751 (0x0839b bytes) 10: 1746134417.287310 TRUNC_UP 0x10004 to 0x30591 11: 1746134417.304339 MAPREAD 0x071b8 thru 0x0e789 (0x075d2 bytes) 12: 1746134417.305298 WRITE 0x2996c thru 0x30724 (0x06db9 bytes) EXTEND 13: 1746134417.316719 FALLOC 0x21d66 thru 0x2423c (0x024d6 bytes)INTERIOR 14: 1746134417.323280 MAPREAD 0x01782 thru 0x0626d (0x04aec bytes) 15: 1746134417.324280 READ 0x19966 thru 0x22017 (0x086b2 bytes) 16: 1746134417.329021 PUNCH 0x26250 thru 0x2de39 (0x07bea bytes) 17: 1746134417.337609 MAPWRITE 0x0737a thru 0x0a8b0 (0x03537 bytes) 18: 1746134417.359487 ZERO 0x20b52 thru 0x2c52e (0x0b9dd bytes) 19: 1746134417.364557 WRITE 0x06f77 thru 0x11bc9 (0x0ac53 bytes) 20: 1746134417.367561 WRITE 0x0264b thru 0x0d9e5 (0x0b39b bytes) 21: 1746134417.368407 MAPWRITE 0x3538c thru 0x3abed (0x05862 bytes) 22: 1746134417.395155 ZERO 0x3664d thru 0x3abed (0x045a1 bytes) 23: 1746134417.403129 FALLOC 0x10803 thru 0x12271 (0x01a6e bytes)INTERIOR 24: 1746134417.406482 ZERO 0x1a825 thru 0x1b7f6 (0x00fd2 bytes) 25: 1746134417.408907 TRUNC_DN 0x3abee to 0x14b8a 26: 1746134417.415358 MAPWRITE 0x24e90 thru 0x33420 (0x0e591 bytes) 27: 1746134417.462272 PUNCH 0x08bbc thru 0x0cdcf (0x04214 bytes) 28: 1746134417.467813 READ 0x2512e thru 0x284b6 (0x03389 bytes) 29: 1746134417.470084 WRITE 0x27d67 thru 0x3643f (0x0e6d9 bytes) EXTEND 30: 1746134417.471252 MAPWRITE 0x3ab36 thru 0x3b0d6 (0x005a1 bytes) 31: 1746134417.504395 MAPWRITE 0x2f366 thru 0x31327 (0x01fc2 bytes) 32: 1746134417.517040 WRITE 0x320d9 thru 0x3ef36 (0x0ce5e bytes) EXTEND 33: 1746134417.518764 MAPWRITE 0x03e19 thru 0x0d133 (0x0931b bytes) 34: 1746134417.535393 ZERO 0x2fa22 thru 0x30b0f (0x010ee bytes) 35: 1746134417.543540 PUNCH 0x1897d thru 0x250f0 (0x0c774 bytes) 36: 1746134417.547316 ZERO 0x075e2 thru 0x12417 (0x0ae36 bytes) 37: 1746134417.553365 READ 0x3d3e4 thru 0x3df80 (0x00b9d bytes) 38: 1746134417.553957 TRUNC_DN 0x3ef37 to 0x28b71 39: 1746134417.567226 MIRROR_EXTEND to 2 mirrors Using seed 4417 Correct content saved for comparison (maybe hexdump "/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr" vs "/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr.fsxgood") sanity-flr test_70a: @@@@@@ FAIL: fsx FLR file /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr failed Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7249:error() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:3351:test_70a() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7622:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7685:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7503:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:3353:main() Dumping lctl log to /tmp/testlogs//sanity-flr.test_70a.*.1746134418.log rsync: chown "/tmp/testlogs/.sanity-flr.test_70a.debug_log.oleg416-server.1746134418.log.Ezq5Ut" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-flr.test_70a.dmesg.oleg416-server.1746134418.log.gC17lO" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2]