== sanity-flr test 70a: flr mode fsx test ================ 15:08:12 (1746126492) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70a.sanity-flr Seed set to 6493 skipping zero length punch hole skipping zero length zero range fallocating to largest ever: 0x14cfe 000003 1746126494.080287 fallocate 0x0081f2 thru 0x014cfd (0xcb0c bytes) 000005 1746126494.093968 write 0x01c87a thru 0x01e5e3 (0x1d6a bytes) truncating to largest ever: 0x33cd9 000006 1746126494.094682 trunc from 0x01e5e4 to 0x033cd8 (0x156f5 bytes) fallocating to largest ever: 0x3df9d 000007 1746126494.109926 fallocate 0x03790f thru 0x03df9c (0x668e bytes) 000008 1746126494.113774 punch from 0x003b59 to 0x00d498 (0x9940 bytes) 000010 1746126494.119524 mapread 0x02574a thru 0x030658 (0xaf0f bytes) 000011 1746126494.162925 trunc from 0x03df9d to 0x01ca27 (0x21575 bytes) 000012 1746126494.183803 fallocate 0x0071d5 thru 0x007426 (0x252 bytes) 000013 1746126494.191194 read 0x013359 thru 0x01ca27 (0x96cf bytes) fallocating to largest ever: 0x40000 000014 1746126494.197418 fallocate 0x03b394 thru 0x03ffff (0x4c6c bytes) 000016 1746126494.201726 trunc from 0x040000 to 0x03c025 (0x3fda bytes) 000017 1746126494.214665 mapread 0x014b6f thru 0x018c48 (0x40da bytes) 000018 1746126494.219967 mapread 0x016bdd thru 0x022de5 (0xc209 bytes) 000019 1746126494.233889 write 0x039a65 thru 0x03ffff (0x659b bytes) 000020 1746126494.238848 punch from 0x000ced to 0x00bcd3 (0xafe7 bytes) 000021 1746126494.241811 punch from 0x01d469 to 0x02870f (0xb2a7 bytes) 000022 1746126494.247908 write 0x02930d thru 0x02cba0 (0x3894 bytes) 000023 1746126494.253660 write 0x00defb thru 0x01964a (0xb750 bytes) 000024 1746126494.259090 trunc from 0x03fec1 to 0x03ffff (0x13f bytes) 000025 1746126494.278914 read 0x0179a5 thru 0x020b16 (0x9172 bytes) 000026 1746126494.284571 trunc from 0x037f11 to 0x03ffff (0x80ef bytes) 000027 1746126494.291104 trunc from 0x040000 to 0x037194 (0x8e6b bytes) 000028 1746126494.304269 punch from 0x032b89 to 0x037194 (0x460c bytes) 000029 1746126494.309956 trunc from 0x037195 to 0x020603 (0x16b91 bytes) 000030 1746126494.324430 read 0x00e597 thru 0x012880 (0x42ea bytes) 000031 1746126494.324877 trunc from 0x013ec5 to 0x01b408 (0x7544 bytes) 000033 1746126494.329921 mapwrite 0x01d164 thru 0x026d2e (0x9bcb bytes) 000034 1746126494.390785 trunc from 0x026d2f to 0x03c27c (0x1554e bytes) 000035 1746126494.412350 fallocate 0x01b63f thru 0x02286e (0x7230 bytes) 000036 1746126494.416491 trunc from 0x030aec to 0x03704d (0x6562 bytes) 000038 1746126494.428669 write 0x022735 thru 0x02dea1 (0xb76d bytes) 000040 1746126494.435137 trunc from 0x03c27d to 0x0353f4 (0x6e88 bytes) 000042 1746126494.448346 mapread 0x01f250 thru 0x023f85 (0x4d36 bytes) 000043 1746126494.454383 write 0x006ad5 thru 0x014f4f (0xe47b bytes) 000044 1746126494.455431 write 0x00a666 thru 0x0138ca (0x9265 bytes) 000045 1746126494.457699 write 0x00b81f thru 0x013257 (0x7a39 bytes) 000047 1746126494.458740 punch from 0x01312d to 0x0186a1 (0x5575 bytes) 000048 1746126494.474604 read 0x031580 thru 0x0353f4 (0x3e75 bytes) 000049 1746126494.480352 mapread 0x01abaa thru 0x01d673 (0x2aca bytes) 000050 1746126494.481469 mapread 0x02e969 thru 0x0353f4 (0x6a8c bytes) 000051 1746126494.481983 trunc from 0x00297c to 0x00934e (0x69d3 bytes) 000052 1746126494.488945 MIRROR_EXTEND to 2 mirrors lfs mirror mirror: fail to pread 0-4194304 of mirror 2: Invalid argument (22) lfs verify: failed to read data from mirror 2: Invalid argument. mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 22 lfs mirror mirror: fail to pread 0-4194304 of mirror 1: Invalid argument (22) lfs verify: failed to read data from mirror 1: Invalid argument. lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x353f5: skipped LOG DUMP (44 total operations): 1: 1746126494.072237 SKIPPED (no operation) 2: 1746126494.079968 SKIPPED (no operation) 3: 1746126494.080287 FALLOC 0x081f2 thru 0x14cfe (0x0cb0c bytes)PAST_EOF 4: 1746126494.093968 WRITE 0x1c87a thru 0x1e5e3 (0x01d6a bytes) HOLE 5: 1746126494.094682 TRUNC_UP 0x1e5e4 to 0x33cd9 6: 1746126494.109926 FALLOC 0x3790f thru 0x3df9d (0x0668e bytes)EXTENDING 7: 1746126494.113774 PUNCH 0x03b59 thru 0x0d498 (0x09940 bytes) 8: 1746126494.119524 MAPREAD 0x2574a thru 0x30658 (0x0af0f bytes) 9: 1746126494.162925 TRUNC_DN 0x3df9d to 0x1ca28 10: 1746126494.183803 FALLOC 0x071d5 thru 0x07427 (0x00252 bytes)INTERIOR 11: 1746126494.191194 READ 0x13359 thru 0x1ca27 (0x096cf bytes) 12: 1746126494.197418 FALLOC 0x3b394 thru 0x40000 (0x04c6c bytes)EXTENDING 13: 1746126494.201726 TRUNC_DN 0x40000 to 0x3c026 14: 1746126494.214665 MAPREAD 0x14b6f thru 0x18c48 (0x040da bytes) 15: 1746126494.219967 MAPREAD 0x16bdd thru 0x22de5 (0x0c209 bytes) 16: 1746126494.233889 WRITE 0x39a65 thru 0x3ffff (0x0659b bytes) EXTEND 17: 1746126494.238848 PUNCH 0x00ced thru 0x0bcd3 (0x0afe7 bytes) 18: 1746126494.241811 PUNCH 0x1d469 thru 0x2870f (0x0b2a7 bytes) 19: 1746126494.247908 WRITE 0x2930d thru 0x2cba0 (0x03894 bytes) 20: 1746126494.253660 WRITE 0x0defb thru 0x1964a (0x0b750 bytes) 21: 1746126494.259090 ZERO 0x3fec1 thru 0x3ffff (0x0013f bytes) 22: 1746126494.278914 READ 0x179a5 thru 0x20b16 (0x09172 bytes) 23: 1746126494.284571 ZERO 0x37f11 thru 0x3ffff (0x080ef bytes) 24: 1746126494.291104 TRUNC_DN 0x40000 to 0x37195 25: 1746126494.304269 PUNCH 0x32b89 thru 0x37194 (0x0460c bytes) 26: 1746126494.309956 TRUNC_DN 0x37195 to 0x20604 27: 1746126494.324430 READ 0x0e597 thru 0x12880 (0x042ea bytes) 28: 1746126494.324877 ZERO 0x13ec5 thru 0x1b408 (0x07544 bytes) 29: 1746126494.329921 MAPWRITE 0x1d164 thru 0x26d2e (0x09bcb bytes) 30: 1746126494.390785 TRUNC_UP 0x26d2f to 0x3c27d 31: 1746126494.412350 FALLOC 0x1b63f thru 0x2286f (0x07230 bytes)INTERIOR 32: 1746126494.416491 ZERO 0x30aec thru 0x3704d (0x06562 bytes) 33: 1746126494.428669 WRITE 0x22735 thru 0x2dea1 (0x0b76d bytes) 34: 1746126494.435137 TRUNC_DN 0x3c27d to 0x353f5 35: 1746126494.448346 MAPREAD 0x1f250 thru 0x23f85 (0x04d36 bytes) 36: 1746126494.454383 WRITE 0x06ad5 thru 0x14f4f (0x0e47b bytes) 37: 1746126494.455431 WRITE 0x0a666 thru 0x138ca (0x09265 bytes) 38: 1746126494.457699 WRITE 0x0b81f thru 0x13257 (0x07a39 bytes) 39: 1746126494.458740 PUNCH 0x1312d thru 0x186a1 (0x05575 bytes) 40: 1746126494.474604 READ 0x31580 thru 0x353f4 (0x03e75 bytes) 41: 1746126494.480352 MAPREAD 0x1abaa thru 0x1d673 (0x02aca bytes) 42: 1746126494.481469 MAPREAD 0x2e969 thru 0x353f4 (0x06a8c bytes) 43: 1746126494.481983 ZERO 0x0297c thru 0x0934e (0x069d3 bytes) 44: 1746126494.488945 MIRROR_EXTEND to 2 mirrors Using seed 6493 Correct content saved for comparison (maybe hexdump "/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr" vs "/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr.fsxgood") sanity-flr test_70a: @@@@@@ FAIL: fsx FLR file /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr failed Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7249:error() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:3351:test_70a() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7622:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7685:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7503:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:3353:main() Dumping lctl log to /tmp/testlogs//sanity-flr.test_70a.*.1746126495.log rsync: chown "/tmp/testlogs/.sanity-flr.test_70a.debug_log.oleg360-server.1746126495.log.XYYhPM" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-flr.test_70a.dmesg.oleg360-server.1746126495.log.ycPDRa" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2]