== sanity-flr test 70a: flr mode fsx test ================ 15:14:29 (1746126869) Seed set to 6870 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported truncating to largest ever: 0x1b885 000002 1746126870.459282 trunc from 00000000 to 0x01b884 (0x1b885 bytes) 000006 1746126870.472609 mapread 0x01895c thru 0x01b884 (0x2f29 bytes) 000012 1746126870.493461 write 0x02be4e thru 0x02da1f (0x1bd2 bytes) 000014 1746126870.494778 mapread 0x02d215 thru 0x02da1f (0x80b bytes) 000015 1746126870.496547 mapwrite 0x01cc9d thru 0x02537c (0x86e0 bytes) 000016 1746126870.641658 mapwrite 0x007b10 thru 0x010de5 (0x92d6 bytes) 000017 1746126870.709959 trunc from 0x02da20 to 0x014958 (0x190c7 bytes) 000018 1746126870.726441 mapwrite 0x03ab22 thru 0x03ffff (0x54de bytes) 000019 1746126870.805813 mapread 0x03b17f thru 0x03b644 (0x4c6 bytes) 000025 1746126870.809609 mapwrite 0x01968d thru 0x01ea30 (0x53a4 bytes) 000027 1746126870.863445 mapwrite 0x02cc57 thru 0x038832 (0xbbdc bytes) 000029 1746126870.943056 write 0x01f1c4 thru 0x02771c (0x8559 bytes) 000034 1746126870.946208 mapread 0x028fbb thru 0x031ff8 (0x903e bytes) truncating to largest ever: 0x2e5c7 000036 1746126870.949296 trunc from 0x040000 to 0x02e5c6 (0x11a39 bytes) 000038 1746126870.965210 write 0x011c15 thru 0x01cf2c (0xb318 bytes) 000039 1746126870.974869 write 0x0160e0 thru 0x01e734 (0x8655 bytes) 000040 1746126870.976774 write 0x027996 thru 0x033c02 (0xc26d bytes) 000042 1746126870.979100 mapread 0x025ea2 thru 0x02bf45 (0x60a4 bytes) 000045 1746126870.982436 write 0x0214f0 thru 0x02753d (0x604e bytes) truncating to largest ever: 0x3e2ea 000047 1746126870.983607 trunc from 0x033c03 to 0x03e2e9 (0xa6e7 bytes) 000049 1746126871.014270 read 0x00565f thru 0x00a66f (0x5011 bytes) 000051 1746126871.015269 write 0x015680 thru 0x01becb (0x684c bytes) 000052 1746126871.016037 mapread 0x024663 thru 0x028a97 (0x4435 bytes) 000054 1746126871.018340 mapread 0x0175d3 thru 0x025155 (0xdb83 bytes) 000055 1746126871.020487 mapwrite 0x01eea9 thru 0x02671f (0x7877 bytes) 000059 1746126871.088196 trunc from 0x03e2ea to 0x02ad01 (0x135e8 bytes) 000060 1746126871.101820 write 0x012130 thru 0x01412a (0x1ffb bytes) 000061 1746126871.103537 read 0x023d6d thru 0x02ad01 (0x6f95 bytes) 000063 1746126871.104494 read 0x004b8a thru 0x011231 (0xc6a8 bytes) 000065 1746126871.105595 mapwrite 0x03a199 thru 0x03b4b0 (0x1318 bytes) 000066 1746126871.196806 mapwrite 0x027a5e thru 0x02a3fb (0x299e bytes) 000068 1746126871.252538 write 0x03b68c thru 0x03ffff (0x4974 bytes) 000069 1746126871.254713 mapwrite 0x006899 thru 0x013c2e (0xd396 bytes) 000071 1746126871.316272 mapwrite 0x0295ce thru 0x036c68 (0xd69b bytes) 000072 1746126871.369968 read 0x01e516 thru 0x02924f (0xad3a bytes) 000073 1746126871.371187 mapread 0x03566c thru 0x03ffff (0xa994 bytes) 000075 1746126871.371801 mapread 0x00104b thru 0x00ba29 (0xa9df bytes) 000079 1746126871.375849 read 0x007e78 thru 0x00d83d (0x59c6 bytes) 000083 1746126871.379018 mapwrite 0x0267c5 thru 0x02f528 (0x8d64 bytes) 000085 1746126871.446438 trunc from 0x040000 to 0x013bb8 (0x2c447 bytes) 000086 1746126871.461346 mapread 0x013189 thru 0x013bb8 (0xa30 bytes) 000087 1746126871.463448 mapread 0x01154a thru 0x013bb8 (0x266f bytes) 000088 1746126871.464908 write 0x02ea66 thru 0x035d50 (0x72eb bytes) 000089 1746126871.467027 mapwrite 0x023f20 thru 0x0288ea (0x49cb bytes) 000091 1746126871.534040 read 0x034ef5 thru 0x035d50 (0xe5c bytes) 000092 1746126871.534545 write 0x01d7eb thru 0x023bb1 (0x63c7 bytes) 000095 1746126871.538140 read 0x013fa4 thru 0x01b06b (0x70c8 bytes) truncating to largest ever: 0x3f63d 000099 1746126871.540551 trunc from 0x035d51 to 0x03f63c (0x98ec bytes) 000100 1746126871.552431 trunc from 0x03f63d to 0x00e148 (0x314f4 bytes) 000101 1746126871.565902 trunc from 0x00e149 to 0x00d424 (0xd24 bytes) 000102 1746126871.586461 trunc from 0x00d425 to 0x021270 (0x13e4c bytes) 000104 1746126871.608920 mapread 0x0125d5 thru 0x01d04e (0xaa7a bytes) 000105 1746126871.624554 trunc from 0x021271 to 0x00df73 (0x132fd bytes) 000106 1746126871.639723 mapread 0x0005b6 thru 0x006f48 (0x6993 bytes) 000108 1746126871.640688 trunc from 0x00df74 to 0x02ab53 (0x1cbe0 bytes) 000109 1746126871.654289 trunc from 0x02ab54 to 0x016bdf (0x13f74 bytes) 000110 1746126871.688852 MIRROR_EXTEND to 2 mirrors lfs mirror mirror: fail to pread 0-4194304 of mirror 2: Invalid argument (22) lfs verify: failed to read data from mirror 2: Invalid argument. mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 22 lfs mirror mirror: fail to pread 0-4194304 of mirror 1: Invalid argument (22) lfs verify: failed to read data from mirror 1: Invalid argument. lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x16be0: skipped LOG DUMP (92 total operations): 1: 1746126870.449142 SKIPPED (no operation) 2: 1746126870.459282 TRUNC_UP 0x00000 to 0x1b885 3: 1746126870.471418 SKIPPED (no operation) 4: 1746126870.472609 MAPREAD 0x1895c thru 0x1b884 (0x02f29 bytes) 5: 1746126870.492215 SKIPPED (no operation) 6: 1746126870.492416 SKIPPED (no operation) 7: 1746126870.492960 SKIPPED (no operation) 8: 1746126870.493280 SKIPPED (no operation) 9: 1746126870.493461 WRITE 0x2be4e thru 0x2da1f (0x01bd2 bytes) HOLE 10: 1746126870.494459 SKIPPED (no operation) 11: 1746126870.494778 MAPREAD 0x2d215 thru 0x2da1f (0x0080b bytes) 12: 1746126870.496547 MAPWRITE 0x1cc9d thru 0x2537c (0x086e0 bytes) 13: 1746126870.641658 MAPWRITE 0x07b10 thru 0x10de5 (0x092d6 bytes) 14: 1746126870.709959 TRUNC_DN 0x2da20 to 0x14959 15: 1746126870.726441 MAPWRITE 0x3ab22 thru 0x3ffff (0x054de bytes) 16: 1746126870.805813 MAPREAD 0x3b17f thru 0x3b644 (0x004c6 bytes) 17: 1746126870.806445 SKIPPED (no operation) 18: 1746126870.807294 SKIPPED (no operation) 19: 1746126870.807650 SKIPPED (no operation) 20: 1746126870.809609 MAPWRITE 0x1968d thru 0x1ea30 (0x053a4 bytes) 21: 1746126870.863445 MAPWRITE 0x2cc57 thru 0x38832 (0x0bbdc bytes) 22: 1746126870.942688 SKIPPED (no operation) 23: 1746126870.943056 WRITE 0x1f1c4 thru 0x2771c (0x08559 bytes) 24: 1746126870.945083 SKIPPED (no operation) 25: 1746126870.945523 SKIPPED (no operation) 26: 1746126870.946208 MAPREAD 0x28fbb thru 0x31ff8 (0x0903e bytes) 27: 1746126870.948864 SKIPPED (no operation) 28: 1746126870.949296 TRUNC_DN 0x40000 to 0x2e5c7 29: 1746126870.965210 WRITE 0x11c15 thru 0x1cf2c (0x0b318 bytes) 30: 1746126870.974869 WRITE 0x160e0 thru 0x1e734 (0x08655 bytes) 31: 1746126870.976774 WRITE 0x27996 thru 0x33c02 (0x0c26d bytes) EXTEND 32: 1746126870.978561 SKIPPED (no operation) 33: 1746126870.979100 MAPREAD 0x25ea2 thru 0x2bf45 (0x060a4 bytes) 34: 1746126870.981537 SKIPPED (no operation) 35: 1746126870.982078 SKIPPED (no operation) 36: 1746126870.982436 WRITE 0x214f0 thru 0x2753d (0x0604e bytes) 37: 1746126870.983261 SKIPPED (no operation) 38: 1746126870.983607 TRUNC_UP 0x33c03 to 0x3e2ea 39: 1746126871.013475 SKIPPED (no operation) 40: 1746126871.014270 READ 0x0565f thru 0x0a66f (0x05011 bytes) 41: 1746126871.015269 WRITE 0x15680 thru 0x1becb (0x0684c bytes) 42: 1746126871.016037 MAPREAD 0x24663 thru 0x28a97 (0x04435 bytes) 43: 1746126871.017938 SKIPPED (no operation) 44: 1746126871.018340 MAPREAD 0x175d3 thru 0x25155 (0x0db83 bytes) 45: 1746126871.020487 MAPWRITE 0x1eea9 thru 0x2671f (0x07877 bytes) 46: 1746126871.084647 SKIPPED (no operation) 47: 1746126871.086761 SKIPPED (no operation) 48: 1746126871.088196 TRUNC_DN 0x3e2ea to 0x2ad02 49: 1746126871.101820 WRITE 0x12130 thru 0x1412a (0x01ffb bytes) 50: 1746126871.103537 READ 0x23d6d thru 0x2ad01 (0x06f95 bytes) 51: 1746126871.104494 READ 0x04b8a thru 0x11231 (0x0c6a8 bytes) 52: 1746126871.105220 SKIPPED (no operation) 53: 1746126871.105595 MAPWRITE 0x3a199 thru 0x3b4b0 (0x01318 bytes) 54: 1746126871.196806 MAPWRITE 0x27a5e thru 0x2a3fb (0x0299e bytes) 55: 1746126871.252538 WRITE 0x3b68c thru 0x3ffff (0x04974 bytes) HOLE 56: 1746126871.254713 MAPWRITE 0x06899 thru 0x13c2e (0x0d396 bytes) 57: 1746126871.315621 SKIPPED (no operation) 58: 1746126871.316272 MAPWRITE 0x295ce thru 0x36c68 (0x0d69b bytes) 59: 1746126871.369968 READ 0x1e516 thru 0x2924f (0x0ad3a bytes) 60: 1746126871.371187 MAPREAD 0x3566c thru 0x3ffff (0x0a994 bytes) 61: 1746126871.371608 SKIPPED (no operation) 62: 1746126871.371801 MAPREAD 0x0104b thru 0x0ba29 (0x0a9df bytes) 63: 1746126871.374846 SKIPPED (no operation) 64: 1746126871.375522 SKIPPED (no operation) 65: 1746126871.375849 READ 0x07e78 thru 0x0d83d (0x059c6 bytes) 66: 1746126871.377854 SKIPPED (no operation) 67: 1746126871.378191 SKIPPED (no operation) 68: 1746126871.378517 SKIPPED (no operation) 69: 1746126871.379018 MAPWRITE 0x267c5 thru 0x2f528 (0x08d64 bytes) 70: 1746126871.446238 SKIPPED (no operation) 71: 1746126871.446438 TRUNC_DN 0x40000 to 0x13bb9 72: 1746126871.461346 MAPREAD 0x13189 thru 0x13bb8 (0x00a30 bytes) 73: 1746126871.463448 MAPREAD 0x1154a thru 0x13bb8 (0x0266f bytes) 74: 1746126871.464908 WRITE 0x2ea66 thru 0x35d50 (0x072eb bytes) HOLE 75: 1746126871.467027 MAPWRITE 0x23f20 thru 0x288ea (0x049cb bytes) 76: 1746126871.534040 READ 0x34ef5 thru 0x35d50 (0x00e5c bytes) 77: 1746126871.534545 WRITE 0x1d7eb thru 0x23bb1 (0x063c7 bytes) 78: 1746126871.538140 READ 0x13fa4 thru 0x1b06b (0x070c8 bytes) 79: 1746126871.538829 SKIPPED (no operation) 80: 1746126871.539421 SKIPPED (no operation) 81: 1746126871.539934 SKIPPED (no operation) 82: 1746126871.540551 TRUNC_UP 0x35d51 to 0x3f63d 83: 1746126871.552431 TRUNC_DN 0x3f63d to 0x0e149 84: 1746126871.565902 TRUNC_DN 0x0e149 to 0x0d425 85: 1746126871.586461 TRUNC_UP 0x0d425 to 0x21271 86: 1746126871.608332 SKIPPED (no operation) 87: 1746126871.608920 MAPREAD 0x125d5 thru 0x1d04e (0x0aa7a bytes) 88: 1746126871.624554 TRUNC_DN 0x21271 to 0x0df74 89: 1746126871.639723 MAPREAD 0x005b6 thru 0x06f48 (0x06993 bytes) 90: 1746126871.640688 TRUNC_UP 0x0df74 to 0x2ab54 91: 1746126871.654289 TRUNC_DN 0x2ab54 to 0x16be0 92: 1746126871.688852 MIRROR_EXTEND to 2 mirrors Using seed 6870 Correct content saved for comparison (maybe hexdump "/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr" vs "/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr.fsxgood") sanity-flr test_70a: @@@@@@ FAIL: fsx FLR file /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr failed Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7249:error() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:3351:test_70a() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7622:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7685:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7503:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:3353:main() Dumping lctl log to /tmp/testlogs//sanity-flr.test_70a.*.1746126873.log rsync: chown "/tmp/testlogs/.sanity-flr.test_70a.debug_log.oleg132-server.1746126873.log.Hu1QUB" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-flr.test_70a.dmesg.oleg132-server.1746126873.log.hxUiLY" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2]