== sanity-flr test 70a: flr mode fsx test ================ 22:04:45 (1746065085) striped dir -i0 -c2 -H all_char /mnt/lustre/d70a.sanity-flr Seed set to 5086 000001 1746065086.589752 write 0x0256ac thru 0x029bfe (0x4553 bytes) fallocating to largest ever: 0x16b74 000002 1746065086.596453 fallocate 0x00c463 thru 0x016b73 (0xa711 bytes) zero_range to largest ever: 0x29bff 000003 1746065086.600561 trunc from 0x023e38 to 0x029bfe (0x5dc7 bytes) 000004 1746065086.609298 fallocate 0x009293 thru 0x0177c9 (0xe537 bytes) 000006 1746065086.612750 mapread 0x0209f2 thru 0x025f41 (0x5550 bytes) 000007 1746065086.626715 mapwrite 0x03fcca thru 0x03ffff (0x336 bytes) 000008 1746065086.670143 trunc from 0x01ba21 to 0x028f8e (0xd56e bytes) truncating to largest ever: 0x3293a 000009 1746065086.674623 trunc from 0x040000 to 0x032939 (0xd6c6 bytes) 000010 1746065086.692103 mapread 0x004240 thru 0x00da1a (0x97db bytes) 000011 1746065086.704249 mapwrite 0x0106a4 thru 0x015af1 (0x544e bytes) 000013 1746065086.720892 mapwrite 0x03cd42 thru 0x03ffff (0x32be bytes) 000014 1746065086.750628 mapwrite 0x00b5fe thru 0x0172d4 (0xbcd7 bytes) 000015 1746065086.772585 mapwrite 0x03fe81 thru 0x03ffff (0x17f bytes) 000016 1746065086.781630 read 0x0244f0 thru 0x02a89b (0x63ac bytes) 000017 1746065086.786279 read 0x01706a thru 0x0171c7 (0x15e bytes) truncating to largest ever: 0x39d06 000019 1746065086.787239 trunc from 0x040000 to 0x039d05 (0x62fa bytes) 000020 1746065086.801022 trunc from 0x00145f to 0x00b410 (0x9fb2 bytes) 000021 1746065086.805956 trunc from 0x039d06 to 0x028a73 (0x11292 bytes) 000022 1746065086.816944 trunc from 0x00f042 to 0x01c5b7 (0xd576 bytes) 000023 1746065086.821350 mapwrite 0x0095b8 thru 0x009fa4 (0x9ed bytes) 000024 1746065086.837633 punch from 0x0231f4 to 0x0236c6 (0x4d3 bytes) 000026 1746065086.841636 write 0x0163c1 thru 0x025348 (0xef88 bytes) 000027 1746065086.842697 read 0x023ff6 thru 0x025c4f (0x1c5a bytes) fallocating to largest ever: 0x40000 000028 1746065086.843085 fallocate 0x03f3c7 thru 0x03ffff (0xc39 bytes) 000029 1746065086.847459 read 0x00e99c thru 0x014308 (0x596d bytes) 000030 1746065086.847850 write 0x0322e0 thru 0x03fd4b (0xda6c bytes) 000031 1746065086.856659 trunc from 0x012f16 to 0x01e089 (0xb174 bytes) 000032 1746065086.872549 mapread 0x01d0ed thru 0x02041d (0x3331 bytes) 000034 1746065086.891188 mapread 0x02a446 thru 0x0370c6 (0xcc81 bytes) 000035 1746065086.891781 trunc from 0x016361 to 0x01789a (0x153a bytes) 000036 1746065086.896264 trunc from 0x00f15e to 0x01e10f (0xefb2 bytes) 000037 1746065086.901367 mapwrite 0x015057 thru 0x01d803 (0x87ad bytes) 000038 1746065086.972269 trunc from 0x014e46 to 0x024330 (0xf4eb bytes) 000039 1746065086.980651 mapwrite 0x02161b thru 0x0287fa (0x71e0 bytes) 000040 1746065087.025780 write 0x01d75c thru 0x01efc4 (0x1869 bytes) 000041 1746065087.031745 fallocate 0x008e4e thru 0x00bec6 (0x3079 bytes) 000042 1746065087.035161 read 0x01ddda thru 0x0214f3 (0x371a bytes) 000043 1746065087.040601 mapread 0x034b53 thru 0x03ffff (0xb4ad bytes) 000044 1746065087.041174 trunc from 0x027b2b to 0x0335c6 (0xba9c bytes) 000045 1746065087.045579 punch from 0x03bd2e to 0x03ffff (0x42d2 bytes) 000046 1746065087.049491 fallocate 0x002594 thru 0x00f408 (0xce75 bytes) 000049 1746065087.054838 fallocate 0x0172ca thru 0x01a8e5 (0x361c bytes) 000050 1746065087.058319 punch from 0x03c9e8 to 0x03cd3c (0x355 bytes) 000051 1746065087.060966 read 0x00e3de thru 0x016481 (0x80a4 bytes) 000052 1746065087.061901 write 0x025321 thru 0x031867 (0xc547 bytes) 000053 1746065087.067805 trunc from 0x040000 to 0x0106be (0x2f941 bytes) 000054 1746065087.082236 read 0x007453 thru 0x007900 (0x4ae bytes) 000055 1746065087.082642 trunc from 0x001658 to 0x00c044 (0xa9ed bytes) 000059 1746065087.088475 trunc from 0x00536d to 0x00cf46 (0x7bda bytes) 000061 1746065087.093842 mapread 0x00e27e thru 0x0106be (0x2441 bytes) 000062 1746065087.094778 trunc from 0x0106bf to 0x00dae4 (0x2bda bytes) 000063 1746065087.108226 write 0x02a39c thru 0x02ae3b (0xaa0 bytes) 000064 1746065087.108915 fallocate 0x0305ab thru 0x03267e (0x20d4 bytes) 000065 1746065087.115507 mapwrite 0x004501 thru 0x006168 (0x1c68 bytes) 000067 1746065087.142913 read 0x01781c thru 0x024112 (0xc8f7 bytes) 000068 1746065087.143449 trunc from 0x02ae3c to 0x01b1e3 (0xfc58 bytes) 000069 1746065087.159371 trunc from 0x01b1e4 to 0x02acd3 (0xfaf0 bytes) 000070 1746065087.171101 trunc from 0x0210c4 to 0x02acd3 (0x9c10 bytes) 000071 1746065087.175107 read 0x017674 thru 0x0268a3 (0xf230 bytes) 000072 1746065087.181003 mapread 0x00550b thru 0x005cba (0x7b0 bytes) 000073 1746065087.182188 punch from 0x025cbf to 0x02acd3 (0x5015 bytes) 000074 1746065087.185735 read 0x010b4c thru 0x013123 (0x25d8 bytes) 000075 1746065087.186495 trunc from 0x01b77d to 0x02154b (0x5dcf bytes) 000076 1746065087.189538 punch from 0x01b2e6 to 0x025338 (0xa053 bytes) 000077 1746065087.192967 mapread 0x01176d thru 0x01bd8c (0xa620 bytes) 000078 1746065087.199805 trunc from 0x02acd4 to 0x027420 (0x38b3 bytes) 000079 1746065087.210689 mapread 0x00c25e thru 0x01b321 (0xf0c4 bytes) 000080 1746065087.212232 mapwrite 0x031303 thru 0x037b9f (0x689d bytes) 000081 1746065087.248280 trunc from 0x0142f6 to 0x015bb8 (0x18c3 bytes) 000082 1746065087.250850 punch from 0x02554c to 0x030761 (0xb216 bytes) 000084 1746065087.254433 write 0x00938e thru 0x00dd3e (0x49b1 bytes) 000085 1746065087.255202 trunc from 0x037ba0 to 0x00043d (0x37762 bytes) 000086 1746065087.269490 mapread 0x000350 thru 0x00043d (0xee bytes) 000087 1746065087.271717 read 0x0001fa thru 0x00043d (0x244 bytes) 000088 1746065087.274778 trunc from 0x0003cd to 0x00043d (0x71 bytes) 000089 1746065087.279189 punch from 0x0002c1 to 0x00043d (0x17d bytes) 000090 1746065087.288083 MIRROR_EXTEND to 2 mirrors lfs mirror mirror: fail to pread 0-4194304 of mirror 2: Invalid argument (22) lfs verify: failed to read data from mirror 2: Invalid argument. mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 22 lfs mirror mirror: fail to pread 0-4194304 of mirror 1: Invalid argument (22) lfs verify: failed to read data from mirror 1: Invalid argument. lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x43e: skipped LOG DUMP (77 total operations): 1: 1746065086.589752 WRITE 0x256ac thru 0x29bfe (0x04553 bytes) HOLE 2: 1746065086.596453 FALLOC 0x0c463 thru 0x16b74 (0x0a711 bytes)INTERIOR 3: 1746065086.600561 ZERO 0x23e38 thru 0x29bfe (0x05dc7 bytes) 4: 1746065086.609298 FALLOC 0x09293 thru 0x177ca (0x0e537 bytes)INTERIOR 5: 1746065086.612750 MAPREAD 0x209f2 thru 0x25f41 (0x05550 bytes) 6: 1746065086.626715 MAPWRITE 0x3fcca thru 0x3ffff (0x00336 bytes) 7: 1746065086.670143 ZERO 0x1ba21 thru 0x28f8e (0x0d56e bytes) 8: 1746065086.674623 TRUNC_DN 0x40000 to 0x3293a 9: 1746065086.692103 MAPREAD 0x04240 thru 0x0da1a (0x097db bytes) 10: 1746065086.704249 MAPWRITE 0x106a4 thru 0x15af1 (0x0544e bytes) 11: 1746065086.720892 MAPWRITE 0x3cd42 thru 0x3ffff (0x032be bytes) 12: 1746065086.750628 MAPWRITE 0x0b5fe thru 0x172d4 (0x0bcd7 bytes) 13: 1746065086.772585 MAPWRITE 0x3fe81 thru 0x3ffff (0x0017f bytes) 14: 1746065086.781630 READ 0x244f0 thru 0x2a89b (0x063ac bytes) 15: 1746065086.786279 READ 0x1706a thru 0x171c7 (0x0015e bytes) 16: 1746065086.787239 TRUNC_DN 0x40000 to 0x39d06 17: 1746065086.801022 ZERO 0x0145f thru 0x0b410 (0x09fb2 bytes) 18: 1746065086.805956 TRUNC_DN 0x39d06 to 0x28a74 19: 1746065086.816944 ZERO 0x0f042 thru 0x1c5b7 (0x0d576 bytes) 20: 1746065086.821350 MAPWRITE 0x095b8 thru 0x09fa4 (0x009ed bytes) 21: 1746065086.837633 PUNCH 0x231f4 thru 0x236c6 (0x004d3 bytes) 22: 1746065086.841636 WRITE 0x163c1 thru 0x25348 (0x0ef88 bytes) 23: 1746065086.842697 READ 0x23ff6 thru 0x25c4f (0x01c5a bytes) 24: 1746065086.843085 FALLOC 0x3f3c7 thru 0x40000 (0x00c39 bytes)EXTENDING 25: 1746065086.847459 READ 0x0e99c thru 0x14308 (0x0596d bytes) 26: 1746065086.847850 WRITE 0x322e0 thru 0x3fd4b (0x0da6c bytes) 27: 1746065086.856659 ZERO 0x12f16 thru 0x1e089 (0x0b174 bytes) 28: 1746065086.872549 MAPREAD 0x1d0ed thru 0x2041d (0x03331 bytes) 29: 1746065086.891188 MAPREAD 0x2a446 thru 0x370c6 (0x0cc81 bytes) 30: 1746065086.891781 ZERO 0x16361 thru 0x1789a (0x0153a bytes) 31: 1746065086.896264 ZERO 0x0f15e thru 0x1e10f (0x0efb2 bytes) 32: 1746065086.901367 MAPWRITE 0x15057 thru 0x1d803 (0x087ad bytes) 33: 1746065086.972269 ZERO 0x14e46 thru 0x24330 (0x0f4eb bytes) 34: 1746065086.980651 MAPWRITE 0x2161b thru 0x287fa (0x071e0 bytes) 35: 1746065087.025780 WRITE 0x1d75c thru 0x1efc4 (0x01869 bytes) 36: 1746065087.031745 FALLOC 0x08e4e thru 0x0bec7 (0x03079 bytes)INTERIOR 37: 1746065087.035161 READ 0x1ddda thru 0x214f3 (0x0371a bytes) 38: 1746065087.040601 MAPREAD 0x34b53 thru 0x3ffff (0x0b4ad bytes) 39: 1746065087.041174 ZERO 0x27b2b thru 0x335c6 (0x0ba9c bytes) 40: 1746065087.045579 PUNCH 0x3bd2e thru 0x3ffff (0x042d2 bytes) 41: 1746065087.049491 FALLOC 0x02594 thru 0x0f409 (0x0ce75 bytes)INTERIOR 42: 1746065087.054838 FALLOC 0x172ca thru 0x1a8e6 (0x0361c bytes)INTERIOR 43: 1746065087.058319 PUNCH 0x3c9e8 thru 0x3cd3c (0x00355 bytes) 44: 1746065087.060966 READ 0x0e3de thru 0x16481 (0x080a4 bytes) 45: 1746065087.061901 WRITE 0x25321 thru 0x31867 (0x0c547 bytes) 46: 1746065087.067805 TRUNC_DN 0x40000 to 0x106bf 47: 1746065087.082236 READ 0x07453 thru 0x07900 (0x004ae bytes) 48: 1746065087.082642 ZERO 0x01658 thru 0x0c044 (0x0a9ed bytes) 49: 1746065087.088475 ZERO 0x0536d thru 0x0cf46 (0x07bda bytes) 50: 1746065087.093842 MAPREAD 0x0e27e thru 0x106be (0x02441 bytes) 51: 1746065087.094778 TRUNC_DN 0x106bf to 0x0dae5 52: 1746065087.108226 WRITE 0x2a39c thru 0x2ae3b (0x00aa0 bytes) HOLE 53: 1746065087.108915 FALLOC 0x305ab thru 0x3267f (0x020d4 bytes)PAST_EOF 54: 1746065087.115507 MAPWRITE 0x04501 thru 0x06168 (0x01c68 bytes) 55: 1746065087.142913 READ 0x1781c thru 0x24112 (0x0c8f7 bytes) 56: 1746065087.143449 TRUNC_DN 0x2ae3c to 0x1b1e4 57: 1746065087.159371 TRUNC_UP 0x1b1e4 to 0x2acd4 58: 1746065087.171101 ZERO 0x210c4 thru 0x2acd3 (0x09c10 bytes) 59: 1746065087.175107 READ 0x17674 thru 0x268a3 (0x0f230 bytes) 60: 1746065087.181003 MAPREAD 0x0550b thru 0x05cba (0x007b0 bytes) 61: 1746065087.182188 PUNCH 0x25cbf thru 0x2acd3 (0x05015 bytes) 62: 1746065087.185735 READ 0x10b4c thru 0x13123 (0x025d8 bytes) 63: 1746065087.186495 ZERO 0x1b77d thru 0x2154b (0x05dcf bytes) 64: 1746065087.189538 PUNCH 0x1b2e6 thru 0x25338 (0x0a053 bytes) 65: 1746065087.192967 MAPREAD 0x1176d thru 0x1bd8c (0x0a620 bytes) 66: 1746065087.199805 TRUNC_DN 0x2acd4 to 0x27421 67: 1746065087.210689 MAPREAD 0x0c25e thru 0x1b321 (0x0f0c4 bytes) 68: 1746065087.212232 MAPWRITE 0x31303 thru 0x37b9f (0x0689d bytes) 69: 1746065087.248280 ZERO 0x142f6 thru 0x15bb8 (0x018c3 bytes) 70: 1746065087.250850 PUNCH 0x2554c thru 0x30761 (0x0b216 bytes) 71: 1746065087.254433 WRITE 0x0938e thru 0x0dd3e (0x049b1 bytes) 72: 1746065087.255202 TRUNC_DN 0x37ba0 to 0x0043e 73: 1746065087.269490 MAPREAD 0x00350 thru 0x0043d (0x000ee bytes) 74: 1746065087.271717 READ 0x001fa thru 0x0043d (0x00244 bytes) 75: 1746065087.274778 ZERO 0x003cd thru 0x0043d (0x00071 bytes) 76: 1746065087.279189 PUNCH 0x002c1 thru 0x0043d (0x0017d bytes) 77: 1746065087.288083 MIRROR_EXTEND to 2 mirrors Using seed 5086 Correct content saved for comparison (maybe hexdump "/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr" vs "/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr.fsxgood") sanity-flr test_70a: @@@@@@ FAIL: fsx FLR file /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr failed Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7249:error() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:3351:test_70a() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7622:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7685:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7503:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:3353:main() Dumping lctl log to /tmp/testlogs//sanity-flr.test_70a.*.1746065088.log rsync: chown "/tmp/testlogs/.sanity-flr.test_70a.debug_log.oleg227-server.1746065088.log.vZ0NGN" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-flr.test_70a.dmesg.oleg227-server.1746065088.log.il23YU" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2]