== sanity-flr test 70a: flr mode fsx test ================ 18:59:20 (1763337560) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70a.sanity-flr Seed set to 7562 000001 1763337562.542140 write 0x02e360 thru 0x03bcad (0xd94e bytes) 000003 1763337562.569247 punch from 0x036197 to 0x03bcad (0x5b17 bytes) 000004 1763337562.581640 write 0x03a81f thru 0x03fba4 (0x5386 bytes) fallocating to largest ever: 0x315b7 000005 1763337562.588701 fallocate 0x02fe56 thru 0x0315b6 (0x1761 bytes) 000006 1763337562.594572 trunc from 0x0221fd to 0x0281da (0x5fde bytes) 000007 1763337562.603072 read 0x0072f9 thru 0x013e25 (0xcb2d bytes) 000008 1763337562.611893 mapwrite 0x018c43 thru 0x027705 (0xeac3 bytes) 000009 1763337562.647393 read 0x02caeb thru 0x0315f2 (0x4b08 bytes) 000010 1763337562.659191 mapread 0x007939 thru 0x010d21 (0x93e9 bytes) 000011 1763337562.662620 mapread 0x00f530 thru 0x01d880 (0xe351 bytes) 000012 1763337562.663747 trunc from 0x02ca32 to 0x036a4f (0xa01e bytes) 000013 1763337562.670441 trunc from 0x03fba5 to 0x014990 (0x2b214 bytes) 000014 1763337562.687560 trunc from 0x00752f to 0x014990 (0xd462 bytes) 000015 1763337562.692606 read 0x013c22 thru 0x014990 (0xd6f bytes) 000017 1763337562.706906 punch from 0x003c19 to 0x00ea73 (0xae5b bytes) 000019 1763337562.716900 punch from 0x0005b4 to 0x004b1b (0x4568 bytes) 000020 1763337562.723754 punch from 0x005b10 to 0x00aeba (0x53ab bytes) 000021 1763337562.731211 write 0x03aeed thru 0x03ffff (0x5113 bytes) 000022 1763337562.733121 write 0x0182b1 thru 0x01f821 (0x7571 bytes) 000023 1763337562.742839 write 0x01ad72 thru 0x02992e (0xebbd bytes) 000024 1763337562.744177 mapwrite 0x0206c6 thru 0x02ee45 (0xe780 bytes) fallocating to largest ever: 0x31691 000025 1763337562.769726 fallocate 0x02a4bd thru 0x031690 (0x71d4 bytes) 000026 1763337562.775950 mapwrite 0x0234c5 thru 0x025e8f (0x29cb bytes) 000027 1763337562.798250 trunc from 0x040000 to 0x01a017 (0x25fe8 bytes) 000029 1763337562.815038 write 0x034ca7 thru 0x03ffff (0xb359 bytes) 000030 1763337562.816440 mapread 0x01aa6b thru 0x0249ec (0x9f82 bytes) 000031 1763337562.914154 mapwrite 0x027600 thru 0x02e01f (0x6a20 bytes) 000032 1763337562.970728 trunc from 0x040000 to 0x0028e4 (0x3d71b bytes) 000033 1763337562.995202 mapread 0x0010a9 thru 0x0028e4 (0x183c bytes) 000034 1763337563.007866 trunc from 0x0028e5 to 0x01f376 (0x1ca92 bytes) 000035 1763337563.032919 trunc from 0x014d57 to 0x01d31e (0x85c8 bytes) 000036 1763337563.037982 mapwrite 0x004de1 thru 0x00e174 (0x9394 bytes) 000037 1763337563.104155 fallocate 0x007ac6 thru 0x009bf1 (0x212c bytes) 000038 1763337563.108674 mapwrite 0x021541 thru 0x028974 (0x7434 bytes) 000039 1763337563.194221 trunc from 0x028975 to 0x01072a (0x1824a bytes) 000040 1763337563.222427 write 0x03aa44 thru 0x03ffff (0x55bc bytes) 000041 1763337563.223998 mapwrite 0x03d794 thru 0x03ffff (0x286c bytes) 000042 1763337563.256786 mapwrite 0x00979e thru 0x010c6f (0x74d2 bytes) fallocating to largest ever: 0x3d5b8 000043 1763337563.309718 fallocate 0x032838 thru 0x03d5b7 (0xad80 bytes) 000044 1763337563.327997 trunc from 0x040000 to 0x0140a3 (0x2bf5c bytes) 000046 1763337563.343124 trunc from 0x0140a4 to 0x033db9 (0x1fd16 bytes) 000047 1763337563.361698 read 0x02f227 thru 0x0315c1 (0x239b bytes) 000048 1763337563.380692 fallocate 0x014231 thru 0x01bc3e (0x7a0e bytes) 000049 1763337563.386825 write 0x010d3f thru 0x0141cc (0x348e bytes) 000050 1763337563.387795 mapread 0x033c7c thru 0x033db9 (0x13e bytes) 000051 1763337563.388579 trunc from 0x02de60 to 0x033db9 (0x5f5a bytes) 000052 1763337563.394540 trunc from 0x033dba to 0x00983b (0x2a57e bytes) 000053 1763337563.419790 fallocate 0x012135 thru 0x01c508 (0xa3d4 bytes) 000054 1763337563.424735 mapwrite 0x01179c thru 0x01b696 (0x9efb bytes) 000055 1763337563.489644 trunc from 0x015e2f to 0x01c508 (0x66da bytes) 000056 1763337563.494327 trunc from 0x01493d to 0x01c508 (0x7bcc bytes) 000057 1763337563.500064 mapread 0x002abc thru 0x011fab (0xf4f0 bytes) fallocating to largest ever: 0x40000 000058 1763337563.531008 fallocate 0x039625 thru 0x03ffff (0x69db bytes) 000059 1763337563.537955 write 0x032fb1 thru 0x03ce3b (0x9e8b bytes) 000060 1763337563.539984 trunc from 0x03ce3c to 0x000c03 (0x3c238 bytes) 000061 1763337563.554097 mapread 0x00042d thru 0x000c03 (0x7d7 bytes) 000063 1763337563.555830 trunc from 0x0003e2 to 0x000c03 (0x822 bytes) 000064 1763337563.562649 trunc from 0x00007f to 0x000c03 (0xb85 bytes) 000065 1763337563.572103 MIRROR_EXTEND to 2 mirrors Size error: expected 0xc04 stat 0xc04 seek 0x1000 LOG DUMP (59 total operations): 1: 1763337562.542140 WRITE 0x2e360 thru 0x3bcad (0x0d94e bytes) HOLE 2: 1763337562.569247 PUNCH 0x36197 thru 0x3bcad (0x05b17 bytes) 3: 1763337562.581640 WRITE 0x3a81f thru 0x3fba4 (0x05386 bytes) EXTEND 4: 1763337562.588701 FALLOC 0x2fe56 thru 0x315b7 (0x01761 bytes)INTERIOR 5: 1763337562.594572 ZERO 0x221fd thru 0x281da (0x05fde bytes) 6: 1763337562.603072 READ 0x072f9 thru 0x13e25 (0x0cb2d bytes) 7: 1763337562.611893 MAPWRITE 0x18c43 thru 0x27705 (0x0eac3 bytes) 8: 1763337562.647393 READ 0x2caeb thru 0x315f2 (0x04b08 bytes) 9: 1763337562.659191 MAPREAD 0x07939 thru 0x10d21 (0x093e9 bytes) 10: 1763337562.662620 MAPREAD 0x0f530 thru 0x1d880 (0x0e351 bytes) 11: 1763337562.663747 ZERO 0x2ca32 thru 0x36a4f (0x0a01e bytes) 12: 1763337562.670441 TRUNC_DN 0x3fba5 to 0x14991 13: 1763337562.687560 ZERO 0x0752f thru 0x14990 (0x0d462 bytes) 14: 1763337562.692606 READ 0x13c22 thru 0x14990 (0x00d6f bytes) 15: 1763337562.706906 PUNCH 0x03c19 thru 0x0ea73 (0x0ae5b bytes) 16: 1763337562.716900 PUNCH 0x005b4 thru 0x04b1b (0x04568 bytes) 17: 1763337562.723754 PUNCH 0x05b10 thru 0x0aeba (0x053ab bytes) 18: 1763337562.731211 WRITE 0x3aeed thru 0x3ffff (0x05113 bytes) HOLE 19: 1763337562.733121 WRITE 0x182b1 thru 0x1f821 (0x07571 bytes) 20: 1763337562.742839 WRITE 0x1ad72 thru 0x2992e (0x0ebbd bytes) 21: 1763337562.744177 MAPWRITE 0x206c6 thru 0x2ee45 (0x0e780 bytes) 22: 1763337562.769726 FALLOC 0x2a4bd thru 0x31691 (0x071d4 bytes)INTERIOR 23: 1763337562.775950 MAPWRITE 0x234c5 thru 0x25e8f (0x029cb bytes) 24: 1763337562.798250 TRUNC_DN 0x40000 to 0x1a018 25: 1763337562.815038 WRITE 0x34ca7 thru 0x3ffff (0x0b359 bytes) HOLE 26: 1763337562.816440 MAPREAD 0x1aa6b thru 0x249ec (0x09f82 bytes) 27: 1763337562.914154 MAPWRITE 0x27600 thru 0x2e01f (0x06a20 bytes) 28: 1763337562.970728 TRUNC_DN 0x40000 to 0x028e5 29: 1763337562.995202 MAPREAD 0x010a9 thru 0x028e4 (0x0183c bytes) 30: 1763337563.007866 TRUNC_UP 0x028e5 to 0x1f377 31: 1763337563.032919 ZERO 0x14d57 thru 0x1d31e (0x085c8 bytes) 32: 1763337563.037982 MAPWRITE 0x04de1 thru 0x0e174 (0x09394 bytes) 33: 1763337563.104155 FALLOC 0x07ac6 thru 0x09bf2 (0x0212c bytes)INTERIOR 34: 1763337563.108674 MAPWRITE 0x21541 thru 0x28974 (0x07434 bytes) 35: 1763337563.194221 TRUNC_DN 0x28975 to 0x1072b 36: 1763337563.222427 WRITE 0x3aa44 thru 0x3ffff (0x055bc bytes) HOLE 37: 1763337563.223998 MAPWRITE 0x3d794 thru 0x3ffff (0x0286c bytes) 38: 1763337563.256786 MAPWRITE 0x0979e thru 0x10c6f (0x074d2 bytes) 39: 1763337563.309718 FALLOC 0x32838 thru 0x3d5b8 (0x0ad80 bytes)INTERIOR 40: 1763337563.327997 TRUNC_DN 0x40000 to 0x140a4 41: 1763337563.343124 TRUNC_UP 0x140a4 to 0x33dba 42: 1763337563.361698 READ 0x2f227 thru 0x315c1 (0x0239b bytes) 43: 1763337563.380692 FALLOC 0x14231 thru 0x1bc3f (0x07a0e bytes)INTERIOR 44: 1763337563.386825 WRITE 0x10d3f thru 0x141cc (0x0348e bytes) 45: 1763337563.387795 MAPREAD 0x33c7c thru 0x33db9 (0x0013e bytes) 46: 1763337563.388579 ZERO 0x2de60 thru 0x33db9 (0x05f5a bytes) 47: 1763337563.394540 TRUNC_DN 0x33dba to 0x0983c 48: 1763337563.419790 FALLOC 0x12135 thru 0x1c509 (0x0a3d4 bytes)EXTENDING 49: 1763337563.424735 MAPWRITE 0x1179c thru 0x1b696 (0x09efb bytes) 50: 1763337563.489644 ZERO 0x15e2f thru 0x1c508 (0x066da bytes) 51: 1763337563.494327 ZERO 0x1493d thru 0x1c508 (0x07bcc bytes) 52: 1763337563.500064 MAPREAD 0x02abc thru 0x11fab (0x0f4f0 bytes) 53: 1763337563.531008 FALLOC 0x39625 thru 0x40000 (0x069db bytes)PAST_EOF 54: 1763337563.537955 WRITE 0x32fb1 thru 0x3ce3b (0x09e8b bytes) HOLE 55: 1763337563.539984 TRUNC_DN 0x3ce3c to 0x00c04 56: 1763337563.554097 MAPREAD 0x0042d thru 0x00c03 (0x007d7 bytes) 57: 1763337563.555830 ZERO 0x003e2 thru 0x00c03 (0x00822 bytes) 58: 1763337563.562649 ZERO 0x0007f thru 0x00c03 (0x00b85 bytes) 59: 1763337563.572103 MIRROR_EXTEND to 2 mirrors Using seed 7562 Correct content saved for comparison (maybe hexdump "/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr" vs "/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr.fsxgood") sanity-flr test_70a: @@@@@@ FAIL: fsx FLR file /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr failed Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7314:error() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:3373:test_70a() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7690:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7753:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7568:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:3375:main() Dumping lctl log to /tmp/testlogs//sanity-flr.test_70a.*.1763337565.log rsync: chown "/tmp/testlogs/.sanity-flr.test_70a.debug_log.oleg407-server.1763337565.log.htmv8H" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-flr.test_70a.dmesg.oleg407-server.1763337565.log.vvE2iC" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1668) [generator=3.1.3]