== sanity-flr test 70a: flr mode fsx test ================ 17:46:23 (1763333183) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70a.sanity-flr Seed set to 3185 skipping zero length zero range skipping zero length punch hole fallocating to largest ever: 0x19f60 000003 1763333186.133536 fallocate 0x00f771 thru 0x019f5f (0xa7ef bytes) 000004 1763333186.148423 mapread 0x019ef3 thru 0x019f5f (0x6d bytes) 000005 1763333186.158959 fallocate 0x006fa6 thru 0x00e44c (0x74a7 bytes) fallocating to largest ever: 0x32d1b 000006 1763333186.165754 fallocate 0x0271b4 thru 0x032d1a (0xbb67 bytes) 000007 1763333186.173954 punch from 0x0185ac to 0x019f5f (0x19b4 bytes) 000008 1763333186.183917 trunc from 0x0102c4 to 0x010c6c (0x9a9 bytes) 000009 1763333186.195125 mapread 0x009884 thru 0x00d301 (0x3a7e bytes) 000011 1763333186.222685 mapread 0x0088d9 thru 0x00ab65 (0x228d bytes) 000013 1763333186.225535 trunc from 0x019f60 to 0x02232f (0x83d0 bytes) 000015 1763333186.253731 punch from 0x01a82c to 0x02232f (0x7b04 bytes) 000016 1763333186.260092 punch from 0x00969f to 0x00ac48 (0x15aa bytes) 000018 1763333186.268138 read 0x021544 thru 0x02232f (0xdec bytes) fallocating to largest ever: 0x377f9 000019 1763333186.278567 fallocate 0x029b0f thru 0x0377f8 (0xdcea bytes) 000020 1763333186.287366 read 0x02429a thru 0x02e614 (0xa37b bytes) 000021 1763333186.304692 trunc from 0x00dcf5 to 0x013896 (0x5ba2 bytes) 000022 1763333186.313507 mapread 0x02cf86 thru 0x0377f8 (0xa873 bytes) 000024 1763333186.315003 fallocate 0x02f40c thru 0x032ecb (0x3ac0 bytes) 000025 1763333186.321563 write 0x02d8cd thru 0x03942a (0xbb5e bytes) 000028 1763333186.328693 punch from 0x025113 to 0x031953 (0xc841 bytes) 000030 1763333186.358958 mapwrite 0x0001a6 thru 0x00d2f9 (0xd154 bytes) 000031 1763333186.405538 read 0x030470 thru 0x03741f (0x6fb0 bytes) 000032 1763333186.428957 trunc from 0x03942b to 0x00c8f9 (0x2cb31 bytes) 000033 1763333186.474886 mapwrite 0x031365 thru 0x03f80d (0xe4a9 bytes) 000034 1763333186.639313 trunc from 0x00b593 to 0x0150a3 (0x9b11 bytes) 000035 1763333186.653663 write 0x01346c thru 0x016661 (0x31f6 bytes) 000036 1763333186.685665 read 0x02810f thru 0x029b60 (0x1a52 bytes) 000037 1763333186.710656 trunc from 0x036ea0 to 0x03b80d (0x496e bytes) 000038 1763333186.725106 punch from 0x03ca1d to 0x03f80d (0x2df1 bytes) 000039 1763333186.740443 trunc from 0x03f80e to 0x028fd6 (0x16837 bytes) 000040 1763333186.765993 write 0x019a60 thru 0x01f79f (0x5d40 bytes) 000041 1763333186.769230 write 0x017a9d thru 0x01ef11 (0x7475 bytes) 000042 1763333186.770554 trunc from 0x028fd7 to 0x00c25b (0x1cd7b bytes) 000043 1763333186.795987 mapwrite 0x035324 thru 0x03b6bc (0x6399 bytes) 000044 1763333186.912218 trunc from 0x01f477 to 0x02b9ed (0xc577 bytes) 000045 1763333186.920602 trunc from 0x03b6bd to 0x00c056 (0x2f666 bytes) 000046 1763333186.947280 fallocate 0x01aa90 thru 0x02057c (0x5aed bytes) 000047 1763333186.966815 trunc from 0x02057d to 0x019412 (0x716a bytes) 000048 1763333186.992185 punch from 0x013bf4 to 0x018c29 (0x5036 bytes) 000049 1763333187.009587 punch from 0x005ab0 to 0x008bf0 (0x3141 bytes) 000050 1763333187.021902 mapwrite 0x010c70 thru 0x01da39 (0xcdca bytes) 000051 1763333187.174324 mapread 0x01b361 thru 0x01da39 (0x26d9 bytes) 000052 1763333187.177365 trunc from 0x003a3d to 0x007edc (0x44a0 bytes) 000053 1763333187.189714 MIRROR_EXTEND to 2 mirrors 000054 1763333187.932517 read 0x00bee9 thru 0x018f00 (0xd018 bytes) 000055 1763333187.947008 mapread 0x00c7aa thru 0x01841b (0xbc72 bytes) 000056 1763333187.961718 trunc from 0x00c20d to 0x0138a4 (0x7698 bytes) 000057 1763333188.007441 trunc from 0x017d1e to 0x01da39 (0x5d1c bytes) 000058 1763333188.014550 read 0x0163a8 thru 0x01c0d9 (0x5d32 bytes) 000060 1763333188.024328 mapread 0x007110 thru 0x00a389 (0x327a bytes) 000061 1763333188.028689 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x1da3a): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1e000): Mirror 2: 0x5dd83eea (stale) CRC-32 checksum value for chunk [0, 0x1e000): Mirror 1: 0x2a5f0424 Mirror 3: 0x2a5f0424 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1e000: skipped Size error: expected 0x1da3a stat 0x1da3a seek 0x1e000 LOG DUMP (52 total operations): 1: 1763333186.123782 SKIPPED (no operation) 2: 1763333186.133048 SKIPPED (no operation) 3: 1763333186.133536 FALLOC 0x0f771 thru 0x19f60 (0x0a7ef bytes)EXTENDING 4: 1763333186.148423 MAPREAD 0x19ef3 thru 0x19f5f (0x0006d bytes) 5: 1763333186.158959 FALLOC 0x06fa6 thru 0x0e44d (0x074a7 bytes)INTERIOR 6: 1763333186.165754 FALLOC 0x271b4 thru 0x32d1b (0x0bb67 bytes)PAST_EOF 7: 1763333186.173954 PUNCH 0x185ac thru 0x19f5f (0x019b4 bytes) 8: 1763333186.183917 ZERO 0x102c4 thru 0x10c6c (0x009a9 bytes) 9: 1763333186.195125 MAPREAD 0x09884 thru 0x0d301 (0x03a7e bytes) 10: 1763333186.222685 MAPREAD 0x088d9 thru 0x0ab65 (0x0228d bytes) 11: 1763333186.225535 TRUNC_UP 0x19f60 to 0x22330 12: 1763333186.253731 PUNCH 0x1a82c thru 0x2232f (0x07b04 bytes) 13: 1763333186.260092 PUNCH 0x0969f thru 0x0ac48 (0x015aa bytes) 14: 1763333186.268138 READ 0x21544 thru 0x2232f (0x00dec bytes) 15: 1763333186.278567 FALLOC 0x29b0f thru 0x377f9 (0x0dcea bytes)EXTENDING 16: 1763333186.287366 READ 0x2429a thru 0x2e614 (0x0a37b bytes) 17: 1763333186.304692 ZERO 0x0dcf5 thru 0x13896 (0x05ba2 bytes) 18: 1763333186.313507 MAPREAD 0x2cf86 thru 0x377f8 (0x0a873 bytes) 19: 1763333186.315003 FALLOC 0x2f40c thru 0x32ecc (0x03ac0 bytes)INTERIOR 20: 1763333186.321563 WRITE 0x2d8cd thru 0x3942a (0x0bb5e bytes) EXTEND 21: 1763333186.328693 PUNCH 0x25113 thru 0x31953 (0x0c841 bytes) 22: 1763333186.358958 MAPWRITE 0x001a6 thru 0x0d2f9 (0x0d154 bytes) 23: 1763333186.405538 READ 0x30470 thru 0x3741f (0x06fb0 bytes) 24: 1763333186.428957 TRUNC_DN 0x3942b to 0x0c8fa 25: 1763333186.474886 MAPWRITE 0x31365 thru 0x3f80d (0x0e4a9 bytes) 26: 1763333186.639313 ZERO 0x0b593 thru 0x150a3 (0x09b11 bytes) 27: 1763333186.653663 WRITE 0x1346c thru 0x16661 (0x031f6 bytes) 28: 1763333186.685665 READ 0x2810f thru 0x29b60 (0x01a52 bytes) 29: 1763333186.710656 ZERO 0x36ea0 thru 0x3b80d (0x0496e bytes) 30: 1763333186.725106 PUNCH 0x3ca1d thru 0x3f80d (0x02df1 bytes) 31: 1763333186.740443 TRUNC_DN 0x3f80e to 0x28fd7 32: 1763333186.765993 WRITE 0x19a60 thru 0x1f79f (0x05d40 bytes) 33: 1763333186.769230 WRITE 0x17a9d thru 0x1ef11 (0x07475 bytes) 34: 1763333186.770554 TRUNC_DN 0x28fd7 to 0x0c25c 35: 1763333186.795987 MAPWRITE 0x35324 thru 0x3b6bc (0x06399 bytes) 36: 1763333186.912218 ZERO 0x1f477 thru 0x2b9ed (0x0c577 bytes) 37: 1763333186.920602 TRUNC_DN 0x3b6bd to 0x0c057 38: 1763333186.947280 FALLOC 0x1aa90 thru 0x2057d (0x05aed bytes)EXTENDING 39: 1763333186.966815 TRUNC_DN 0x2057d to 0x19413 40: 1763333186.992185 PUNCH 0x13bf4 thru 0x18c29 (0x05036 bytes) 41: 1763333187.009587 PUNCH 0x05ab0 thru 0x08bf0 (0x03141 bytes) 42: 1763333187.021902 MAPWRITE 0x10c70 thru 0x1da39 (0x0cdca bytes) 43: 1763333187.174324 MAPREAD 0x1b361 thru 0x1da39 (0x026d9 bytes) 44: 1763333187.177365 ZERO 0x03a3d thru 0x07edc (0x044a0 bytes) 45: 1763333187.189714 MIRROR_EXTEND to 2 mirrors 46: 1763333187.932517 READ 0x0bee9 thru 0x18f00 (0x0d018 bytes) 47: 1763333187.947008 MAPREAD 0x0c7aa thru 0x1841b (0x0bc72 bytes) 48: 1763333187.961718 ZERO 0x0c20d thru 0x138a4 (0x07698 bytes) 49: 1763333188.007441 ZERO 0x17d1e thru 0x1da39 (0x05d1c bytes) 50: 1763333188.014550 READ 0x163a8 thru 0x1c0d9 (0x05d32 bytes) 51: 1763333188.024328 MAPREAD 0x07110 thru 0x0a389 (0x0327a bytes) 52: 1763333188.028689 MIRROR_EXTEND to 3 mirrors Using seed 3185 Correct content saved for comparison (maybe hexdump "/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr" vs "/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr.fsxgood") sanity-flr test_70a: @@@@@@ FAIL: fsx FLR file /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr failed Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7314:error() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:3373:test_70a() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7690:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7753:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7568:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:3375:main() Dumping lctl log to /tmp/testlogs//sanity-flr.test_70a.*.1763333190.log rsync: chown "/tmp/testlogs/.sanity-flr.test_70a.debug_log.oleg428-server.1763333190.log.7VT08G" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-flr.test_70a.dmesg.oleg428-server.1763333190.log.zcO0VR" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1668) [generator=3.1.3]