== sanity-flr test 70a: flr mode fsx test ================ 23:48:01 (1746157681) Seed set to 7682 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported truncating to largest ever: 0x197cc 000004 1746157682.917231 trunc from 00000000 to 0x0197cb (0x197cc bytes) 000005 1746157682.935185 write 0x026181 thru 0x02d9ca (0x784a bytes) 000007 1746157682.938710 read 0x009716 thru 0x00975a (0x45 bytes) 000008 1746157682.950712 write 0x035fcb thru 0x03ffff (0xa035 bytes) 000010 1746157682.959013 MIRROR_EXTEND to 2 mirrors 000011 1746157683.731595 read 0x0317ac thru 0x03ffff (0xe854 bytes) 000012 1746157683.764049 write 0x0349f2 thru 0x03fed4 (0xb4e3 bytes) 000013 1746157683.840064 mapwrite 0x01b071 thru 0x01fa7b (0x4a0b bytes) 000014 1746157683.922722 mapwrite 0x027b5f thru 0x035fc0 (0xe462 bytes) 000017 1746157684.028995 mapwrite 0x00f092 thru 0x01729f (0x820e bytes) 000019 1746157684.106458 MIRROR_RESYNC 2 mirrors truncating to largest ever: 0x28533 000023 1746157684.542508 trunc from 0x040000 to 0x028532 (0x17acd bytes) 000025 1746157684.649030 mapread 0x02667d thru 0x028532 (0x1eb6 bytes) 000026 1746157684.672362 write 0x0387e4 thru 0x03ffff (0x781c bytes) 000027 1746157684.673341 write 0x03dd7e thru 0x03ffff (0x2282 bytes) 000028 1746157684.674216 read 0x009f78 thru 0x0146b6 (0xa73f bytes) 000029 1746157684.686790 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000403:0x2a4:0x0]: Invalid argument (22) 000030 1746157684.843949 trunc from 0x040000 to 0x022f9c (0x1d063 bytes) 000031 1746157684.857748 read 0x014ef7 thru 0x018d12 (0x3e1c bytes) 000032 1746157684.867451 read 0x007cd4 thru 0x012f7d (0xb2aa bytes) truncating to largest ever: 0x3f62c 000033 1746157684.878391 trunc from 0x022f9d to 0x03f62b (0x1c68f bytes) 000034 1746157684.896830 write 0x019e07 thru 0x022038 (0x8232 bytes) 000037 1746157684.899162 read 0x0179fc thru 0x0264b3 (0xeab8 bytes) 000038 1746157684.939518 MIRROR_EXTEND to 2 mirrors lfs mirror mirror: fail to pread 0-4194304 of mirror 2: Invalid argument (22) lfs verify: failed to read data from mirror 2: Invalid argument. mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 22 lfs mirror mirror: fail to pread 0-4194304 of mirror 1: Invalid argument (22) lfs verify: failed to read data from mirror 1: Invalid argument. lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f62c: skipped LOG DUMP (34 total operations): 1: 1746157682.917231 TRUNC_UP 0x00000 to 0x197cc 2: 1746157682.935185 WRITE 0x26181 thru 0x2d9ca (0x0784a bytes) HOLE 3: 1746157682.938710 READ 0x09716 thru 0x0975a (0x00045 bytes) 4: 1746157682.950712 WRITE 0x35fcb thru 0x3ffff (0x0a035 bytes) HOLE 5: 1746157682.953344 SKIPPED (no operation) 6: 1746157682.959013 MIRROR_EXTEND to 2 mirrors 7: 1746157683.731595 READ 0x317ac thru 0x3ffff (0x0e854 bytes) 8: 1746157683.764049 WRITE 0x349f2 thru 0x3fed4 (0x0b4e3 bytes) 9: 1746157683.840064 MAPWRITE 0x1b071 thru 0x1fa7b (0x04a0b bytes) 10: 1746157683.922722 MAPWRITE 0x27b5f thru 0x35fc0 (0x0e462 bytes) 11: 1746157684.027638 SKIPPED (no operation) 12: 1746157684.028011 SKIPPED (no operation) 13: 1746157684.028995 MAPWRITE 0x0f092 thru 0x1729f (0x0820e bytes) 14: 1746157684.095955 SKIPPED (no operation) 15: 1746157684.106458 MIRROR_RESYNC 2 mirrors 16: 1746157684.542122 SKIPPED (no operation) 17: 1746157684.542275 SKIPPED (no operation) 18: 1746157684.542384 SKIPPED (no operation) 19: 1746157684.542508 TRUNC_DN 0x40000 to 0x28533 20: 1746157684.648638 SKIPPED (no operation) 21: 1746157684.649030 MAPREAD 0x2667d thru 0x28532 (0x01eb6 bytes) 22: 1746157684.672362 WRITE 0x387e4 thru 0x3ffff (0x0781c bytes) HOLE 23: 1746157684.673341 WRITE 0x3dd7e thru 0x3ffff (0x02282 bytes) 24: 1746157684.674216 READ 0x09f78 thru 0x146b6 (0x0a73f bytes) 25: 1746157684.686790 MIRROR_SPLIT mirror 1 to 1 mirrors 26: 1746157684.843949 TRUNC_DN 0x40000 to 0x22f9d 27: 1746157684.857748 READ 0x14ef7 thru 0x18d12 (0x03e1c bytes) 28: 1746157684.867451 READ 0x07cd4 thru 0x12f7d (0x0b2aa bytes) 29: 1746157684.878391 TRUNC_UP 0x22f9d to 0x3f62c 30: 1746157684.896830 WRITE 0x19e07 thru 0x22038 (0x08232 bytes) 31: 1746157684.898296 SKIPPED (no operation) 32: 1746157684.898824 SKIPPED (no operation) 33: 1746157684.899162 READ 0x179fc thru 0x264b3 (0x0eab8 bytes) 34: 1746157684.939518 MIRROR_EXTEND to 2 mirrors Using seed 7682 Correct content saved for comparison (maybe hexdump "/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr" vs "/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr.fsxgood") sanity-flr test_70a: @@@@@@ FAIL: fsx FLR file /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr failed Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7249:error() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:3351:test_70a() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7622:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7685:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7503:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:3353:main() Dumping lctl log to /tmp/testlogs//sanity-flr.test_70a.*.1746157686.log rsync: chown "/tmp/testlogs/.sanity-flr.test_70a.debug_log.oleg101-server.1746157686.log.pPjZws" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-flr.test_70a.dmesg.oleg101-server.1746157686.log.bqTfia" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2]