== sanity-flr test 70a: flr mode fsx test ================ 19:44:57 (1763340297) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70a.sanity-flr Seed set to 299 skipping zero length punch hole fallocating to largest ever: 0x40000 000002 1763340299.527977 fallocate 0x03bf77 thru 0x03ffff (0x4089 bytes) 000005 1763340299.534249 trunc from 0x01aef3 to 0x02796e (0xca7c bytes) 000006 1763340299.539958 mapread 0x002a8a thru 0x00f3c9 (0xc940 bytes) 000008 1763340299.595945 MIRROR_EXTEND to 2 mirrors 000010 1763340300.193605 trunc from 0x027e08 to 0x034178 (0xc371 bytes) 000012 1763340300.234731 punch from 0x03e11b to 0x03f824 (0x170a bytes) 000013 1763340300.241525 trunc from 0x040000 to 0x02c393 (0x13c6c bytes) 000014 1763340300.287933 mapread 0x010130 thru 0x010558 (0x429 bytes) 000015 1763340300.295164 read 0x02c143 thru 0x02c393 (0x251 bytes) 000016 1763340300.305155 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1db:0x0]: Invalid argument (22) 000017 1763340300.484298 fallocate 0x031360 thru 0x033675 (0x2316 bytes) 000018 1763340300.489485 punch from 0x008221 to 0x015626 (0xd406 bytes) 000019 1763340300.494460 write 0x0318a5 thru 0x03cb38 (0xb294 bytes) 000020 1763340300.510246 punch from 0x02c422 to 0x03b027 (0xec06 bytes) 000021 1763340300.533616 read 0x01bbca thru 0x025ea5 (0xa2dc bytes) 000022 1763340300.550645 write 0x0160ba thru 0x01726e (0x11b5 bytes) 000023 1763340300.580643 trunc from 0x03cb39 to 0x02564f (0x174e9 bytes) 000025 1763340300.617874 mapwrite 0x02585e thru 0x026d59 (0x14fc bytes) 000027 1763340300.658149 mapwrite 0x0022d0 thru 0x009af5 (0x7826 bytes) 000029 1763340300.684142 punch from 0x002888 to 0x004ac3 (0x223c bytes) 000030 1763340300.695064 mapwrite 0x009d9e thru 0x00f035 (0x5298 bytes) 000031 1763340300.723436 mapread 0x004b5c thru 0x00d7a5 (0x8c4a bytes) 000034 1763340300.732587 trunc from 0x006c5e to 0x00b535 (0x48d8 bytes) 000036 1763340300.739558 trunc from 0x0233c0 to 0x025266 (0x1ea7 bytes) 000037 1763340300.746821 punch from 0x00cf7b to 0x01418b (0x7211 bytes) 000038 1763340300.751976 trunc from 0x026d5a to 0x02edf4 (0x809b bytes) 000039 1763340300.775558 punch from 0x01dc46 to 0x02a7d5 (0xcb90 bytes) 000040 1763340300.782187 read 0x018dd7 thru 0x01a6a9 (0x18d3 bytes) 000041 1763340300.782840 mapread 0x027798 thru 0x02b9fd (0x4266 bytes) 000042 1763340300.794141 punch from 0x00c8f9 to 0x0135e9 (0x6cf1 bytes) 000043 1763340300.802216 read 0x00f2f9 thru 0x014c2f (0x5937 bytes) 000044 1763340300.811692 read 0x02cb70 thru 0x02e911 (0x1da2 bytes) 000046 1763340300.819152 punch from 0x029a5d to 0x02edf4 (0x5398 bytes) 000047 1763340300.831703 trunc from 0x01c7a9 to 0x028a0f (0xc267 bytes) 000048 1763340300.836018 trunc from 0x023be4 to 0x02e91b (0xad38 bytes) 000049 1763340300.851955 punch from 0x0283af to 0x02ba17 (0x3669 bytes) 000050 1763340300.860648 mapread 0x01b29f thru 0x023622 (0x8384 bytes) 000051 1763340300.884775 MIRROR_EXTEND to 2 mirrors Size error: expected 0x2edf5 stat 0x2edf5 seek 0x2f000 LOG DUMP (39 total operations): 1: 1763340299.522686 SKIPPED (no operation) 2: 1763340299.527977 FALLOC 0x3bf77 thru 0x40000 (0x04089 bytes)EXTENDING 3: 1763340299.534249 ZERO 0x1aef3 thru 0x2796e (0x0ca7c bytes) 4: 1763340299.539958 MAPREAD 0x02a8a thru 0x0f3c9 (0x0c940 bytes) 5: 1763340299.595945 MIRROR_EXTEND to 2 mirrors 6: 1763340300.193605 ZERO 0x27e08 thru 0x34178 (0x0c371 bytes) 7: 1763340300.234731 PUNCH 0x3e11b thru 0x3f824 (0x0170a bytes) 8: 1763340300.241525 TRUNC_DN 0x40000 to 0x2c394 9: 1763340300.287933 MAPREAD 0x10130 thru 0x10558 (0x00429 bytes) 10: 1763340300.295164 READ 0x2c143 thru 0x2c393 (0x00251 bytes) 11: 1763340300.305155 MIRROR_SPLIT mirror 1 to 1 mirrors 12: 1763340300.484298 FALLOC 0x31360 thru 0x33676 (0x02316 bytes)EXTENDING 13: 1763340300.489485 PUNCH 0x08221 thru 0x15626 (0x0d406 bytes) 14: 1763340300.494460 WRITE 0x318a5 thru 0x3cb38 (0x0b294 bytes) EXTEND 15: 1763340300.510246 PUNCH 0x2c422 thru 0x3b027 (0x0ec06 bytes) 16: 1763340300.533616 READ 0x1bbca thru 0x25ea5 (0x0a2dc bytes) 17: 1763340300.550645 WRITE 0x160ba thru 0x1726e (0x011b5 bytes) 18: 1763340300.580643 TRUNC_DN 0x3cb39 to 0x25650 19: 1763340300.617874 MAPWRITE 0x2585e thru 0x26d59 (0x014fc bytes) 20: 1763340300.658149 MAPWRITE 0x022d0 thru 0x09af5 (0x07826 bytes) 21: 1763340300.684142 PUNCH 0x02888 thru 0x04ac3 (0x0223c bytes) 22: 1763340300.695064 MAPWRITE 0x09d9e thru 0x0f035 (0x05298 bytes) 23: 1763340300.723436 MAPREAD 0x04b5c thru 0x0d7a5 (0x08c4a bytes) 24: 1763340300.732587 ZERO 0x06c5e thru 0x0b535 (0x048d8 bytes) 25: 1763340300.739558 ZERO 0x233c0 thru 0x25266 (0x01ea7 bytes) 26: 1763340300.746821 PUNCH 0x0cf7b thru 0x1418b (0x07211 bytes) 27: 1763340300.751976 TRUNC_UP 0x26d5a to 0x2edf5 28: 1763340300.775558 PUNCH 0x1dc46 thru 0x2a7d5 (0x0cb90 bytes) 29: 1763340300.782187 READ 0x18dd7 thru 0x1a6a9 (0x018d3 bytes) 30: 1763340300.782840 MAPREAD 0x27798 thru 0x2b9fd (0x04266 bytes) 31: 1763340300.794141 PUNCH 0x0c8f9 thru 0x135e9 (0x06cf1 bytes) 32: 1763340300.802216 READ 0x0f2f9 thru 0x14c2f (0x05937 bytes) 33: 1763340300.811692 READ 0x2cb70 thru 0x2e911 (0x01da2 bytes) 34: 1763340300.819152 PUNCH 0x29a5d thru 0x2edf4 (0x05398 bytes) 35: 1763340300.831703 ZERO 0x1c7a9 thru 0x28a0f (0x0c267 bytes) 36: 1763340300.836018 ZERO 0x23be4 thru 0x2e91b (0x0ad38 bytes) 37: 1763340300.851955 PUNCH 0x283af thru 0x2ba17 (0x03669 bytes) 38: 1763340300.860648 MAPREAD 0x1b29f thru 0x23622 (0x08384 bytes) 39: 1763340300.884775 MIRROR_EXTEND to 2 mirrors Using seed 299 Correct content saved for comparison (maybe hexdump "/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr" vs "/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr.fsxgood") sanity-flr test_70a: @@@@@@ FAIL: fsx FLR file /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr failed Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7314:error() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:3373:test_70a() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7690:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7753:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7568:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:3375:main() Dumping lctl log to /tmp/testlogs//sanity-flr.test_70a.*.1763340303.log rsync: chown "/tmp/testlogs/.sanity-flr.test_70a.debug_log.oleg342-server.1763340303.log.KQpURC" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-flr.test_70a.dmesg.oleg342-server.1763340303.log.objGh1" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1668) [generator=3.1.3]