== sanity-flr test 70a: flr mode fsx test ================ 23:11:46 (1746155506) Seed set to 5507 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000002 1746155507.616913 MIRROR_EXTEND to 2 mirrors 000003 1746155507.824634 mapwrite 0x01c26e thru 0x026562 (0xa2f5 bytes) 000007 1746155507.994045 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000403:0x354:0x0]: Invalid argument (22) 000008 1746155508.083855 mapread 0x01ac5b thru 0x026562 (0xb908 bytes) 000010 1746155508.101616 write 0x009551 thru 0x0124da (0x8f8a bytes) 000011 1746155508.113683 mapwrite 0x01bd37 thru 0x01d275 (0x153f bytes) 000012 1746155508.169574 mapread 0x003e5d thru 0x00d541 (0x96e5 bytes) 000013 1746155508.173097 read 0x008ddb thru 0x0161ad (0xd3d3 bytes) truncating to largest ever: 0x22c30 000015 1746155508.187529 trunc from 0x026563 to 0x022c2f (0x3933 bytes) 000017 1746155508.202741 mapread 0x021e49 thru 0x022c2f (0xde7 bytes) 000019 1746155508.203801 read 0x01ddcd thru 0x022c2f (0x4e63 bytes) 000020 1746155508.204304 mapread 0x017e26 thru 0x021eeb (0xa0c6 bytes) 000024 1746155508.205875 mapwrite 0x01a74b thru 0x02613d (0xb9f3 bytes) 000025 1746155508.273199 read 0x00a1a0 thru 0x015c0a (0xba6b bytes) 000026 1746155508.273663 mapwrite 0x015b12 thru 0x01d1c8 (0x76b7 bytes) 000027 1746155508.320123 mapread 0x017e27 thru 0x01d06d (0x5247 bytes) 000029 1746155508.321952 mapread 0x01b0dd thru 0x02613d (0xb061 bytes) 000030 1746155508.323402 mapread 0x01576d thru 0x021df3 (0xc687 bytes) 000031 1746155508.323946 trunc from 0x02613e to 0x009dd9 (0x1c364 bytes) 000032 1746155508.337473 trunc from 0x009dda to 0x0021cb (0x7c0e bytes) 000033 1746155508.346112 write 0x025194 thru 0x02b0df (0x5f4c bytes) 000036 1746155508.347827 trunc from 0x02b0e0 to 0x010d4a (0x1a395 bytes) 000037 1746155508.359184 mapread 0x00170b thru 0x0059e5 (0x42db bytes) 000039 1746155508.364563 mapread 0x00dc05 thru 0x010d4a (0x3146 bytes) 000040 1746155508.365125 read 0x0079ef thru 0x008390 (0x9a2 bytes) 000042 1746155508.366223 mapread 0x003f0c thru 0x008995 (0x4a8a bytes) 000045 1746155508.371624 MIRROR_EXTEND to 2 mirrors lfs mirror mirror: fail to pread 0-4194304 of mirror 2: Invalid argument (22) lfs verify: failed to read data from mirror 2: Invalid argument. mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 22 lfs mirror mirror: fail to pread 0-4194304 of mirror 1: Invalid argument (22) lfs verify: failed to read data from mirror 1: Invalid argument. lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x10d4b: skipped LOG DUMP (39 total operations): 1: 1746155507.616913 MIRROR_EXTEND to 2 mirrors 2: 1746155507.824634 MAPWRITE 0x1c26e thru 0x26562 (0x0a2f5 bytes) 3: 1746155507.987541 SKIPPED (no operation) 4: 1746155507.988671 SKIPPED (no operation) 5: 1746155507.994045 MIRROR_SPLIT mirror 1 to 1 mirrors 6: 1746155508.083855 MAPREAD 0x1ac5b thru 0x26562 (0x0b908 bytes) 7: 1746155508.101616 WRITE 0x09551 thru 0x124da (0x08f8a bytes) 8: 1746155508.113683 MAPWRITE 0x1bd37 thru 0x1d275 (0x0153f bytes) 9: 1746155508.169574 MAPREAD 0x03e5d thru 0x0d541 (0x096e5 bytes) 10: 1746155508.173097 READ 0x08ddb thru 0x161ad (0x0d3d3 bytes) 11: 1746155508.187123 SKIPPED (no operation) 12: 1746155508.187529 TRUNC_DN 0x26563 to 0x22c30 13: 1746155508.202268 SKIPPED (no operation) 14: 1746155508.202741 MAPREAD 0x21e49 thru 0x22c2f (0x00de7 bytes) 15: 1746155508.203524 SKIPPED (no operation) 16: 1746155508.203801 READ 0x1ddcd thru 0x22c2f (0x04e63 bytes) 17: 1746155508.204304 MAPREAD 0x17e26 thru 0x21eeb (0x0a0c6 bytes) 18: 1746155508.204904 SKIPPED (no operation) 19: 1746155508.205216 SKIPPED (no operation) 20: 1746155508.205571 SKIPPED (no operation) 21: 1746155508.205875 MAPWRITE 0x1a74b thru 0x2613d (0x0b9f3 bytes) 22: 1746155508.273199 READ 0x0a1a0 thru 0x15c0a (0x0ba6b bytes) 23: 1746155508.273663 MAPWRITE 0x15b12 thru 0x1d1c8 (0x076b7 bytes) 24: 1746155508.320123 MAPREAD 0x17e27 thru 0x1d06d (0x05247 bytes) 25: 1746155508.321621 SKIPPED (no operation) 26: 1746155508.321952 MAPREAD 0x1b0dd thru 0x2613d (0x0b061 bytes) 27: 1746155508.323402 MAPREAD 0x1576d thru 0x21df3 (0x0c687 bytes) 28: 1746155508.323946 TRUNC_DN 0x2613e to 0x09dda 29: 1746155508.337473 TRUNC_DN 0x09dda to 0x021cc 30: 1746155508.346112 WRITE 0x25194 thru 0x2b0df (0x05f4c bytes) HOLE 31: 1746155508.346986 SKIPPED (no operation) 32: 1746155508.347827 TRUNC_DN 0x2b0e0 to 0x10d4b 33: 1746155508.359184 MAPREAD 0x0170b thru 0x059e5 (0x042db bytes) 34: 1746155508.364563 MAPREAD 0x0dc05 thru 0x10d4a (0x03146 bytes) 35: 1746155508.365125 READ 0x079ef thru 0x08390 (0x009a2 bytes) 36: 1746155508.365643 SKIPPED (no operation) 37: 1746155508.366223 MAPREAD 0x03f0c thru 0x08995 (0x04a8a bytes) 38: 1746155508.367480 SKIPPED (no operation) 39: 1746155508.371624 MIRROR_EXTEND to 2 mirrors Using seed 5507 Correct content saved for comparison (maybe hexdump "/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr" vs "/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr.fsxgood") sanity-flr test_70a: @@@@@@ FAIL: fsx FLR file /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr failed Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7249:error() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:3351:test_70a() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7622:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7685:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7503:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:3353:main() Dumping lctl log to /tmp/testlogs//sanity-flr.test_70a.*.1746155509.log rsync: chown "/tmp/testlogs/.sanity-flr.test_70a.debug_log.oleg251-server.1746155509.log.yLmHSN" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-flr.test_70a.dmesg.oleg251-server.1746155509.log.0QN1H0" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2]