== sanity-flr test 70a: flr mode fsx test ================ 16:21:16 (1746130876) Seed set to 877 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000002 1746130877.331616 write 0x00245b thru 0x006a11 (0x45b7 bytes) 000005 1746130877.337877 write 0x010bf2 thru 0x02047b (0xf88a bytes) 000006 1746130877.347037 read 0x00d362 thru 0x0190a1 (0xbd40 bytes) truncating to largest ever: 0x78ce 000009 1746130877.352365 trunc from 0x02047c to 0x0078cd (0x18bae bytes) 000014 1746130877.372802 mapread 0x00037e thru 0x00628d (0x5f10 bytes) 000015 1746130877.382242 mapwrite 0x003ac5 thru 0x0046f4 (0xc30 bytes) truncating to largest ever: 0x31a91 000018 1746130877.449544 trunc from 0x0078ce to 0x031a90 (0x2a1c3 bytes) 000019 1746130877.466384 read 0x0251a1 thru 0x031a90 (0xc8f0 bytes) 000021 1746130877.488525 read 0x007621 thru 0x0154c2 (0xdea2 bytes) 000022 1746130877.489722 trunc from 0x031a91 to 0x02eb81 (0x2f0f bytes) 000023 1746130877.509210 read 0x01d156 thru 0x0247b8 (0x7663 bytes) 000024 1746130877.509686 mapread 0x025323 thru 0x02eb81 (0x985f bytes) 000025 1746130877.511260 trunc from 0x02eb82 to 0x02c95f (0x2222 bytes) 000028 1746130877.532882 mapread 0x02c598 thru 0x02c95f (0x3c8 bytes) 000032 1746130877.536054 read 0x00dc01 thru 0x01a9f6 (0xcdf6 bytes) 000033 1746130877.536656 mapwrite 0x02ea24 thru 0x03e210 (0xf7ed bytes) 000034 1746130877.634990 mapwrite 0x031961 thru 0x03a1ba (0x885a bytes) 000036 1746130877.689776 mapwrite 0x02266e thru 0x02a696 (0x8029 bytes) 000038 1746130877.746878 trunc from 0x03e211 to 0x018249 (0x25fc7 bytes) 000041 1746130877.761018 read 0x009dfc thru 0x00fd81 (0x5f86 bytes) 000044 1746130877.762790 write 0x02b969 thru 0x030de6 (0x547e bytes) 000047 1746130877.765205 read 0x00cc19 thru 0x01699c (0x9d84 bytes) 000048 1746130877.769352 read 0x026089 thru 0x030de6 (0xad5e bytes) 000049 1746130877.776940 mapread 0x00c83e thru 0x01186f (0x5032 bytes) 000050 1746130877.779869 mapwrite 0x01b43f thru 0x022a5c (0x761e bytes) 000052 1746130877.826398 read 0x0137d7 thru 0x018c9c (0x54c6 bytes) 000053 1746130877.826894 trunc from 0x030de7 to 0x01b343 (0x15aa3 bytes) 000058 1746130877.847653 write 0x02358c thru 0x0288a0 (0x5315 bytes) 000059 1746130877.849725 write 0x00a013 thru 0x00b77f (0x176d bytes) 000061 1746130877.851276 read 0x00166d thru 0x00a1cc (0x8b60 bytes) 000065 1746130877.853033 mapread 0x0191f8 thru 0x01de5d (0x4c66 bytes) 000066 1746130877.860355 read 0x007b32 thru 0x0153ac (0xd87b bytes) 000067 1746130877.862227 mapread 0x01df8d thru 0x0288a0 (0xa914 bytes) 000068 1746130877.864256 trunc from 0x0288a1 to 0x016e95 (0x11a0b bytes) 000069 1746130877.879812 mapwrite 0x0358c7 thru 0x03ffff (0xa739 bytes) 000070 1746130877.964042 read 0x03de2e thru 0x03ffff (0x21d2 bytes) 000071 1746130877.965247 trunc from 0x040000 to 0x02964f (0x169b0 bytes) 000072 1746130877.984490 trunc from 0x029650 to 0x02a7e9 (0x119a bytes) truncating to largest ever: 0x3836d 000074 1746130878.002547 trunc from 0x02a7ea to 0x03836c (0xdb83 bytes) 000076 1746130878.014610 read 0x023b96 thru 0x02afbf (0x742a bytes) 000077 1746130878.020985 mapread 0x00258f thru 0x009ec8 (0x793a bytes) 000079 1746130878.022159 mapwrite 0x02e6fc thru 0x034835 (0x613a bytes) 000080 1746130878.088177 write 0x0209e2 thru 0x030170 (0xf78f bytes) 000081 1746130878.089791 write 0x01b570 thru 0x022c64 (0x76f5 bytes) 000085 1746130878.093821 mapwrite 0x0073c1 thru 0x0134cc (0xc10c bytes) 000086 1746130878.159569 read 0x00aa9d thru 0x014569 (0x9acd bytes) 000087 1746130878.160336 mapread 0x02f666 thru 0x03836c (0x8d07 bytes) 000090 1746130878.164168 write 0x02738a thru 0x02a6fa (0x3371 bytes) 000091 1746130878.166414 mapwrite 0x00c799 thru 0x014ada (0x8342 bytes) 000093 1746130878.216280 trunc from 0x03836d to 0x02ed38 (0x9634 bytes) 000097 1746130878.233706 mapread 0x01b675 thru 0x0298fa (0xe286 bytes) 000099 1746130878.236230 mapwrite 0x03e65c thru 0x03ffff (0x19a4 bytes) 000101 1746130878.332674 trunc from 0x040000 to 0x01c9f3 (0x2360c bytes) 000105 1746130878.353619 read 0x00f287 thru 0x01c9f3 (0xd76d bytes) 000106 1746130878.354367 trunc from 0x01c9f4 to 0x01f79d (0x2daa bytes) 000107 1746130878.366428 mapwrite 0x033d6f thru 0x03ffff (0xc291 bytes) 000109 1746130878.514339 write 0x02cf9d thru 0x032eeb (0x5f4f bytes) 000112 1746130878.515429 trunc from 0x040000 to 0x008d4c (0x372b3 bytes) 000117 1746130878.536642 mapread 0x002b29 thru 0x006791 (0x3c69 bytes) 000118 1746130878.541554 trunc from 0x008d4d to 0x00890b (0x441 bytes) 000125 1746130878.555971 trunc from 0x00890c to 0x006251 (0x26ba bytes) 000127 1746130878.573510 trunc from 0x006252 to 0x02210e (0x1bebd bytes) 000129 1746130878.597815 write 0x038150 thru 0x03ffff (0x7eb0 bytes) 000130 1746130878.598779 mapwrite 0x0065df thru 0x00ad1b (0x473d bytes) 000131 1746130878.668752 mapread 0x015463 thru 0x01ad88 (0x5926 bytes) 000132 1746130878.689267 mapread 0x03aec4 thru 0x03ffff (0x513c bytes) 000133 1746130878.690304 trunc from 0x040000 to 0x009a22 (0x365dd bytes) 000134 1746130878.700983 mapwrite 0x0247a4 thru 0x02b5bc (0x6e19 bytes) 000136 1746130878.861252 read 0x004571 thru 0x0141a7 (0xfc37 bytes) 000139 1746130878.875608 trunc from 0x02b5bd to 0x015f20 (0x1569c bytes) 000142 1746130878.895501 mapread 0x00982e thru 0x010f2e (0x7701 bytes) 000143 1746130878.897389 read 0x00c94a thru 0x00f2ec (0x29a3 bytes) 000146 1746130878.899301 mapread 0x004663 thru 0x00ab76 (0x6514 bytes) 000148 1746130878.900427 read 0x0153d7 thru 0x015f20 (0xb4a bytes) 000150 1746130878.905602 trunc from 0x015f21 to 0x013bf1 (0x232f bytes) 000153 1746130878.925035 read 0x013a8f thru 0x013bf1 (0x163 bytes) 000154 1746130878.926115 mapread 0x00bd3c thru 0x013bf1 (0x7eb6 bytes) 000156 1746130878.929218 trunc from 0x013bf2 to 0x01c480 (0x888f bytes) 000157 1746130878.953938 write 0x0232a1 thru 0x02656a (0x32ca bytes) 000159 1746130878.957386 write 0x0269de thru 0x031c67 (0xb28a bytes) 000160 1746130878.967378 MIRROR_EXTEND to 2 mirrors lfs mirror mirror: fail to pread 0-4194304 of mirror 2: Invalid argument (22) lfs verify: failed to read data from mirror 2: Invalid argument. mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 22 lfs mirror mirror: fail to pread 0-4194304 of mirror 1: Invalid argument (22) lfs verify: failed to read data from mirror 1: Invalid argument. lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31c68: skipped LOG DUMP (127 total operations): 1: 1746130877.326153 SKIPPED (no operation) 2: 1746130877.331616 WRITE 0x0245b thru 0x06a11 (0x045b7 bytes) HOLE 3: 1746130877.337877 WRITE 0x10bf2 thru 0x2047b (0x0f88a bytes) HOLE 4: 1746130877.347037 READ 0x0d362 thru 0x190a1 (0x0bd40 bytes) 5: 1746130877.351361 SKIPPED (no operation) 6: 1746130877.352042 SKIPPED (no operation) 7: 1746130877.352365 TRUNC_DN 0x2047c to 0x078ce 8: 1746130877.370565 SKIPPED (no operation) 9: 1746130877.371106 SKIPPED (no operation) 10: 1746130877.372802 MAPREAD 0x0037e thru 0x0628d (0x05f10 bytes) 11: 1746130877.382242 MAPWRITE 0x03ac5 thru 0x046f4 (0x00c30 bytes) 12: 1746130877.448368 SKIPPED (no operation) 13: 1746130877.449544 TRUNC_UP 0x078ce to 0x31a91 14: 1746130877.466384 READ 0x251a1 thru 0x31a90 (0x0c8f0 bytes) 15: 1746130877.487646 SKIPPED (no operation) 16: 1746130877.488525 READ 0x07621 thru 0x154c2 (0x0dea2 bytes) 17: 1746130877.489722 TRUNC_DN 0x31a91 to 0x2eb82 18: 1746130877.509210 READ 0x1d156 thru 0x247b8 (0x07663 bytes) 19: 1746130877.509686 MAPREAD 0x25323 thru 0x2eb81 (0x0985f bytes) 20: 1746130877.511260 TRUNC_DN 0x2eb82 to 0x2c960 21: 1746130877.532882 MAPREAD 0x2c598 thru 0x2c95f (0x003c8 bytes) 22: 1746130877.535185 SKIPPED (no operation) 23: 1746130877.535386 SKIPPED (no operation) 24: 1746130877.535743 SKIPPED (no operation) 25: 1746130877.536054 READ 0x0dc01 thru 0x1a9f6 (0x0cdf6 bytes) 26: 1746130877.536656 MAPWRITE 0x2ea24 thru 0x3e210 (0x0f7ed bytes) 27: 1746130877.634990 MAPWRITE 0x31961 thru 0x3a1ba (0x0885a bytes) 28: 1746130877.689483 SKIPPED (no operation) 29: 1746130877.689776 MAPWRITE 0x2266e thru 0x2a696 (0x08029 bytes) 30: 1746130877.746878 TRUNC_DN 0x3e211 to 0x1824a 31: 1746130877.760041 SKIPPED (no operation) 32: 1746130877.761018 READ 0x09dfc thru 0x0fd81 (0x05f86 bytes) 33: 1746130877.761761 SKIPPED (no operation) 34: 1746130877.762343 SKIPPED (no operation) 35: 1746130877.762790 WRITE 0x2b969 thru 0x30de6 (0x0547e bytes) HOLE 36: 1746130877.764444 SKIPPED (no operation) 37: 1746130877.764823 SKIPPED (no operation) 38: 1746130877.765205 READ 0x0cc19 thru 0x1699c (0x09d84 bytes) 39: 1746130877.769352 READ 0x26089 thru 0x30de6 (0x0ad5e bytes) 40: 1746130877.776940 MAPREAD 0x0c83e thru 0x1186f (0x05032 bytes) 41: 1746130877.779869 MAPWRITE 0x1b43f thru 0x22a5c (0x0761e bytes) 42: 1746130877.826398 READ 0x137d7 thru 0x18c9c (0x054c6 bytes) 43: 1746130877.826894 TRUNC_DN 0x30de7 to 0x1b344 44: 1746130877.845661 SKIPPED (no operation) 45: 1746130877.846339 SKIPPED (no operation) 46: 1746130877.847266 SKIPPED (no operation) 47: 1746130877.847477 SKIPPED (no operation) 48: 1746130877.847653 WRITE 0x2358c thru 0x288a0 (0x05315 bytes) HOLE 49: 1746130877.849725 WRITE 0x0a013 thru 0x0b77f (0x0176d bytes) 50: 1746130877.850955 SKIPPED (no operation) 51: 1746130877.851276 READ 0x0166d thru 0x0a1cc (0x08b60 bytes) 52: 1746130877.851673 SKIPPED (no operation) 53: 1746130877.852472 SKIPPED (no operation) 54: 1746130877.853033 MAPREAD 0x191f8 thru 0x1de5d (0x04c66 bytes) 55: 1746130877.860355 READ 0x07b32 thru 0x153ac (0x0d87b bytes) 56: 1746130877.862227 MAPREAD 0x1df8d thru 0x288a0 (0x0a914 bytes) 57: 1746130877.864256 TRUNC_DN 0x288a1 to 0x16e96 58: 1746130877.879812 MAPWRITE 0x358c7 thru 0x3ffff (0x0a739 bytes) 59: 1746130877.964042 READ 0x3de2e thru 0x3ffff (0x021d2 bytes) 60: 1746130877.965247 TRUNC_DN 0x40000 to 0x29650 61: 1746130877.984490 TRUNC_UP 0x29650 to 0x2a7ea 62: 1746130878.001614 SKIPPED (no operation) 63: 1746130878.002547 TRUNC_UP 0x2a7ea to 0x3836d 64: 1746130878.014256 SKIPPED (no operation) 65: 1746130878.014610 READ 0x23b96 thru 0x2afbf (0x0742a bytes) 66: 1746130878.020985 MAPREAD 0x0258f thru 0x09ec8 (0x0793a bytes) 67: 1746130878.021754 SKIPPED (no operation) 68: 1746130878.022159 MAPWRITE 0x2e6fc thru 0x34835 (0x0613a bytes) 69: 1746130878.088177 WRITE 0x209e2 thru 0x30170 (0x0f78f bytes) 70: 1746130878.089791 WRITE 0x1b570 thru 0x22c64 (0x076f5 bytes) 71: 1746130878.092304 SKIPPED (no operation) 72: 1746130878.092627 SKIPPED (no operation) 73: 1746130878.093821 MAPWRITE 0x073c1 thru 0x134cc (0x0c10c bytes) 74: 1746130878.159569 READ 0x0aa9d thru 0x14569 (0x09acd bytes) 75: 1746130878.160336 MAPREAD 0x2f666 thru 0x3836c (0x08d07 bytes) 76: 1746130878.162749 SKIPPED (no operation) 77: 1746130878.164168 WRITE 0x2738a thru 0x2a6fa (0x03371 bytes) 78: 1746130878.166414 MAPWRITE 0x0c799 thru 0x14ada (0x08342 bytes) 79: 1746130878.216280 TRUNC_DN 0x3836d to 0x2ed39 80: 1746130878.232752 SKIPPED (no operation) 81: 1746130878.233078 SKIPPED (no operation) 82: 1746130878.233366 SKIPPED (no operation) 83: 1746130878.233706 MAPREAD 0x1b675 thru 0x298fa (0x0e286 bytes) 84: 1746130878.236230 MAPWRITE 0x3e65c thru 0x3ffff (0x019a4 bytes) 85: 1746130878.332674 TRUNC_DN 0x40000 to 0x1c9f4 86: 1746130878.352430 SKIPPED (no operation) 87: 1746130878.352979 SKIPPED (no operation) 88: 1746130878.353619 READ 0x0f287 thru 0x1c9f3 (0x0d76d bytes) 89: 1746130878.354367 TRUNC_UP 0x1c9f4 to 0x1f79e 90: 1746130878.366428 MAPWRITE 0x33d6f thru 0x3ffff (0x0c291 bytes) 91: 1746130878.514339 WRITE 0x2cf9d thru 0x32eeb (0x05f4f bytes) 92: 1746130878.515085 SKIPPED (no operation) 93: 1746130878.515260 SKIPPED (no operation) 94: 1746130878.515429 TRUNC_DN 0x40000 to 0x08d4d 95: 1746130878.535362 SKIPPED (no operation) 96: 1746130878.536642 MAPREAD 0x02b29 thru 0x06791 (0x03c69 bytes) 97: 1746130878.541554 TRUNC_DN 0x08d4d to 0x0890c 98: 1746130878.553286 SKIPPED (no operation) 99: 1746130878.555733 SKIPPED (no operation) 100: 1746130878.555971 TRUNC_DN 0x0890c to 0x06252 101: 1746130878.572684 SKIPPED (no operation) 102: 1746130878.573510 TRUNC_UP 0x06252 to 0x2210f 103: 1746130878.597475 SKIPPED (no operation) 104: 1746130878.597815 WRITE 0x38150 thru 0x3ffff (0x07eb0 bytes) HOLE 105: 1746130878.598779 MAPWRITE 0x065df thru 0x0ad1b (0x0473d bytes) 106: 1746130878.668752 MAPREAD 0x15463 thru 0x1ad88 (0x05926 bytes) 107: 1746130878.689267 MAPREAD 0x3aec4 thru 0x3ffff (0x0513c bytes) 108: 1746130878.690304 TRUNC_DN 0x40000 to 0x09a23 109: 1746130878.700983 MAPWRITE 0x247a4 thru 0x2b5bc (0x06e19 bytes) 110: 1746130878.861252 READ 0x04571 thru 0x141a7 (0x0fc37 bytes) 111: 1746130878.875608 TRUNC_DN 0x2b5bd to 0x15f21 112: 1746130878.894760 SKIPPED (no operation) 113: 1746130878.895501 MAPREAD 0x0982e thru 0x10f2e (0x07701 bytes) 114: 1746130878.897389 READ 0x0c94a thru 0x0f2ec (0x029a3 bytes) 115: 1746130878.898600 SKIPPED (no operation) 116: 1746130878.899301 MAPREAD 0x04663 thru 0x0ab76 (0x06514 bytes) 117: 1746130878.900427 READ 0x153d7 thru 0x15f20 (0x00b4a bytes) 118: 1746130878.905265 SKIPPED (no operation) 119: 1746130878.905602 TRUNC_DN 0x15f21 to 0x13bf2 120: 1746130878.924338 SKIPPED (no operation) 121: 1746130878.925035 READ 0x13a8f thru 0x13bf1 (0x00163 bytes) 122: 1746130878.926115 MAPREAD 0x0bd3c thru 0x13bf1 (0x07eb6 bytes) 123: 1746130878.929218 TRUNC_UP 0x13bf2 to 0x1c481 124: 1746130878.953938 WRITE 0x232a1 thru 0x2656a (0x032ca bytes) HOLE 125: 1746130878.957043 SKIPPED (no operation) 126: 1746130878.957386 WRITE 0x269de thru 0x31c67 (0x0b28a bytes) HOLE 127: 1746130878.967378 MIRROR_EXTEND to 2 mirrors Using seed 877 Correct content saved for comparison (maybe hexdump "/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr" vs "/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr.fsxgood") sanity-flr test_70a: @@@@@@ FAIL: fsx FLR file /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr failed Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7249:error() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:3351:test_70a() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7622:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7685:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7503:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:3353:main() Dumping lctl log to /tmp/testlogs//sanity-flr.test_70a.*.1746130880.log rsync: chown "/tmp/testlogs/.sanity-flr.test_70a.debug_log.oleg152-server.1746130880.log.1RrF0P" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-flr.test_70a.dmesg.oleg152-server.1746130880.log.519iQT" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2]