== sanity-flr test 70a: flr mode fsx test ================ 15:14:24 (1746126864) Seed set to 6864 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000001 1746126865.051134 MIRROR_EXTEND to 2 mirrors skipping zero size read skipping zero size read 000006 1746126865.259415 MIRROR_EXTEND to 3 mirrors skipping zero size read skipping zero size read skipping zero size read skipping zero size read skipping zero size read skipping zero size read 000015 1746126865.416581 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000403:0x490:0x0]: Invalid argument (22) 000016 1746126865.538418 MIRROR_EXTEND to 3 mirrors 000019 1746126865.711267 write 0x03df21 thru 0x03e7f7 (0x8d7 bytes) truncating to largest ever: 0x23e9a 000020 1746126865.739526 trunc from 0x03e7f8 to 0x023e99 (0x1a95e bytes) 000021 1746126865.760896 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000403:0x490:0x0]: Invalid argument (22) 000023 1746126865.859808 mapread 0x0110da thru 0x011ea3 (0xdca bytes) 000029 1746126865.865251 mapwrite 0x01b3eb thru 0x01b8ec (0x502 bytes) 000030 1746126865.898581 mapread 0x014ccc thru 0x015059 (0x38e bytes) 000031 1746126865.905568 write 0x00155f thru 0x00b03a (0x9adc bytes) 000032 1746126865.912386 mapwrite 0x01ad93 thru 0x02794f (0xcbbd bytes) 000033 1746126865.981960 mapread 0x0118af thru 0x01fad4 (0xe226 bytes) 000035 1746126865.983557 write 0x01dad3 thru 0x027d1f (0xa24d bytes) 000039 1746126865.992386 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000403:0x490:0x0]: Invalid argument (22) 000041 1746126866.065904 trunc from 0x027d20 to 0x0017ee (0x26531 bytes) truncating to largest ever: 0x300a6 000044 1746126866.074469 trunc from 0x0017ef to 0x0300a5 (0x2e8b7 bytes) 000046 1746126866.084732 read 0x029a07 thru 0x0300a5 (0x669f bytes) 000049 1746126866.091029 mapread 0x002240 thru 0x00c297 (0xa058 bytes) 000051 1746126866.102763 trunc from 0x0300a6 to 0x005fab (0x2a0fa bytes) 000052 1746126866.110239 read 0x0013e8 thru 0x004072 (0x2c8b bytes) 000053 1746126866.110691 write 0x01be5d thru 0x0267c2 (0xa966 bytes) 000054 1746126866.111487 read 0x00b776 thru 0x015f76 (0xa801 bytes) 000061 1746126866.118382 write 0x037d7c thru 0x03f997 (0x7c1c bytes) 000062 1746126866.119287 mapwrite 0x00abf7 thru 0x014e70 (0xa27a bytes) 000063 1746126866.161490 write 0x02874b thru 0x0295ff (0xeb5 bytes) 000064 1746126866.168473 mapwrite 0x03b5a5 thru 0x03ffff (0x4a5b bytes) 000065 1746126866.211932 write 0x039c3f thru 0x03ffff (0x63c1 bytes) 000068 1746126866.213316 mapwrite 0x004a73 thru 0x00ed53 (0xa2e1 bytes) 000069 1746126866.246600 mapread 0x01aa80 thru 0x022d42 (0x82c3 bytes) 000070 1746126866.247247 mapread 0x03c81b thru 0x03d5f3 (0xdd9 bytes) 000071 1746126866.247739 trunc from 0x040000 to 0x015bf5 (0x2a40a bytes) 000074 1746126866.256569 mapread 0x0104e8 thru 0x012910 (0x2429 bytes) 000076 1746126866.257686 trunc from 0x015bf6 to 0x0014df (0x14716 bytes) 000079 1746126866.266954 mapread 0x000b0f thru 0x0014df (0x9d1 bytes) 000080 1746126866.267544 mapread 0x0002e1 thru 0x0014df (0x11ff bytes) 000081 1746126866.268220 write 0x036cfe thru 0x03ffff (0x9302 bytes) 000085 1746126866.271580 mapwrite 0x00c7ff thru 0x011ae0 (0x52e2 bytes) 000087 1746126866.321739 write 0x025d1a thru 0x032bde (0xcec5 bytes) 000089 1746126866.330869 write 0x02c97c thru 0x032010 (0x5695 bytes) 000090 1746126866.331603 write 0x009d71 thru 0x015177 (0xb407 bytes) 000091 1746126866.340158 mapread 0x017146 thru 0x0203b4 (0x926f bytes) 000092 1746126866.369451 mapread 0x00425c thru 0x013b0d (0xf8b2 bytes) 000093 1746126866.380987 write 0x008118 thru 0x011394 (0x927d bytes) 000094 1746126866.382032 trunc from 0x040000 to 0x029753 (0x168ac bytes) 000095 1746126866.390137 trunc from 0x029754 to 0x020484 (0x92cf bytes) 000098 1746126866.395699 read 0x00c1db thru 0x014d4a (0x8b70 bytes) 000099 1746126866.396378 mapwrite 0x0273d7 thru 0x036d9b (0xf9c5 bytes) truncating to largest ever: 0x32bf4 000101 1746126866.452442 trunc from 0x036d9c to 0x032bf3 (0x41a8 bytes) 000103 1746126866.460004 mapread 0x002db7 thru 0x0032ea (0x534 bytes) 000105 1746126866.460984 trunc from 0x032bf4 to 0x02bb6a (0x7089 bytes) 000106 1746126866.470645 read 0x025994 thru 0x0269a4 (0x1011 bytes) 000108 1746126866.471494 mapread 0x016752 thru 0x0239fa (0xd2a9 bytes) 000109 1746126866.472248 read 0x00fae8 thru 0x0171a6 (0x76bf bytes) truncating to largest ever: 0x33bf8 000110 1746126866.472685 trunc from 0x02bb6b to 0x033bf7 (0x808d bytes) 000112 1746126866.484334 write 0x02dc65 thru 0x02fcf2 (0x208e bytes) 000114 1746126866.494845 MIRROR_EXTEND to 2 mirrors lfs mirror mirror: fail to pread 0-4194304 of mirror 2: Invalid argument (22) lfs verify: failed to read data from mirror 2: Invalid argument. mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 22 lfs mirror mirror: fail to pread 0-4194304 of mirror 1: Invalid argument (22) lfs verify: failed to read data from mirror 1: Invalid argument. lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33bf8: skipped LOG DUMP (100 total operations): 1: 1746126865.051134 MIRROR_EXTEND to 2 mirrors 2: 1746126865.237009 SKIPPED (no operation) 3: 1746126865.243166 SKIPPED (no operation) 4: 1746126865.246256 SKIPPED (no operation) 5: 1746126865.259415 MIRROR_EXTEND to 3 mirrors 6: 1746126865.413272 SKIPPED (no operation) 7: 1746126865.413523 SKIPPED (no operation) 8: 1746126865.413965 SKIPPED (no operation) 9: 1746126865.414135 SKIPPED (no operation) 10: 1746126865.414291 SKIPPED (no operation) 11: 1746126865.414469 SKIPPED (no operation) 12: 1746126865.414713 SKIPPED (no operation) 13: 1746126865.416581 MIRROR_SPLIT mirror 1 to 2 mirrors 14: 1746126865.538418 MIRROR_EXTEND to 3 mirrors 15: 1746126865.710828 SKIPPED (no operation) 16: 1746126865.711064 SKIPPED (no operation) 17: 1746126865.711267 WRITE 0x3df21 thru 0x3e7f7 (0x008d7 bytes) HOLE 18: 1746126865.739526 TRUNC_DN 0x3e7f8 to 0x23e9a 19: 1746126865.760896 MIRROR_SPLIT mirror 1 to 2 mirrors 20: 1746126865.859496 SKIPPED (no operation) 21: 1746126865.859808 MAPREAD 0x110da thru 0x11ea3 (0x00dca bytes) 22: 1746126865.863388 SKIPPED (no operation) 23: 1746126865.863783 SKIPPED (no operation) 24: 1746126865.864256 SKIPPED (no operation) 25: 1746126865.864587 SKIPPED (no operation) 26: 1746126865.864933 SKIPPED (no operation) 27: 1746126865.865251 MAPWRITE 0x1b3eb thru 0x1b8ec (0x00502 bytes) 28: 1746126865.898581 MAPREAD 0x14ccc thru 0x15059 (0x0038e bytes) 29: 1746126865.905568 WRITE 0x0155f thru 0x0b03a (0x09adc bytes) 30: 1746126865.912386 MAPWRITE 0x1ad93 thru 0x2794f (0x0cbbd bytes) 31: 1746126865.981960 MAPREAD 0x118af thru 0x1fad4 (0x0e226 bytes) 32: 1746126865.983251 SKIPPED (no operation) 33: 1746126865.983557 WRITE 0x1dad3 thru 0x27d1f (0x0a24d bytes) EXTEND 34: 1746126865.984386 SKIPPED (no operation) 35: 1746126865.984583 SKIPPED (no operation) 36: 1746126865.985044 SKIPPED (no operation) 37: 1746126865.992386 MIRROR_SPLIT mirror 1 to 1 mirrors 38: 1746126866.065513 SKIPPED (no operation) 39: 1746126866.065904 TRUNC_DN 0x27d20 to 0x017ef 40: 1746126866.074469 TRUNC_UP 0x017ef to 0x300a6 41: 1746126866.084376 SKIPPED (no operation) 42: 1746126866.084732 READ 0x29a07 thru 0x300a5 (0x0669f bytes) 43: 1746126866.090636 SKIPPED (no operation) 44: 1746126866.091029 MAPREAD 0x02240 thru 0x0c297 (0x0a058 bytes) 45: 1746126866.102536 SKIPPED (no operation) 46: 1746126866.102763 TRUNC_DN 0x300a6 to 0x05fac 47: 1746126866.110239 READ 0x013e8 thru 0x04072 (0x02c8b bytes) 48: 1746126866.110691 WRITE 0x1be5d thru 0x267c2 (0x0a966 bytes) HOLE 49: 1746126866.111487 READ 0x0b776 thru 0x15f76 (0x0a801 bytes) 50: 1746126866.116310 SKIPPED (no operation) 51: 1746126866.117470 SKIPPED (no operation) 52: 1746126866.117780 SKIPPED (no operation) 53: 1746126866.118089 SKIPPED (no operation) 54: 1746126866.118382 WRITE 0x37d7c thru 0x3f997 (0x07c1c bytes) HOLE 55: 1746126866.119287 MAPWRITE 0x0abf7 thru 0x14e70 (0x0a27a bytes) 56: 1746126866.161490 WRITE 0x2874b thru 0x295ff (0x00eb5 bytes) 57: 1746126866.168473 MAPWRITE 0x3b5a5 thru 0x3ffff (0x04a5b bytes) 58: 1746126866.211932 WRITE 0x39c3f thru 0x3ffff (0x063c1 bytes) 59: 1746126866.213101 SKIPPED (no operation) 60: 1746126866.213316 MAPWRITE 0x04a73 thru 0x0ed53 (0x0a2e1 bytes) 61: 1746126866.246600 MAPREAD 0x1aa80 thru 0x22d42 (0x082c3 bytes) 62: 1746126866.247247 MAPREAD 0x3c81b thru 0x3d5f3 (0x00dd9 bytes) 63: 1746126866.247739 TRUNC_DN 0x40000 to 0x15bf6 64: 1746126866.255996 SKIPPED (no operation) 65: 1746126866.256288 SKIPPED (no operation) 66: 1746126866.256569 MAPREAD 0x104e8 thru 0x12910 (0x02429 bytes) 67: 1746126866.257686 TRUNC_DN 0x15bf6 to 0x014e0 68: 1746126866.266954 MAPREAD 0x00b0f thru 0x014df (0x009d1 bytes) 69: 1746126866.267544 MAPREAD 0x002e1 thru 0x014df (0x011ff bytes) 70: 1746126866.268220 WRITE 0x36cfe thru 0x3ffff (0x09302 bytes) HOLE 71: 1746126866.270971 SKIPPED (no operation) 72: 1746126866.271272 SKIPPED (no operation) 73: 1746126866.271580 MAPWRITE 0x0c7ff thru 0x11ae0 (0x052e2 bytes) 74: 1746126866.321403 SKIPPED (no operation) 75: 1746126866.321739 WRITE 0x25d1a thru 0x32bde (0x0cec5 bytes) 76: 1746126866.330869 WRITE 0x2c97c thru 0x32010 (0x05695 bytes) 77: 1746126866.331603 WRITE 0x09d71 thru 0x15177 (0x0b407 bytes) 78: 1746126866.340158 MAPREAD 0x17146 thru 0x203b4 (0x0926f bytes) 79: 1746126866.369451 MAPREAD 0x0425c thru 0x13b0d (0x0f8b2 bytes) 80: 1746126866.380987 WRITE 0x08118 thru 0x11394 (0x0927d bytes) 81: 1746126866.382032 TRUNC_DN 0x40000 to 0x29754 82: 1746126866.390137 TRUNC_DN 0x29754 to 0x20485 83: 1746126866.394980 SKIPPED (no operation) 84: 1746126866.395297 SKIPPED (no operation) 85: 1746126866.395699 READ 0x0c1db thru 0x14d4a (0x08b70 bytes) 86: 1746126866.396378 MAPWRITE 0x273d7 thru 0x36d9b (0x0f9c5 bytes) 87: 1746126866.452071 SKIPPED (no operation) 88: 1746126866.452442 TRUNC_DN 0x36d9c to 0x32bf4 89: 1746126866.459576 SKIPPED (no operation) 90: 1746126866.460004 MAPREAD 0x02db7 thru 0x032ea (0x00534 bytes) 91: 1746126866.460566 SKIPPED (no operation) 92: 1746126866.460984 TRUNC_DN 0x32bf4 to 0x2bb6b 93: 1746126866.470645 READ 0x25994 thru 0x269a4 (0x01011 bytes) 94: 1746126866.471144 SKIPPED (no operation) 95: 1746126866.471494 MAPREAD 0x16752 thru 0x239fa (0x0d2a9 bytes) 96: 1746126866.472248 READ 0x0fae8 thru 0x171a6 (0x076bf bytes) 97: 1746126866.472685 TRUNC_UP 0x2bb6b to 0x33bf8 98: 1746126866.484050 SKIPPED (no operation) 99: 1746126866.484334 WRITE 0x2dc65 thru 0x2fcf2 (0x0208e bytes) 100: 1746126866.494845 MIRROR_EXTEND to 2 mirrors Using seed 6864 Correct content saved for comparison (maybe hexdump "/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr" vs "/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr.fsxgood") sanity-flr test_70a: @@@@@@ FAIL: fsx FLR file /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr failed Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7249:error() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:3351:test_70a() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7622:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7685:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7503:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:3353:main() Dumping lctl log to /tmp/testlogs//sanity-flr.test_70a.*.1746126867.log rsync: chown "/tmp/testlogs/.sanity-flr.test_70a.debug_log.oleg404-server.1746126867.log.gStYgA" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-flr.test_70a.dmesg.oleg404-server.1746126867.log.rGf3f9" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2]