-----============= acceptance-small: sanityn ============----- Wed Oct 22 04:17:46 EDT 2025 oleg416-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanityn.*ex': No such file or directory excepting tests: 27 28 102 skipping tests SLOW=no: 33a Starting client oleg416-client.virtnet: -o user_xattr,flock oleg416-server@tcp:/lustre /mnt/lustre2 Started clients oleg416-client.virtnet: 192.168.204.116@tcp:/lustre on /mnt/lustre2 type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt) oleg416-client.virtnet: executing check_config_client /mnt/lustre oleg416-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg416-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9a01c7fbe000.idle_timeout=debug osc.lustre-OST0000-osc-ffff9a01d93a6000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9a01c7fbe000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9a01d93a6000.idle_timeout=debug disable quota as required oleg416-server: oleg416-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0175514 s, 59.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: [true] running as uid/gid/euid/egid 500/500/500/500, groups: [touch] [/mnt/lustre/d0_runas_test/f7143] == sanityn test 1: Check attribute updates on 2 mount points ========================================================== 04:18:13 (1761121093) /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0777 OK /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0666 OK /mnt/lustre/f1.sanityn: absent OK PASS 1 (4s) == sanityn test 2a: check cached attribute updates on 2 mtpt's ================================================================== 04:18:18 (1761121098) -rw-r--r-- 1 root root 0 Oct 22 04:18 /mnt/lustre2/f2a /mnt/lustre/f2a has type file OK /mnt/lustre/f2a has perms 0777 OK PASS 2a (5s) == sanityn test 2b: check cached attribute updates on 2 mtpt's ================================================================== 04:18:23 (1761121103) -rw-r--r-- 1 root root 0 Oct 22 04:18 /mnt/lustre2/f2b /mnt/lustre2/f2b has type file OK /mnt/lustre2/f2b has perms 0777 OK PASS 2b (6s) == sanityn test 2c: check cached attribute updates on 2 mtpt's root ============================================================= 04:18:29 (1761121109) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0777 OK PASS 2c (6s) == sanityn test 2d: check cached attribute updates on 2 mtpt's root ============================================================= 04:18:35 (1761121115) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0755 OK PASS 2d (7s) == sanityn test 2e: check chmod on root is propagated to others ========================================================== 04:18:42 (1761121122) total 1 -rwxrwxrwx 1 root root 0 Oct 22 04:18 f2a -rwxrwxrwx 1 root root 0 Oct 22 04:18 f2b total 1 -rwxrwxrwx 1 root root 0 Oct 22 04:18 f2a -rwxrwxrwx 1 root root 0 Oct 22 04:18 f2b running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/dev/zero] [of=/mnt/lustre2/f2e.sanityn] [count=1] 1+0 records in 1+0 records out 512 bytes copied, 0.0245048 s, 20.9 kB/s PASS 2e (7s) == sanityn test 2f: check attr/owner updates on DNE with 2 mtpt's ========================================================== 04:18:49 (1761121129) SKIP: sanityn test_2f needs >= 2 MDTs SKIP 2f (4s) == sanityn test 2g: check blocks update on sync write ==== 04:18:53 (1761121133) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.452142 s, 4.6 MB/s /mnt/lustre/f2g.sanityn has 4103 blocks /mnt/lustre2/f2g.sanityn has 4103 blocks PASS 2g (7s) == sanityn test 3: symlink on one mtpt, readlink on another ===================================================================== 04:19:01 (1761121141) PASS 3 (8s) == sanityn test 4: fstat validation on multiple mount points ==================================================================== 04:19:09 (1761121149) PASS 4 (9s) == sanityn test 5: create a file on one mount, truncate it on the other ========================================================== 04:19:18 (1761121158) /mnt/lustre/f5 has type file OK /mnt/lustre/f5 has size 100 OK PASS 5 (7s) == sanityn test 6: remove of open file on other node ============================================================================ 04:19:25 (1761121165) opening writing unlinking /mnt/lustre2/f6.sanityn accessing (1) seeking (1) accessing (2) fstat... reading comparing data truncating seeking (2) writing again seeking (3) reading again comparing data again closing SUCCESS - goto beer PASS 6 (8s) == sanityn test 7: remove of open directory on other node ======================================================================= 04:19:33 (1761121173) creating directory /mnt/lustre/d7 opening directory unlinking /mnt/lustre/d7 Ok, everything goes well. PASS 7 (6s) == sanityn test 8: remove of open special file on other node ==================================================================== 04:19:39 (1761121179) creating special file /mnt/lustre/f8.sanityn opening file unlinking /mnt/lustre/f8.sanityn Ok, everything goes well. PASS 8 (6s) == sanityn test 9a: append of file with sub-page size on multiple mounts ========================================================== 04:19:45 (1761121185) PASS 9a (6s) == sanityn test 9b: append to striped sparse file ======== 04:19:51 (1761121191) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0904506 s, 11.6 MB/s 3+0 records in 3+0 records out 3 bytes copied, 0.0856522 s, 0.0 kB/s Data read (expecting 'foo'): foo PASS 9b (5s) == sanityn test 10a: write of file with sub-page size on multiple mounts ========================================================== 04:19:56 (1761121196) 1+0 records in 1+0 records out 1 byte copied, 0.00740025 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00845708 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00708474 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.005915 s, 0.2 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00502718 s, 0.2 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00608667 s, 0.2 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00885099 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00419905 s, 0.2 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0028106 s, 0.4 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00623168 s, 0.2 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00590474 s, 0.2 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00743342 s, 0.1 kB/s PASS 10a (5s) == sanityn test 10b: write of file with sub-page size on multiple mounts ========================================================== 04:20:01 (1761121201) 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.0147972 s, 208 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0314443 s, 130 kB/s 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.000288004 s, 10.7 MB/s PASS 10b (4s) == sanityn test 11: execution of file opened for write should return error ============================================================== 04:20:05 (1761121205) multiop /mnt/lustre/d11/f vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7143 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 273: /mnt/lustre2/d11/f: Text file busy PASS 11 (2s) == sanityn test 12: test lock ordering (link, stat, unlink) ========================================================== 04:20:07 (1761121207) warning: '-runtime' deprecated, use '-t runtime' instead - link 746 (time 1761121218.84 total 10.04 last 74.30) - link 1135 (time 1761121228.84 total 20.04 last 38.90) - link 1596 (time 1761121238.86 total 30.06 last 46.02) - link 2045 (time 1761121248.86 total 40.06 last 44.88) - link 2769 (time 1761121258.87 total 50.07 last 72.38) - link 4284 (time 1761121268.87 total 60.07 last 151.38) - link 6127 (time 1761121278.88 total 70.08 last 184.22) - link 7768 (time 1761121288.89 total 80.09 last 163.97) - link 9400 (time 1761121298.89 total 90.09 last 163.14) - link 10000 (time 1761121302.60 total 93.80 last 161.66) total: 11092 link in 100.01 seconds: 110.91 ops/second using seed 2963355471 running for 100 seconds - stat 10000 (time 1761121214 ; total 5 ; last 5) - stat 20000 (time 1761121222 ; total 13 ; last 8) - stat 30000 (time 1761121231 ; total 22 ; last 9) - stat 40000 (time 1761121239 ; total 30 ; last 8) - stat 50000 (time 1761121246 ; total 37 ; last 7) - stat 60000 (time 1761121254 ; total 45 ; last 8) - stat 70000 (time 1761121259 ; total 50 ; last 5) - stat 80000 (time 1761121269 ; total 60 ; last 10) - stat 89951 (time 1761121280 ; total 71 ; last 11) - stat 90000 (time 1761121280 ; total 71 ; last 0) - stat 99775 (time 1761121291 ; total 82 ; last 11) - stat 100000 (time 1761121291 ; total 82 ; last 0) - stat 109786 (time 1761121302 ; total 93 ; last 11) - stat 110000 (time 1761121302 ; total 93 ; last 0) total: 116338 stats in 100 seconds: 1163.380005 stats/second - unlinked 0 (time 1761121259 ; total 0 ; last 0) - unlinked 10000 (time 1761121323 ; total 64 ; last 64) unlink(/mnt/lustre2/lockdir/lockfile11092) error: No such file or directory total: 11091 unlinks in 67 seconds: 165.537308 unlinks/second /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 77: kill: (19715) - No such process /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 78: kill: (19718) - No such process PASS 12 (120s) == sanityn test 13: test directory page revocation ======= 04:22:07 (1761121327) f13 f13 PASS 13 (2s) == sanityn test 14aa: execution of file open for write returns -ETXTBSY ========================================================== 04:22:10 (1761121330) multiop /mnt/lustre/d14aa.sanityn/f14aa.sanityn vOw_c TMPPIPE=/tmp/multiop_open_wait_pipe.7143 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 308: /mnt/lustre2/d14aa.sanityn/f14aa.sanityn: Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4298: 20886 Terminated $MULTIOP_PROG $FILE v$ARGS > $TMPPIPE (wd: ~) PASS 14aa (2s) == sanityn test 14ab: open(RDWR) of executing file returns -ETXTBSY ========================================================== 04:22:12 (1761121332) open(O_RDWR|O_CREAT): Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4298: 21460 Terminated $DIR1/$tdir/sleep 60 (wd: ~) PASS 14ab (2s) == sanityn test 14b: truncate of executing file returns -ETXTBSY ================================================================ 04:22:14 (1761121334) truncate: cannot truncate '/mnt/lustre2/d14b.sanityn/sleep' to length 60: Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 323: 22032 Terminated $DIR1/$tdir/sleep 60 PASS 14b (2s) == sanityn test 14c: open(O_TRUNC) of executing file return -ETXTBSY ============================================================ 04:22:16 (1761121336) cp: cannot create regular file '/mnt/lustre2/d14c.sanityn/sleep': Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 335: 22607 Terminated $DIR1/$tdir/sleep 60 PASS 14c (2s) == sanityn test 14d: chmod of executing file is still possible ================================================================== 04:22:18 (1761121338) chmod /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 346: 23182 Terminated $DIR1/$tdir/sleep 60 PASS 14d (3s) == sanityn test 15: test out-of-space with multiple writers ===================================================================== 04:22:21 (1761121341) PATH=/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin: STRIPECOUNT=2 ORIGFREE=7519232 MAXFREE=800000 SKIP: oos2 test_15 oos2.sh: 7519232kB free gt MAXFREE 800000kB, increase 800000 (or reduce test fs size) to proceed sleep 5 for ZFS MDS Waiting for MDT destroys to complete SKIP 15 (11s) == sanityn test 16a: 500 iterations of dual-mount fsx ==== 04:22:33 (1761121353) fallocate on zfs doesn't consume space fallocate not supported 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.100108 s, 20.9 MB/s 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0713391 s, 29.4 MB/s Chance of close/open is 1 in 50 Seed set to 1354 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported truncating to largest ever: 0x175d4a 000001[1] 1761121354.378540 trunc from 00000000 to 0x175d49 (0x175d4a bytes) 000005[1] 1761121354.391250 mapread 0x083f07 thru 0x08e1d0 (0xa2ca bytes) truncating to largest ever: 0x9ab9fe 000006[1] 1761121354.407396 trunc from 0x175d4a to 0x9ab9fd (0x835cb4 bytes) 000007[1] 1761121354.415625 trunc from 0x9ab9fe to 0x086779 (0x925284 bytes) 000008[0] 1761121354.425074 mapread 0x0837f8 thru 0x086779 (0x2f82 bytes) 000009[1] 1761121354.440403 trunc from 0x08677a to 0x5ffdd5 (0x57965c bytes) truncating to largest ever: 0x9d61d1 000010[0] 1761121354.457612 trunc from 0x5ffdd6 to 0x9d61d0 (0x3d63fb bytes) 000011[1] 1761121354.476443 read 0x9c92f8 thru 0x9d1f28 (0x8c31 bytes) 000013[1] 1761121354.493456 mapread 0x445cf7 thru 0x455abc (0xfdc6 bytes) 000014[1] 1761121354.511448 read 0x38922f thru 0x38bbff (0x29d1 bytes) 000016[1] 1761121354.545994 trunc from 0x9d61d1 to 0x7c5375 (0x210e5b bytes) 000017[0] 1761121354.569994 mapread 0x3cea86 thru 0x3da9de (0xbf59 bytes) 000019[0] 1761121354.596336 mapread 0x4c2d9c thru 0x4d0bb1 (0xde16 bytes) 000020[0] 1761121354.610473 write 0x596002 thru 0x59a126 (0x4125 bytes) 000022[0] 1761121354.632451 write 0x4bbeff thru 0x4c5e48 (0x9f4a bytes) 000024[0] 1761121354.647122 write 0x2d753d thru 0x2e5ddb (0xe89f bytes) 000025[0] 1761121354.658647 mapread 0x201c1c thru 0x202673 (0xa58 bytes) 000026[0] 1761121354.672030 mapread 0x3663da thru 0x375e50 (0xfa77 bytes) 000028[1] 1761121354.687540 write 0x1fe294 thru 0x203e0a (0x5b77 bytes) 000029[0] 1761121354.705801 write 0x684ae3 thru 0x68e01e (0x953c bytes) 000030[1] 1761121354.720066 mapwrite 0x15941d thru 0x165227 (0xbe0b bytes) 000032[0] 1761121354.785020 write 0x035475 thru 0x036fb5 (0x1b41 bytes) 000033[1] 1761121354.790580 read 0x47a0e9 thru 0x487261 (0xd179 bytes) 000035[1] 1761121354.820430 trunc from 0x7c5376 to 0x8ccdcb (0x107a56 bytes) 000037[1] 1761121354.838191 trunc from 0x8ccdcc to 0x0748bc (0x85850f bytes) 000040[0] 1761121354.859048 mapread 0x01a470 thru 0x029e7b (0xfa0c bytes) 000042[1] 1761121354.877015 trunc from 0x0748bd to 0x105be6 (0x9132a bytes) 000045[1] 1761121354.894687 write 0x7e19d1 thru 0x7ed3e1 (0xba11 bytes) 000049[1] 1761121354.918328 read 0x24be80 thru 0x2504b6 (0x4637 bytes) 000052[1] 1761121354.972698 mapwrite 0x13bd04 thru 0x13bd8b (0x88 bytes) 000056[0] 1761121355.007516 mapread 0x7c30f8 thru 0x7d1cad (0xebb6 bytes) 000058[0] 1761121355.036463 mapwrite 0x889216 thru 0x89594f (0xc73a bytes) 000062[1] 1761121355.083243 write 0x10b6cb thru 0x1184ae (0xcde4 bytes) 000064[0] 1761121355.101834 trunc from 0x895950 to 0x470fc9 (0x424986 bytes) 000065[0] 1761121355.115490 read 0x1e1696 thru 0x1e3e43 (0x27ae bytes) 000066[0] 1761121355.118457 trunc from 0x470fca to 0x5a9417 (0x13844e bytes) 000067[1] 1761121355.125170 mapread 0x1ecb27 thru 0x1fa46e (0xd948 bytes) 000072[0] 1761121355.164698 trunc from 0x5a9418 to 0x466e4d (0x1425ca bytes) 000073[1] 1761121355.173481 trunc from 0x466e4e to 0x178b00 (0x2ee34d bytes) 000075[1] 1761121355.193380 write 0x6ffad4 thru 0x70afbb (0xb4e8 bytes) 000077[1] 1761121355.203637 write 0x54ca9f thru 0x54e1a6 (0x1708 bytes) 000078[1] 1761121355.207674 mapwrite 0x9aba5b thru 0x9b4f35 (0x94db bytes) 000080[0] 1761121355.251842 mapwrite 0x18e095 thru 0x19ae1e (0xcd8a bytes) 000082[0] 1761121355.310298 mapwrite 0x6df1c9 thru 0x6ec8d0 (0xd708 bytes) 000083[1] 1761121355.356965 mapwrite 0x01dd75 thru 0x02bcbd (0xdf49 bytes) 000084[1] 1761121355.410060 trunc from 0x9b4f36 to 0x7b4c7a (0x2002bb bytes) 000087[1] 1761121355.419517 trunc from 0x7b4c7b to 0x92598c (0x170d12 bytes) 000090[1] 1761121355.439554 trunc from 0x92598d to 0x63307c (0x2f2910 bytes) 000091[0] 1761121355.452000 mapread 0x5ba7ea thru 0x5c0fb9 (0x67d0 bytes) 000092[1] 1761121355.467317 read 0x16873b thru 0x16bc06 (0x34cc bytes) 000094[1] 1761121355.502135 trunc from 0x63307d to 0x702003 (0xcef87 bytes) 000095[1] 1761121355.514598 read 0x1bb373 thru 0x1c0a03 (0x5691 bytes) 000096[1] 1761121355.537774 mapread 0x2753ea thru 0x27bac4 (0x66db bytes) 000097[1] 1761121355.546288 write 0x6073d8 thru 0x60e3ef (0x7018 bytes) 000098[0] 1761121355.552971 mapwrite 0x611f5a thru 0x6194f6 (0x759d bytes) 000099[0] 1761121355.590032 trunc from 0x702004 to 0x650eca (0xb1139 bytes) 000100[0] 1761121355.602361 trunc from 0x650ecb to 0x149bfd (0x5072cd bytes) 000101[1] 1761121355.611294 read 0x02831f thru 0x033c45 (0xb927 bytes) 000103[0] 1761121355.626975 mapwrite 0x1416e6 thru 0x144db5 (0x36d0 bytes) 000105[1] 1761121355.666407 mapwrite 0x2a181c thru 0x2acd1c (0xb501 bytes) 000106[0] 1761121355.714385 read 0x04b80d thru 0x05024b (0x4a3f bytes) 000107[1] 1761121355.729570 write 0x87d5c9 thru 0x889d20 (0xc758 bytes) 000108[0] 1761121355.740745 read 0x0110ea thru 0x019e0b (0x8d22 bytes) 000110[0] 1761121355.769471 trunc from 0x889d21 to 0x886374 (0x39ac bytes) 000111[0] 1761121355.789649 write 0x9970e5 thru 0x998268 (0x1184 bytes) 000113[1] 1761121355.793004 write 0x0f61f6 thru 0x101375 (0xb180 bytes) 000114[1] 1761121355.813701 mapread 0x3a92ea thru 0x3aabc3 (0x18da bytes) truncating to largest ever: 0x9f053b 000115[1] 1761121355.822808 trunc from 0x998269 to 0x9f053a (0x582d2 bytes) 000121[0] 1761121355.858740 read 0x7f3672 thru 0x7f6c58 (0x35e7 bytes) 000124[0] 1761121355.867127 mapwrite 0x605775 thru 0x610116 (0xa9a2 bytes) 000125[0] 1761121355.934509 mapwrite 0x3247ed thru 0x3297cb (0x4fdf bytes) 000127[1] 1761121355.963945 trunc from 0x9f053b to 0x1517a0 (0x89ed9a bytes) 000129[0] 1761121355.978985 mapread 0x101c11 thru 0x103766 (0x1b56 bytes) 000130[1] 1761121355.991076 read 0x070a0b thru 0x071b74 (0x116a bytes) 000131[1] 1761121355.999979 mapwrite 0x61e6b3 thru 0x62066d (0x1fbb bytes) 000132[0] 1761121356.064488 trunc from 0x62066e to 0x4a0938 (0x17fd35 bytes) 000135[1] 1761121356.097225 mapwrite 0x56e239 thru 0x57cd46 (0xeb0e bytes) 000136[0] 1761121356.171974 trunc from 0x57cd47 to 0x4eae7e (0x91ec8 bytes) 000140[1] 1761121356.192653 mapread 0x01574d thru 0x02020d (0xaac1 bytes) 000143[0] 1761121356.226028 read 0x07d114 thru 0x07fe0d (0x2cfa bytes) 000145[1] 1761121356.262594 trunc from 0x4eae7f to 0x4c0df8 (0x2a086 bytes) 000148[0] 1761121356.292072 mapread 0x21f8fa thru 0x22a72e (0xae35 bytes) 000152[1] 1761121356.335762 read 0x1fce3b thru 0x203854 (0x6a1a bytes) 000154[0] 1761121356.365448 mapread 0x224f8a thru 0x232108 (0xd17f bytes) 000155[0] 1761121356.385753 mapwrite 0x6ae205 thru 0x6b1514 (0x3310 bytes) 000156[0] 1761121356.450557 mapwrite 0x9b912d thru 0x9ba520 (0x13f4 bytes) 000157[1] 1761121356.484360 write 0x3dfd33 thru 0x3e8654 (0x8922 bytes) 000158[0] 1761121356.502430 trunc from 0x9ba521 to 0x0f73f1 (0x8c312f bytes) 000159[1] 1761121356.523209 read 0x02c9ff thru 0x03317d (0x677f bytes) 000160[0] 1761121356.537462 read 0x04d7cf thru 0x05b5cd (0xddff bytes) 000161[0] 1761121356.545175 read 0x0579d5 thru 0x065a70 (0xe09c bytes) 000163[1] 1761121356.550504 write 0x21dc11 thru 0x21fb69 (0x1f59 bytes) 000165[1] 1761121356.564637 mapread 0x13c349 thru 0x143f12 (0x7bca bytes) 000166[0] 1761121356.585729 mapread 0x2146f4 thru 0x21fb69 (0xb476 bytes) 000167[0] 1761121356.613727 read 0x0ca3bb thru 0x0ca474 (0xba bytes) 000169[1] 1761121356.623191 mapread 0x00ac8e thru 0x00e03b (0x33ae bytes) 000171[1] 1761121356.631130 mapwrite 0x35993f thru 0x35dc48 (0x430a bytes) 000172[1] 1761121356.667902 mapwrite 0x036325 thru 0x036f68 (0xc44 bytes) 000173[0] 1761121356.691115 write 0x07b011 thru 0x083583 (0x8573 bytes) 000174[1] 1761121356.711356 write 0x0bbd9a thru 0x0c0c62 (0x4ec9 bytes) 000175[0] 1761121356.721981 trunc from 0x35dc49 to 0x6dda5b (0x37fe13 bytes) 000176[0] 1761121356.743454 write 0x7a2f11 thru 0x7ac920 (0x9a10 bytes) 000177[0] 1761121356.745915 write 0x093b2b thru 0x09677c (0x2c52 bytes) 000179[0] 1761121356.786431 trunc from 0x7ac921 to 0x065bc3 (0x746d5d bytes) 000180[0] 1761121356.800898 write 0x92d461 thru 0x9398ab (0xc44b bytes) 000181[0] 1761121356.812575 mapwrite 0x853fff thru 0x85ec6e (0xac70 bytes) 000185[1] 1761121356.845459 write 0x6e953f thru 0x6f4a33 (0xb4f5 bytes) 000186[1] 1761121356.860408 trunc from 0x9398ac to 0x1447d7 (0x7f50d4 bytes) 000188[0] 1761121356.884871 write 0x9a1524 thru 0x9ad679 (0xc156 bytes) 000189[1] 1761121356.897119 write 0x690070 thru 0x69e90f (0xe8a0 bytes) 000190[1] 1761121356.903160 write 0x57272e thru 0x581f87 (0xf85a bytes) 000191[1] 1761121356.912403 read 0x1204b8 thru 0x12cba4 (0xc6ed bytes) 000192[0] 1761121356.966176 trunc from 0x9ad67a to 0x8e5c51 (0xc7a28 bytes) 000193[0] 1761121356.994864 read 0x3d9c94 thru 0x3daad6 (0xe43 bytes) 000194[1] 1761121357.032497 write 0x3a5760 thru 0x3a7f15 (0x27b6 bytes) 000195[0] 1761121357.045156 mapread 0x1128f5 thru 0x11d4bb (0xabc7 bytes) 000197[1] 1761121357.068376 read 0x5fd2fd thru 0x602381 (0x5085 bytes) 000200[1] 1761121357.098876 trunc from 0x8e5c52 to 0x1650c6 (0x780b8b bytes) 000202[1] 1761121357.115363 trunc from 0x1650c7 to 0x414b0c (0x2afa46 bytes) 000203[1] 1761121357.120887 read 0x298a7f thru 0x2a72b6 (0xe838 bytes) 000204[1] 1761121357.132439 read 0x1f39a0 thru 0x1f68b5 (0x2f16 bytes) 000207[0] 1761121357.144849 trunc from 0x414b0d to 0x6381ec (0x2236e0 bytes) 000210[0] 1761121357.173808 read 0x555c11 thru 0x5592a3 (0x3693 bytes) 000213[0] 1761121357.187466 trunc from 0x6381ed to 0x0104c3 (0x627d29 bytes) 000216[0] 1761121357.207571 read 0x00c1ef thru 0x0104c3 (0x42d5 bytes) 000219[1] 1761121357.223578 read 0x00e052 thru 0x0104c3 (0x2472 bytes) 000220[0] 1761121357.231525 write 0x9c9dd7 thru 0x9cde15 (0x403f bytes) 000222[0] 1761121357.241348 mapread 0x72de0e thru 0x736808 (0x89fb bytes) 000227[0] 1761121357.311956 write 0x34ffcb thru 0x358c69 (0x8c9f bytes) 000228[0] 1761121357.312822 mapwrite 0x8c5891 thru 0x8cf3ef (0x9b5f bytes) 000229[1] 1761121357.381105 mapwrite 0x373e8b thru 0x37d102 (0x9278 bytes) 000230[1] 1761121357.439446 trunc from 0x9cde16 to 0x97643c (0x579d9 bytes) 000233[1] 1761121357.459434 mapread 0x216c6a thru 0x223bde (0xcf75 bytes) 000237[0] 1761121357.528106 mapwrite 0x98e33f thru 0x99a90d (0xc5cf bytes) 000238[1] 1761121357.590008 read 0x121575 thru 0x12e010 (0xca9c bytes) 000239[0] 1761121357.605169 trunc from 0x99a90e to 0x4e7889 (0x4b3084 bytes) 000240[0] 1761121357.621842 write 0x2fe4de thru 0x30c683 (0xe1a6 bytes) 000241[1] 1761121357.628790 read 0x1b0c97 thru 0x1bf8be (0xec28 bytes) 000242[0] 1761121357.646484 mapwrite 0x22f66b thru 0x230066 (0x9fc bytes) 000244[1] 1761121357.673667 read 0x0e4fe3 thru 0x0efa56 (0xaa74 bytes) 000249[1] 1761121357.704281 trunc from 0x4e788a to 0x7837ba (0x29bf31 bytes) 000250[1] 1761121357.726429 trunc from 0x7837bb to 0x9ec81a (0x269060 bytes) 000255[0] 1761121357.745888 read 0x8f047b thru 0x8faf4c (0xaad2 bytes) 000262[1] 1761121357.764930 mapwrite 0x06560f thru 0x06c089 (0x6a7b bytes) 000267[1] 1761121357.836149 mapread 0x764c1f thru 0x766850 (0x1c32 bytes) 000268[0] 1761121357.836628 mapread 0x9983b6 thru 0x9a08c6 (0x8511 bytes) 000269[0] 1761121357.856569 write 0x12a5ba thru 0x12a6c3 (0x10a bytes) 000273[0] 1761121357.876487 mapread 0x643f0c thru 0x64fcb0 (0xbda5 bytes) 000274[1] 1761121357.898252 trunc from 0x9ec81b to 0x9d028e (0x1c58c bytes) 000275[0] 1761121357.915041 mapwrite 0x3507f7 thru 0x357e8b (0x7695 bytes) 000276[1] 1761121357.955635 read 0x91a584 thru 0x91c158 (0x1bd5 bytes) 000277[0] 1761121357.974345 mapwrite 0x8a4514 thru 0x8a4a5c (0x549 bytes) 000279[1] 1761121357.995959 read 0x48ed9a thru 0x49083a (0x1aa1 bytes) 000280[0] 1761121358.032617 mapread 0x466756 thru 0x474ded (0xe698 bytes) 000281[0] 1761121358.057405 read 0x4b61e2 thru 0x4bbaf3 (0x5912 bytes) 000282[0] 1761121358.098011 mapwrite 0x31ce83 thru 0x3202bc (0x343a bytes) 000287[1] 1761121358.139133 mapwrite 0x4273f8 thru 0x42b8bf (0x44c8 bytes) 000288[0] 1761121358.193236 write 0x93f10a thru 0x940614 (0x150b bytes) 000292[1] 1761121358.217920 write 0x752725 thru 0x753349 (0xc25 bytes) 000293[1] 1761121358.226040 write 0x0bb510 thru 0x0bcc74 (0x1765 bytes) 000294[1] 1761121358.235745 mapread 0x0d4a4d thru 0x0d5565 (0xb19 bytes) 000295[0] 1761121358.240061 trunc from 0x9d028f to 0x8e4310 (0xebf7e bytes) 000297[0] 1761121358.257488 write 0x6038d9 thru 0x60c853 (0x8f7b bytes) 000299[0] 1761121358.262904 write 0x074174 thru 0x0821d4 (0xe061 bytes) 000302[1] 1761121358.270622 mapwrite 0x6f5be7 thru 0x6f8427 (0x2841 bytes) 000303[0] 1761121358.320557 mapread 0x224ee0 thru 0x226096 (0x11b7 bytes) 000307[0] 1761121358.349420 write 0x51b3ce thru 0x52130f (0x5f42 bytes) 000308[0] 1761121358.360679 write 0x861d77 thru 0x867fc9 (0x6253 bytes) 000309[1] 1761121358.374698 write 0x8832e6 thru 0x88a5c6 (0x72e1 bytes) 000310[0] 1761121358.389597 read 0x4a8980 thru 0x4b4aa1 (0xc122 bytes) 000314[1] 1761121358.433297 write 0x5c79fc thru 0x5ce818 (0x6e1d bytes) 000315[1] 1761121358.445512 read 0x05a9e6 thru 0x0642cc (0x98e7 bytes) 000318[0] 1761121358.517894 mapwrite 0x86f1c8 thru 0x874bfe (0x5a37 bytes) 000319[0] 1761121358.573212 write 0x1128d4 thru 0x113925 (0x1052 bytes) 000321[0] 1761121358.586826 mapwrite 0x2d7488 thru 0x2e22a0 (0xae19 bytes) 000323[0] 1761121358.636579 read 0x7ba8d4 thru 0x7c182f (0x6f5c bytes) 000325[1] 1761121358.650644 read 0x46d2de thru 0x47ac30 (0xd953 bytes) 000327[1] 1761121358.667751 mapread 0x176c0e thru 0x1795a8 (0x299b bytes) 000329[0] 1761121358.703347 mapwrite 0x08dc92 thru 0x097261 (0x95d0 bytes) 000330[1] 1761121358.740108 mapread 0x4fb3fd thru 0x504643 (0x9247 bytes) 000331[1] 1761121358.753676 mapread 0x32f319 thru 0x339635 (0xa31d bytes) 000336[0] 1761121358.774374 read 0x6d0c91 thru 0x6dbadd (0xae4d bytes) 000337[1] 1761121358.794038 read 0x6862a9 thru 0x690af5 (0xa84d bytes) 000338[1] 1761121358.800849 write 0x66e7ea thru 0x67bd14 (0xd52b bytes) 000340[1] 1761121358.816146 mapread 0x4a3dcb thru 0x4b2199 (0xe3cf bytes) 000342[1] 1761121358.851686 mapwrite 0x6754aa thru 0x681245 (0xbd9c bytes) 000343[1] 1761121358.881381 trunc from 0x8e4311 to 0x552ca1 (0x39166f bytes) 000345[0] 1761121358.895172 trunc from 0x552ca2 to 0x947b2b (0x3f4e8a bytes) 000348[1] 1761121358.922873 mapwrite 0x576ace thru 0x57f23f (0x8772 bytes) 000349[1] 1761121358.961679 write 0x3949f4 thru 0x3a2def (0xe3fc bytes) 000351[0] 1761121358.973337 mapread 0x692e30 thru 0x69ab85 (0x7d56 bytes) 000352[1] 1761121358.996722 trunc from 0x947b2c to 0x829703 (0x11e428 bytes) 000354[0] 1761121359.011652 mapwrite 0x50a730 thru 0x519116 (0xe9e7 bytes) 000357[0] 1761121359.071673 mapwrite 0x4527e8 thru 0x455745 (0x2f5e bytes) 000359[1] 1761121359.099811 mapwrite 0x70cce7 thru 0x711ab6 (0x4dd0 bytes) 000360[1] 1761121359.139753 trunc from 0x829704 to 0x3f7957 (0x431dac bytes) 000361[0] 1761121359.151356 read 0x1f74cb thru 0x205508 (0xe03e bytes) 000362[0] 1761121359.181751 mapread 0x0b8f65 thru 0x0bc2a2 (0x333e bytes) 000363[1] 1761121359.194024 read 0x00e515 thru 0x012fc9 (0x4ab5 bytes) 000364[1] 1761121359.224459 trunc from 0x3f7958 to 0x5f72a4 (0x1ff94d bytes) 000365[0] 1761121359.248667 trunc from 0x5f72a5 to 0x39a35e (0x25cf46 bytes) 000366[1] 1761121359.261967 write 0x1fd007 thru 0x1ff1ea (0x21e4 bytes) 000367[0] 1761121359.270603 read 0x1f0b43 thru 0x1f1722 (0xbe0 bytes) 000369[1] 1761121359.294343 mapread 0x023d78 thru 0x029db9 (0x6042 bytes) 000370[1] 1761121359.312907 mapread 0x1e7356 thru 0x1f2b3f (0xb7ea bytes) 000372[0] 1761121359.335060 mapwrite 0x365083 thru 0x36d535 (0x84b3 bytes) 000373[1] 1761121359.380024 mapwrite 0x6694ae thru 0x674aba (0xb60d bytes) 000375[0] 1761121359.435343 read 0x41b581 thru 0x4212b0 (0x5d30 bytes) 000383[0] 1761121359.476753 read 0x013dcd thru 0x018d82 (0x4fb6 bytes) 000384[1] 1761121359.487574 mapread 0x2bbab6 thru 0x2c1796 (0x5ce1 bytes) 000386[1] 1761121359.510006 write 0x04f6c3 thru 0x05a813 (0xb151 bytes) 000391[1] 1761121359.535479 read 0x161b54 thru 0x16b7af (0x9c5c bytes) 000397[1] 1761121359.580483 mapread 0x03de36 thru 0x04c04c (0xe217 bytes) 000398[1] 1761121359.604548 mapread 0x3125dd thru 0x3179d3 (0x53f7 bytes) 000400[0] 1761121359.610315 mapread 0x16914b thru 0x171f09 (0x8dbf bytes) 000401[0] 1761121359.627592 read 0x4d7228 thru 0x4e62aa (0xf083 bytes) 000402[1] 1761121359.647608 mapwrite 0x73d5fe thru 0x74b6eb (0xe0ee bytes) 000404[0] 1761121359.692965 mapread 0x3b0820 thru 0x3b473c (0x3f1d bytes) 000406[0] 1761121359.709803 trunc from 0x74b6ec to 0x7a5c81 (0x5a596 bytes) 000408[1] 1761121359.730463 read 0x2a941a thru 0x2b81f7 (0xedde bytes) 000409[1] 1761121359.741533 read 0x5e3928 thru 0x5f265a (0xed33 bytes) 000410[0] 1761121359.759641 write 0x3e6c0c thru 0x3f345e (0xc853 bytes) 000414[0] 1761121359.783657 mapread 0x39c822 thru 0x3aa276 (0xda55 bytes) 000415[0] 1761121359.813497 mapread 0x237291 thru 0x24285a (0xb5ca bytes) 000416[0] 1761121359.834216 trunc from 0x7a5c82 to 0x25a8b7 (0x54b3ca bytes) 000417[1] 1761121359.852942 mapwrite 0x781403 thru 0x787c94 (0x6892 bytes) 000418[1] 1761121359.905218 mapread 0x1a8006 thru 0x1a87ad (0x7a8 bytes) 000419[1] 1761121359.913278 write 0x0f275f thru 0x0f6deb (0x468d bytes) 000420[0] 1761121359.921390 read 0x0cbd0d thru 0x0d3383 (0x7677 bytes) 000422[0] 1761121359.936991 read 0x1875a6 thru 0x188435 (0xe90 bytes) 000423[1] 1761121359.985009 mapwrite 0x88f8c6 thru 0x8916d0 (0x1e0b bytes) 000424[0] 1761121360.037991 trunc from 0x8916d1 to 0x410353 (0x48137d bytes) 000425[1] 1761121360.053459 mapwrite 0x1570dc thru 0x160094 (0x8fb9 bytes) 000426[1] 1761121360.096430 mapwrite 0x7f762e thru 0x7fc245 (0x4c18 bytes) 000427[0] 1761121360.134830 mapwrite 0x455244 thru 0x4586a0 (0x345d bytes) 000428[0] 1761121360.166322 read 0x381c2b thru 0x38596d (0x3d43 bytes) 000432[0] 1761121360.213181 trunc from 0x7fc246 to 0x5512d6 (0x2aaf6f bytes) 000436[1] 1761121360.240315 write 0x884b78 thru 0x884db9 (0x242 bytes) 000437[0] 1761121360.250871 trunc from 0x884dba to 0x2416e5 (0x6436d4 bytes) 000438[0] 1761121360.267461 read 0x218e1c thru 0x221687 (0x886c bytes) 000439[0] 1761121360.272871 mapwrite 0x263679 thru 0x270450 (0xcdd8 bytes) 000441[1] 1761121360.297804 trunc from 0x270451 to 0x40f370 (0x19ef20 bytes) 000442[1] 1761121360.315645 mapwrite 0x79539c thru 0x799fad (0x4c12 bytes) 000444[0] 1761121360.352862 mapread 0x71e1c3 thru 0x72d27d (0xf0bb bytes) 000446[0] 1761121360.385178 read 0x43f078 thru 0x4499d0 (0xa959 bytes) 000447[0] 1761121360.409952 trunc from 0x799fae to 0x53b5da (0x25e9d3 bytes) 000451[1] 1761121360.432113 write 0x4b6642 thru 0x4b8e2b (0x27ea bytes) 000452[0] 1761121360.442340 mapwrite 0x0b855a thru 0x0c5f85 (0xda2c bytes) 000454[0] 1761121360.483572 mapwrite 0x712098 thru 0x7125fe (0x567 bytes) 000455[1] 1761121360.510622 mapwrite 0x6734a7 thru 0x677127 (0x3c81 bytes) 000456[1] 1761121360.546218 mapwrite 0x147779 thru 0x14b9fc (0x4284 bytes) 000457[0] 1761121360.574044 trunc from 0x7125ff to 0x720c71 (0xe673 bytes) 000459[0] 1761121360.591761 mapread 0x59d7e2 thru 0x5aab1e (0xd33d bytes) 000460[0] 1761121360.621534 trunc from 0x720c72 to 0x2fccb5 (0x423fbc bytes) 000462[1] 1761121360.635540 read 0x18f6ec thru 0x1909c9 (0x12de bytes) 000463[0] 1761121360.645531 trunc from 0x2fccb6 to 0x56dd68 (0x2710b3 bytes) 000465[1] 1761121360.660175 trunc from 0x56dd69 to 0x535f46 (0x37e22 bytes) 000466[1] 1761121360.670541 trunc from 0x535f47 to 0x20f371 (0x326bd5 bytes) 000467[1] 1761121360.674917 mapread 0x1e90b6 thru 0x1e9868 (0x7b3 bytes) 000468[1] 1761121360.677374 write 0x5787d8 thru 0x5841fe (0xba27 bytes) 000469[1] 1761121360.679055 mapwrite 0x9a939e thru 0x9b005d (0x6cc0 bytes) 000470[0] 1761121360.749247 trunc from 0x9b005e to 0x1278bf (0x88879e bytes) 000471[0] 1761121360.774903 mapwrite 0x776f74 thru 0x77e38c (0x7419 bytes) 000472[0] 1761121360.812316 write 0x42553d thru 0x433a0d (0xe4d1 bytes) 000474[0] 1761121360.821621 trunc from 0x77e38d to 0x1dcb80 (0x5a180c bytes) 000478[0] 1761121360.837431 read 0x15969c thru 0x1625ee (0x8f53 bytes) 000481[0] 1761121360.848236 trunc from 0x1dcb81 to 0x59e39f (0x3c181f bytes) 000482[0] 1761121360.858024 write 0x27e3ce thru 0x2855f7 (0x722a bytes) 000484[1] 1761121360.892476 read 0x35b7fb thru 0x3604ba (0x4cc0 bytes) 000485[1] 1761121360.909670 write 0x5bb6bd thru 0x5c8d62 (0xd6a6 bytes) 000487[1] 1761121360.923525 mapread 0x46fdf7 thru 0x473c25 (0x3e2f bytes) 000490[1] 1761121360.954251 mapwrite 0x53bf7b thru 0x5412c2 (0x5348 bytes) 000491[1] 1761121360.975488 mapwrite 0x261efe thru 0x2646c6 (0x27c9 bytes) 000495[1] 1761121361.008070 write 0x653beb thru 0x655636 (0x1a4c bytes) 000496[1] 1761121361.008800 read 0x4e443b thru 0x4f1ebe (0xda84 bytes) 000497[1] 1761121361.021323 trunc from 0x655637 to 0x40c8bf (0x248d77 bytes) 000500[1] 1761121361.031464 mapread 0x10759e thru 0x10de23 (0x6886 bytes) All operations completed A-OK! Chance of close/open is 1 in 50 Seed set to 1361 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported truncating to largest ever: 0x38fc0 000001[0] 1761121361.301403 trunc from 00000000 to 0x038fbf (0x38fc0 bytes) 000004[0] 1761121361.308047 mapread 0x00a000 thru 0x00ce3d (0x2e3e bytes) 000005[1] 1761121361.314380 read_OD 0x02d000 thru 0x031fff (0x5000 bytes) 000006[0] 1761121361.321315 read 0x025000 thru 0x0266b9 (0x16ba bytes) 000008[1] 1761121361.325242 mapwrite 0x66f000 thru 0x67d2a0 (0xe2a1 bytes) 000013[0] 1761121361.388811 mapwrite 0x7ea000 thru 0x7f9399 (0xf39a bytes) 000014[1] 1761121361.428316 read_OD 0x788000 thru 0x792fff (0xb000 bytes) 000015[1] 1761121361.436203 mapread 0x266000 thru 0x275625 (0xf626 bytes) 000018[1] 1761121361.466668 mapread 0x445000 thru 0x44ed48 (0x9d49 bytes) 000019[0] 1761121361.479717 mapread 0x1ea000 thru 0x1ef59b (0x559c bytes) truncating to largest ever: 0x3b517e 000020[0] 1761121361.485417 trunc from 0x7f939a to 0x3b517d (0x44421c bytes) 000022[1] 1761121361.502614 write_OD 0x754000 thru 0x75ffff (0xc000 bytes) 000031[1] 1761121361.523681 mapread 0x162000 thru 0x16f719 (0xd71a bytes) 000032[0] 1761121361.546578 mapread 0x141000 thru 0x14baa1 (0xaaa2 bytes) 000033[0] 1761121361.552785 read 0x683000 thru 0x68e42e (0xb42f bytes) 000034[0] 1761121361.561930 mapread 0x5b8000 thru 0x5b82e2 (0x2e3 bytes) 000039[0] 1761121361.565342 mapwrite 0x467000 thru 0x46fa91 (0x8a92 bytes) 000040[1] 1761121361.602202 write_OD 0x863000 thru 0x86afff (0x8000 bytes) 000041[1] 1761121361.616353 read_OD 0x44f000 thru 0x454fff (0x6000 bytes) 000042[1] 1761121361.619991 mapwrite 0x910000 thru 0x917328 (0x7329 bytes) 000043[1] 1761121361.680074 read_OD 0x05e000 thru 0x05efff (0x1000 bytes) 000047[1] 1761121361.689568 mapwrite 0x843000 thru 0x84743b (0x443c bytes) 000048[0] 1761121361.768156 read 0x30a000 thru 0x317e21 (0xde22 bytes) 000049[0] 1761121361.823564 mapwrite 0x2e2000 thru 0x2e9a69 (0x7a6a bytes) 000053[1] 1761121361.868444 write_OD 0x5e2000 thru 0x5ecfff (0xb000 bytes) 000055[1] 1761121361.884460 mapwrite 0x27c000 thru 0x280be7 (0x4be8 bytes) 000057[1] 1761121361.920461 mapwrite 0x44e000 thru 0x45b302 (0xd303 bytes) 000063[1] 1761121361.964447 write_OD 0x97c000 thru 0x97ffff (0x4000 bytes) 000064[0] 1761121361.979661 mapwrite 0x581000 thru 0x58ac82 (0x9c83 bytes) 000065[0] 1761121362.015746 mapwrite 0x780000 thru 0x782be1 (0x2be2 bytes) 000066[1] 1761121362.033731 read_OD 0x1b2000 thru 0x1b5fff (0x4000 bytes) truncating to largest ever: 0x7a34c7 000069[0] 1761121362.043394 trunc from 0x980000 to 0x7a34c6 (0x1dcb39 bytes) 000070[0] 1761121362.057189 mapwrite 0x904000 thru 0x909bbe (0x5bbf bytes) 000071[1] 1761121362.096790 write_OD 0x542000 thru 0x543fff (0x2000 bytes) 000073[1] 1761121362.114132 mapwrite 0x330000 thru 0x33a128 (0xa129 bytes) 000075[0] 1761121362.186581 mapread 0x07d000 thru 0x08ce92 (0xfe93 bytes) 000079[1] 1761121362.220756 read_OD 0x075000 thru 0x083fff (0xf000 bytes) 000081[0] 1761121362.224842 trunc from 0x909bbf to 0x3c0313 (0x5498ab bytes) 000082[0] 1761121362.237533 mapwrite 0x250000 thru 0x2584c1 (0x84c2 bytes) 000084[0] 1761121362.272436 trunc from 0x3c0314 to 0x012b84 (0x3ad78f bytes) 000085[0] 1761121362.278185 write 0x1c5000 thru 0x1cc655 (0x7656 bytes) truncating to largest ever: 0x96d942 000086[0] 1761121362.279314 trunc from 0x1cc656 to 0x96d941 (0x7a12ec bytes) 000087[0] 1761121362.289693 mapwrite 0x645000 thru 0x654dee (0xfdef bytes) 000088[0] 1761121362.336648 mapread 0x5d4000 thru 0x5d704a (0x304b bytes) 000089[0] 1761121362.343129 mapwrite 0x088000 thru 0x08ddc4 (0x5dc5 bytes) 000092[0] 1761121362.373204 read 0x4e7000 thru 0x4e800a (0x100b bytes) 000096[1] 1761121362.410431 read_OD 0x813000 thru 0x816fff (0x4000 bytes) 000097[0] 1761121362.423711 read 0x110000 thru 0x11b617 (0xb618 bytes) 000099[1] 1761121362.472048 read_OD 0x7db000 thru 0x7e1fff (0x7000 bytes) 000102[1] 1761121362.475999 read_OD 0x089000 thru 0x08bfff (0x3000 bytes) 000104[1] 1761121362.478965 mapread 0x407000 thru 0x40eb25 (0x7b26 bytes) 000105[0] 1761121362.547861 trunc from 0x96d942 to 0x50b6f8 (0x462249 bytes) 000106[1] 1761121362.579370 read_OD 0x18f000 thru 0x19cfff (0xe000 bytes) 000107[1] 1761121362.587234 read_OD 0x0ce000 thru 0x0cefff (0x1000 bytes) 000109[1] 1761121362.591727 read_OD 0x49b000 thru 0x49ffff (0x5000 bytes) 000112[0] 1761121362.594891 mapwrite 0x632000 thru 0x63c9d8 (0xa9d9 bytes) 000113[1] 1761121362.640571 mapwrite 0x4a5000 thru 0x4b36b9 (0xe6ba bytes) 000114[0] 1761121362.689455 mapwrite 0x823000 thru 0x82ae64 (0x7e65 bytes) 000116[1] 1761121362.732531 mapread 0x48f000 thru 0x49a124 (0xb125 bytes) 000117[1] 1761121362.753439 write_OD 0x783000 thru 0x787fff (0x5000 bytes) 000119[1] 1761121362.775103 trunc from 0x82ae65 to 0x1b8fbf (0x671ea5 bytes) 000120[1] 1761121362.782772 read_OD 0x136000 thru 0x137fff (0x2000 bytes) 000121[0] 1761121362.787433 read 0x12f000 thru 0x137fbc (0x8fbd bytes) 000122[0] 1761121362.794714 read 0x129000 thru 0x137ff9 (0xeffa bytes) 000124[0] 1761121362.798153 write 0x5f2000 thru 0x5f8bd1 (0x6bd2 bytes) 000125[0] 1761121362.808489 mapwrite 0x1f7000 thru 0x1f84e7 (0x14e8 bytes) 000126[1] 1761121362.849883 read_OD 0x2b9000 thru 0x2c4fff (0xc000 bytes) 000127[1] 1761121362.861808 write_OD 0x650000 thru 0x657fff (0x8000 bytes) 000128[1] 1761121362.878182 mapwrite 0x975000 thru 0x978873 (0x3874 bytes) 000130[1] 1761121362.949086 trunc from 0x978874 to 0x0e42c9 (0x8945aa bytes) 000131[0] 1761121362.965146 trunc from 0x0e42ca to 0x2de86c (0x1fa5a3 bytes) 000134[1] 1761121362.977220 mapread 0x038000 thru 0x046a8e (0xea8f bytes) 000135[1] 1761121363.006307 mapread 0x213000 thru 0x21cada (0x9adb bytes) 000136[1] 1761121363.023393 write_OD 0x39b000 thru 0x39cfff (0x2000 bytes) 000137[0] 1761121363.041285 trunc from 0x39d000 to 0x446934 (0xa9935 bytes) 000144[1] 1761121363.067462 trunc from 0x446935 to 0x64792b (0x200ff7 bytes) 000147[0] 1761121363.086401 read 0x134000 thru 0x13abab (0x6bac bytes) 000148[0] 1761121363.122417 mapwrite 0x977000 thru 0x980976 (0x9977 bytes) 000149[1] 1761121363.180405 mapread 0x206000 thru 0x20f98a (0x998b bytes) 000155[0] 1761121363.203039 write 0x3ff000 thru 0x4078fc (0x88fd bytes) 000157[0] 1761121363.207863 read 0x761000 thru 0x768ee6 (0x7ee7 bytes) 000158[1] 1761121363.232450 read_OD 0x343000 thru 0x34cfff (0xa000 bytes) 000159[0] 1761121363.245574 write 0x3d6000 thru 0x3d80ad (0x20ae bytes) 000160[1] 1761121363.250782 mapwrite 0x111000 thru 0x120cf6 (0xfcf7 bytes) 000164[0] 1761121363.353186 read 0x787000 thru 0x78d47a (0x647b bytes) 000165[1] 1761121363.383798 mapwrite 0x935000 thru 0x9428f4 (0xd8f5 bytes) 000167[1] 1761121363.446906 mapwrite 0x34e000 thru 0x35bf34 (0xdf35 bytes) 000168[1] 1761121363.495799 read_OD 0x3df000 thru 0x3e4fff (0x6000 bytes) 000170[1] 1761121363.502080 mapread 0x7a6000 thru 0x7aaf3a (0x4f3b bytes) 000171[0] 1761121363.537880 read 0x876000 thru 0x885565 (0xf566 bytes) 000172[0] 1761121363.549188 trunc from 0x980977 to 0x8071ff (0x179777 bytes) 000174[1] 1761121363.571494 write_OD 0x58a000 thru 0x58cfff (0x3000 bytes) 000175[0] 1761121363.585901 mapwrite 0x0c7000 thru 0x0c70e2 (0xe3 bytes) truncating to largest ever: 0x98c35f 000176[1] 1761121363.604913 trunc from 0x807200 to 0x98c35e (0x18515f bytes) 000177[1] 1761121363.621410 trunc from 0x98c35f to 0x1d5bc4 (0x7b679a bytes) 000178[1] 1761121363.629501 read_OD 0x111000 thru 0x113fff (0x3000 bytes) 000181[1] 1761121363.638997 mapwrite 0x810000 thru 0x812473 (0x2474 bytes) 000182[0] 1761121363.711879 write 0x210000 thru 0x2116cb (0x16cc bytes) 000183[0] 1761121363.732358 read 0x240000 thru 0x247bfa (0x7bfb bytes) 000185[1] 1761121363.788438 read_OD 0x395000 thru 0x395fff (0x1000 bytes) 000186[1] 1761121363.802821 mapread 0x09b000 thru 0x0a32f4 (0x82f5 bytes) 000187[0] 1761121363.845681 read 0x008000 thru 0x00b6cd (0x36ce bytes) 000188[1] 1761121363.894367 mapread 0x0de000 thru 0x0e1c60 (0x3c61 bytes) 000189[1] 1761121363.906715 read_OD 0x152000 thru 0x154fff (0x3000 bytes) 000192[1] 1761121363.909216 mapwrite 0x689000 thru 0x68d332 (0x4333 bytes) 000193[0] 1761121363.998521 read 0x773000 thru 0x774848 (0x1849 bytes) 000194[0] 1761121364.011973 write 0x636000 thru 0x638f9f (0x2fa0 bytes) 000195[0] 1761121364.021215 write 0x6fa000 thru 0x6fdaf8 (0x3af9 bytes) 000196[1] 1761121364.032817 write_OD 0x4b1000 thru 0x4b8fff (0x8000 bytes) 000197[0] 1761121364.059389 read 0x6d3000 thru 0x6dc951 (0x9952 bytes) 000199[1] 1761121364.075688 mapread 0x517000 thru 0x524ca5 (0xdca6 bytes) 000200[1] 1761121364.101040 mapread 0x013000 thru 0x0214f9 (0xe4fa bytes) 000201[0] 1761121364.127398 trunc from 0x812474 to 0x92843e (0x115fcb bytes) 000202[0] 1761121364.137586 write 0x11e000 thru 0x12c4af (0xe4b0 bytes) 000208[0] 1761121364.165488 trunc from 0x92843f to 0x8d1786 (0x56cb8 bytes) 000210[0] 1761121364.176963 mapwrite 0x41e000 thru 0x42bddf (0xdde0 bytes) 000216[0] 1761121364.230634 mapread 0x12b000 thru 0x12f8d6 (0x48d7 bytes) 000217[0] 1761121364.235643 mapwrite 0x26f000 thru 0x273dac (0x4dad bytes) 000218[1] 1761121364.271581 trunc from 0x8d1787 to 0x88054e (0x51238 bytes) 000219[0] 1761121364.284091 mapread 0x149000 thru 0x1502c1 (0x72c2 bytes) 000220[0] 1761121364.301837 mapread 0x0bc000 thru 0x0c654c (0xa54d bytes) 000221[1] 1761121364.319173 write_OD 0x767000 thru 0x767fff (0x1000 bytes) 000223[0] 1761121364.337887 mapread 0x5e6000 thru 0x5e7c5d (0x1c5e bytes) 000224[1] 1761121364.341706 write_OD 0x9a0000 thru 0x9a0fff (0x1000 bytes) 000225[1] 1761121364.363343 mapwrite 0x19a000 thru 0x1a421b (0xa21c bytes) 000227[0] 1761121364.420821 mapwrite 0x5ac000 thru 0x5b9ff4 (0xdff5 bytes) 000231[1] 1761121364.488698 read_OD 0x3e3000 thru 0x3eefff (0xc000 bytes) 000233[1] 1761121364.497708 read_OD 0x041000 thru 0x04afff (0xa000 bytes) 000235[1] 1761121364.500419 mapread 0x045000 thru 0x04de72 (0x8e73 bytes) 000238[1] 1761121364.559540 mapread 0x073000 thru 0x082cc0 (0xfcc1 bytes) 000240[1] 1761121364.560249 write_OD 0x0e8000 thru 0x0effff (0x8000 bytes) 000241[1] 1761121364.589924 trunc from 0x9a1000 to 0x38807c (0x618f83 bytes) 000242[1] 1761121364.603821 write_OD 0x48f000 thru 0x49afff (0xc000 bytes) 000245[0] 1761121364.628610 read 0x32a000 thru 0x338913 (0xe914 bytes) 000246[0] 1761121364.643006 mapread 0x0d8000 thru 0x0e670d (0xe70e bytes) 000248[0] 1761121364.671018 trunc from 0x49b000 to 0x7d790b (0x33c90c bytes) 000250[0] 1761121364.682764 read 0x393000 thru 0x39d94e (0xa94f bytes) 000251[1] 1761121364.711199 trunc from 0x7d790c to 0x465387 (0x372584 bytes) 000252[0] 1761121364.726780 mapwrite 0x0af000 thru 0x0b6197 (0x7198 bytes) 000253[1] 1761121364.765657 write_OD 0x531000 thru 0x53cfff (0xc000 bytes) 000254[0] 1761121364.786832 mapread 0x2ce000 thru 0x2d6b70 (0x8b71 bytes) 000256[0] 1761121364.805532 trunc from 0x53d000 to 0x034517 (0x508ae8 bytes) 000258[1] 1761121364.823793 mapwrite 0x923000 thru 0x92adcd (0x7dce bytes) 000259[1] 1761121364.868372 mapread 0x5a0000 thru 0x5ae8f2 (0xe8f3 bytes) 000260[1] 1761121364.897564 mapwrite 0x69a000 thru 0x6a92ea (0xf2eb bytes) 000261[1] 1761121364.942110 trunc from 0x92adce to 0x5dc4d1 (0x34e8fc bytes) 000262[0] 1761121364.952926 read 0x5b5000 thru 0x5b74f6 (0x24f7 bytes) 000263[0] 1761121364.962026 mapwrite 0x611000 thru 0x61308e (0x208f bytes) 000265[0] 1761121364.999163 mapread 0x346000 thru 0x353408 (0xd409 bytes) 000270[1] 1761121365.036451 mapwrite 0x7ed000 thru 0x7f9f2c (0xcf2d bytes) 000271[1] 1761121365.085156 mapwrite 0x772000 thru 0x7766b5 (0x46b6 bytes) 000274[1] 1761121365.111933 trunc from 0x7f9f2d to 0x80b324 (0x113f8 bytes) 000275[1] 1761121365.119356 trunc from 0x80b325 to 0x735e1c (0xd5508 bytes) 000277[0] 1761121365.124503 mapwrite 0x057000 thru 0x05994f (0x2950 bytes) 000278[0] 1761121365.150655 mapread 0x312000 thru 0x313dc5 (0x1dc6 bytes) 000281[0] 1761121365.158078 mapread 0x1e8000 thru 0x1ec3a0 (0x43a1 bytes) 000283[0] 1761121365.165076 trunc from 0x735e1d to 0x8ee6a8 (0x1b888c bytes) 000284[1] 1761121365.170746 read_OD 0x485000 thru 0x491fff (0xd000 bytes) 000285[1] 1761121365.177813 trunc from 0x8ee6a9 to 0x82b1ba (0xc34ee bytes) 000287[1] 1761121365.187880 mapwrite 0x4d5000 thru 0x4ddd66 (0x8d67 bytes) 000289[1] 1761121365.225799 trunc from 0x82b1bb to 0x375e07 (0x4b53b3 bytes) 000291[1] 1761121365.234987 trunc from 0x375e08 to 0x4897aa (0x1139a3 bytes) 000292[0] 1761121365.243859 write 0x68e000 thru 0x693974 (0x5975 bytes) 000294[1] 1761121365.254489 trunc from 0x693975 to 0x814ba9 (0x181235 bytes) 000296[0] 1761121365.269632 read 0x067000 thru 0x06f05d (0x805e bytes) 000297[1] 1761121365.309448 read_OD 0x716000 thru 0x723fff (0xe000 bytes) 000298[1] 1761121365.319543 read_OD 0x102000 thru 0x10bfff (0xa000 bytes) 000299[1] 1761121365.323487 mapwrite 0x869000 thru 0x874056 (0xb057 bytes) 000307[1] 1761121365.410545 trunc from 0x874057 to 0x0da1cc (0x799e8a bytes) 000309[1] 1761121365.423134 write_OD 0x972000 thru 0x97ffff (0xe000 bytes) 000311[0] 1761121365.443692 trunc from 0x980000 to 0x04c11e (0x933ee1 bytes) 000312[1] 1761121365.452094 write_OD 0x236000 thru 0x243fff (0xe000 bytes) 000313[1] 1761121365.469086 mapwrite 0x465000 thru 0x46c8d9 (0x78da bytes) 000314[0] 1761121365.512294 trunc from 0x46c8da to 0x59bdc4 (0x12f4eb bytes) 000315[0] 1761121365.523914 mapread 0x405000 thru 0x4067d0 (0x17d1 bytes) 000316[0] 1761121365.532396 read 0x398000 thru 0x39a768 (0x2769 bytes) 000321[1] 1761121365.573355 write_OD 0x911000 thru 0x918fff (0x8000 bytes) 000322[0] 1761121365.595384 mapwrite 0x9ab000 thru 0x9b558a (0xa58b bytes) 000324[0] 1761121365.654866 trunc from 0x9b558b to 0x758c3c (0x25c94e bytes) 000326[1] 1761121365.663683 mapread 0x0fe000 thru 0x1064cd (0x84ce bytes) 000328[1] 1761121365.685634 mapread 0x5c6000 thru 0x5cd2c8 (0x72c9 bytes) 000329[0] 1761121365.696885 write 0x3ba000 thru 0x3bf91b (0x591c bytes) 000330[1] 1761121365.701419 mapwrite 0x3c7000 thru 0x3d5794 (0xe795 bytes) 000331[0] 1761121365.752829 read 0x563000 thru 0x570f11 (0xdf12 bytes) 000332[1] 1761121365.775153 mapwrite 0x43c000 thru 0x446ed8 (0xaed9 bytes) 000333[0] 1761121365.822216 mapread 0x378000 thru 0x37d47a (0x547b bytes) 000334[1] 1761121365.834722 mapread 0x5c0000 thru 0x5cd123 (0xd124 bytes) 000337[1] 1761121365.858991 mapwrite 0x9ad000 thru 0x9ba312 (0xd313 bytes) 000338[0] 1761121365.904777 read 0x6bc000 thru 0x6c0197 (0x4198 bytes) 000339[1] 1761121365.928167 write_OD 0x3e7000 thru 0x3ebfff (0x5000 bytes) 000341[0] 1761121365.953028 write 0x486000 thru 0x49339e (0xd39f bytes) 000343[1] 1761121365.962655 trunc from 0x9ba313 to 0x0627d6 (0x957b3c bytes) 000346[1] 1761121365.986656 trunc from 0x0627d7 to 0x01fda3 (0x42a33 bytes) 000348[0] 1761121366.001128 write_OD 0x8d5000 thru 0x8e3fff (0xf000 bytes) 000350[1] 1761121366.019711 trunc from 0x8e4000 to 0x01a494 (0x8c9b6b bytes) 000352[0] 1761121366.032922 mapread 0x004000 thru 0x013c86 (0xfc87 bytes) 000354[0] 1761121366.046849 read_OD 0x016000 thru 0x019fff (0x4000 bytes) 000356[0] 1761121366.049116 mapread 0x00b000 thru 0x0199d0 (0xe9d1 bytes) 000358[1] 1761121366.049898 mapwrite 0x859000 thru 0x85965f (0x660 bytes) 000363[0] 1761121366.086423 trunc from 0x859660 to 0x81407a (0x455e5 bytes) 000364[1] 1761121366.097987 read_OD 0x1b5000 thru 0x1bbfff (0x7000 bytes) 000365[0] 1761121366.105898 read_OD 0x697000 thru 0x69dfff (0x7000 bytes) 000371[0] 1761121366.110360 mapwrite 0x556000 thru 0x55a93a (0x493b bytes) 000372[1] 1761121366.148908 mapwrite 0x349000 thru 0x357c08 (0xec09 bytes) 000373[1] 1761121366.211646 mapread 0x4f4000 thru 0x503fd5 (0xffd6 bytes) 000374[1] 1761121366.218716 trunc from 0x81407b to 0x65fd1c (0x1b435e bytes) 000378[1] 1761121366.237115 mapwrite 0x7b6000 thru 0x7b654c (0x54d bytes) 000379[1] 1761121366.265113 trunc from 0x7b654d to 0x6ba9c7 (0xfbb85 bytes) 000382[0] 1761121366.283344 trunc from 0x6ba9c8 to 0x498f1c (0x221aab bytes) 000385[1] 1761121366.303105 trunc from 0x498f1d to 0x93c5f8 (0x4a36dc bytes) 000386[0] 1761121366.321116 mapwrite 0x3f8000 thru 0x40558b (0xd58c bytes) 000387[1] 1761121366.378752 trunc from 0x93c5f9 to 0x4f318a (0x44946e bytes) 000389[1] 1761121366.392780 write_OD 0x6c8000 thru 0x6cffff (0x8000 bytes) 000391[1] 1761121366.409167 read_OD 0x4ca000 thru 0x4d2fff (0x9000 bytes) truncating to largest ever: 0x9af522 000393[1] 1761121366.413046 trunc from 0x6d0000 to 0x9af521 (0x2df522 bytes) 000396[0] 1761121366.429710 mapwrite 0x08d000 thru 0x098e5d (0xbe5e bytes) 000399[0] 1761121366.487433 write_OD 0x09f000 thru 0x0a2fff (0x4000 bytes) 000403[1] 1761121366.504769 write_OD 0x5de000 thru 0x5defff (0x1000 bytes) 000405[1] 1761121366.522179 write_OD 0x632000 thru 0x63efff (0xd000 bytes) 000406[0] 1761121366.538587 trunc from 0x9af522 to 0x5d5f40 (0x3d95e1 bytes) 000407[0] 1761121366.551223 write_OD 0x27d000 thru 0x288fff (0xc000 bytes) 000408[0] 1761121366.570233 trunc from 0x5d5f41 to 0x776da6 (0x1a0e66 bytes) 000412[1] 1761121366.589408 mapwrite 0x394000 thru 0x39f970 (0xb971 bytes) 000413[0] 1761121366.641039 mapwrite 0x195000 thru 0x19ef1e (0x9f1f bytes) 000416[0] 1761121366.681692 read_OD 0x30e000 thru 0x317fff (0xa000 bytes) 000417[0] 1761121366.686638 mapwrite 0x273000 thru 0x274a2e (0x1a2f bytes) 000418[1] 1761121366.731411 trunc from 0x776da7 to 0x78b878 (0x14ad2 bytes) 000419[1] 1761121366.746023 trunc from 0x78b879 to 0x1fad8a (0x590aee bytes) 000420[1] 1761121366.751104 read_OD 0x061000 thru 0x06efff (0xe000 bytes) 000421[0] 1761121366.755922 read_OD 0x1cc000 thru 0x1d2fff (0x7000 bytes) 000423[1] 1761121366.760581 trunc from 0x1fad8b to 0x4bf7af (0x2c4a25 bytes) 000425[1] 1761121366.773975 write_OD 0x305000 thru 0x311fff (0xd000 bytes) 000426[0] 1761121366.792776 mapread 0x0a9000 thru 0x0a9529 (0x52a bytes) 000428[1] 1761121366.817846 mapread 0x20e000 thru 0x2133e5 (0x53e6 bytes) 000430[0] 1761121366.846568 mapread 0x2f9000 thru 0x2fcc30 (0x3c31 bytes) 000434[0] 1761121366.847664 trunc from 0x4bf7b0 to 0x6d531f (0x215b70 bytes) 000435[1] 1761121366.877224 mapwrite 0x089000 thru 0x0966e0 (0xd6e1 bytes) 000437[1] 1761121366.932561 mapwrite 0x120000 thru 0x129cd4 (0x9cd5 bytes) 000444[1] 1761121366.983673 read_OD 0x371000 thru 0x371fff (0x1000 bytes) 000445[1] 1761121366.989666 write_OD 0x299000 thru 0x2a7fff (0xf000 bytes) 000448[0] 1761121367.006990 write_OD 0x69a000 thru 0x6a1fff (0x8000 bytes) 000449[0] 1761121367.022086 mapwrite 0x3c6000 thru 0x3ce614 (0x8615 bytes) 000450[0] 1761121367.054382 mapwrite 0x368000 thru 0x37381f (0xb820 bytes) 000456[0] 1761121367.099408 read_OD 0x21d000 thru 0x21ffff (0x3000 bytes) 000459[0] 1761121367.105184 mapwrite 0x6a9000 thru 0x6aa279 (0x127a bytes) 000460[1] 1761121367.154960 trunc from 0x6d5320 to 0x5fcae7 (0xd8838 bytes) 000463[1] 1761121367.181992 mapwrite 0x668000 thru 0x67460f (0xc610 bytes) 000466[0] 1761121367.238470 mapwrite 0x9d0000 thru 0x9da0bd (0xa0be bytes) 000467[0] 1761121367.287048 write_OD 0x7a7000 thru 0x7affff (0x9000 bytes) 000470[0] 1761121367.320616 read_OD 0x4ed000 thru 0x4f8fff (0xc000 bytes) 000472[1] 1761121367.324127 trunc from 0x9da0be to 0x5a6d4f (0x43336e bytes) 000473[0] 1761121367.337930 write_OD 0x124000 thru 0x124fff (0x1000 bytes) 000474[1] 1761121367.357250 read_OD 0x4fd000 thru 0x504fff (0x8000 bytes) 000475[1] 1761121367.362582 trunc from 0x5a6d50 to 0x5d7a28 (0x30cd9 bytes) 000477[1] 1761121367.376019 trunc from 0x5d7a29 to 0x7af588 (0x1d7b60 bytes) 000478[0] 1761121367.385726 mapwrite 0x785000 thru 0x79075f (0xb760 bytes) 000481[0] 1761121367.437353 trunc from 0x7af589 to 0x090654 (0x71ef34 bytes) 000489[1] 1761121367.488510 trunc from 0x090655 to 0x839698 (0x7a9044 bytes) 000491[1] 1761121367.498633 write_OD 0x4f3000 thru 0x4f6fff (0x4000 bytes) 000493[0] 1761121367.519089 mapread 0x509000 thru 0x51705a (0xe05b bytes) 000494[1] 1761121367.539378 mapread 0x173000 thru 0x177d75 (0x4d76 bytes) 000495[1] 1761121367.574416 trunc from 0x839699 to 0x481234 (0x3b8464 bytes) 000496[1] 1761121367.586724 mapwrite 0x402000 thru 0x409f3e (0x7f3f bytes) 000499[1] 1761121367.622125 write_OD 0x0a0000 thru 0x0aefff (0xf000 bytes) All operations completed A-OK! PASS 16a (16s) == sanityn test 16b: 500 iterations of dual-mount fsx at small size ========================================================== 04:22:49 (1761121369) fallocate on zfs doesn't consume space fallocate not supported 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0988702 s, 21.2 MB/s 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0845095 s, 24.8 MB/s Chance of close/open is 1 in 50 Seed set to 1370 fd 0: /mnt/lustre/f16b.sanityn fd 1: /mnt/lustre2/f16b.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported truncating to largest ever: 0x1d94b7 000001[0] 1761121370.573648 trunc from 00000000 to 0x1d94b6 (0x1d94b7 bytes) 000004[0] 1761121370.586710 write 0x561896 thru 0x562b08 (0x1273 bytes) 000005[0] 1761121370.588147 mapread 0x1160d0 thru 0x11681c (0x74d bytes) 000006[1] 1761121370.597442 write 0x4f0f89 thru 0x4f126a (0x2e2 bytes) 000007[1] 1761121370.605637 mapread 0x1d8ace thru 0x1d8af0 (0x23 bytes) 000009[0] 1761121370.622749 mapwrite 0x5f9d1f thru 0x5fb775 (0x1a57 bytes) 000010[1] 1761121370.694082 read 0x3185dc thru 0x318b5a (0x57f bytes) 000013[1] 1761121370.719481 mapwrite 0x0e6e97 thru 0x0e7ff1 (0x115b bytes) 000014[0] 1761121370.772150 read 0x3b070a thru 0x3b21d0 (0x1ac7 bytes) 000016[1] 1761121370.780950 write 0x17fde9 thru 0x181c39 (0x1e51 bytes) 000017[0] 1761121370.794903 read 0x5bae66 thru 0x5bc5e4 (0x177f bytes) 000021[1] 1761121370.820016 write 0x518531 thru 0x519ce5 (0x17b5 bytes) truncating to largest ever: 0x9bb8e2 000022[0] 1761121370.826964 trunc from 0x5fb776 to 0x9bb8e1 (0x3c016c bytes) 000023[0] 1761121370.846529 write 0x87a33b thru 0x87a5a2 (0x268 bytes) 000024[1] 1761121370.878003 read 0x86dbb1 thru 0x86f531 (0x1981 bytes) 000025[0] 1761121370.895240 read 0x8c8600 thru 0x8c94fb (0xefc bytes) 000026[0] 1761121370.901555 mapread 0x711ff3 thru 0x7139ad (0x19bb bytes) 000027[1] 1761121370.909279 mapread 0x4585a8 thru 0x45941e (0xe77 bytes) 000028[0] 1761121370.925762 mapwrite 0x3157fe thru 0x315a4d (0x250 bytes) 000030[1] 1761121370.968214 trunc from 0x9bb8e2 to 0x0d9b7a (0x8e1d67 bytes) 000031[0] 1761121370.996197 mapread 0x03f724 thru 0x0414c8 (0x1da5 bytes) 000032[0] 1761121371.017539 mapwrite 0x4aae10 thru 0x4abc7a (0xe6b bytes) 000033[1] 1761121371.060148 write 0x5cd883 thru 0x5cdfe6 (0x764 bytes) 000034[0] 1761121371.074858 trunc from 0x5cdfe7 to 0x04e68f (0x57f957 bytes) 000036[0] 1761121371.108282 trunc from 0x04e690 to 0x0de872 (0x901e3 bytes) 000037[1] 1761121371.127568 read 0x018e95 thru 0x019f58 (0x10c4 bytes) 000039[1] 1761121371.155086 read 0x029926 thru 0x02a89f (0xf7a bytes) 000040[1] 1761121371.155660 write 0x513d4f thru 0x514c55 (0xf07 bytes) 000044[0] 1761121371.189220 trunc from 0x514c56 to 0x04f13f (0x4c5b16 bytes) 000046[1] 1761121371.218029 mapread 0x01cf9c thru 0x01ebf3 (0x1c58 bytes) 000049[1] 1761121371.255926 mapread 0x000c3f thru 0x001fe5 (0x13a7 bytes) 000050[0] 1761121371.264921 write 0x404806 thru 0x404c3a (0x435 bytes) 000052[1] 1761121371.279861 mapwrite 0x9cb395 thru 0x9cd326 (0x1f92 bytes) 000054[0] 1761121371.386934 mapwrite 0x31749c thru 0x317cc4 (0x829 bytes) 000055[0] 1761121371.423325 write 0x3c7b3e thru 0x3c7c57 (0x11a bytes) 000056[1] 1761121371.430313 mapwrite 0x44b07b thru 0x44bab9 (0xa3f bytes) 000057[0] 1761121371.455204 read 0x1aa472 thru 0x1abbb8 (0x1747 bytes) 000059[1] 1761121371.471142 read 0x056d9d thru 0x05889e (0x1b02 bytes) 000060[0] 1761121371.477540 trunc from 0x9cd327 to 0x13e007 (0x88f31f bytes) 000064[1] 1761121371.503743 trunc from 0x13e008 to 0x666fc1 (0x528fba bytes) 000065[1] 1761121371.518165 mapread 0x097748 thru 0x098303 (0xbbc bytes) 000066[1] 1761121371.522518 trunc from 0x666fc2 to 0x0dbd6a (0x58b257 bytes) 000067[0] 1761121371.533488 read 0x05cf78 thru 0x05eb81 (0x1c0a bytes) 000068[1] 1761121371.546633 trunc from 0x0dbd6b to 0x40edc3 (0x333059 bytes) 000069[1] 1761121371.563640 write 0x7fd5b7 thru 0x7fde52 (0x89c bytes) 000070[0] 1761121371.567857 mapwrite 0x4bc480 thru 0x4bcefb (0xa7c bytes) 000072[1] 1761121371.612462 trunc from 0x7fde53 to 0x2fb556 (0x5028fc bytes) 000075[1] 1761121371.635170 mapread 0x11abb5 thru 0x11baa1 (0xeed bytes) 000077[1] 1761121371.644502 mapwrite 0x7e16a2 thru 0x7e26a5 (0x1004 bytes) 000078[0] 1761121371.678041 trunc from 0x7e26a6 to 0x5cc603 (0x2160a2 bytes) 000079[0] 1761121371.696321 read 0x463974 thru 0x465949 (0x1fd6 bytes) 000080[0] 1761121371.711297 trunc from 0x5cc604 to 0x9683c6 (0x39bdc3 bytes) 000084[1] 1761121371.735138 write 0x48a05b thru 0x48ae53 (0xdf9 bytes) 000086[0] 1761121371.753798 mapwrite 0x853a42 thru 0x855855 (0x1e14 bytes) 000087[0] 1761121371.821678 mapread 0x5a7e06 thru 0x5a966f (0x186a bytes) 000088[0] 1761121371.828353 trunc from 0x9683c7 to 0x4ef39c (0x47902a bytes) 000089[0] 1761121371.840512 trunc from 0x4ef39d to 0x268526 (0x286e76 bytes) 000094[1] 1761121371.860505 mapread 0x186904 thru 0x187d3e (0x143b bytes) 000096[0] 1761121371.876596 trunc from 0x268527 to 0x208312 (0x60214 bytes) 000097[0] 1761121371.893851 mapread 0x050f26 thru 0x051219 (0x2f4 bytes) 000098[1] 1761121371.906639 mapwrite 0x1d0766 thru 0x1d2130 (0x19cb bytes) 000099[0] 1761121371.942466 read 0x1893c7 thru 0x1895aa (0x1e4 bytes) 000100[1] 1761121371.961058 trunc from 0x208313 to 0x1a4799 (0x63b79 bytes) 000101[1] 1761121371.979073 read 0x12832e thru 0x129dc9 (0x1a9c bytes) 000102[1] 1761121371.991055 mapread 0x105802 thru 0x1061c5 (0x9c4 bytes) 000103[0] 1761121371.995925 trunc from 0x1a479a to 0x39050b (0x1ebd72 bytes) 000107[0] 1761121372.010818 read 0x084bcd thru 0x085c7b (0x10af bytes) 000110[0] 1761121372.044748 mapread 0x27fb7f thru 0x280c5a (0x10dc bytes) 000111[0] 1761121372.045516 mapread 0x32c910 thru 0x32cf08 (0x5f9 bytes) 000113[0] 1761121372.060285 mapwrite 0x88f3af thru 0x88fe07 (0xa59 bytes) 000114[1] 1761121372.090883 mapwrite 0x7b883d thru 0x7ba614 (0x1dd8 bytes) 000119[1] 1761121372.159376 mapwrite 0x0e2227 thru 0x0e2789 (0x563 bytes) 000120[1] 1761121372.241000 trunc from 0x88fe08 to 0x03b5e7 (0x854820 bytes) 000121[0] 1761121372.260058 mapread 0x00c10d thru 0x00c2b7 (0x1ab bytes) 000122[0] 1761121372.273196 mapwrite 0x7f6224 thru 0x7f7a6f (0x184c bytes) 000123[1] 1761121372.309857 mapread 0x41e43e thru 0x41eb7f (0x742 bytes) 000125[1] 1761121372.331460 trunc from 0x7f7a70 to 0x11face (0x6d7fa1 bytes) 000126[0] 1761121372.350317 trunc from 0x11facf to 0x276ace (0x157000 bytes) 000127[1] 1761121372.361299 write 0x06f365 thru 0x06fd57 (0x9f3 bytes) 000131[0] 1761121372.383916 read 0x262971 thru 0x264480 (0x1b10 bytes) 000132[0] 1761121372.397550 trunc from 0x276acf to 0x56b6a1 (0x2f4bd3 bytes) 000135[0] 1761121372.423248 read 0x1e93fb thru 0x1e958a (0x190 bytes) 000139[0] 1761121372.459372 write 0x32a75b thru 0x32bc94 (0x153a bytes) 000145[1] 1761121372.487925 mapread 0x25180e thru 0x2527a0 (0xf93 bytes) 000146[1] 1761121372.506107 mapwrite 0x10f327 thru 0x10f695 (0x36f bytes) 000150[1] 1761121372.551655 mapread 0x2e0612 thru 0x2e1c0a (0x15f9 bytes) 000151[0] 1761121372.561214 write 0x49cf41 thru 0x49e1aa (0x126a bytes) 000153[1] 1761121372.580956 mapwrite 0x8a8e7d thru 0x8aad2d (0x1eb1 bytes) 000154[1] 1761121372.615614 trunc from 0x8aad2e to 0x63f03e (0x26bcef bytes) 000156[0] 1761121372.631456 write 0x1be700 thru 0x1bfb71 (0x1472 bytes) 000157[1] 1761121372.647457 mapwrite 0x9e3430 thru 0x9e3c13 (0x7e4 bytes) 000159[1] 1761121372.690040 mapwrite 0x6167df thru 0x617fe1 (0x1803 bytes) 000162[1] 1761121372.722148 read 0x75ed6c thru 0x760a11 (0x1ca6 bytes) 000163[1] 1761121372.749022 mapwrite 0x3f8e97 thru 0x3fa0c0 (0x122a bytes) 000164[1] 1761121372.768801 trunc from 0x9e3c14 to 0x69ca1b (0x3471f8 bytes) 000166[0] 1761121372.786365 read 0x50b639 thru 0x50b7c6 (0x18e bytes) 000168[0] 1761121372.806898 write 0x0892a6 thru 0x08acc1 (0x1a1c bytes) 000171[1] 1761121372.826238 mapread 0x064c2c thru 0x06565f (0xa34 bytes) 000172[1] 1761121372.838524 write 0x0f974c thru 0x0f9d58 (0x60d bytes) 000173[0] 1761121372.847360 trunc from 0x69ca1c to 0x3e46a6 (0x2b8375 bytes) 000175[0] 1761121372.862758 mapwrite 0x552df7 thru 0x5530af (0x2b9 bytes) 000176[1] 1761121372.890251 read 0x0c8047 thru 0x0c8e1d (0xdd7 bytes) 000177[0] 1761121372.900444 trunc from 0x5530b0 to 0x2753db (0x2ddcd4 bytes) 000178[0] 1761121372.910955 trunc from 0x2753dc to 0x5314e0 (0x2bc105 bytes) 000184[1] 1761121372.937923 trunc from 0x5314e1 to 0x621ff1 (0xf0b11 bytes) 000185[1] 1761121372.953079 mapwrite 0x674940 thru 0x676466 (0x1b27 bytes) 000187[0] 1761121373.029909 mapwrite 0x926d24 thru 0x928a00 (0x1cdd bytes) 000190[0] 1761121373.096709 mapread 0x04d7cf thru 0x04e8b6 (0x10e8 bytes) 000191[0] 1761121373.106362 write 0x5e903e thru 0x5e9404 (0x3c7 bytes) 000193[1] 1761121373.116133 trunc from 0x928a01 to 0x033a88 (0x8f4f78 bytes) 000194[0] 1761121373.138499 read 0x016f2d thru 0x018eaa (0x1f7e bytes) 000200[1] 1761121373.165447 read 0x004856 thru 0x005aa8 (0x1253 bytes) 000203[1] 1761121373.177932 mapread 0x0318eb thru 0x032f14 (0x162a bytes) 000206[1] 1761121373.194096 mapwrite 0x790988 thru 0x792702 (0x1d7b bytes) 000207[1] 1761121373.238900 mapread 0x133dd5 thru 0x134515 (0x741 bytes) 000208[1] 1761121373.253050 mapread 0x1c1dcb thru 0x1c2096 (0x2cc bytes) 000209[1] 1761121373.263469 trunc from 0x792703 to 0x4847f7 (0x30df0b bytes) 000210[0] 1761121373.270033 trunc from 0x4847f8 to 0x2a0211 (0x1e45e6 bytes) 000213[1] 1761121373.297044 mapread 0x25b661 thru 0x25baf2 (0x492 bytes) truncating to largest ever: 0x9fbcdc 000214[1] 1761121373.307872 trunc from 0x2a0212 to 0x9fbcdb (0x75baca bytes) 000215[0] 1761121373.327094 trunc from 0x9fbcdc to 0x8fac07 (0x1010d4 bytes) 000217[0] 1761121373.350118 write 0x005709 thru 0x00670b (0x1003 bytes) 000218[0] 1761121373.413133 write 0x5bce58 thru 0x5be7af (0x1958 bytes) 000219[1] 1761121373.420017 write 0x1916d4 thru 0x192235 (0xb62 bytes) 000220[0] 1761121373.439698 write 0x6ef182 thru 0x6f06c0 (0x153f bytes) 000224[1] 1761121373.463357 mapwrite 0x34c7e3 thru 0x34e210 (0x1a2e bytes) 000229[0] 1761121373.515026 write 0x848d48 thru 0x84ac3c (0x1ef5 bytes) 000233[1] 1761121373.540021 trunc from 0x8fac08 to 0x4a0cf6 (0x459f11 bytes) 000236[1] 1761121373.567917 write 0x457914 thru 0x459301 (0x19ee bytes) 000237[1] 1761121373.571773 write 0x44d075 thru 0x44d10a (0x96 bytes) 000240[0] 1761121373.579327 write 0x22231f thru 0x2232d8 (0xfba bytes) 000241[1] 1761121373.593589 write 0x77761a thru 0x778af3 (0x14da bytes) 000245[1] 1761121373.618112 mapwrite 0x354dba thru 0x3556bc (0x903 bytes) 000247[1] 1761121373.652438 mapwrite 0x63ed00 thru 0x6408f2 (0x1bf3 bytes) 000248[1] 1761121373.693727 read 0x51e502 thru 0x51e9fc (0x4fb bytes) 000250[1] 1761121373.725831 write 0x8cc8ef thru 0x8ce868 (0x1f7a bytes) 000251[0] 1761121373.733473 write 0x53aae3 thru 0x53b4cb (0x9e9 bytes) 000253[1] 1761121373.757787 read 0x1de54f thru 0x1dfcec (0x179e bytes) 000257[0] 1761121373.820532 mapread 0x2d8c36 thru 0x2da1bd (0x1588 bytes) 000258[1] 1761121373.843848 mapwrite 0x6c36ed thru 0x6c3828 (0x13c bytes) 000261[0] 1761121373.891966 write 0x73b8d1 thru 0x73b9f8 (0x128 bytes) 000262[1] 1761121373.907444 trunc from 0x8ce869 to 0x5fc165 (0x2d2703 bytes) 000263[1] 1761121373.926806 trunc from 0x5fc166 to 0x730371 (0x13420c bytes) 000266[0] 1761121373.934720 read 0x1bbc9d thru 0x1bbcdd (0x41 bytes) 000268[1] 1761121373.950041 write 0x7e4a0a thru 0x7e68e0 (0x1ed7 bytes) 000270[0] 1761121373.963233 write 0x353ee1 thru 0x355c74 (0x1d94 bytes) 000272[1] 1761121373.980830 mapwrite 0x232a3c thru 0x233ef5 (0x14ba bytes) 000273[1] 1761121374.009064 mapread 0x732006 thru 0x733b3e (0x1b39 bytes) 000280[0] 1761121374.045704 trunc from 0x7e68e1 to 0x88903a (0xa275a bytes) 000281[1] 1761121374.058640 trunc from 0x88903b to 0x087a65 (0x8015d5 bytes) 000282[0] 1761121374.073317 write 0x24f43d thru 0x2506d5 (0x1299 bytes) 000283[0] 1761121374.083204 read 0x02c0c9 thru 0x02d808 (0x1740 bytes) 000286[1] 1761121374.105539 write 0x02a3e9 thru 0x02bafc (0x1714 bytes) 000288[0] 1761121374.128728 trunc from 0x2506d6 to 0x2d8b42 (0x8846d bytes) 000290[0] 1761121374.154814 trunc from 0x2d8b43 to 0x17ef5a (0x159be8 bytes) 000291[0] 1761121374.168247 trunc from 0x17ef5b to 0x962197 (0x7e323d bytes) 000292[0] 1761121374.184375 trunc from 0x962198 to 0x7d2910 (0x18f887 bytes) 000293[1] 1761121374.198329 write 0x9e6ff3 thru 0x9e851e (0x152c bytes) 000294[1] 1761121374.209549 mapread 0x237b91 thru 0x237c26 (0x96 bytes) 000295[1] 1761121374.232053 trunc from 0x9e851f to 0x91cba8 (0xcb976 bytes) 000296[0] 1761121374.251867 write 0x39ca02 thru 0x39e34d (0x194c bytes) 000298[1] 1761121374.288345 mapread 0x5b496c thru 0x5b4ad6 (0x16b bytes) 000299[0] 1761121374.311594 read 0x53cd2e thru 0x53e51b (0x17ee bytes) 000301[0] 1761121374.332338 read 0x320408 thru 0x321bd7 (0x17d0 bytes) 000302[0] 1761121374.348738 trunc from 0x91cba9 to 0x3a7040 (0x575b68 bytes) 000304[1] 1761121374.369664 read 0x207334 thru 0x207ccc (0x999 bytes) 000305[1] 1761121374.389912 mapread 0x112b9d thru 0x11331e (0x782 bytes) 000309[0] 1761121374.408383 write 0x33e418 thru 0x33f20a (0xdf3 bytes) 000310[1] 1761121374.415228 mapread 0x25fae6 thru 0x25fcb1 (0x1cc bytes) 000311[0] 1761121374.422229 write 0x8de0fa thru 0x8dfbe0 (0x1ae7 bytes) 000312[1] 1761121374.433608 read 0x634031 thru 0x63588f (0x185f bytes) 000313[1] 1761121374.449033 read 0x06d0e4 thru 0x06edef (0x1d0c bytes) 000315[0] 1761121374.456826 mapwrite 0x1e70da thru 0x1e7b51 (0xa78 bytes) 000316[1] 1761121374.502160 trunc from 0x8dfbe1 to 0x1ffcbb (0x6dff25 bytes) 000320[1] 1761121374.547422 mapread 0x1ad968 thru 0x1adae6 (0x17f bytes) 000322[1] 1761121374.572877 read 0x113173 thru 0x113e1c (0xcaa bytes) 000323[1] 1761121374.579073 mapread 0x1a7ce4 thru 0x1a99fe (0x1d1b bytes) 000324[0] 1761121374.579725 mapwrite 0x560346 thru 0x560629 (0x2e4 bytes) 000326[1] 1761121374.628078 write 0x2165df thru 0x216e6a (0x88c bytes) 000327[1] 1761121374.639532 mapwrite 0x539e95 thru 0x539f56 (0xc2 bytes) 000331[1] 1761121374.668876 mapread 0x001b4c thru 0x0021b3 (0x668 bytes) 000333[1] 1761121374.676337 trunc from 0x56062a to 0x3ea47b (0x1761ae bytes) 000337[1] 1761121374.690638 mapread 0x218040 thru 0x219518 (0x14d9 bytes) 000339[0] 1761121374.701722 trunc from 0x3ea47c to 0x8870d8 (0x49cc5d bytes) 000342[0] 1761121374.717317 mapwrite 0x212790 thru 0x21369b (0xf0c bytes) 000343[0] 1761121374.746571 mapwrite 0x571b64 thru 0x572405 (0x8a2 bytes) 000344[0] 1761121374.765702 trunc from 0x8870d9 to 0x9fb69f (0x1745c7 bytes) 000346[1] 1761121374.785294 trunc from 0x9fb6a0 to 0x698a82 (0x362c1d bytes) 000347[1] 1761121374.795294 read 0x314e6e thru 0x315a51 (0xbe4 bytes) 000349[0] 1761121374.819936 write 0x0ddd4e thru 0x0dfa7a (0x1d2d bytes) 000350[1] 1761121374.832923 mapread 0x24685c thru 0x246f57 (0x6fc bytes) 000351[1] 1761121374.851047 write 0x734cc1 thru 0x735b40 (0xe80 bytes) 000352[1] 1761121374.852041 read 0x664cbd thru 0x666c12 (0x1f56 bytes) 000356[0] 1761121374.871793 read 0x553996 thru 0x5547ab (0xe16 bytes) 000359[1] 1761121374.898851 trunc from 0x735b41 to 0x515bbc (0x21ff84 bytes) 000368[1] 1761121374.960747 mapwrite 0x164575 thru 0x165533 (0xfbf bytes) 000370[0] 1761121374.987812 read 0x061f7d thru 0x063358 (0x13dc bytes) 000371[1] 1761121375.001440 read 0x208686 thru 0x208d1c (0x697 bytes) 000372[1] 1761121375.028012 mapread 0x50b10b thru 0x50d0d9 (0x1fcf bytes) 000377[0] 1761121375.041757 mapread 0x12b758 thru 0x12c143 (0x9ec bytes) 000379[0] 1761121375.082377 write 0x096891 thru 0x097a3e (0x11ae bytes) 000380[1] 1761121375.098361 mapwrite 0x3d4475 thru 0x3d44ba (0x46 bytes) 000381[1] 1761121375.135764 read 0x0e6f5f thru 0x0e80b2 (0x1154 bytes) 000382[1] 1761121375.166360 mapwrite 0x71c1af thru 0x71e186 (0x1fd8 bytes) 000383[1] 1761121375.223489 mapread 0x2ee824 thru 0x2ef873 (0x1050 bytes) 000385[0] 1761121375.238299 write 0x9659b3 thru 0x9672c8 (0x1916 bytes) 000386[1] 1761121375.252458 trunc from 0x9672c9 to 0x557bdf (0x40f6e9 bytes) 000387[1] 1761121375.272321 mapread 0x17a566 thru 0x17a597 (0x32 bytes) 000388[0] 1761121375.276493 trunc from 0x557be0 to 0x3de9ff (0x1791e0 bytes) 000392[0] 1761121375.308162 mapread 0x0d8441 thru 0x0d971d (0x12dd bytes) 000393[1] 1761121375.312464 trunc from 0x3dea00 to 0x982cdd (0x5a42de bytes) 000394[1] 1761121375.327351 write 0x0ba94c thru 0x0bbb7f (0x1234 bytes) 000396[0] 1761121375.332008 trunc from 0x982cde to 0x1dc3d3 (0x7a690a bytes) 000398[0] 1761121375.354210 mapwrite 0x1ce8f1 thru 0x1ce8fb (0xb bytes) 000399[0] 1761121375.372838 mapread 0x02890d thru 0x028d30 (0x424 bytes) 000402[0] 1761121375.392573 write 0x99525d thru 0x995a16 (0x7ba bytes) 000403[0] 1761121375.401708 mapwrite 0x3f30ef thru 0x3f4fb6 (0x1ec8 bytes) 000404[0] 1761121375.423826 mapwrite 0x63a46b thru 0x63a907 (0x49d bytes) 000405[1] 1761121375.444208 write 0x3abb96 thru 0x3aca33 (0xe9e bytes) 000407[0] 1761121375.466524 write 0x136e16 thru 0x137963 (0xb4e bytes) 000408[1] 1761121375.476607 mapwrite 0x2d1ede thru 0x2d1fe0 (0x103 bytes) 000410[1] 1761121375.511319 write 0x6fab00 thru 0x6fb2f4 (0x7f5 bytes) 000414[0] 1761121375.530700 trunc from 0x995a17 to 0x56fdb4 (0x425c62 bytes) 000415[1] 1761121375.546472 trunc from 0x56fdb5 to 0x449392 (0x126a22 bytes) 000419[0] 1761121375.561158 mapread 0x1da5e0 thru 0x1db68c (0x10ad bytes) 000421[0] 1761121375.582109 write 0x0c9e01 thru 0x0cb8df (0x1adf bytes) 000426[1] 1761121375.596375 write 0x0436e0 thru 0x0451b8 (0x1ad9 bytes) 000427[0] 1761121375.607243 write 0x563123 thru 0x5636da (0x5b8 bytes) 000428[1] 1761121375.617300 mapread 0x2a6044 thru 0x2a6f3c (0xef9 bytes) 000429[0] 1761121375.627796 read 0x2d5e3c thru 0x2d6074 (0x239 bytes) 000431[1] 1761121375.656031 mapwrite 0x0dc633 thru 0x0de2e4 (0x1cb2 bytes) 000432[1] 1761121375.688722 mapread 0x05e6d8 thru 0x05eedb (0x804 bytes) 000434[0] 1761121375.702408 read 0x2f3e15 thru 0x2f57ce (0x19ba bytes) 000435[0] 1761121375.721146 write 0x57bf0f thru 0x57c97a (0xa6c bytes) 000436[1] 1761121375.726820 trunc from 0x57c97b to 0x612898 (0x95f1e bytes) 000438[0] 1761121375.754015 write 0x0b9258 thru 0x0b9c06 (0x9af bytes) 000439[1] 1761121375.765424 mapread 0x28b567 thru 0x28d0b7 (0x1b51 bytes) 000441[1] 1761121375.793149 mapwrite 0x57c313 thru 0x57e06e (0x1d5c bytes) 000442[0] 1761121375.824931 mapread 0x503e4a thru 0x504e7b (0x1032 bytes) 000444[1] 1761121375.842889 read 0x49363f thru 0x494ba2 (0x1564 bytes) 000445[1] 1761121375.854699 read 0x178568 thru 0x1787f7 (0x290 bytes) 000446[1] 1761121375.878282 mapwrite 0x42df66 thru 0x42f71a (0x17b5 bytes) 000450[1] 1761121375.926042 trunc from 0x612899 to 0x6555b5 (0x42d1d bytes) 000451[1] 1761121375.944720 mapwrite 0x0dfc9b thru 0x0e1927 (0x1c8d bytes) 000452[1] 1761121375.965985 mapread 0x4f5f2a thru 0x4f762a (0x1701 bytes) 000454[1] 1761121375.978489 mapread 0x218754 thru 0x2187d0 (0x7d bytes) 000455[1] 1761121375.980729 trunc from 0x6555b6 to 0x2db58f (0x37a026 bytes) 000457[1] 1761121375.989054 mapwrite 0x84b8cc thru 0x84cf19 (0x164e bytes) 000459[0] 1761121376.020871 read 0x74f9ee thru 0x7504f1 (0xb04 bytes) 000460[0] 1761121376.034643 mapwrite 0x5d2deb thru 0x5d3594 (0x7aa bytes) 000461[0] 1761121376.069341 read 0x076793 thru 0x077e04 (0x1672 bytes) 000462[1] 1761121376.124511 write 0x8889c3 thru 0x8891b3 (0x7f1 bytes) 000467[1] 1761121376.157393 mapread 0x693d51 thru 0x69419c (0x44c bytes) 000468[1] 1761121376.167977 write 0x971a9a thru 0x971b8a (0xf1 bytes) 000469[1] 1761121376.183700 write 0x57058f thru 0x570fbb (0xa2d bytes) 000470[0] 1761121376.191414 mapread 0x3fd727 thru 0x3feb30 (0x140a bytes) 000471[1] 1761121376.208019 trunc from 0x971b8b to 0x56fcb9 (0x401ed1 bytes) 000473[0] 1761121376.227152 mapwrite 0x8e21fa thru 0x8e35c3 (0x13ca bytes) 000474[1] 1761121376.271927 write 0x1f8082 thru 0x1f8fc6 (0xf45 bytes) 000475[1] 1761121376.286111 mapread 0x4e4f6c thru 0x4e5a9e (0xb33 bytes) 000477[0] 1761121376.297079 read 0x09509c thru 0x096af4 (0x1a59 bytes) 000478[1] 1761121376.309973 mapwrite 0x1518b5 thru 0x1520bd (0x809 bytes) 000479[1] 1761121376.334822 write 0x06c564 thru 0x06c74d (0x1ea bytes) 000482[1] 1761121376.353032 read 0x2566c2 thru 0x2584eb (0x1e2a bytes) 000483[0] 1761121376.400992 read 0x24d7f2 thru 0x24f79e (0x1fad bytes) 000484[0] 1761121376.426631 mapread 0x638e64 thru 0x63acea (0x1e87 bytes) 000487[0] 1761121376.452665 write 0x6217fa thru 0x621b59 (0x360 bytes) 000488[1] 1761121376.469443 mapread 0x4879d4 thru 0x4884bb (0xae8 bytes) 000489[1] 1761121376.482422 read 0x4e20bd thru 0x4e24e1 (0x425 bytes) 000490[1] 1761121376.510020 trunc from 0x8e35c4 to 0x81ce44 (0xc677f bytes) 000493[0] 1761121376.523936 trunc from 0x81ce45 to 0x4ba748 (0x3626fc bytes) 000499[0] 1761121376.569427 write 0x061673 thru 0x06255d (0xeeb bytes) 000500[1] 1761121376.594745 trunc from 0x4ba749 to 0x058e58 (0x4618f0 bytes) All operations completed A-OK! PASS 16b (9s) == sanityn test 16c: verify data consistency on ldiskfs with cache disabled (b=17397) ========================================================== 04:22:58 (1761121378) SKIP: sanityn test_16c dio on ldiskfs only SKIP 16c (1s) == sanityn test 16d: Verify DIO and buffer IO with two clients ========================================================== 04:22:59 (1761121379) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.109748 s, 19.1 MB/s 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0731887 s, 28.7 MB/s ldlm.namespaces.MGC192.168.204.116@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff9a01c7fbe000.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff9a01d93a6000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff9a01c7fbe000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff9a01d93a6000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff9a01c7fbe000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff9a01d93a6000.lru_size=clear 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 2.87473 s, 36.5 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 0.809968 s, 129 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 0.844937 s, 124 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 1.70059 s, 61.7 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 1.97659 s, 53.0 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 1.27369 s, 82.3 MB/s PASS 16d (14s) == sanityn test 16e: Verify size consistency for O_DIRECT write ========================================================== 04:23:13 (1761121393) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.10094 s, 104 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0432112 s, 24.3 MB/s PASS 16e (3s) == sanityn test 16f: rw sequential consistency vs drop_caches ========================================================== 04:23:16 (1761121396) TIMEOUT PASS 16f (22s) == sanityn test 16g: mmap rw sequential consistency vs drop_caches ========================================================== 04:23:38 (1761121418) TIMEOUT PASS 16g (23s) == sanityn test 16i: read after truncate file ============ 04:24:01 (1761121441) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16i (2s) == sanityn test 16k: Parallel FSX and drop caches should not panic ========================================================== 04:24:03 (1761121443) + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.1 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.2 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.3 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.4 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.5 -l 67108864 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported truncating to largest ever: 0x2325403 fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported truncating to largest ever: 0x282e29b truncating to largest ever: 0x2325403 truncating to largest ever: 0x2325403 truncating to largest ever: 0x2d8bfaa truncating to largest ever: 0x2325403 truncating to largest ever: 0x282e29b truncating to largest ever: 0x2325403 truncating to largest ever: 0x282e29b truncating to largest ever: 0x282e29b truncating to largest ever: 0x2d8bfaa truncating to largest ever: 0x2d8bfaa truncating to largest ever: 0x2d8bfaa truncating to largest ever: 0x282e29b truncating to largest ever: 0x2d8bfaa truncating to largest ever: 0x3bdf026 truncating to largest ever: 0x3bdf026 truncating to largest ever: 0x3bdf026 truncating to largest ever: 0x3bdf026 truncating to largest ever: 0x3bdf026 truncating to largest ever: 0x3eb1faf truncating to largest ever: 0x3eb1faf truncating to largest ever: 0x3eb1faf truncating to largest ever: 0x3eb1faf truncating to largest ever: 0x3eb1faf truncating to largest ever: 0x3f056a0 truncating to largest ever: 0x3f056a0 truncating to largest ever: 0x3f056a0 truncating to largest ever: 0x3f056a0 truncating to largest ever: 0x3f056a0 truncating to largest ever: 0x3fdc431 truncating to largest ever: 0x3fdc431 truncating to largest ever: 0x3fdc431 truncating to largest ever: 0x3fdc431 truncating to largest ever: 0x3fdc431 All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! + fsx 1 done: rc=0 Wed Oct 22 04:24:20 EDT 2025 + fsx 2 done: rc=0 Wed Oct 22 04:24:20 EDT 2025 All operations completed A-OK! + fsx 3 done: rc=0 Wed Oct 22 04:24:20 EDT 2025 + fsx 4 done: rc=0 Wed Oct 22 04:24:20 EDT 2025 + fsx 5 done: rc=0 Wed Oct 22 04:24:20 EDT 2025 PASS 16k (19s) == sanityn test 17: resource creation/LVB creation race ========================================================================= 04:24:22 (1761121462) fail_loc=0x8000030a PASS 17 (5s) == sanityn test 18: mmap sanity check =========================================================================================== 04:24:27 (1761121467) mmap test1: basic mmap operation (PASS, 0.021985s) mmap test2: MAP_PRIVATE not write back (PASS, 0.025287s) mmap test3: concurrent mmap ops on two nodes (PASS, 14.953s) mmap test4: c1 write to f1 from mmapped f2, c2 write to f1 from mmapped f1 (PASS, 2.1009s) mmap test5: read/write file to/from the buffer which mmapped to just this file (PASS, 0.036514s) mmap test6: check mmap write/read content on two nodes (PASS, 0.03919s) mmap test7: file i/o with an unmapped buffer (SKIPPED, 0s) mmap test8: SIGBUS for beyond file size (SKIPPED, 0s) mmap test9: SIGBUS for negative file offset (SKIPPED, 0s) mmap test10: mtime not change for readonly mmap access (PASS, 1.0182s) PASS 18 (22s) == sanityn test 19: test concurrent uncached read races ========================================================================= 04:24:49 (1761121489) oleg416-server: error: get_param: param_path 'osd-*/lustre-OST*/read_cache_enable': No such file or directory pdsh@oleg416-client: oleg416-server: ssh exited with exit code 2 SKIP: sanityn test_19 not cache-capable obdfilter SKIP 19 (1s) == sanityn test 20: test extra readahead page left in cache ============================================================== 04:24:50 (1761121490) PASS 20 (3s) == sanityn test 21: Try to remove mountpoint on another dir ============================================================== 04:24:53 (1761121493) rmdir: removing directory, '/mnt/lustre/d21.sanityn' rmdir: failed to remove '/mnt/lustre/d21.sanityn': Device or resource busy rmdir: removing directory, '/mnt/lustre2/d21.sanityn' Removed mounted directory from another mountpoint, needs to be fixed PASS 21 (2s) == sanityn test 23: others should see updated atime while another read============================================================== 04:24:55 (1761121495) now is 1761121496 starting reads multiop /mnt/lustre/f23.sanityn vor20_c TMPPIPE=/tmp/multiop_open_wait_pipe.7143 new atime is 1761121557 PASS 23 (64s) == sanityn test 24a: lfs df [-ih] [path] test =================================================================================== 04:25:59 (1761121559) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 10240 3759104 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 6144 3763200 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 16384 7522304 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 10240 3759104 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 6144 3763200 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 16384 7522304 1% /mnt/lustre2 UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 490.1K 383 489.7K 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 115.1K 374 114.8K 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 115.3K 375 114.9K 1% /mnt/lustre[OST:1] filesystem_summary: 230.1K 383 229.7K 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 490.1K 383 489.7K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.1K 374 114.8K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 115.3K 375 114.9K 1% /mnt/lustre2[OST:1] filesystem_summary: 230.1K 383 229.7K 1% /mnt/lustre2 UUID bytes Used Available Use% Mounted on lustre-MDT0000_UUID 2.1G 3.2M 2.1G 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3.6G 10.0M 3.6G 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3.6G 6.0M 3.6G 1% /mnt/lustre[OST:1] filesystem_summary: 7.2G 16.0M 7.2G 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 501835 383 501452 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 117910 374 117536 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 118039 375 117664 1% /mnt/lustre2[OST:1] filesystem_summary: 235583 383 235200 1% /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 10240 3759104 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 6144 3763200 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 16384 7522304 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 490.1K 383 489.7K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.1K 374 114.8K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 115.3K 375 114.9K 1% /mnt/lustre2[OST:1] filesystem_summary: 230.1K 383 229.7K 1% /mnt/lustre2 No device found for name osc: Invalid argument UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 501835 383 501452 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 117910 374 117536 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 118039 375 117664 1% /mnt/lustre[OST:1] filesystem_summary: 235583 383 235200 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 501835 383 501452 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 117910 374 117536 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 118039 375 117664 1% /mnt/lustre2[OST:1] filesystem_summary: 235583 383 235200 1% /mnt/lustre2 No device found for name osc: Invalid argument UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 10240 3759104 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 6144 3763200 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 16384 7522304 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 10240 3759104 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 6144 3763200 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 16384 7522304 1% /mnt/lustre2 PASS 24a (2s) == sanityn test 24b: lfs df should show both filesystems ========================================================================= 04:26:01 (1761121561) PASS 24b (2s) == sanityn test 25a: change ACL on one mountpoint be seen on another ============================================================= 04:26:03 (1761121563) running as uid/gid/euid/egid 500/500/500/500, groups: [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] PASS 25a (3s) == sanityn test 25b: change ACL under remote dir on one mountpoint be seen on another ========================================================== 04:26:06 (1761121566) SKIP: sanityn test_25b needs >= 2 MDTs SKIP 25b (1s) == sanityn test 26a: allow mtime to get older ============ 04:26:07 (1761121567) utime: good mknod times 1761121566 <= 1761121567 <= 1761121567 for /mnt/lustre/f26a utime: good mknod times 1761121566 <= 1761121567 <= 1761121567 for /mnt/lustre2/f26a utime: good utime mtimes 100000, atime 200000 utime: updated times for /mnt/lustre2/f26a PASS 26a (3s) == sanityn test 26b: sync mtime between ost and mds ====== 04:26:10 (1761121570) PASS 26b (4s) SKIP: sanityn test_27 skipping excluded test 27 SKIP: sanityn test_28 skipping ALWAYS excluded test 28 == sanityn test 30: recreate file race =================== 04:26:15 (1761121575) 0 PASS 30 (4s) == sanityn test 31a: voluntary cancel / blocking ast race======================================================================== 04:26:19 (1761121579) fail_loc=0x314 PASS 31a (4s) == sanityn test 31b: voluntary OST cancel / blocking ast race======================================================================== 04:26:23 (1761121583) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg416-server mds-ost sync done. sleep 5 for ZFS MDS Waiting for MDT destroys to complete fail_loc=0x314 fail_loc=0x316 fail_loc=0 fail_loc=0 Filesystem 1K-blocks Used Available Use% Mounted on 192.168.204.116@tcp:/lustre 7542784 17408 7519232 1% /mnt/lustre2 PASS 31b (13s) == sanityn test 31r: open-rename(replace) race =========== 04:26:36 (1761121596) -rw-r--r-- 1 root root 0 Oct 22 04:26 /mnt/lustre/f31r.sanityn.target fail_loc=0x1419 fail_val=3 PASS 31r (5s) SKIP: sanityn test_33a skipping SLOW test 33a == sanityn test 33b: COS: cross create/delete, 2 clients, benchmark under remote dir ========================================================== 04:26:41 (1761121601) SKIP: sanityn test_33b Need two or more clients, have 1 SKIP 33b (1s) == sanityn test 33c: Cancel cross-MDT lock should trigger Sync-Lock-Cancel ========================================================== 04:26:43 (1761121603) SKIP: sanityn test_33c needs >= 2 MDTs SKIP 33c (1s) == sanityn test 33d: DNE distributed operation should trigger COS ========================================================== 04:26:44 (1761121604) SKIP: sanityn test_33d needs >= 2 MDTs SKIP 33d (1s) == sanityn test 33e: DNE local operation shouldn't trigger COS ========================================================== 04:26:45 (1761121605) SKIP: sanityn test_33e Need two or more clients, have 1 SKIP 33e (1s) == sanityn test 34: no lock timeout under IO ============= 04:26:46 (1761121606) fail_loc=0x512 fail_loc=0x512 lock should not expire writing on client1 reading on client2 success fail_loc=0x511 fail_loc=0x511 lock should expire writing on client1 reading on client2 success oleg416-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9a01c7fbe000.ost_server_uuid,osc.lustre-OST0000-osc-ffff9a01d93a6000.ost_server_uuid 40 osc.lustre-OST0000-osc-ffff9a01c7fbe000.ost_server_uuid in IDLE state after 0 sec osc.lustre-OST0000-osc-ffff9a01d93a6000.ost_server_uuid in IDLE state after 0 sec oleg416-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9a01c7fbe000.ost_server_uuid,osc.lustre-OST0001-osc-ffff9a01d93a6000.ost_server_uuid 40 osc.lustre-OST0001-osc-ffff9a01c7fbe000.ost_server_uuid in FULL state after 0 sec osc.lustre-OST0001-osc-ffff9a01d93a6000.ost_server_uuid in FULL state after 0 sec oleg416-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9a01c7fbe000.ost_server_uuid,osc.lustre-OST0000-osc-ffff9a01d93a6000.ost_server_uuid 40 osc.lustre-OST0000-osc-ffff9a01c7fbe000.ost_server_uuid in IDLE state after 0 sec osc.lustre-OST0000-osc-ffff9a01d93a6000.ost_server_uuid in IDLE state after 0 sec oleg416-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9a01c7fbe000.ost_server_uuid,osc.lustre-OST0001-osc-ffff9a01d93a6000.ost_server_uuid 40 osc.lustre-OST0001-osc-ffff9a01c7fbe000.ost_server_uuid in FULL state after 0 sec osc.lustre-OST0001-osc-ffff9a01d93a6000.ost_server_uuid in IDLE state after 0 sec oleg416-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9a01c7fbe000.ost_server_uuid,osc.lustre-OST0000-osc-ffff9a01d93a6000.ost_server_uuid 40 osc.lustre-OST0000-osc-ffff9a01c7fbe000.ost_server_uuid in IDLE state after 0 sec osc.lustre-OST0000-osc-ffff9a01d93a6000.ost_server_uuid in IDLE state after 0 sec oleg416-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9a01c7fbe000.ost_server_uuid,osc.lustre-OST0001-osc-ffff9a01d93a6000.ost_server_uuid 40 osc.lustre-OST0001-osc-ffff9a01c7fbe000.ost_server_uuid in FULL state after 0 sec osc.lustre-OST0001-osc-ffff9a01d93a6000.ost_server_uuid in IDLE state after 0 sec PASS 34 (79s) == sanityn test 35: -EINTR cp_ast vs. bl_ast race does not evict client ========================================================== 04:28:05 (1761121685) fail_loc=0x80000317 Race attempt 0 Wait for 45235 45365 for 60 sec... total: 4000 open/close in 8.79 seconds: 455.07 ops/second fail_loc=0x0 Filesystem Size Used Avail Use% Mounted on 192.168.204.116@tcp:/lustre 7.2G 18M 7.2G 1% /mnt/lustre 192.168.204.116@tcp:/lustre 7.2G 18M 7.2G 1% /mnt/lustre2 PASS 35 (65s) == sanityn test 36: handle ESTALE/open-unlink correctly == 04:29:10 (1761121750) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg416-server mds-ost sync done. Waiting for MDT destroys to complete 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.407546 s, 129 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7143 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(0) *** before(7520256) after_dd(7469056) after(7520256) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.355165 s, 148 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7143 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(1) *** before(7520256) after_dd(7469056) after(7520256) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.386802 s, 136 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7143 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(2) *** before(7520256) after_dd(7469056) after(7520256) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.37841 s, 139 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7143 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(3) *** before(7520256) after_dd(7469056) after(7520256) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.397985 s, 132 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7143 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(4) *** before(7520256) after_dd(7469056) after(7520256) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.384415 s, 136 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7143 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(5) *** before(7520256) after_dd(7469056) after(7520256) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.39492 s, 133 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7143 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(6) *** before(7520256) after_dd(7469056) after(7520256) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.363605 s, 144 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7143 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(7) *** before(7520256) after_dd(7469056) after(7520256) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.398375 s, 132 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7143 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(8) *** before(7520256) after_dd(7469056) after(7520256) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.325836 s, 161 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7143 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(9) *** before(7520256) after_dd(7469056) after(7520256) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.323524 s, 162 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7143 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(10) *** before(7520256) after_dd(7469056) after(7520256) PASS 36 (233s) == sanityn test 37: check i_size is not updated for directory on close (bug 18695) ======================================================================== 04:33:03 (1761121983) multiop /mnt/lustre/d37.sanityn vD_c TMPPIPE=/tmp/multiop_open_wait_pipe.7143 - create 8397 (time 1761121994.39 total 10.00 last 839.68) total: 10000 create in 11.96 seconds: 836.26 ops/second PASS 37 (15s) == sanityn test 39a: test from 11063 ============================================================================================ 04:33:18 (1761121998) repeat after cancel_lru_locks PASS 39a (3s) == sanityn test 39b: 11063 problem 1 ============================================================================================ 04:33:21 (1761122001) repeat after cancel_lru_locks PASS 39b (3s) == sanityn test 39c: check truncate mtime update ================================================================================ 04:33:24 (1761122004) repeat after cancel_lru_locks PASS 39c (3s) == sanityn test 39d: sync write should update mtime ====== 04:33:27 (1761122007) fail_loc=0x411 fail_loc=0 PASS 39d (2s) == sanityn test 40a: pdirops: create vs others ======================================================================== 04:33:29 (1761122009) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40a (8s) == sanityn test 40b: pdirops: open|create and others ======================================================================== 04:33:37 (1761122017) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40b (8s) == sanityn test 40c: pdirops: link and others ======================================================================== 04:33:45 (1761122025) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40c (9s) == sanityn test 40d: pdirops: unlink and others ======================================================================== 04:33:54 (1761122034) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40d (9s) == sanityn test 40e: pdirops: rename and others ======================================================================== 04:34:03 (1761122043) No conflict No conflict No conflict No conflict No conflict No conflict PASS 40e (8s) == sanityn test 41a: pdirops: create vs mkdir ======================================================================== 04:34:11 (1761122051) mkdir: cannot create directory '/mnt/lustre2/f41a.sanityn': File exists Conflict PASS 41a (5s) == sanityn test 41b: pdirops: create vs create ======================================================================== 04:34:16 (1761122056) open: File exists Conflict PASS 41b (5s) == sanityn test 41c: pdirops: create vs link ======================================================================== 04:34:21 (1761122061) link: cannot create link '/mnt/lustre2/f41c.sanityn' to '/mnt/lustre2/f41c.sanityn-2': File exists Conflict PASS 41c (5s) == sanityn test 41d: pdirops: create vs unlink ======================================================================== 04:34:26 (1761122066) Conflict PASS 41d (5s) == sanityn test 41e: pdirops: create and rename (tgt) ======================================================================== 04:34:31 (1761122071) Conflict PASS 41e (5s) == sanityn test 41f: pdirops: create and rename (src) ======================================================================== 04:34:37 (1761122077) Conflict PASS 41f (5s) == sanityn test 41g: pdirops: create vs getattr ======================================================================== 04:34:42 (1761122082) Conflict PASS 41g (5s) == sanityn test 41h: pdirops: create vs readdir ======================================================================== 04:34:47 (1761122087) Conflict PASS 41h (6s) == sanityn test 41i: reint_open: create vs create ======== 04:34:53 (1761122093) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 41i (767s) == sanityn test 42a: pdirops: mkdir vs mkdir ======================================================================== 04:47:40 (1761122860) mkdir: cannot create directory '/mnt/lustre2/f42a.sanityn': File exists Conflict PASS 42a (6s) == sanityn test 42b: pdirops: mkdir vs create ======================================================================== 04:47:46 (1761122866) open: File exists Conflict PASS 42b (7s) == sanityn test 42c: pdirops: mkdir vs link ======================================================================== 04:47:53 (1761122873) link: cannot create link '/mnt/lustre2/f42c.sanityn' to '/mnt/lustre2/f42c.sanityn-2': File exists Conflict PASS 42c (7s) == sanityn test 42d: pdirops: mkdir vs unlink ======================================================================== 04:48:00 (1761122880) Conflict PASS 42d (7s) == sanityn test 42e: pdirops: mkdir and rename (tgt) ======================================================================== 04:48:07 (1761122887) mv: cannot overwrite directory '/mnt/lustre2/f42e.sanityn' with non-directory Conflict PASS 42e (6s) == sanityn test 42f: pdirops: mkdir and rename (src) ======================================================================== 04:48:14 (1761122894) Conflict PASS 42f (6s) == sanityn test 42g: pdirops: mkdir vs getattr ======================================================================== 04:48:20 (1761122900) Conflict PASS 42g (6s) == sanityn test 42h: pdirops: mkdir vs readdir ======================================================================== 04:48:26 (1761122906) Conflict PASS 42h (7s) == sanityn test 43a: rmdir,mkdir doesn't return -EEXIST ======================================================================== 04:48:33 (1761122913) PASS 43a (25s) == sanityn test 43b: pdirops: unlink vs create ======================================================================== 04:48:58 (1761122938) Conflict PASS 43b (6s) == sanityn test 43c: pdirops: unlink vs link ======================================================================== 04:49:04 (1761122944) Conflict PASS 43c (7s) == sanityn test 43d: pdirops: unlink vs unlink ======================================================================== 04:49:11 (1761122951) rm: cannot remove '/mnt/lustre2/f43d.sanityn': No such file or directory Conflict PASS 43d (6s) == sanityn test 43e: pdirops: unlink and rename (tgt) ======================================================================== 04:49:17 (1761122957) Conflict PASS 43e (7s) == sanityn test 43f: pdirops: unlink and rename (src) ======================================================================== 04:49:24 (1761122964) mv: cannot stat '/mnt/lustre2/f43f.sanityn': No such file or directory Conflict PASS 43f (7s) == sanityn test 43g: pdirops: unlink vs getattr ======================================================================== 04:49:31 (1761122971) stat: cannot statx '/mnt/lustre2/f43g.sanityn': No such file or directory Conflict PASS 43g (7s) == sanityn test 43h: pdirops: unlink vs readdir ======================================================================== 04:49:38 (1761122978) Conflict PASS 43h (7s) == sanityn test 43i: pdirops: unlink vs remote mkdir ===== 04:49:45 (1761122985) SKIP: sanityn test_43i needs >= 2 MDTs SKIP 43i (1s) == sanityn test 43j: racy mkdir return EEXIST ======================================================================== 04:49:46 (1761122986) mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists PASS 43j (77s) == sanityn test 43k: unlink vs create ==================== 04:51:03 (1761123063) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 43k (638s) == sanityn test 44a: pdirops: rename tgt vs mkdir ======================================================================== 05:01:41 (1761123701) mkdir: cannot create directory '/mnt/lustre2/f44a.sanityn': File exists Conflict Wed Oct 22 05:01:46 EDT 2025 PASS 44a (7s) == sanityn test 44b: pdirops: rename tgt vs create ======================================================================== 05:01:48 (1761123708) open: File exists Conflict PASS 44b (6s) == sanityn test 44c: pdirops: rename tgt vs link ======================================================================== 05:01:54 (1761123714) link: cannot create link '/mnt/lustre2/f44c.sanityn' to '/mnt/lustre2/f44c.sanityn-3': File exists Conflict PASS 44c (7s) == sanityn test 44d: pdirops: rename tgt vs unlink ======================================================================== 05:02:01 (1761123721) Conflict PASS 44d (6s) == sanityn test 44e: pdirops: rename tgt and rename (tgt) ======================================================================== 05:02:07 (1761123727) Conflict PASS 44e (7s) == sanityn test 44f: pdirops: rename tgt and rename (src) ======================================================================== 05:02:14 (1761123734) Conflict PASS 44f (7s) == sanityn test 44g: pdirops: rename tgt vs getattr ======================================================================== 05:02:21 (1761123741) Conflict PASS 44g (6s) == sanityn test 44h: pdirops: rename tgt vs readdir ======================================================================== 05:02:27 (1761123747) Conflict PASS 44h (6s) == sanityn test 44i: pdirops: rename tgt vs remote mkdir ========================================================== 05:02:33 (1761123753) SKIP: sanityn test_44i needs >= 2 MDTs SKIP 44i (1s) == sanityn test 45a: rename,mkdir doesn't return -EEXIST ======================================================================== 05:02:35 (1761123755) PASS 45a (80s) == sanityn test 45b: pdirops: rename src vs create ======================================================================== 05:03:54 (1761123834) Conflict PASS 45b (13s) == sanityn test 45c: pdirops: rename src vs link ======================================================================== 05:04:07 (1761123847) Conflict PASS 45c (11s) == sanityn test 45d: pdirops: rename src vs unlink ======================================================================== 05:04:18 (1761123858) rm: cannot remove '/mnt/lustre2/f45d.sanityn': No such file or directory Conflict PASS 45d (11s) == sanityn test 45e: pdirops: rename src and rename (tgt) ======================================================================== 05:04:29 (1761123869) Conflict PASS 45e (10s) == sanityn test 45f: pdirops: rename src and rename (src) ======================================================================== 05:04:39 (1761123879) mv: cannot stat '/mnt/lustre2/f45f.sanityn': No such file or directory Conflict PASS 45f (10s) == sanityn test 45g: pdirops: rename src vs getattr ======================================================================== 05:04:49 (1761123889) stat: cannot statx '/mnt/lustre2/f45g.sanityn': No such file or directory Conflict PASS 45g (11s) == sanityn test 45h: pdirops: unlink vs readdir ======================================================================== 05:05:00 (1761123900) Conflict PASS 45h (10s) == sanityn test 45i: pdirops: rename src vs remote mkdir ========================================================== 05:05:10 (1761123910) SKIP: sanityn test_45i needs >= 2 MDTs SKIP 45i (2s) == sanityn test 45j: read vs rename ====================== 05:05:13 (1761123913) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 45j (757s) == sanityn test 46a: pdirops: link vs mkdir ======================================================================== 05:17:50 (1761124670) mkdir: cannot create directory '/mnt/lustre2/f46a.sanityn': File exists Conflict PASS 46a (8s) == sanityn test 46b: pdirops: link vs create ======================================================================== 05:17:58 (1761124678) open: File exists Conflict PASS 46b (7s) == sanityn test 46c: pdirops: link vs link ======================================================================== 05:18:05 (1761124685) link: cannot create link '/mnt/lustre2/f46c.sanityn' to '/mnt/lustre2/f46c.sanityn': File exists Conflict PASS 46c (7s) == sanityn test 46d: pdirops: link vs unlink ======================================================================== 05:18:12 (1761124692) Conflict PASS 46d (7s) == sanityn test 46e: pdirops: link and rename (tgt) ======================================================================== 05:18:20 (1761124700) Conflict PASS 46e (6s) == sanityn test 46f: pdirops: link and rename (src) ======================================================================== 05:18:26 (1761124706) Conflict PASS 46f (7s) == sanityn test 46g: pdirops: link vs getattr ======================================================================== 05:18:33 (1761124713) Conflict PASS 46g (7s) == sanityn test 46h: pdirops: link vs readdir ======================================================================== 05:18:40 (1761124720) Conflict PASS 46h (6s) == sanityn test 46i: pdirops: link vs remote mkdir ======= 05:18:46 (1761124726) SKIP: sanityn test_46i needs >= 2 MDTs SKIP 46i (1s) == sanityn test 47a: pdirops: remote mkdir vs mkdir ====== 05:18:47 (1761124727) SKIP: sanityn test_47a needs >= 2 MDTs SKIP 47a (2s) == sanityn test 47b: pdirops: remote mkdir vs create ===== 05:18:49 (1761124729) SKIP: sanityn test_47b needs >= 2 MDTs SKIP 47b (1s) == sanityn test 47c: pdirops: remote mkdir vs link ======= 05:18:50 (1761124730) SKIP: sanityn test_47c needs >= 2 MDTs SKIP 47c (1s) == sanityn test 47d: pdirops: remote mkdir vs unlink ===== 05:18:51 (1761124731) SKIP: sanityn test_47d needs >= 2 MDTs SKIP 47d (1s) == sanityn test 47e: pdirops: remote mkdir and rename (tgt) ========================================================== 05:18:53 (1761124733) SKIP: sanityn test_47e needs >= 2 MDTs SKIP 47e (1s) == sanityn test 47f: pdirops: remote mkdir and rename (src) ========================================================== 05:18:54 (1761124734) SKIP: sanityn test_47f needs >= 2 MDTs SKIP 47f (1s) == sanityn test 47g: pdirops: remote mkdir vs getattr ==== 05:18:55 (1761124735) SKIP: sanityn test_47g needs >= 2 MDTs SKIP 47g (1s) == sanityn test 50: osc lvb attrs: enqueue vs. CP AST ======================================================================== 05:18:56 (1761124736) 10+0 records in 10+0 records out 10240 bytes (10 kB, 10 KiB) copied, 0.00767105 s, 1.3 MB/s fail_loc=0x410 fail_loc=0x0 PASS 50 (8s) == sanityn test 51a: layout lock: refresh layout should work ========================================================== 05:19:04 (1761124744) 0+1 records in 0+1 records out 158 bytes copied, 0.00861139 s, 18.3 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 3178: kill: (243134) - No such process PASS 51a (5s) == sanityn test 51b: layout lock: glimpse should be able to restart if layout changed ========================================================== 05:19:09 (1761124749) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00933575 s, 110 kB/s fail_loc=0x1404 fail_val=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0271966 s, 38.6 MB/s 3145728 PASS 51b (15s) == sanityn test 51c: layout lock: IT_LAYOUT blocked and correct layout can be returned ========================================================== 05:19:24 (1761124764) fail_loc=0x172 Setting layout to have 2 stripes ... 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00861142 s, 119 kB/s PASS 51c (7s) == sanityn test 51d: layout lock: losing layout lock should clean up memory map region ========================================================== 05:19:32 (1761124772) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0263447 s, 39.8 MB/s Before revoking layout lock: 1024 KB mapped PASS 51d (5s) == sanityn test 51e: lfs getstripe does not break leases, part 2 ========================================================== 05:19:36 (1761124776) /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 2437 0x985 0 write lease(2) has applied. /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 2437 0x985 0 read lease(1) has applied. PASS 51e (5s) == sanityn test 54: rename locking ======================= 05:19:41 (1761124781) llite.lustre-ffff9a01c7fbe000.xattr_cache=1 llite.lustre-ffff9a01d93a6000.xattr_cache=1 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 512 Blocks: 1 IO Block: 131072 directory Device: 2c54f966h/743766374d Inode: 144115205272504977 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-10-22 05:19:41.000000000 -0400 Modify: 2025-10-22 05:19:41.000000000 -0400 Change: 2025-10-22 05:19:41.000000000 -0400 Birth: 2025-10-22 05:19:40.000000000 -0400 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000154 File: /mnt/lustre/d1/d2 Size: 11776 Blocks: 23 IO Block: 131072 directory Device: 2c54f966h/743766374d Inode: 144115205272504980 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-10-22 05:19:47.000000000 -0400 Modify: 2025-10-22 05:19:47.000000000 -0400 Change: 2025-10-22 05:19:47.000000000 -0400 Birth: 2025-10-22 05:19:46.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 11776 Blocks: 23 IO Block: 131072 directory Device: 2c54f966h/743766374d Inode: 144115205272504983 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-10-22 05:19:53.000000000 -0400 Modify: 2025-10-22 05:19:53.000000000 -0400 Change: 2025-10-22 05:19:53.000000000 -0400 Birth: 2025-10-22 05:19:52.000000000 -0400 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1 Size: 11776 Blocks: 23 IO Block: 131072 directory Device: 2c54f966h/743766374d Inode: 144115205272504982 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-10-22 05:19:53.000000000 -0400 Modify: 2025-10-22 05:19:53.000000000 -0400 Change: 2025-10-22 05:19:53.000000000 -0400 Birth: 2025-10-22 05:19:52.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000154 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1/d2 Size: 11776 Blocks: 23 IO Block: 131072 directory Device: 2c54f966h/743766374d Inode: 144115205272504986 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-10-22 05:19:58.000000000 -0400 Modify: 2025-10-22 05:19:59.000000000 -0400 Change: 2025-10-22 05:19:59.000000000 -0400 Birth: 2025-10-22 05:19:57.000000000 -0400 File: /mnt/lustre/d1 Size: 11776 Blocks: 23 IO Block: 131072 directory Device: 2c54f966h/743766374d Inode: 144115205272504985 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-10-22 05:19:58.000000000 -0400 Modify: 2025-10-22 05:19:59.000000000 -0400 Change: 2025-10-22 05:19:59.000000000 -0400 Birth: 2025-10-22 05:19:57.000000000 -0400 PASS 54 (25s) == sanityn test 55a: rename vs unlink target dir ========= 05:20:06 (1761124806) fail_loc=0x80000156 mv: cannot move '/mnt/lustre/d55a.sanityn/d1/d2' to '/mnt/lustre/d55a.sanityn/d3/d2': No such file or directory PASS 55a (8s) == sanityn test 55b: rename vs unlink source dir ========= 05:20:14 (1761124814) fail_loc=0x80000156 mv: cannot stat '/mnt/lustre/d55b.sanityn/d1/d2': No such file or directory PASS 55b (9s) == sanityn test 55c: rename vs unlink orphan target dir == 05:20:23 (1761124823) fail_loc=0x156 mv: cannot move '/mnt/lustre/d55c.sanityn/d1/d2' to '/mnt/lustre/d55c.sanityn/d3/d2': No such file or directory PASS 55c (14s) == sanityn test 55d: rename file vs link ================= 05:20:37 (1761124837) fail_loc=0x155 mv: '/mnt/lustre/d55d.sanityn/f1' and '/mnt/lustre/d55d.sanityn/d55d.sanityn/f1' are the same file PASS 55d (11s) == sanityn test 60: Verify data_version behaviour ======== 05:20:48 (1761124848) PASS 60 (3s) == sanityn test 70a: cd directory && rm directory ======== 05:20:52 (1761124852) PASS 70a (3s) == sanityn test 70b: remove files after calling rm_entry ========================================================== 05:20:55 (1761124855) PASS 70b (3s) == sanityn test 71a: correct file map just after write operation is finished ========================================================== 05:20:58 (1761124858) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.00490413 s, 13.4 MB/s SKIP: sanityn test_71a ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71a (2s) == sanityn test 71b: check fiemap support for stripecount > 1 ========================================================== 05:21:00 (1761124860) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.00718406 s, 9.1 MB/s SKIP: sanityn test_71b ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71b (1s) == sanityn test 71c: check FIEMAP_EXTENT_LAST flag with different extents number ========================================================== 05:21:01 (1761124861) SKIP: sanityn test_71c support only ldiskfs ost SKIP 71c (2s) == sanityn test 71d: fiemap corruption test with fm_extent_count=0 ========================================================== 05:21:03 (1761124863) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.00691915 s, 9.5 MB/s SKIP: sanityn test_71d ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71d (2s) == sanityn test 72: getxattr/setxattr cache should be consistent between nodes ========================================================== 05:21:05 (1761124865) llite.lustre-ffff9a01c7fbe000.xattr_cache=1 llite.lustre-ffff9a01d93a6000.xattr_cache=1 getfattr: Removing leading '/' from absolute path names user.attr1="value1" getfattr: Removing leading '/' from absolute path names user.attr1="value2" getfattr: Removing leading '/' from absolute path names getfattr: Removing leading '/' from absolute path names b362740ad371eae580e7f85452e8830b - 870e072da1ebfc20ea8c0cd3d6bdf3b6 - PASS 72 (3s) == sanityn test 73: getxattr should not cause xattr lock cancellation ========================================================== 05:21:08 (1761124868) llite.lustre-ffff9a01c7fbe000.xattr_cache=1 llite.lustre-ffff9a01d93a6000.xattr_cache=1 getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" PASS 73 (2s) == sanityn test 74: flock deadlock: different mounts ======================================================================== 05:21:10 (1761124870) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00635645 s, 161 kB/s 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0137707 s, 74.4 kB/s 253179: get lock2 253180: get lock1 253179: done 253179: get lock1 253180: done 253180: get lock2 fcntl cmd 7 failed: Resource deadlock avoided 253180: cannot get lock2: Resource deadlock avoided 253180: exit rc=0 fcntl cmd 7 failed: Resource deadlock avoided 253179: cannot get lock1: Resource deadlock avoided 253179: done 253179: exit rc=0 PASS 74 (7s) == sanityn test 75: osc: upcall after unuse lock============================================================================= 05:21:17 (1761124877) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0279367 s, 75.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0117631 s, 89.1 MB/s fail_loc=0x31d /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 3746: 253753 Killed stat -c %s $DIR1/$tfile fail_loc=0 PASS 75 (8s) == sanityn test 76: Verify MDT open_files listing ======== 05:21:25 (1761124885) opening files: ........ closing files: ........ PASS 76 (26s) == sanityn test 77a: check FIFO NRS policy =============== 05:21:51 (1761124911) ost.OSS.ost.nrs_policies=fifo ost.OSS.ost_create.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_out.nrs_policies=fifo ost.OSS.ost_seq.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.159747 s, 105 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0315374 s, 33.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0339409 s, 30.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1228 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134302 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113973 s, 9.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9a01d93a6000.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff9a01d93a6000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12503 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138329 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1409 s, 7.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17289 s, 6.1 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177491 s, 5.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166142 s, 6.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16529 s, 6.3 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169471 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18002 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183273 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174935 s, 6.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9a01d93a6000.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffff9a01d93a6000.lock_unused_count=3 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0249221 s, 42.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0278129 s, 37.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0346441 s, 30.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0425301 s, 24.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00477324 s, 220 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0307515 s, 34.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00477189 s, 220 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00544578 s, 193 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0304894 s, 34.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0373665 s, 28.1 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0306694 s, 34.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183465 s, 57.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0381182 s, 27.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0434711 s, 24.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0589739 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0413699 s, 25.3 MB/s removed '/mnt/lustre/d77a.sanityn/nrs_w_oleg416-client.virtnet' removed '/mnt/lustre/d77a.sanityn/nrs_r_oleg416-client.virtnet' removed directory '/mnt/lustre/d77a.sanityn' PASS 77a (4s) == sanityn test 77b: check CRR-N NRS policy ============== 05:21:55 (1761124915) ost.OSS.ost.nrs_policies=crrn ost.OSS.ost_create.nrs_policies=crrn ost.OSS.ost_io.nrs_policies=crrn ost.OSS.ost_out.nrs_policies=crrn ost.OSS.ost_seq.nrs_policies=crrn ost.OSS.ost.nrs_crrn_quantum=1 ost.OSS.ost_create.nrs_crrn_quantum=1 ost.OSS.ost_io.nrs_crrn_quantum=1 ost.OSS.ost_out.nrs_crrn_quantum=1 ost.OSS.ost_seq.nrs_crrn_quantum=1 policy: crr-n, crrn_quantum 1 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.190749 s, 88.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0330441 s, 31.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0384326 s, 27.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.098734 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10852 s, 9.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9a01d93a6000.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111717 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120905 s, 8.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116078 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112094 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116316 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12978 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132891 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131847 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12556 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107246 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120835 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126706 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0359377 s, 29.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0368181 s, 28.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0328223 s, 31.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0384707 s, 27.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0365429 s, 28.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0258609 s, 40.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0298993 s, 35.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0327016 s, 32.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0347455 s, 30.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0329425 s, 31.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0315884 s, 33.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0366705 s, 28.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0361868 s, 29.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.04311 s, 24.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0386771 s, 27.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9a01d93a6000.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff9a01d93a6000.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0495658 s, 21.2 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg416-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg416-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost.nrs_crrn_quantum=64 ost.OSS.ost_create.nrs_crrn_quantum=64 ost.OSS.ost_io.nrs_crrn_quantum=64 ost.OSS.ost_out.nrs_crrn_quantum=64 ost.OSS.ost_seq.nrs_crrn_quantum=64 policy: crr-n, crrn_quantum 64 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.159301 s, 105 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0307638 s, 34.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0310643 s, 33.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0392317 s, 26.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0898635 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0801067 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.059187 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0876966 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0990789 s, 10.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9a01d93a6000.lock_unused_count=3 ldlm.namespaces.lustre-OST0001-osc-ffff9a01d93a6000.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104339 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1123 s, 9.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0813885 s, 12.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102013 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118672 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0998836 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104717 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122951 s, 8.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0195197 s, 53.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158282 s, 66.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0302356 s, 34.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0234119 s, 44.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0238847 s, 43.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0244852 s, 42.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016508 s, 63.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0191457 s, 54.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0196368 s, 53.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0271506 s, 38.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out ldlm.namespaces.lustre-OST0001-osc-ffff9a01d93a6000.lock_unused_count=1 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0298095 s, 35.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0044541 s, 235 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0357452 s, 29.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0281459 s, 37.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0322454 s, 32.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0361523 s, 29.0 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg416-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg416-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77b (6s) == sanityn test 77c: check ORR NRS policy ================ 05:22:01 (1761124921) ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_orr_quantum=1 ost.OSS.ost_io.nrs_orr_offset_type=physical ost.OSS.ost_io.nrs_orr_supported=reads policy: orr, orr_quantum 1, orr_offset_type physical, orr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.152979 s, 110 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0392582 s, 26.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0303314 s, 34.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0669105 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0595871 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0579285 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0726966 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0683276 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0770195 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0552155 s, 19.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0784925 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108653 s, 9.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128182 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11639 s, 9.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0988443 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139229 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143059 s, 7.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0360376 s, 29.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0366144 s, 28.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0299708 s, 35.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0209739 s, 50.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0262367 s, 40.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0218821 s, 47.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0274974 s, 38.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0229307 s, 45.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0229646 s, 45.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0232142 s, 45.2 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0260096 s, 40.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.026128 s, 40.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0274837 s, 38.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.028954 s, 36.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0322157 s, 32.5 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9a01d93a6000.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0331117 s, 31.7 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg416-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg416-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=writes ost.OSS.ost_io.nrs_orr_quantum=64 policy: orr, orr_quantum 64, orr_offset_type physical, orr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.173843 s, 96.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0614525 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0807935 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0729387 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0847829 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0870245 s, 12.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0912703 s, 11.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0811616 s, 12.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.098207 s, 10.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101897 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0594543 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0969509 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110078 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106029 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0978272 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0898194 s, 11.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9a01d93a6000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9a01d93a6000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0917509 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.031031 s, 33.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.023154 s, 45.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.022761 s, 46.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0269095 s, 39.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183724 s, 57.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.029133 s, 36.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0204722 s, 51.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144357 s, 72.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0322508 s, 32.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00999284 s, 105 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131905 s, 79.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0274088 s, 38.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173841 s, 60.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0322541 s, 32.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0337989 s, 31.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00560814 s, 187 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9a01d93a6000.lock_unused_count=1 removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg416-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg416-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=reads_and_writes ost.OSS.ost_io.nrs_orr_offset_type=logical policy: orr, orr_quantum 64, orr_offset_type logical, orr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.154832 s, 108 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0341583 s, 30.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0346338 s, 30.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0505732 s, 20.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.048353 s, 21.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0510447 s, 20.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0558987 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0736189 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0737425 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0663702 s, 15.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9a01d93a6000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117885 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10561 s, 9.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112469 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118824 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146364 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0250022 s, 41.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015073 s, 69.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0295712 s, 35.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12903 s, 8.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0115231 s, 91.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0255432 s, 41.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00539563 s, 194 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0298681 s, 35.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0180623 s, 58.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00573763 s, 183 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0446564 s, 23.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0120109 s, 87.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167845 s, 62.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.049624 s, 21.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00591761 s, 177 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147489 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00302336 s, 347 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0023428 s, 448 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg416-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg416-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77c (7s) == sanityn test 77d: check TRR nrs policy ================ 05:22:08 (1761124928) ost.OSS.ost_io.nrs_policies=trr ost.OSS.ost_io.nrs_trr_quantum=1 ost.OSS.ost_io.nrs_trr_offset_type=physical ost.OSS.ost_io.nrs_trr_supported=reads policy: trr, trr_quantum 1, trr_offset_type physical, trr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.150431 s, 112 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0390435 s, 26.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0500187 s, 21.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0808814 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0862502 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.082206 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0813419 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103563 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.089151 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100178 s, 10.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0896056 s, 11.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9a01d93a6000.lock_unused_count=3 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0823113 s, 12.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110162 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0844614 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104435 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122983 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101445 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0123695 s, 84.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0123178 s, 85.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136276 s, 76.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0237851 s, 44.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0176655 s, 59.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0188231 s, 55.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0205719 s, 51.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0190383 s, 55.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.02473 s, 42.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0228418 s, 45.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.037567 s, 27.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0298057 s, 35.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0270756 s, 38.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.026451 s, 39.6 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9a01d93a6000.lock_unused_count=2 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0306397 s, 34.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0308714 s, 34.0 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg416-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg416-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=writes ost.OSS.ost_io.nrs_trr_quantum=64 policy: trr, trr_quantum 64, trr_offset_type physical, trr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.155705 s, 108 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0559382 s, 18.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0605432 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0647107 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.057329 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.071097 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0697593 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0915851 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0886073 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0843694 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.082261 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.070824 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104496 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0969437 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0731217 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0913646 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0742029 s, 14.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0293047 s, 35.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0230635 s, 45.5 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0208588 s, 50.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0215724 s, 48.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0228946 s, 45.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0255746 s, 41.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0251715 s, 41.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.026892 s, 39.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0200946 s, 52.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.034004 s, 30.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9a01d93a6000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9a01d93a6000.lock_unused_count=1 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0304361 s, 34.5 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0261284 s, 40.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0248768 s, 42.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0316476 s, 33.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0303567 s, 34.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0313989 s, 33.4 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg416-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg416-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=reads_and_writes ost.OSS.ost_io.nrs_trr_offset_type=logical policy: trr, trr_quantum 64, trr_offset_type logical, trr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.213961 s, 78.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0482057 s, 21.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0608356 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0642178 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0638395 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0904905 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0905956 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.087966 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0705891 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0823599 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11297 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0926218 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0836358 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0735057 s, 14.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111702 s, 9.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123718 s, 8.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9a01d93a6000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9a01d93a6000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.099054 s, 10.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0337706 s, 31.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0442964 s, 23.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0328774 s, 31.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.038556 s, 27.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0311755 s, 33.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0392146 s, 26.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0364368 s, 28.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0329912 s, 31.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0377449 s, 27.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0433448 s, 24.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0359855 s, 29.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0460326 s, 22.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.038647 s, 27.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0484505 s, 21.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0489339 s, 21.4 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9a01d93a6000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0464087 s, 22.6 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg416-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg416-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77d (6s) == sanityn test 77e: check TBF NID nrs policy ============ 05:22:14 (1761124934) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_tbf_rule=start localhost nid={0@lo} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start clients nid={192.168.204.16@tcp} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start others nid={*.*.*.*@tcp} rate=50 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.190386 s, 88.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0221402 s, 47.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0330259 s, 31.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0737436 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0470122 s, 22.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0833226 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0668556 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0839699 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0526864 s, 19.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0884385 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0815219 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0946434 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0930934 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0827692 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0850158 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0385445 s, 27.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0842398 s, 12.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0316285 s, 33.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0112628 s, 93.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0360903 s, 29.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0222306 s, 47.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0279423 s, 37.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139848 s, 75.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156404 s, 67.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161382 s, 65.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0373606 s, 28.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0240774 s, 43.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0206917 s, 50.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0207834 s, 50.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0225789 s, 46.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0222119 s, 47.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0235577 s, 44.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0411673 s, 25.5 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg416-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg416-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change localhost rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change clients rate=101 ost.OSS.ost_io.nrs_tbf_rule=change others rate=51 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.151986 s, 110 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0409701 s, 25.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0305272 s, 34.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0464209 s, 22.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0411122 s, 25.5 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0576731 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0341272 s, 30.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0285139 s, 36.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0412663 s, 25.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.045719 s, 22.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0946805 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105451 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114336 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0915197 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.095174 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102518 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114344 s, 9.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0298482 s, 35.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012863 s, 81.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142101 s, 73.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0229924 s, 45.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00965799 s, 109 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0103406 s, 101 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0241746 s, 43.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142967 s, 73.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0209029 s, 50.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00413121 s, 254 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0174981 s, 59.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00426284 s, 246 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00502245 s, 209 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00431987 s, 243 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00401012 s, 261 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00469562 s, 223 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg416-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg416-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop localhost ost.OSS.ost_io.nrs_tbf_rule=stop clients ost.OSS.ost_io.nrs_tbf_rule=stop others 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.173784 s, 96.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0298359 s, 35.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0841409 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.087476 s, 12.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.073621 s, 14.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0965737 s, 10.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0648143 s, 16.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0855787 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106974 s, 9.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100682 s, 10.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0947682 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0800919 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11483 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.09927 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117305 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0886888 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111833 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0199192 s, 52.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0279539 s, 37.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00296201 s, 354 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9a01d93a6000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00307223 s, 341 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0899683 s, 11.7 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0953156 s, 11.0 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0308003 s, 34.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0385724 s, 27.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0469614 s, 22.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00292068 s, 359 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0484844 s, 21.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0248658 s, 42.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0174381 s, 60.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0104999 s, 99.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00867103 s, 121 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00974544 s, 108 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg416-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg416-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.196452 s, 85.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.04415 s, 23.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0424896 s, 24.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0623386 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0707914 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0540204 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.067157 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0605033 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0727997 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0659847 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0873685 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104296 s, 10.1 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124828 s, 8.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108109 s, 9.7 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108105 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122587 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141538 s, 7.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0197339 s, 53.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0171259 s, 61.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0274325 s, 38.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0324121 s, 32.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0221345 s, 47.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0200518 s, 52.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0214246 s, 48.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0231097 s, 45.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0225562 s, 46.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0220609 s, 47.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00709008 s, 148 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00455812 s, 230 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0254609 s, 41.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0258611 s, 40.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0281248 s, 37.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0402997 s, 26.0 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9a01d93a6000.lock_unused_count=4 removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg416-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg416-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' PASS 77e (11s) == sanityn test 77f: check TBF JobID nrs policy ========== 05:22:25 (1761124945) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start runas jobid={iozone.500 dd.500 tiotest.500} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start iozone_runas jobid={iozone.500} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg416-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.207479 s, 80.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: [declare] [-a] [pids_w] execvp fails running declare (2): No such file or directory 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0427219 s, 24.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0480319 s, 21.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0434892 s, 24.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0744142 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0744492 s, 14.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0840375 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0744334 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0722574 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0701704 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0885051 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107354 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0819382 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0886243 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0746442 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0801086 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103773 s, 10.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9a01d93a6000.lock_unused_count=1 running as uid/gid/euid/egid 500/500/500/500, groups: [declare] [-a] [pids_r] execvp fails running declare (2): No such file or directory 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0267174 s, 39.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0227057 s, 46.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0185365 s, 56.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0265477 s, 39.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0199851 s, 52.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0204404 s, 51.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0208554 s, 50.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0236968 s, 44.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.034522 s, 30.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0251646 s, 41.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0365721 s, 28.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0279782 s, 37.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0268668 s, 39.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0284386 s, 36.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0287836 s, 36.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0323993 s, 32.4 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg416-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg416-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change runas rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change iozone_runas rate=101 ost.OSS.ost_io.nrs_tbf_rule=change dd_runas rate=51 running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg416-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.175499 s, 95.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: [declare] [-a] [pids_w] execvp fails running declare (2): No such file or directory 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0208382 s, 50.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0324978 s, 32.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0398161 s, 26.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.042337 s, 24.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105757 s, 9.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119394 s, 8.8 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122049 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111895 s, 9.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145654 s, 7.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137713 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137222 s, 7.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135089 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142965 s, 7.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150336 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139516 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155401 s, 6.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9a01d93a6000.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffff9a01d93a6000.lock_unused_count=3 running as uid/gid/euid/egid 500/500/500/500, groups: [declare] [-a] [pids_r] execvp fails running declare (2): No such file or directory 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0207384 s, 50.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0364231 s, 28.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142084 s, 73.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0185843 s, 56.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0245752 s, 42.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.021819 s, 48.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0253456 s, 41.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0269801 s, 38.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0271911 s, 38.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0283768 s, 37.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0264488 s, 39.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0295581 s, 35.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0285086 s, 36.8 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0288407 s, 36.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0306246 s, 34.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0383624 s, 27.3 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg416-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg416-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop runas ost.OSS.ost_io.nrs_tbf_rule=stop iozone_runas ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg416-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.160466 s, 105 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: [declare] [-a] [pids_w] execvp fails running declare (2): No such file or directory 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0516566 s, 20.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.098217 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.092308 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0896602 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0926502 s, 11.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0842116 s, 12.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0973995 s, 10.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9a01d93a6000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10949 s, 9.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126356 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112837 s, 9.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124617 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121872 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10516 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136772 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146261 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141052 s, 7.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: [declare] [-a] [pids_r] execvp fails running declare (2): No such file or directory 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0110223 s, 95.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0121081 s, 86.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0120085 s, 87.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0205022 s, 51.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0202228 s, 51.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0103034 s, 102 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0415197 s, 25.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00599852 s, 175 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9a01d93a6000.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff9a01d93a6000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00427237 s, 245 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00449914 s, 233 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00500606 s, 209 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0398534 s, 26.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00929848 s, 113 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0079472 s, 132 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0327473 s, 32.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00448836 s, 234 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg416-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg416-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_policies=fifo running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg416-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.233513 s, 71.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: [declare] [-a] [pids_w] execvp fails running declare (2): No such file or directory 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0441876 s, 23.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0472474 s, 22.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0553959 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0513097 s, 20.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0348807 s, 30.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0509365 s, 20.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0859776 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0745032 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0779102 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102866 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113601 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0892662 s, 11.7 MB/s 1+0 records in 1+0 records out ldlm.namespaces.lustre-OST0000-osc-ffff9a01d93a6000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9a01d93a6000.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115198 s, 9.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10253 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120872 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108473 s, 9.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: [declare] [-a] [pids_r] execvp fails running declare (2): No such file or directory 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0176436 s, 59.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00694465 s, 151 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0223178 s, 47.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00304638 s, 344 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139094 s, 75.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00407052 s, 258 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00405727 s, 258 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00428836 s, 245 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0042686 s, 246 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00343469 s, 305 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00445398 s, 235 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00434742 s, 241 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173342 s, 60.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0512949 s, 20.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00348291 s, 301 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0054462 s, 193 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg416-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg416-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' PASS 77f (10s) == sanityn test 77g: Change TBF type directly ============ 05:22:35 (1761124955) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 ost.OSS.ost_io.nrs_policies=fifo PASS 77g (5s) == sanityn test 77h: Wrong policy name should report error, not LBUG ========================================================== 05:22:40 (1761124960) oleg416-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=abc: No such file or directory pdsh@oleg416-client: oleg416-server: ssh exited with exit code 2 oleg416-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc: Unknown error 524 pdsh@oleg416-client: oleg416-server: ssh exited with exit code 12 oleg416-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf reg abc: Unknown error 524 pdsh@oleg416-client: oleg416-server: ssh exited with exit code 12 oleg416-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc efg: Unknown error 524 pdsh@oleg416-client: oleg416-server: ssh exited with exit code 12 PASS 77h (5s) == sanityn test 77i: Change rank of TBF rule ============= 05:22:45 (1761124965) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start before jobid={jobid} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start after jobid={jobid} rate=1000 rank=default ost.OSS.ost_io.nrs_tbf_rule=start target jobid={jobid} rate=1000 rank=after Move before itself ost.OSS.ost_io.nrs_tbf_rule=change target rank=target Move to higher rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=before Move to lower rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=after Move before default ost.OSS.ost_io.nrs_tbf_rule=change target rank=default ost.OSS.ost_io.nrs_policies=fifo PASS 77i (10s) == sanityn test 77j: check TBF-OPCode NRS policy ========= 05:22:55 (1761124975) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf opcode ost.OSS.ost_io.nrs_tbf_rule=start ost_r opcode={ost_read} rate=5 ost.OSS.ost_io.nrs_tbf_rule=start ost_w opcode={ost_write} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.224737 s, 74.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0321529 s, 32.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0431424 s, 24.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.084866 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0765684 s, 13.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0820572 s, 12.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109158 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109588 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101806 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105716 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102805 s, 10.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124049 s, 8.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139779 s, 7.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165958 s, 6.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179534 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223561 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.247102 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0207064 s, 50.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0263624 s, 39.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00437289 s, 240 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135593 s, 77.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0185503 s, 56.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169595 s, 61.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157045 s, 66.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164081 s, 63.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00449807 s, 233 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0184525 s, 56.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0260339 s, 40.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0288365 s, 36.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0258518 s, 40.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00352131 s, 298 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00545135 s, 192 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00733818 s, 143 MB/s removed '/mnt/lustre/d77j.sanityn/nrs_w_oleg416-client.virtnet' removed '/mnt/lustre/d77j.sanityn/nrs_r_oleg416-client.virtnet' removed directory '/mnt/lustre/d77j.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 5 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.03144 s, 20.8 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.4895 s, 5.4 MB/s Read runtime is 20 s, speed is 5.000000 IOPS cleanup_tbf /mnt/lustre/d77j.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_r ost.OSS.ost_io.nrs_tbf_rule=stop ost_w ost.OSS.ost_io.nrs_policies=fifo PASS 77j (48s) == sanityn test 77ja: check TBF-UID/GID NRS policy ======= 05:23:43 (1761125023) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg416-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.494288 s, 33.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: [declare] [-a] [pids_w] execvp fails running declare (2): No such file or directory 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0595037 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0653653 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0758334 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0513093 s, 20.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0943789 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0586922 s, 17.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0781934 s, 13.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0675092 s, 15.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.305939 s, 3.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.323426 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.329781 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.319346 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.341502 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.330506 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.339909 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.336274 s, 3.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: [declare] [-a] [pids_r] execvp fails running declare (2): No such file or directory 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13301 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124792 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12996 s, 8.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128112 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1315 s, 8.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131687 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133184 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13404 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132967 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140173 s, 7.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13591 s, 7.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142931 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136222 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134269 s, 7.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137135 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137864 s, 7.6 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg416-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg416-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8081 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9576 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg416-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.675797 s, 24.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: [declare] [-a] [pids_w] execvp fails running declare (2): No such file or directory 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132528 s, 7.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126874 s, 8.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138662 s, 7.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143436 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147311 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146589 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153428 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158042 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.328236 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.356063 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.365924 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.351273 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.355157 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.373811 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.372549 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.393485 s, 2.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: [declare] [-a] [pids_r] execvp fails running declare (2): No such file or directory 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12451 s, 8.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135958 s, 7.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122674 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13305 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14301 s, 7.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140379 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127129 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136648 s, 7.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129445 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148477 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15102 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135396 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136941 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138036 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138845 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139764 s, 7.5 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg416-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg416-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.7169 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9665 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77ja (138s) == sanityn test 77k: check TBF policy with NID/JobID/OPCode expression ========================================================== 05:26:01 (1761125161) ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg416-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.246146 s, 68.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: [declare] [-a] [pids_w] execvp fails running declare (2): No such file or directory 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0371118 s, 28.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0268582 s, 39.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0544495 s, 19.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0683257 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0570268 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0399721 s, 26.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0692461 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0867757 s, 12.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109221 s, 9.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0967421 s, 10.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109447 s, 9.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0866865 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117451 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109489 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118106 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0985893 s, 10.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: [declare] [-a] [pids_r] execvp fails running declare (2): No such file or directory 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0180184 s, 58.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00567186 s, 185 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153358 s, 68.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0637784 s, 16.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0604628 s, 17.3 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0546291 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0325142 s, 32.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0285865 s, 36.7 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0251017 s, 41.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00275857 s, 380 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0385365 s, 27.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0237739 s, 44.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147085 s, 71.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131914 s, 79.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0100634 s, 104 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0198835 s, 52.7 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg416-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg416-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.97831 s, 21.1 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.76281 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.204.16@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.204.16@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.172958 s, 97.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0331485 s, 31.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0402148 s, 26.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0716897 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0595155 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0878112 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0872315 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.091991 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0892032 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0843244 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0803516 s, 13.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135429 s, 7.7 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118385 s, 8.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22707 s, 4.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.219726 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.311252 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.311385 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0121671 s, 86.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127694 s, 82.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0203032 s, 51.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0180546 s, 58.1 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0188324 s, 55.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0191688 s, 54.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0199469 s, 52.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159197 s, 65.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.018596 s, 56.4 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0194331 s, 54.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183488 s, 57.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00613081 s, 171 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00387885 s, 270 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00613995 s, 171 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0217903 s, 48.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00773741 s, 136 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg416-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg416-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.94425 s, 21.2 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.74874 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.204.16@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg416-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.212889 s, 78.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: [declare] [-a] [pids_w] execvp fails running declare (2): No such file or directory 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0415469 s, 25.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0505232 s, 20.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0946521 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0943948 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113535 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106584 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0959965 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111306 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0958671 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115539 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101419 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106163 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112958 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109659 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110595 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129536 s, 8.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: [declare] [-a] [pids_r] execvp fails running declare (2): No such file or directory 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0204544 s, 51.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0114328 s, 91.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0200746 s, 52.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0200863 s, 52.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141129 s, 74.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166543 s, 63.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173576 s, 60.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0217285 s, 48.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0214791 s, 48.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0211906 s, 49.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0232765 s, 45.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0234046 s, 44.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0228727 s, 45.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00414434 s, 253 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.025047 s, 41.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0254995 s, 41.1 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg416-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg416-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.95827 s, 21.1 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.90362 s, 21.4 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg416-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.301481 s, 55.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: [declare] [-a] [pids_w] execvp fails running declare (2): No such file or directory 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0313989 s, 33.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0745352 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0755688 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0884173 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0906462 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0748453 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0749287 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0856903 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110849 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127743 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.226739 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23977 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.300096 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.286541 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.299894 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.290858 s, 3.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: [declare] [-a] [pids_r] execvp fails running declare (2): No such file or directory 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.023146 s, 45.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0391281 s, 26.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128371 s, 81.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00665882 s, 157 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00602607 s, 174 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00335215 s, 313 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.010345 s, 101 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00444349 s, 236 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00393279 s, 267 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127651 s, 82.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0433679 s, 24.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0304907 s, 34.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0349754 s, 30.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00925236 s, 113 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00640515 s, 164 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00447892 s, 234 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg416-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg416-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.88518 s, 10.6 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.76058 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.96054 s, 21.1 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.7923 s, 10.7 MB/s Read runtime is 10 s, speed is 10.000000 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={1000} rate=5 running as uid/gid/euid/egid 500/1000/500/1000, groups: [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg416-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.821387 s, 20.4 MB/s running as uid/gid/euid/egid 500/1000/500/1000, groups: [declare] [-a] [pids_w] execvp fails running declare (2): No such file or directory 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0736458 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126598 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150059 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164442 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205639 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18846 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207134 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197138 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.26271 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.318821 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.302633 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.351069 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.334716 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.358564 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.313959 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.396656 s, 2.6 MB/s running as uid/gid/euid/egid 500/1000/500/1000, groups: [declare] [-a] [pids_r] execvp fails running declare (2): No such file or directory 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0243822 s, 43.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0818049 s, 12.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0565444 s, 18.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0247166 s, 42.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0570929 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0648019 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0763493 s, 13.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0738176 s, 14.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0496942 s, 21.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00942274 s, 111 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00648795 s, 162 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105905 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.030609 s, 34.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00557041 s, 188 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013361 s, 78.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0293596 s, 35.7 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg416-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg416-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/1000/500/1000, groups: [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.0267 s, 5.2 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/1000/500/1000, groups: [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.6923 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_ug ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg416-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.335628 s, 50.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: [declare] [-a] [pids_w] execvp fails running declare (2): No such file or directory 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0466112 s, 22.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0542796 s, 19.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0802884 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12749 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123453 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127595 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108811 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126182 s, 8.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145413 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132116 s, 7.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149716 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149646 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128298 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14201 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196828 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165095 s, 6.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: [declare] [-a] [pids_r] execvp fails running declare (2): No such file or directory 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00469103 s, 224 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0472293 s, 22.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00338113 s, 310 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00493736 s, 212 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00406683 s, 258 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0699727 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.002898 s, 362 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0874288 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0639379 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0806363 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0956792 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140633 s, 74.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0318822 s, 32.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114625 s, 9.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.025993 s, 40.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0641258 s, 16.4 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg416-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg416-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.14509 s, 17.1 MB/s Write runtime is 7 s, speed is 14.285714 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.86499 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg416-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.395511 s, 42.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: [declare] [-a] [pids_w] execvp fails running declare (2): No such file or directory 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0287106 s, 36.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0447362 s, 23.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120864 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10714 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135775 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140578 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15662 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160543 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157009 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168929 s, 6.2 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177128 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202496 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.321109 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.322083 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.449829 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.452314 s, 2.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: [declare] [-a] [pids_r] execvp fails running declare (2): No such file or directory 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0462265 s, 22.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00256025 s, 410 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.092973 s, 11.3 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0730165 s, 14.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0872514 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0801849 s, 13.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00699072 s, 150 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103857 s, 10.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0531811 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.044218 s, 23.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110975 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106382 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0410243 s, 25.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0267583 s, 39.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0324538 s, 32.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0212159 s, 49.4 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg416-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg416-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0239 s, 10.5 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.91002 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0305 s, 10.5 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.85547 s, 10.6 MB/s Read runtime is 10 s, speed is 10.000000 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_policies=fifo PASS 77k (390s) == sanityn test 77l: check the output of NRS policies for generic TBF ========================================================== 05:32:31 (1761125551) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_policies=fifo PASS 77l (7s) == sanityn test 77m: check NRS Delay slows write RPC processing ========================================================== 05:32:39 (1761125559) ost.OSS.ost_io.nrs_policies=delay ost.OSS.ost_io.nrs_delay_min=4 ost.OSS.ost_io.nrs_delay_max=4 ost.OSS.ost_io.nrs_delay_pct=100 running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg416-client.virtnet] [bs=1M] [count=1] [oflag=direct] [conv=fdatasync] 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 3.56913 s, 294 kB/s running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg416-client.virtnet] [bs=1M] [count=10] [oflag=direct] [conv=fdatasync] 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 40.0902 s, 262 kB/s ost.OSS.ost_io.nrs_policies=fifo PASS 77m (57s) == sanityn test 77n: check wildcard support for TBF JobID NRS policy ========================================================== 05:33:35 (1761125615) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={*.500} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.498309 s, 33.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0576703 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0518273 s, 20.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0401445 s, 26.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109954 s, 9.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0471814 s, 22.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0895828 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0850914 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145092 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104159 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.288775 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190461 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.305989 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.334081 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.296512 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.267178 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0638269 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00244224 s, 429 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0991094 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210663 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0681597 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.396966 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0688845 s, 15.2 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208852 s, 5.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0710563 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.214877 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213538 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.218555 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108513 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222963 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144072 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.240634 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.2392 s, 4.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9a01d93a6000.lock_unused_count=8 removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg416-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg416-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/dev/zero] [of=/mnt/lustre/d77n.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.24514 s, 11.3 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/mnt/lustre/d77n.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.08899 s, 20.6 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.*} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.311133 s, 53.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0280683 s, 37.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0574777 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0508489 s, 20.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129776 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126931 s, 8.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124604 s, 8.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14864 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116436 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151217 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165858 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171379 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15669 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168451 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181384 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168479 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190729 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0219737 s, 47.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0550318 s, 19.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0435717 s, 24.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0193359 s, 54.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0591661 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0214075 s, 49.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0630816 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0265224 s, 39.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0269332 s, 38.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0307648 s, 34.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0326506 s, 32.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0770665 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0377544 s, 27.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0383477 s, 27.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0394321 s, 26.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0607163 s, 17.3 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg416-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg416-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.2822 s, 12.7 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.21353 s, 20.1 MB/s Read runtime is 7 s, speed is 14.285714 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_policies=fifo PASS 77n (89s) == sanityn test 77o: Changing rank should not panic ====== 05:35:04 (1761125704) mds.MDS.mdt.nrs_policies=tbf nid mds.MDS.mdt.nrs_tbf_rule=start name nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=start name1 nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=change name1 rank=name mds.MDS.mdt.nrs_tbf_rule=stop name mds.MDS.mdt.nrs_policies=fifo PASS 77o (10s) == sanityn test 77q: Parallel TBF rule definitions should not panic ========================================================== 05:35:14 (1761125714) mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_policies=fifo PASS 77q (81s) == sanityn test 77p: Check validity of rule names for TBF policies ========================================================== 05:36:35 (1761125795) mds.MDS.mdt.nrs_policies=tbf oleg416-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_overflo uid={500} rate=500: File name too long pdsh@oleg416-client: oleg416-server: ssh exited with exit code 36 oleg416-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start uid={500} rate=500: Invalid argument pdsh@oleg416-client: oleg416-server: ssh exited with exit code 22 oleg416-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_empty: Invalid argument pdsh@oleg416-client: oleg416-server: ssh exited with exit code 22 oleg416-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p@spec uid={500} rate=500: Invalid argument pdsh@oleg416-client: oleg416-server: ssh exited with exit code 22 oleg416-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p.spec uid={500} rate=500: Invalid argument pdsh@oleg416-client: oleg416-server: ssh exited with exit code 22 oleg416-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p~spec uid={500} rate=500: Invalid argument pdsh@oleg416-client: oleg416-server: ssh exited with exit code 22 oleg416-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p#spec uid={500} rate=500: Invalid argument pdsh@oleg416-client: oleg416-server: ssh exited with exit code 22 oleg416-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p/spec uid={500} rate=500: Invalid argument pdsh@oleg416-client: oleg416-server: ssh exited with exit code 22 oleg416-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p^spec uid={500} rate=500: Invalid argument pdsh@oleg416-client: oleg416-server: ssh exited with exit code 22 oleg416-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p%spec uid={500} rate=500: Invalid argument pdsh@oleg416-client: oleg416-server: ssh exited with exit code 22 oleg416-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p*spec uid={500} rate=500: Invalid argument pdsh@oleg416-client: oleg416-server: ssh exited with exit code 22 oleg416-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p;spec uid={500} rate=500: Invalid argument pdsh@oleg416-client: oleg416-server: ssh exited with exit code 22 oleg416-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p,spec uid={500} rate=500: Invalid argument pdsh@oleg416-client: oleg416-server: ssh exited with exit code 22 oleg416-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p?spec uid={500} rate=500: Invalid argument pdsh@oleg416-client: oleg416-server: ssh exited with exit code 22 oleg416-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77pspec uid={500} rate=500: Invalid argument pdsh@oleg416-client: oleg416-server: ssh exited with exit code 22 oleg416-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p:spec uid={500} rate=500: Invalid argument pdsh@oleg416-client: oleg416-server: ssh exited with exit code 22 oleg416-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p+spec uid={500} rate=500: Invalid argument pdsh@oleg416-client: oleg416-server: ssh exited with exit code 22 oleg416-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p=spec uid={500} rate=500: Invalid argument pdsh@oleg416-client: oleg416-server: ssh exited with exit code 22 oleg416-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p)spec uid={500} rate=500: Invalid argument pdsh@oleg416-client: oleg416-server: ssh exited with exit code 22 oleg416-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p(spec uid={500} rate=500: Invalid argument pdsh@oleg416-client: oleg416-server: ssh exited with exit code 22 oleg416-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p{spec uid={500} rate=500: Invalid argument pdsh@oleg416-client: oleg416-server: ssh exited with exit code 22 oleg416-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p}spec uid={500} rate=500: Invalid argument pdsh@oleg416-client: oleg416-server: ssh exited with exit code 22 oleg416-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p|spec uid={500} rate=500: Invalid argument pdsh@oleg416-client: oleg416-server: ssh exited with exit code 22 oleg416-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p[spec uid={500} rate=500: Invalid argument pdsh@oleg416-client: oleg416-server: ssh exited with exit code 22 oleg416-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p]spec uid={500} rate=500: Invalid argument pdsh@oleg416-client: oleg416-server: ssh exited with exit code 22 oleg416-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p!spec uid={500} rate=500: Invalid argument pdsh@oleg416-client: oleg416-server: ssh exited with exit code 22 oleg416-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p&spec uid={500} rate=500: Invalid argument pdsh@oleg416-client: oleg416-server: ssh exited with exit code 22 oleg416-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p$spec uid={500} rate=500: Invalid argument pdsh@oleg416-client: oleg416-server: ssh exited with exit code 22 oleg416-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p`spec uid={500} rate=500: Invalid argument pdsh@oleg416-client: oleg416-server: ssh exited with exit code 22 oleg416-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p\spec uid={500} rate=500: Invalid argument pdsh@oleg416-client: oleg416-server: ssh exited with exit code 22 mds.MDS.mdt.nrs_policies=fifo PASS 77p (24s) == sanityn test 78: Enable policy and specify tunings right away ========================================================== 05:36:59 (1761125819) oleg416-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_orr_quantum=1: Resource temporarily unavailable pdsh@oleg416-client: oleg416-server: ssh exited with exit code 11 ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_policies=fifo PASS 78 (7s) == sanityn test 79: xattr: intent error ================== 05:37:06 (1761125826) File: /mnt/lustre/d79.sanityn Size: 512 Blocks: 1 IO Block: 131072 directory Device: 2c54f966h/743766374d Inode: 144115205272505445 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-10-22 05:37:08.000000000 -0400 Modify: 2025-10-22 05:37:08.000000000 -0400 Change: 2025-10-22 05:37:08.000000000 -0400 Birth: 2025-10-22 05:37:07.000000000 -0400 fail_val=0 fail_loc=0x80000160 fail_val=0 fail_loc=0x80000131 # file: mnt/lustre/d79.sanityn trusted.name1="value1" PASS 79 (17s) == sanityn test 80a: migrate directory when some children is being opened ========================================================== 05:37:23 (1761125843) SKIP: sanityn test_80a needs >= 2 MDTs SKIP 80a (2s) == sanityn test 80b: Accessing directory during migration ========================================================== 05:37:25 (1761125845) SKIP: sanityn test_80b needs >= 2 MDTs SKIP 80b (2s) == sanityn test 81a: rename and stat under striped directory ========================================================== 05:37:27 (1761125847) SKIP: sanityn test_81a needs >= 2 MDTs SKIP 81a (2s) == sanityn test 81b: rename under striped directory doesn't deadlock ========================================================== 05:37:29 (1761125849) SKIP: sanityn test_81b We need at least 2 MDTs for this test SKIP 81b (2s) == sanityn test 81c: rename revoke LOOKUP lock for remote object ========================================================== 05:37:31 (1761125851) SKIP: sanityn test_81c needs >= 4 MDTs SKIP 81c (2s) == sanityn test 82: fsetxattr and fgetxattr on orphan files ========================================================== 05:37:33 (1761125853) multiop /mnt/lustre/f82.sanityn vO_ac TMPPIPE=/tmp/multiop_open_wait_pipe.7143 multiop /mnt/lustre2/f82.sanityn vo_Ac TMPPIPE=/tmp/multiop_open_wait_pipe.7143 PASS 82 (5s) == sanityn test 83: access striped directory while it is being created/unlinked ========================================================== 05:37:38 (1761125858) SKIP: sanityn test_83 needs >= 2 MDTs SKIP 83 (2s) == sanityn test 84: 0-nlink race in lu_object_find() ===== 05:37:41 (1761125861) fail_loc=0x8000060b PASS 84 (10s) == sanityn test 90: open/create and unlink striped directory ========================================================== 05:37:51 (1761125871) SKIP: sanityn test_90 needs >= 2 MDTs SKIP 90 (2s) == sanityn test 91: chmod and unlink striped directory === 05:37:54 (1761125874) SKIP: sanityn test_91 needs >= 2 MDTs SKIP 91 (2s) == sanityn test 92: create remote directory under orphan directory ========================================================== 05:37:56 (1761125876) SKIP: sanityn test_92 needs >= 2 MDTs SKIP 92 (2s) == sanityn test 93: alloc_rr should not allocate on same ost ========================================================== 05:37:58 (1761125878) fail_loc=0x00000163 fail_loc=0x0 /mnt/lustre/f93.sanityn-1/file1 lmm_stripe_count: 2 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 2649 0xa59 0 1 2633 0xa49 0 /mnt/lustre2/f93.sanityn-2/file2 lmm_stripe_count: 2 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 2650 0xa5a 0 1 2634 0xa4a 0 rc1=0 and rc2=0 PASS 93 (13s) == sanityn test 94: signal vs CP callback race =========== 05:38:11 (1761125891) osc.lustre-OST0000-osc-ffff9a01c7fbe000.idle_timeout=0 osc.lustre-OST0000-osc-ffff9a01d93a6000.idle_timeout=0 osc.lustre-OST0001-osc-ffff9a01c7fbe000.idle_timeout=0 osc.lustre-OST0001-osc-ffff9a01d93a6000.idle_timeout=0 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.191398 s, 42.8 kB/s fail_val=5 fail_loc=0x80000312 fail_val=6 fail_loc=0x80000329 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 3.12947 s, 1.3 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 5167: 284053 Alarm clock dd if=/dev/zero of=$DIR/$tfile conv=notrunc oflag=append bs=4k count=1 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0687497 s, 59.6 kB/s osc.lustre-OST0000-osc-ffff9a01c7fbe000.idle_timeout=debug osc.lustre-OST0000-osc-ffff9a01d93a6000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9a01c7fbe000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9a01d93a6000.idle_timeout=debug PASS 94 (14s) == sanityn test 95b: Check readpage() on a page that is no longer uptodate ========================================================== 05:38:25 (1761125905) llite.lustre-ffff9a01c7fbe000.fast_read=0 llite.lustre-ffff9a01d93a6000.fast_read=0 1+0 records in 1+0 records out 12288 bytes (12 kB, 12 KiB) copied, 0.00637225 s, 1.9 MB/s fail_loc=0x80001424 fail_val=5 fail_loc=0 llite.lustre-ffff9a01c7fbe000.fast_read=1 llite.lustre-ffff9a01d93a6000.fast_read=1 PASS 95b (11s) == sanityn test 100a: DoM: glimpse RPCs for stat without IO lock (DoM only file) ========================================================== 05:38:36 (1761125916) SKIP: sanityn test_100a Reserved for glimpse-ahead SKIP 100a (3s) == sanityn test 100b: DoM: no glimpse RPC for stat with IO lock (DoM only file) ========================================================== 05:38:39 (1761125919) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00924816 s, 443 kB/s /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK PASS 100b (5s) == sanityn test 100c: DoM: write vs stat without IO lock (combined file) ========================================================== 05:38:44 (1761125924) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0689169 s, 30.4 MB/s /mnt/lustre/d100c.sanityn/dom has type file OK /mnt/lustre/d100c.sanityn/dom has size 2097152 OK PASS 100c (5s) == sanityn test 100d: DoM: write+truncate vs stat without IO lock (combined file) ========================================================== 05:38:49 (1761125929) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0690055 s, 30.4 MB/s /mnt/lustre/d100d.sanityn/dom has type file OK /mnt/lustre/d100d.sanityn/dom has size 4096 OK PASS 100d (5s) == sanityn test 100e: DoM: read on open and file size ==== 05:38:54 (1761125934) 1+0 records in 1+0 records out 12000 bytes (12 kB, 12 KiB) copied, 0.00513179 s, 2.3 MB/s PASS 100e (4s) == sanityn test 101a: Discard DoM data on unlink ========= 05:38:59 (1761125939) /mnt/lustre/f101a.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00188265 s, 2.2 MB/s PASS 101a (5s) == sanityn test 101b: Discard DoM data on rename ========= 05:39:04 (1761125944) /mnt/lustre/f101b.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00364171 s, 1.1 MB/s PASS 101b (6s) == sanityn test 101c: Discard DoM data on close-unlink === 05:39:10 (1761125950) /mnt/lustre/f101c.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00526043 s, 779 kB/s PASS 101c (6s) SKIP: sanityn test_102 skipping ALWAYS excluded test 102 == sanityn test 103: Test size correctness with lockahead ========================================================== 05:39:18 (1761125958) fail_loc=0x415 fail_loc=0x415 fail_val=2 Incorrect size expected (no glimpse fix): Starting test test23 at 1761125961 lockahead_test: lockahead_test.c:1186: test23: assertion 'sb.st_size == sb2.st_size' failed: size on /mnt/lustre/d103.sanityn/f103.sanityn and /mnt/lustre2/d103.sanityn/f103.sanityn differs: 0 vs 1048576 fail_loc=0 fail_loc=0x214 fail_val=2 Starting test test23 at 1761125964 Finishing test test23 at 1761125964 PASS 103 (12s) == sanityn test 104: Verify that MDS stores atime/mtime/ctime during close ========================================================== 05:39:30 (1761125970) SKIP: sanityn test_104 ldiskfs only test SKIP 104 (2s) == sanityn test 105: Glimpse and lock cancel race ======== 05:39:32 (1761125972) fail_loc=0x416 PASS 105 (52s) == sanityn test 106a: Verify the btime via statx() ======= 05:40:24 (1761126024) SKIP: sanityn test_106a Test only for ldiskfs and statx() supported SKIP 106a (2s) == sanityn test 106b: Glimpse RPCs test for statx ======== 05:40:26 (1761126026) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0141524 s, 72.4 kB/s File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205272505473 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-10-22 05:40:27.000000000 -0400 Modify: 2025-10-22 05:40:27.000000000 -0400 Change: 2025-10-22 05:40:27.000000000 -0400 Birth: 2025-10-22 05:40:26.000000000 -0400 /mnt/lustre/f106b.sanityn 144115205272505473 -rw-r--r-- 1 0 0 1761126026 1761126027 1761126027 File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205272505473 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-10-22 05:40:27.000000000 -0400 Modify: 2025-10-22 05:40:27.000000000 -0400 Change: 2025-10-22 05:40:27.000000000 -0400 Birth: 2025-10-22 05:40:26.000000000 -0400 1761126027 1024 1 PASS 106b (6s) == sanityn test 106c: Verify statx attributes mask ======= 05:40:32 (1761126032) PASS 106c (5s) == sanityn test 107a: Basic grouplock conflict =========== 05:40:37 (1761126037) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.175207 s, 59.8 MB/s /mnt/lustre/f107a.sanityn lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 2658 0xa62 0 multiop /mnt/lustre/f107a.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7143 multiop /mnt/lustre2/f107a.sanityn vO_G16022000r10g16022000c TMPPIPE=/tmp/multiop_open_wait_pipe.7143 First grouplock blocks second one PASS 107a (8s) == sanityn test 107b: Grouplock is added to the head of waiting list ========================================================== 05:40:45 (1761126045) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.199208 s, 52.6 MB/s /mnt/lustre/f107b.sanityn lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 2639 0xa4f 0 multiop /mnt/lustre/f107b.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7143 Grouplock blocks IO First grouplock blocks second one Second grouplock blocks IO PASS 107b (10s) == sanityn test 108a: lseek: parallel updates ============ 05:40:56 (1761126056) 1+0 records in 1+0 records out 10000 bytes (10 kB, 9.8 KiB) copied, 0.00300855 s, 3.3 MB/s fail_val=4 fail_loc=0x80000414 1+0 records in 1+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 4.59761 s, 1.8 MB/s PASS 108a (9s) == sanityn test 109: Race with several mount instances on 1 node ========================================================== 05:41:05 (1761126065) 192.168.204.116@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt 0 0 Stopping client oleg416-client.virtnet /mnt/lustre (opts:) 192.168.204.116@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt 0 0 Stopping client oleg416-client.virtnet /mnt/lustre2 (opts:) Starting race between client mount instances (50 iterations): Iteration 1 Starting client: oleg416-client.virtnet: -o user_xattr,flock oleg416-server@tcp:/lustre /mnt/lustre2 Starting client: oleg416-client.virtnet: -o user_xattr,flock oleg416-server@tcp:/lustre /mnt/lustre 192.168.204.116@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt 0 0 192.168.204.116@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt 0 0 Stopping client oleg416-client.virtnet /mnt/lustre (opts:) Stopping client oleg416-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg416-server' oleg416-server: oleg416-server.virtnet: executing load_modules_local oleg416-server: Loading modules from /home/green/git/lustre-release/lustre oleg416-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg416-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg416-server: detected 4 online CPUs by sysfs oleg416-server: Force libcfs to create 2 CPU partitions Iteration 2 Starting client: oleg416-client.virtnet: -o user_xattr,flock oleg416-server@tcp:/lustre /mnt/lustre2 Starting client: oleg416-client.virtnet: -o user_xattr,flock oleg416-server@tcp:/lustre /mnt/lustre 192.168.204.116@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt 0 0 192.168.204.116@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt 0 0 Stopping client oleg416-client.virtnet /mnt/lustre (opts:) Stopping client oleg416-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg416-server' oleg416-server: oleg416-server.virtnet: executing load_modules_local oleg416-server: Loading modules from /home/green/git/lustre-release/lustre oleg416-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg416-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg416-server: detected 4 online CPUs by sysfs oleg416-server: Force libcfs to create 2 CPU partitions Iteration 3 Starting client: oleg416-client.virtnet: -o user_xattr,flock oleg416-server@tcp:/lustre /mnt/lustre2 Starting client: oleg416-client.virtnet: -o user_xattr,flock oleg416-server@tcp:/lustre /mnt/lustre 192.168.204.116@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt 0 0 192.168.204.116@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt 0 0 Stopping client oleg416-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg416-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg416-server' oleg416-server: oleg416-server.virtnet: executing load_modules_local oleg416-server: Loading modules from /home/green/git/lustre-release/lustre oleg416-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg416-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg416-server: detected 4 online CPUs by sysfs oleg416-server: Force libcfs to create 2 CPU partitions Iteration 4 Starting client: oleg416-client.virtnet: -o user_xattr,flock oleg416-server@tcp:/lustre /mnt/lustre Starting client: oleg416-client.virtnet: -o user_xattr,flock oleg416-server@tcp:/lustre /mnt/lustre2 192.168.204.116@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt 0 0 192.168.204.116@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt 0 0 Stopping client oleg416-client.virtnet /mnt/lustre (opts:) Stopping client oleg416-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg416-server' oleg416-server: oleg416-server.virtnet: executing load_modules_local oleg416-server: Loading modules from /home/green/git/lustre-release/lustre oleg416-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg416-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg416-server: detected 4 online CPUs by sysfs oleg416-server: Force libcfs to create 2 CPU partitions Iteration 5 Starting client: oleg416-client.virtnet: -o user_xattr,flock oleg416-server@tcp:/lustre /mnt/lustre Starting client: oleg416-client.virtnet: -o user_xattr,flock oleg416-server@tcp:/lustre /mnt/lustre2 192.168.204.116@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt 0 0 192.168.204.116@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt 0 0 Stopping client oleg416-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg416-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg416-server' oleg416-server: oleg416-server.virtnet: executing load_modules_local oleg416-server: Loading modules from /home/green/git/lustre-release/lustre oleg416-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg416-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg416-server: detected 4 online CPUs by sysfs oleg416-server: Force libcfs to create 2 CPU partitions Iteration 6 Starting client: oleg416-client.virtnet: -o user_xattr,flock oleg416-server@tcp:/lustre /mnt/lustre2 Starting client: oleg416-client.virtnet: -o user_xattr,flock oleg416-server@tcp:/lustre /mnt/lustre 192.168.204.116@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt 0 0 192.168.204.116@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt 0 0 Stopping client oleg416-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg416-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg416-server' oleg416-server: oleg416-server.virtnet: executing load_modules_local oleg416-server: Loading modules from /home/green/git/lustre-release/lustre oleg416-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg416-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg416-server: detected 4 online CPUs by sysfs oleg416-server: Force libcfs to create 2 CPU partitions Iteration 7 Starting client: oleg416-client.virtnet: -o user_xattr,flock oleg416-server@tcp:/lustre /mnt/lustre Starting client: oleg416-client.virtnet: -o user_xattr,flock oleg416-server@tcp:/lustre /mnt/lustre2 192.168.204.116@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt 0 0 192.168.204.116@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt 0 0 Stopping client oleg416-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg416-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg416-server' oleg416-server: oleg416-server.virtnet: executing load_modules_local oleg416-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg416-server: Loading modules from /home/green/git/lustre-release/lustre oleg416-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg416-server: detected 4 online CPUs by sysfs oleg416-server: Force libcfs to create 2 CPU partitions Iteration 8 Starting client: oleg416-client.virtnet: -o user_xattr,flock oleg416-server@tcp:/lustre /mnt/lustre Starting client: oleg416-client.virtnet: -o user_xattr,flock oleg416-server@tcp:/lustre /mnt/lustre2 192.168.204.116@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt 0 0 192.168.204.116@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt 0 0 Stopping client oleg416-client.virtnet /mnt/lustre (opts:) Stopping client oleg416-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg416-server' oleg416-server: oleg416-server.virtnet: executing load_modules_local oleg416-server: Loading modules from /home/green/git/lustre-release/lustre oleg416-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg416-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg416-server: detected 4 online CPUs by sysfs oleg416-server: Force libcfs to create 2 CPU partitions Iteration 9 Starting client: oleg416-client.virtnet: -o user_xattr,flock oleg416-server@tcp:/lustre /mnt/lustre Starting client: oleg416-client.virtnet: -o user_xattr,flock oleg416-server@tcp:/lustre /mnt/lustre2 192.168.204.116@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt 0 0 192.168.204.116@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt 0 0 Stopping client oleg416-client.virtnet /mnt/lustre (opts:) Stopping client oleg416-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg416-server' oleg416-server: oleg416-server.virtnet: executing load_modules_local oleg416-server: Loading modules from /home/green/git/lustre-release/lustre oleg416-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg416-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg416-server: detected 4 online CPUs by sysfs oleg416-server: Force libcfs to create 2 CPU partitions Iteration 10 Starting client: oleg416-client.virtnet: -o user_xattr,flock oleg416-server@tcp:/lustre /mnt/lustre2 Starting client: oleg416-client.virtnet: -o user_xattr,flock oleg416-server@tcp:/lustre /mnt/lustre 192.168.204.116@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt 0 0 192.168.204.116@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt 0 0 Stopping client oleg416-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg416-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg416-server' oleg416-server: oleg416-server.virtnet: executing load_modules_local oleg416-server: Loading modules from /home/green/git/lustre-release/lustre oleg416-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg416-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg416-server: detected 4 online CPUs by sysfs oleg416-server: Force libcfs to create 2 CPU partitions Iteration 11 Starting client: oleg416-client.virtnet: -o user_xattr,flock oleg416-server@tcp:/lustre /mnt/lustre Starting client: oleg416-client.virtnet: -o user_xattr,flock oleg416-server@tcp:/lustre /mnt/lustre2 192.168.204.116@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt 0 0 192.168.204.116@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt 0 0 Stopping client oleg416-client.virtnet /mnt/lustre (opts:) Stopping client oleg416-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg416-server' oleg416-server: oleg416-server.virtnet: executing load_modules_local oleg416-server: Loading modules from /home/green/git/lustre-release/lustre oleg416-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg416-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg416-server: detected 4 online CPUs by sysfs oleg416-server: Force libcfs to create 2 CPU partitions Iteration 12 Starting client: oleg416-client.virtnet: -o user_xattr,flock oleg416-server@tcp:/lustre /mnt/lustre Starting client: oleg416-client.virtnet: -o user_xattr,flock oleg416-server@tcp:/lustre /mnt/lustre2 192.168.204.116@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt 0 0 192.168.204.116@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt 0 0 Stopping client oleg416-client.virtnet /mnt/lustre (opts:) Stopping client oleg416-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg416-server' oleg416-server: oleg416-server.virtnet: executing load_modules_local oleg416-server: Loading modules from /home/green/git/lustre-release/lustre oleg416-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg416-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg416-server: detected 4 online CPUs by sysfs oleg416-server: Force libcfs to create 2 CPU partitions Iteration 13 Starting client: oleg416-client.virtnet: -o user_xattr,flock oleg416-server@tcp:/lustre /mnt/lustre Starting client: oleg416-client.virtnet: -o user_xattr,flock oleg416-server@tcp:/lustre /mnt/lustre2 192.168.204.116@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt 0 0 192.168.204.116@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt 0 0 Stopping client oleg416-client.virtnet /mnt/lustre (opts:) Stopping client oleg416-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg416-server' oleg416-server: oleg416-server.virtnet: executing load_modules_local oleg416-server: Loading modules from /home/green/git/lustre-release/lustre oleg416-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg416-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg416-server: detected 4 online CPUs by sysfs oleg416-server: Force libcfs to create 2 CPU partitions Iteration 14 Starting client: oleg416-client.virtnet: -o user_xattr,flock oleg416-server@tcp:/lustre /mnt/lustre Starting client: oleg416-client.virtnet: -o user_xattr,flock oleg416-server@tcp:/lustre /mnt/lustre2 192.168.204.116@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt 0 0 192.168.204.116@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt 0 0 Stopping client oleg416-client.virtnet /mnt/lustre (opts:) Stopping client oleg416-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg416-server' oleg416-server: oleg416-server.virtnet: executing load_modules_local oleg416-server: Loading modules from /home/green/git/lustre-release/lustre oleg416-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg416-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg416-server: detected 4 online CPUs by sysfs oleg416-server: Force libcfs to create 2 CPU partitions Iteration 15 Starting client: oleg416-client.virtnet: -o user_xattr,flock oleg416-server@tcp:/lustre /mnt/lustre2 Starting client: oleg416-client.virtnet: -o user_xattr,flock oleg416-server@tcp:/lustre /mnt/lustre 192.168.204.116@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt 0 0 192.168.204.116@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt 0 0 Stopping client oleg416-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg416-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg416-server' oleg416-server: oleg416-server.virtnet: executing load_modules_local oleg416-server: Loading modules from /home/green/git/lustre-release/lustre oleg416-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg416-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg416-server: detected 4 online CPUs by sysfs oleg416-server: Force libcfs to create 2 CPU partitions Iteration 16 Starting client: oleg416-client.virtnet: -o user_xattr,flock oleg416-server@tcp:/lustre /mnt/lustre Starting client: oleg416-client.virtnet: -o user_xattr,flock oleg416-server@tcp:/lustre /mnt/lustre2 192.168.204.116@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt 0 0 192.168.204.116@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt 0 0 Stopping client oleg416-client.virtnet /mnt/lustre (opts:) Stopping client oleg416-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg416-server' oleg416-server: oleg416-server.virtnet: executing load_modules_local oleg416-server: Loading modules from /home/green/git/lustre-release/lustre oleg416-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg416-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg416-server: detected 4 online CPUs by sysfs oleg416-server: Force libcfs to create 2 CPU partitions Iteration 17 Starting client: oleg416-client.virtnet: -o user_xattr,flock oleg416-server@tcp:/lustre /mnt/lustre Starting client: oleg416-client.virtnet: -o user_xattr,flock oleg416-server@tcp:/lustre /mnt/lustre2 192.168.204.116@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt 0 0 192.168.204.116@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt 0 0 Stopping client oleg416-client.virtnet /mnt/lustre (opts:) Stopping client oleg416-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg416-server' oleg416-server: oleg416-server.virtnet: executing load_modules_local oleg416-server: Loading modules from /home/green/git/lustre-release/lustre oleg416-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg416-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg416-server: detected 4 online CPUs by sysfs oleg416-server: Force libcfs to create 2 CPU partitions Iteration 18 Starting client: oleg416-client.virtnet: -o user_xattr,flock oleg416-server@tcp:/lustre /mnt/lustre2 Starting client: oleg416-client.virtnet: -o user_xattr,flock oleg416-server@tcp:/lustre /mnt/lustre 192.168.204.116@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt 0 0 192.168.204.116@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt 0 0 Stopping client oleg416-client.virtnet /mnt/lustre (opts:) Stopping client oleg416-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg416-server' oleg416-server: oleg416-server.virtnet: executing load_modules_local oleg416-server: Loading modules from /home/green/git/lustre-release/lustre oleg416-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg416-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg416-server: detected 4 online CPUs by sysfs oleg416-server: Force libcfs to create 2 CPU partitions Iteration 19 Starting client: oleg416-client.virtnet: -o user_xattr,flock oleg416-server@tcp:/lustre /mnt/lustre Starting client: oleg416-client.virtnet: -o user_xattr,flock oleg416-server@tcp:/lustre /mnt/lustre2 192.168.204.116@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt 0 0 192.168.204.116@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt 0 0 Stopping client oleg416-client.virtnet /mnt/lustre (opts:) Stopping client oleg416-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg416-server' oleg416-server: oleg416-server.virtnet: executing load_modules_local oleg416-server: Loading modules from /home/green/git/lustre-release/lustre oleg416-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg416-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg416-server: detected 4 online CPUs by sysfs oleg416-server: Force libcfs to create 2 CPU partitions Iteration 20 Starting client: oleg416-client.virtnet: -o user_xattr,flock oleg416-server@tcp:/lustre /mnt/lustre2 Starting client: oleg416-client.virtnet: -o user_xattr,flock oleg416-server@tcp:/lustre /mnt/lustre 192.168.204.116@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt 0 0 192.168.204.116@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt 0 0 Stopping client oleg416-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg416-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg416-server' oleg416-server: oleg416-server.virtnet: executing load_modules_local oleg416-server: Loading modules from /home/green/git/lustre-release/lustre oleg416-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg416-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg416-server: detected 4 online CPUs by sysfs oleg416-server: Force libcfs to create 2 CPU partitions Iteration 21 Starting client: oleg416-client.virtnet: -o user_xattr,flock oleg416-server@tcp:/lustre /mnt/lustre Starting client: oleg416-client.virtnet: -o user_xattr,flock oleg416-server@tcp:/lustre /mnt/lustre2 192.168.204.116@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt 0 0 192.168.204.116@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt 0 0 Stopping client oleg416-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg416-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg416-server' oleg416-server: oleg416-server.virtnet: executing load_modules_local oleg416-server: Loading modules from /home/green/git/lustre-release/lustre oleg416-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg416-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg416-server: detected 4 online CPUs by sysfs oleg416-server: Force libcfs to create 2 CPU partitions Iteration 22 Starting client: oleg416-client.virtnet: -o user_xattr,flock oleg416-server@tcp:/lustre /mnt/lustre2 Starting client: oleg416-client.virtnet: -o user_xattr,flock oleg416-server@tcp:/lustre /mnt/lustre 192.168.204.116@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt 0 0 192.168.204.116@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt 0 0 Stopping client oleg416-client.virtnet /mnt/lustre (opts:) Stopping client oleg416-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg416-server' oleg416-server: oleg416-server.virtnet: executing load_modules_local oleg416-server: Loading modules from /home/green/git/lustre-release/lustre oleg416-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg416-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg416-server: detected 4 online CPUs by sysfs oleg416-server: Force libcfs to create 2 CPU partitions Iteration 23 Starting client: oleg416-client.virtnet: -o user_xattr,flock oleg416-server@tcp:/lustre /mnt/lustre Starting client: oleg416-client.virtnet: -o user_xattr,flock oleg416-server@tcp:/lustre /mnt/lustre2 192.168.204.116@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt 0 0 192.168.204.116@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt 0 0 Stopping client oleg416-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg416-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg416-server' oleg416-server: oleg416-server.virtnet: executing load_modules_local oleg416-server: Loading modules from /home/green/git/lustre-release/lustre oleg416-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg416-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg416-server: detected 4 online CPUs by sysfs oleg416-server: Force libcfs to create 2 CPU partitions Iteration 24 Starting client: oleg416-client.virtnet: -o user_xattr,flock oleg416-server@tcp:/lustre /mnt/lustre2 Starting client: oleg416-client.virtnet: -o user_xattr,flock oleg416-server@tcp:/lustre /mnt/lustre 192.168.204.116@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt 0 0 192.168.204.116@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt 0 0 Stopping client oleg416-client.virtnet /mnt/lustre (opts:) Stopping client oleg416-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg416-server' oleg416-server: oleg416-server.virtnet: executing load_modules_local oleg416-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg416-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg416-server: Loading modules from /home/green/git/lustre-release/lustre oleg416-server: detected 4 online CPUs by sysfs oleg416-server: Force libcfs to create 2 CPU partitions Iteration 25 Starting client: oleg416-client.virtnet: -o user_xattr,flock oleg416-server@tcp:/lustre /mnt/lustre Starting client: oleg416-client.virtnet: -o user_xattr,flock oleg416-server@tcp:/lustre /mnt/lustre2 192.168.204.116@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt 0 0 192.168.204.116@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt 0 0 Stopping client oleg416-client.virtnet /mnt/lustre (opts:) Stopping client oleg416-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg416-server' oleg416-server: oleg416-server.virtnet: executing load_modules_local oleg416-server: Loading modules from /home/green/git/lustre-release/lustre oleg416-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg416-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg416-server: detected 4 online CPUs by sysfs oleg416-server: Force libcfs to create 2 CPU partitions Iteration 26 Starting client: oleg416-client.virtnet: -o user_xattr,flock oleg416-server@tcp:/lustre /mnt/lustre2 Starting client: oleg416-client.virtnet: -o user_xattr,flock oleg416-server@tcp:/lustre /mnt/lustre 192.168.204.116@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt 0 0 192.168.204.116@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt 0 0 Stopping client oleg416-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg416-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg416-server' oleg416-server: oleg416-server.virtnet: executing load_modules_local oleg416-server: Loading modules from /home/green/git/lustre-release/lustre oleg416-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg416-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg416-server: detected 4 online CPUs by sysfs oleg416-server: Force libcfs to create 2 CPU partitions Iteration 27 Starting client: oleg416-client.virtnet: -o user_xattr,flock oleg416-server@tcp:/lustre /mnt/lustre Starting client: oleg416-client.virtnet: -o user_xattr,flock oleg416-server@tcp:/lustre /mnt/lustre2 192.168.204.116@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt 0 0 192.168.204.116@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt 0 0 Stopping client oleg416-client.virtnet /mnt/lustre (opts:) Stopping client oleg416-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg416-server' oleg416-server: oleg416-server.virtnet: executing load_modules_local oleg416-server: Loading modules from /home/green/git/lustre-release/lustre oleg416-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg416-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg416-server: detected 4 online CPUs by sysfs oleg416-server: Force libcfs to create 2 CPU partitions Iteration 28 Starting client: oleg416-client.virtnet: -o user_xattr,flock oleg416-server@tcp:/lustre /mnt/lustre2 Starting client: oleg416-client.virtnet: -o user_xattr,flock oleg416-server@tcp:/lustre /mnt/lustre 192.168.204.116@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt 0 0 192.168.204.116@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt 0 0 Stopping client oleg416-client.virtnet /mnt/lustre (opts:) Stopping client oleg416-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg416-server' oleg416-server: oleg416-server.virtnet: executing load_modules_local oleg416-server: Loading modules from /home/green/git/lustre-release/lustre oleg416-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg416-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg416-server: detected 4 online CPUs by sysfs oleg416-server: Force libcfs to create 2 CPU partitions Iteration 29 Starting client: oleg416-client.virtnet: -o user_xattr,flock oleg416-server@tcp:/lustre /mnt/lustre Starting client: oleg416-client.virtnet: -o user_xattr,flock oleg416-server@tcp:/lustre /mnt/lustre2 192.168.204.116@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt 0 0 192.168.204.116@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt 0 0 Stopping client oleg416-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg416-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg416-server' oleg416-server: oleg416-server.virtnet: executing load_modules_local oleg416-server: Loading modules from /home/green/git/lustre-release/lustre oleg416-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg416-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg416-server: detected 4 online CPUs by sysfs oleg416-server: Force libcfs to create 2 CPU partitions Iteration 30 Starting client: oleg416-client.virtnet: -o user_xattr,flock oleg416-server@tcp:/lustre /mnt/lustre Starting client: oleg416-client.virtnet: -o user_xattr,flock oleg416-server@tcp:/lustre /mnt/lustre2 192.168.204.116@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt 0 0 192.168.204.116@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt 0 0 Stopping client oleg416-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg416-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg416-server' oleg416-server: oleg416-server.virtnet: executing load_modules_local oleg416-server: Loading modules from /home/green/git/lustre-release/lustre oleg416-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg416-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg416-server: detected 4 online CPUs by sysfs oleg416-server: Force libcfs to create 2 CPU partitions Iteration 31 Starting client: oleg416-client.virtnet: -o user_xattr,flock oleg416-server@tcp:/lustre /mnt/lustre2 Starting client: oleg416-client.virtnet: -o user_xattr,flock oleg416-server@tcp:/lustre /mnt/lustre 192.168.204.116@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt 0 0 192.168.204.116@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt 0 0 Stopping client oleg416-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg416-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg416-server' oleg416-server: oleg416-server.virtnet: executing load_modules_local oleg416-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg416-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg416-server: Loading modules from /home/green/git/lustre-release/lustre oleg416-server: detected 4 online CPUs by sysfs oleg416-server: Force libcfs to create 2 CPU partitions Iteration 32 Starting client: oleg416-client.virtnet: -o user_xattr,flock oleg416-server@tcp:/lustre /mnt/lustre Starting client: oleg416-client.virtnet: -o user_xattr,flock oleg416-server@tcp:/lustre /mnt/lustre2 192.168.204.116@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt 0 0 192.168.204.116@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt 0 0 Stopping client oleg416-client.virtnet /mnt/lustre (opts:) Stopping client oleg416-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg416-server' oleg416-server: oleg416-server.virtnet: executing load_modules_local oleg416-server: Loading modules from /home/green/git/lustre-release/lustre oleg416-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg416-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg416-server: detected 4 online CPUs by sysfs oleg416-server: Force libcfs to create 2 CPU partitions Iteration 33 Starting client: oleg416-client.virtnet: -o user_xattr,flock oleg416-server@tcp:/lustre /mnt/lustre2 Starting client: oleg416-client.virtnet: -o user_xattr,flock oleg416-server@tcp:/lustre /mnt/lustre 192.168.204.116@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt 0 0 192.168.204.116@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt 0 0 Stopping client oleg416-client.virtnet /mnt/lustre (opts:) Stopping client oleg416-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg416-server' oleg416-server: oleg416-server.virtnet: executing load_modules_local oleg416-server: Loading modules from /home/green/git/lustre-release/lustre oleg416-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg416-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg416-server: detected 4 online CPUs by sysfs oleg416-server: Force libcfs to create 2 CPU partitions Iteration 34 Starting client: oleg416-client.virtnet: -o user_xattr,flock oleg416-server@tcp:/lustre /mnt/lustre Starting client: oleg416-client.virtnet: -o user_xattr,flock oleg416-server@tcp:/lustre /mnt/lustre2 192.168.204.116@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt 0 0 192.168.204.116@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt 0 0 Stopping client oleg416-client.virtnet /mnt/lustre (opts:) Stopping client oleg416-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg416-server' oleg416-server: oleg416-server.virtnet: executing load_modules_local oleg416-server: Loading modules from /home/green/git/lustre-release/lustre oleg416-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg416-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg416-server: detected 4 online CPUs by sysfs oleg416-server: Force libcfs to create 2 CPU partitions Iteration 35 Starting client: oleg416-client.virtnet: -o user_xattr,flock oleg416-server@tcp:/lustre /mnt/lustre2 Starting client: oleg416-client.virtnet: -o user_xattr,flock oleg416-server@tcp:/lustre /mnt/lustre 192.168.204.116@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt 0 0 192.168.204.116@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt 0 0 Stopping client oleg416-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg416-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg416-server' oleg416-server: oleg416-server.virtnet: executing load_modules_local oleg416-server: Loading modules from /home/green/git/lustre-release/lustre oleg416-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg416-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg416-server: detected 4 online CPUs by sysfs oleg416-server: Force libcfs to create 2 CPU partitions Iteration 36 Starting client: oleg416-client.virtnet: -o user_xattr,flock oleg416-server@tcp:/lustre /mnt/lustre Starting client: oleg416-client.virtnet: -o user_xattr,flock oleg416-server@tcp:/lustre /mnt/lustre2 192.168.204.116@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt 0 0 192.168.204.116@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt 0 0 Stopping client oleg416-client.virtnet /mnt/lustre (opts:) Stopping client oleg416-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg416-server' oleg416-server: oleg416-server.virtnet: executing load_modules_local oleg416-server: Loading modules from /home/green/git/lustre-release/lustre oleg416-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg416-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg416-server: detected 4 online CPUs by sysfs oleg416-server: Force libcfs to create 2 CPU partitions Iteration 37 Starting client: oleg416-client.virtnet: -o user_xattr,flock oleg416-server@tcp:/lustre /mnt/lustre2 Starting client: oleg416-client.virtnet: -o user_xattr,flock oleg416-server@tcp:/lustre /mnt/lustre 192.168.204.116@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt 0 0 192.168.204.116@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt 0 0 Stopping client oleg416-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg416-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg416-server' oleg416-server: oleg416-server.virtnet: executing load_modules_local oleg416-server: Loading modules from /home/green/git/lustre-release/lustre oleg416-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg416-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg416-server: detected 4 online CPUs by sysfs oleg416-server: Force libcfs to create 2 CPU partitions Iteration 38 Starting client: oleg416-client.virtnet: -o user_xattr,flock oleg416-server@tcp:/lustre /mnt/lustre Starting client: oleg416-client.virtnet: -o user_xattr,flock oleg416-server@tcp:/lustre /mnt/lustre2 192.168.204.116@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt 0 0 192.168.204.116@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt 0 0 Stopping client oleg416-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg416-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg416-server' oleg416-server: oleg416-server.virtnet: executing load_modules_local oleg416-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg416-server: Loading modules from /home/green/git/lustre-release/lustre oleg416-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg416-server: detected 4 online CPUs by sysfs oleg416-server: Force libcfs to create 2 CPU partitions Iteration 39 Starting client: oleg416-client.virtnet: -o user_xattr,flock oleg416-server@tcp:/lustre /mnt/lustre2 Starting client: oleg416-client.virtnet: -o user_xattr,flock oleg416-server@tcp:/lustre /mnt/lustre 192.168.204.116@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt 0 0 192.168.204.116@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt 0 0 Stopping client oleg416-client.virtnet /mnt/lustre (opts:) Stopping client oleg416-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg416-server' oleg416-server: oleg416-server.virtnet: executing load_modules_local oleg416-server: Loading modules from /home/green/git/lustre-release/lustre oleg416-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg416-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg416-server: detected 4 online CPUs by sysfs oleg416-server: Force libcfs to create 2 CPU partitions Iteration 40 Starting client: oleg416-client.virtnet: -o user_xattr,flock oleg416-server@tcp:/lustre /mnt/lustre2 Starting client: oleg416-client.virtnet: -o user_xattr,flock oleg416-server@tcp:/lustre /mnt/lustre 192.168.204.116@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt 0 0 192.168.204.116@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt 0 0 Stopping client oleg416-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg416-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg416-server' oleg416-server: oleg416-server.virtnet: executing load_modules_local oleg416-server: Loading modules from /home/green/git/lustre-release/lustre oleg416-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg416-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg416-server: detected 4 online CPUs by sysfs oleg416-server: Force libcfs to create 2 CPU partitions Iteration 41 Starting client: oleg416-client.virtnet: -o user_xattr,flock oleg416-server@tcp:/lustre /mnt/lustre Starting client: oleg416-client.virtnet: -o user_xattr,flock oleg416-server@tcp:/lustre /mnt/lustre2 192.168.204.116@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt 0 0 192.168.204.116@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt 0 0 Stopping client oleg416-client.virtnet /mnt/lustre (opts:) Stopping client oleg416-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg416-server' oleg416-server: oleg416-server.virtnet: executing load_modules_local oleg416-server: Loading modules from /home/green/git/lustre-release/lustre oleg416-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg416-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg416-server: detected 4 online CPUs by sysfs oleg416-server: Force libcfs to create 2 CPU partitions Iteration 42 Starting client: oleg416-client.virtnet: -o user_xattr,flock oleg416-server@tcp:/lustre /mnt/lustre Starting client: oleg416-client.virtnet: -o user_xattr,flock oleg416-server@tcp:/lustre /mnt/lustre2 192.168.204.116@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt 0 0 192.168.204.116@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt 0 0 Stopping client oleg416-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg416-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg416-server' oleg416-server: oleg416-server.virtnet: executing load_modules_local oleg416-server: Loading modules from /home/green/git/lustre-release/lustre oleg416-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg416-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg416-server: detected 4 online CPUs by sysfs oleg416-server: Force libcfs to create 2 CPU partitions Iteration 43 Starting client: oleg416-client.virtnet: -o user_xattr,flock oleg416-server@tcp:/lustre /mnt/lustre Starting client: oleg416-client.virtnet: -o user_xattr,flock oleg416-server@tcp:/lustre /mnt/lustre2 192.168.204.116@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt 0 0 192.168.204.116@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt 0 0 Stopping client oleg416-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg416-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg416-server' oleg416-server: oleg416-server.virtnet: executing load_modules_local oleg416-server: Loading modules from /home/green/git/lustre-release/lustre oleg416-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg416-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg416-server: detected 4 online CPUs by sysfs oleg416-server: Force libcfs to create 2 CPU partitions Iteration 44 Starting client: oleg416-client.virtnet: -o user_xattr,flock oleg416-server@tcp:/lustre /mnt/lustre2 Starting client: oleg416-client.virtnet: -o user_xattr,flock oleg416-server@tcp:/lustre /mnt/lustre 192.168.204.116@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt 0 0 192.168.204.116@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt 0 0 Stopping client oleg416-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg416-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg416-server' oleg416-server: oleg416-server.virtnet: executing load_modules_local oleg416-server: Loading modules from /home/green/git/lustre-release/lustre oleg416-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg416-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg416-server: detected 4 online CPUs by sysfs oleg416-server: Force libcfs to create 2 CPU partitions Iteration 45 Starting client: oleg416-client.virtnet: -o user_xattr,flock oleg416-server@tcp:/lustre /mnt/lustre Starting client: oleg416-client.virtnet: -o user_xattr,flock oleg416-server@tcp:/lustre /mnt/lustre2 192.168.204.116@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt 0 0 192.168.204.116@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt 0 0 Stopping client oleg416-client.virtnet /mnt/lustre (opts:) Stopping client oleg416-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg416-server' oleg416-server: oleg416-server.virtnet: executing load_modules_local oleg416-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg416-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg416-server: Loading modules from /home/green/git/lustre-release/lustre oleg416-server: detected 4 online CPUs by sysfs oleg416-server: Force libcfs to create 2 CPU partitions Iteration 46 Starting client: oleg416-client.virtnet: -o user_xattr,flock oleg416-server@tcp:/lustre /mnt/lustre2 Starting client: oleg416-client.virtnet: -o user_xattr,flock oleg416-server@tcp:/lustre /mnt/lustre 192.168.204.116@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt 0 0 192.168.204.116@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt 0 0 Stopping client oleg416-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg416-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg416-server' oleg416-server: oleg416-server.virtnet: executing load_modules_local oleg416-server: Loading modules from /home/green/git/lustre-release/lustre oleg416-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg416-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg416-server: detected 4 online CPUs by sysfs oleg416-server: Force libcfs to create 2 CPU partitions Iteration 47 Starting client: oleg416-client.virtnet: -o user_xattr,flock oleg416-server@tcp:/lustre /mnt/lustre Starting client: oleg416-client.virtnet: -o user_xattr,flock oleg416-server@tcp:/lustre /mnt/lustre2 192.168.204.116@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt 0 0 192.168.204.116@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt 0 0 Stopping client oleg416-client.virtnet /mnt/lustre (opts:) Stopping client oleg416-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg416-server' oleg416-server: oleg416-server.virtnet: executing load_modules_local oleg416-server: Loading modules from /home/green/git/lustre-release/lustre oleg416-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg416-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg416-server: detected 4 online CPUs by sysfs oleg416-server: Force libcfs to create 2 CPU partitions Iteration 48 Starting client: oleg416-client.virtnet: -o user_xattr,flock oleg416-server@tcp:/lustre /mnt/lustre2 Starting client: oleg416-client.virtnet: -o user_xattr,flock oleg416-server@tcp:/lustre /mnt/lustre 192.168.204.116@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt 0 0 192.168.204.116@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt 0 0 Stopping client oleg416-client.virtnet /mnt/lustre (opts:) Stopping client oleg416-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg416-server' oleg416-server: oleg416-server.virtnet: executing load_modules_local oleg416-server: Loading modules from /home/green/git/lustre-release/lustre oleg416-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg416-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg416-server: detected 4 online CPUs by sysfs oleg416-server: Force libcfs to create 2 CPU partitions Iteration 49 Starting client: oleg416-client.virtnet: -o user_xattr,flock oleg416-server@tcp:/lustre /mnt/lustre2 Starting client: oleg416-client.virtnet: -o user_xattr,flock oleg416-server@tcp:/lustre /mnt/lustre 192.168.204.116@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt 0 0 192.168.204.116@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt 0 0 Stopping client oleg416-client.virtnet /mnt/lustre (opts:) Stopping client oleg416-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg416-server' oleg416-server: oleg416-server.virtnet: executing load_modules_local oleg416-server: Loading modules from /home/green/git/lustre-release/lustre oleg416-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg416-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg416-server: detected 4 online CPUs by sysfs oleg416-server: Force libcfs to create 2 CPU partitions Iteration 50 Starting client: oleg416-client.virtnet: -o user_xattr,flock oleg416-server@tcp:/lustre /mnt/lustre2 Starting client: oleg416-client.virtnet: -o user_xattr,flock oleg416-server@tcp:/lustre /mnt/lustre 192.168.204.116@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt 0 0 192.168.204.116@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt 0 0 Stopping client oleg416-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg416-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg416-server' oleg416-server: oleg416-server.virtnet: executing load_modules_local oleg416-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg416-server: Loading modules from /home/green/git/lustre-release/lustre oleg416-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg416-server: detected 4 online CPUs by sysfs oleg416-server: Force libcfs to create 2 CPU partitions Starting client: oleg416-client.virtnet: -o user_xattr,flock oleg416-server@tcp:/lustre /mnt/lustre Starting client: oleg416-client.virtnet: -o user_xattr,flock oleg416-server@tcp:/lustre /mnt/lustre2 PASS 109 (1146s) == sanityn test 110: do not grant another lock on resend ========================================================== 06:00:11 (1761127211) fail_loc=0x525 fail_val=3 fail_loc=0x534 File: /mnt/lustre/f110.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205306056707 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-10-22 06:00:13.000000000 -0400 Modify: 2025-10-22 06:00:13.000000000 -0400 Change: 2025-10-22 06:00:13.000000000 -0400 Birth: 2025-10-22 06:00:12.000000000 -0400 fail_loc=0 fail_val=0 Connected clients: oleg416-client.virtnet oleg416-client.virtnet PASS 110 (41s) == sanityn test 111: A racy rename/link an open file should not cause fs corruption ========================================================== 06:00:52 (1761127252) SKIP: sanityn test_111 needs >= 2 MDTs SKIP 111 (3s) == sanityn test 112: update max-inherit in default LMV === 06:00:55 (1761127255) SKIP: sanityn test_112 We need at least 2 MDTs for this test SKIP 112 (3s) == sanityn test 113: check servers of specified fs ======= 06:00:59 (1761127259) lustre-OST0000-osc-ffff9a01ee758000 active. lustre-OST0001-osc-ffff9a01ee758000 active. lustre-OST0000-osc-ffff9a01ee758000 active. lustre-OST0001-osc-ffff9a01ee758000 active. MGC192.168.204.116@tcp active. PASS 113 (5s) cleanup: ====================================================== == sanityn test complete, duration 6198 sec ============== 06:01:05 (1761127265) Stopping clients: oleg416-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg416-client.virtnet /mnt/lustre2 opts: