-----============= acceptance-small: sanityn ============----- Mon Sep 7 19:08:50 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg407-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanityn.*ex': No such file or directory excepting tests: 27 40a skipping tests SLOW=no: 33a === sanityn: start setup 19:09:09 (1788822549) === Starting client oleg407-client.virtnet: -o user_xattr,flock 192.168.204.107@tcp:/lustre /mnt/lustre2 Mount client oleg407-client.virtnet: mount -t lustre -o user_xattr,flock 192.168.204.107@tcp:/lustre /mnt/lustre2 Started clients oleg407-client.virtnet: 192.168.204.107@tcp:/lustre on /mnt/lustre2 type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) oleg407-client.virtnet: executing check_config_client /mnt/lustre oleg407-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg407-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff92ca0a087000.idle_timeout=debug osc.lustre-OST0000-osc-ffff92ca100da000.idle_timeout=debug osc.lustre-OST0001-osc-ffff92ca0a087000.idle_timeout=debug osc.lustre-OST0001-osc-ffff92ca100da000.idle_timeout=debug disable quota as required oleg407-server: oleg407-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanityn: finish setup 19:09:49 (1788822589) === 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0320317 s, 32.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7612] == sanityn test 0: do_node_vp() and do_facet_vp() do the right thing ========================================================== 19:09:52 (1788822592) PASS 0 (8s) == sanityn test 1: Check attribute updates on 2 mount points ========================================================== 19:10:00 (1788822600) /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0777 OK /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0666 OK /mnt/lustre/f1.sanityn: absent OK PASS 1 (7s) == sanityn test 2a: check cached attribute updates on 2 mtpt's ================================================================== 19:10:07 (1788822607) -rw-r--r-- 1 root root 0 Sep 7 19:10 /mnt/lustre2/f2a /mnt/lustre/f2a has type file OK /mnt/lustre/f2a has perms 0777 OK PASS 2a (7s) == sanityn test 2b: check cached attribute updates on 2 mtpt's ================================================================== 19:10:14 (1788822614) -rw-r--r-- 1 root root 0 Sep 7 19:10 /mnt/lustre2/f2b /mnt/lustre2/f2b has type file OK /mnt/lustre2/f2b has perms 0777 OK PASS 2b (7s) == sanityn test 2c: check cached attribute updates on 2 mtpt's root ============================================================= 19:10:21 (1788822621) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0777 OK PASS 2c (6s) == sanityn test 2d: check cached attribute updates on 2 mtpt's root ============================================================= 19:10:27 (1788822627) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0755 OK PASS 2d (6s) == sanityn test 2e: check chmod on root is propagated to others ========================================================== 19:10:33 (1788822633) total 1 -rwxrwxrwx 1 root root 0 Sep 7 19:10 f2a -rwxrwxrwx 1 root root 0 Sep 7 19:10 f2b total 1 -rwxrwxrwx 1 root root 0 Sep 7 19:10 f2a -rwxrwxrwx 1 root root 0 Sep 7 19:10 f2b running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre2/f2e.sanityn] [count=1] 1+0 records in 1+0 records out 512 bytes copied, 0.0118209 s, 43.3 kB/s PASS 2e (8s) == sanityn test 2f: check attr/owner updates on DNE with 2 mtpt's ========================================================== 19:10:41 (1788822641) SKIP: sanityn test_2f needs >= 2 MDTs SKIP 2f (4s) == sanityn test 2g: check blocks update on sync write ==== 19:10:45 (1788822645) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.361842 s, 5.8 MB/s /mnt/lustre/f2g.sanityn has 4103 blocks /mnt/lustre2/f2g.sanityn has 4103 blocks PASS 2g (8s) == sanityn test 3: symlink on one mtpt, readlink on another ===================================================================== 19:10:54 (1788822654) PASS 3 (8s) == sanityn test 4: fstat validation on multiple mount points ==================================================================== 19:11:02 (1788822662) PASS 4 (8s) == sanityn test 5: create a file on one mount, truncate it on the other ========================================================== 19:11:11 (1788822671) /mnt/lustre/f5 has type file OK /mnt/lustre/f5 has size 100 OK PASS 5 (10s) == sanityn test 6: remove of open file on other node ============================================================================ 19:11:20 (1788822680) opening writing unlinking /mnt/lustre2/f6.sanityn accessing (1) seeking (1) accessing (2) fstat... reading comparing data truncating seeking (2) writing again seeking (3) reading again comparing data again closing SUCCESS - goto beer PASS 6 (7s) == sanityn test 7: remove of open directory on other node ======================================================================= 19:11:27 (1788822687) creating directory /mnt/lustre/d7 opening directory unlinking /mnt/lustre/d7 Ok, everything goes well. PASS 7 (7s) == sanityn test 8: remove of open special file on other node ==================================================================== 19:11:34 (1788822694) creating special file /mnt/lustre/f8.sanityn opening file unlinking /mnt/lustre/f8.sanityn Ok, everything goes well. PASS 8 (8s) == sanityn test 9a: append of file with sub-page size on multiple mounts ========================================================== 19:11:42 (1788822702) PASS 9a (7s) == sanityn test 9b: append to striped sparse file ======== 19:11:50 (1788822710) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.06419 s, 16.3 MB/s 3+0 records in 3+0 records out 3 bytes copied, 0.0366319 s, 0.1 kB/s Data read (expecting 'foo'): foo PASS 9b (8s) == sanityn test 10a: write of file with sub-page size on multiple mounts ========================================================== 19:11:58 (1788822718) 1+0 records in 1+0 records out 1 byte copied, 0.00552349 s, 0.2 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00670289 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.026443 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0192959 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00711986 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00887197 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00799379 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0156808 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00941433 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00938377 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0331829 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0102594 s, 0.1 kB/s PASS 10a (9s) == sanityn test 10b: write of file with sub-page size on multiple mounts ========================================================== 19:12:07 (1788822727) 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.0174723 s, 176 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0487175 s, 84.1 kB/s 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.000554626 s, 5.5 MB/s PASS 10b (8s) == sanityn test 11: execution of file opened for write should return error ============================================================== 19:12:15 (1788822735) multiop /mnt/lustre/d11/f vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7612 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 298: /mnt/lustre2/d11/f: Text file busy PASS 11 (9s) == sanityn test 12: test lock ordering (link, stat, unlink) ========================================================== 19:12:24 (1788822744) warning: '-runtime' deprecated, use '-t runtime' instead - link 361 (time 1788822756.38 total 10.04 last 35.94) - link 725 (time 1788822766.41 total 20.07 last 36.30) - link 1090 (time 1788822776.43 total 30.09 last 36.44) - link 1442 (time 1788822786.43 total 40.09 last 35.18) - link 1846 (time 1788822796.46 total 50.12 last 40.27) - link 2214 (time 1788822806.47 total 60.13 last 36.79) - link 2567 (time 1788822816.47 total 70.14 last 35.27) - link 2933 (time 1788822826.48 total 80.15 last 36.56) - link 3347 (time 1788822836.52 total 90.18 last 41.26) total: 3723 link in 100.01 seconds: 37.23 ops/second using seed 164302503 running for 100 seconds - stat 4700 (time 1788822758 ; total 11 ; last 11) - stat 10000 (time 1788822768 ; total 21 ; last 10) - stat 15209 (time 1788822779 ; total 32 ; last 11) - stat 20000 (time 1788822788 ; total 41 ; last 9) - stat 25466 (time 1788822799 ; total 52 ; last 11) - stat 28201 (time 1788822810 ; total 63 ; last 11) - stat 30000 (time 1788822817 ; total 70 ; last 7) - stat 32548 (time 1788822828 ; total 81 ; last 11) - stat 35919 (time 1788822839 ; total 92 ; last 11) total: 38145 stats in 100 seconds: 381.450012 stats/second - unlinked 0 (time 1788822797 ; total 0 ; last 0) unlink(/mnt/lustre2/lockdir/lockfile3723) error: No such file or directory total: 3722 unlinks in 90 seconds: 41.355556 unlinks/second /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 77: kill: (21418) - No such process /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 78: kill: (21420) - No such process PASS 12 (148s) == sanityn test 13: test directory page revocation ======= 19:14:52 (1788822892) f13 f13 PASS 13 (8s) == sanityn test 14aa: execution of file open for write returns -ETXTBSY ========================================================== 19:15:01 (1788822901) multiop /mnt/lustre/d14aa.sanityn/f14aa.sanityn vOw_c TMPPIPE=/tmp/multiop_open_wait_pipe.7612 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 333: /mnt/lustre2/d14aa.sanityn/f14aa.sanityn: Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 22604 Terminated $MULTIOP_PROG $FILE v$ARGS > $TMPPIPE (wd: ~) PASS 14aa (7s) == sanityn test 14ab: open(RDWR) of executing file returns -ETXTBSY ========================================================== 19:15:07 (1788822907) open(O_RDWR|O_CREAT): Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 23184 Terminated $DIR1/$tdir/sleep 60 (wd: ~) PASS 14ab (8s) == sanityn test 14b: truncate of executing file returns -ETXTBSY ================================================================ 19:15:15 (1788822915) truncate: cannot truncate '/mnt/lustre2/d14b.sanityn/sleep' to length 60: Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 348: 23762 Terminated $DIR1/$tdir/sleep 60 PASS 14b (6s) == sanityn test 14c: open(O_TRUNC) of executing file return -ETXTBSY ============================================================ 19:15:22 (1788822922) cp: cannot create regular file '/mnt/lustre2/d14c.sanityn/sleep': Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 360: 24344 Terminated $DIR1/$tdir/sleep 60 PASS 14c (6s) == sanityn test 14d: chmod of executing file is still possible ================================================================== 19:15:28 (1788822928) chmod /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 371: 24927 Terminated $DIR1/$tdir/sleep 60 PASS 14d (9s) == sanityn test 15: test out-of-space with multiple writers ===================================================================== 19:15:37 (1788822937) PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin: MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky sleep 5 for ZFS MDS Waiting for MDT destroys to complete fallocate on zfs doesn't consume space STRIPECOUNT=2 ORIGFREE=7531520 MAXFREE=800000 SKIP: oos2 test_15 oos2.sh: 7531520KB free > 800000KB, increase MAXFREE (or reduce fs size) sleep 5 for ZFS MDS Waiting for MDT destroys to complete SKIP 15 (56s) == sanityn test 16a: 500 iterations of dual-mount fsx ==== 19:16:33 (1788822993) keep default fallocate mode: 2 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.574576 s, 14.6 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.412337 s, 20.3 MB/s Chance of close/open is 1 in 50 Seed set to 2997 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000001[1] 1788822998.284921 write 0x542c5e thru 0x54c3b8 (0x975b bytes) 000002[1] 1788822998.319515 read 0x10358a thru 0x1081b0 (0x4c27 bytes) truncating to largest ever: 0x82f48a 000004[1] 1788822998.350311 trunc from 0x54c3b9 to 0x82f489 (0x2e30d1 bytes) 000005[0] 1788822998.402899 mapread 0x09f2c3 thru 0x0ac589 (0xd2c7 bytes) 000009[1] 1788822998.557604 mapwrite 0x775368 thru 0x783a35 (0xe6ce bytes) 000013[0] 1788822998.822626 mapwrite 0x3db07b thru 0x3e2956 (0x78dc bytes) 000017[1] 1788822999.030621 mapwrite 0x4a9b09 thru 0x4b5c7d (0xc175 bytes) 000018[0] 1788822999.189552 read 0x0067d2 thru 0x00d78d (0x6fbc bytes) 000019[1] 1788822999.222958 write 0x6ab980 thru 0x6b1c0a (0x628b bytes) 000021[1] 1788822999.268028 write 0x62c221 thru 0x634dd5 (0x8bb5 bytes) 000023[0] 1788822999.307340 punch from 0x4fecb2 to 0x4ff12c (0x47b bytes) truncating to largest ever: 0x88c44f 000024[1] 1788822999.354499 trunc from 0x82f48a to 0x88c44e (0x5cfc5 bytes) 000026[0] 1788822999.403477 mapread 0x4200bb thru 0x424d0d (0x4c53 bytes) 000027[1] 1788822999.458093 punch from 0x318823 to 0x328725 (0xff03 bytes) 000028[0] 1788822999.462924 mapwrite 0x477e00 thru 0x4814fb (0x96fc bytes) 000029[1] 1788822999.605484 write 0x68647c thru 0x6954ba (0xf03f bytes) 000030[0] 1788822999.622883 write 0x502a68 thru 0x50bc94 (0x922d bytes) 000031[1] 1788822999.669595 mapread 0x7943e6 thru 0x798776 (0x4391 bytes) 000033[1] 1788822999.724027 punch from 0x59be6e to 0x5a7173 (0xb306 bytes) 000034[0] 1788822999.749636 mapwrite 0x2f76d0 thru 0x2f8805 (0x1136 bytes) 000036[0] 1788822999.847026 trunc from 0x88c44f to 0x33adb8 (0x551696 bytes) 000037[1] 1788822999.880207 write 0x35a0b7 thru 0x369ddf (0xfd29 bytes) 000038[0] 1788822999.907081 trunc from 0x369de0 to 0x5ec849 (0x282a6a bytes) 000039[0] 1788822999.965028 mapread 0x096148 thru 0x0a52e6 (0xf19f bytes) 000040[1] 1788823000.009798 write 0x219141 thru 0x21db14 (0x49d4 bytes) 000042[1] 1788823000.062150 mapwrite 0x55e63c thru 0x56ded9 (0xf89e bytes) truncating to largest ever: 0x8ce070 000043[0] 1788823000.177608 trunc from 0x5ec84a to 0x8ce06f (0x2e1826 bytes) truncating to largest ever: 0x9c7ba9 000045[1] 1788823000.254988 trunc from 0x8ce070 to 0x9c7ba8 (0xf9b39 bytes) 000049[0] 1788823000.344902 mapread 0x31d633 thru 0x320402 (0x2dd0 bytes) 000051[0] 1788823000.417725 mapwrite 0x91ed91 thru 0x921ec0 (0x3130 bytes) 000053[1] 1788823000.519169 mapwrite 0x8d9ccf thru 0x8e51bb (0xb4ed bytes) 000056[1] 1788823000.723903 mapread 0x5f4253 thru 0x5faa69 (0x6817 bytes) 000057[0] 1788823000.750592 mapread 0x144f9f thru 0x14d62b (0x868d bytes) 000058[1] 1788823000.819603 write 0x3cc390 thru 0x3ce0a0 (0x1d11 bytes) 000063[1] 1788823000.885712 punch from 0x85ddec to 0x85f7b0 (0x19c5 bytes) 000064[1] 1788823000.907494 mapwrite 0x0ccb7b thru 0x0d5de2 (0x9268 bytes) 000065[1] 1788823001.004507 punch from 0x07cf7d to 0x07d71e (0x7a2 bytes) 000066[1] 1788823001.047243 mapwrite 0x133eb2 thru 0x141e3d (0xdf8c bytes) 000067[0] 1788823001.244831 mapwrite 0x024775 thru 0x030fe1 (0xc86d bytes) 000071[1] 1788823001.483244 trunc from 0x9c7ba9 to 0x47ef9f (0x548c09 bytes) 000072[1] 1788823001.592995 read 0x3a230e thru 0x3a6522 (0x4215 bytes) 000074[0] 1788823001.642026 read 0x2b5392 thru 0x2c2d1b (0xd98a bytes) 000075[0] 1788823001.673830 write 0x30d19f thru 0x319879 (0xc6db bytes) 000076[0] 1788823001.709474 write 0x66327e thru 0x666bd0 (0x3953 bytes) 000079[1] 1788823001.767201 trunc from 0x666bd1 to 0x5b03b8 (0xb6818 bytes) 000080[0] 1788823001.830925 write 0x4568bb thru 0x4580d7 (0x181d bytes) 000082[1] 1788823001.906882 punch from 0x157454 to 0x163801 (0xc3ae bytes) 000083[0] 1788823001.967612 mapread 0x2b4211 thru 0x2b4256 (0x46 bytes) 000084[1] 1788823002.012938 mapwrite 0x690236 thru 0x69cdcd (0xcb98 bytes) 000085[0] 1788823002.202516 trunc from 0x69cdce to 0x578b0a (0x1242c3 bytes) 000087[1] 1788823002.282816 mapwrite 0x0b4df0 thru 0x0ba69d (0x58ae bytes) 000088[0] 1788823002.374788 punch from 0x15e3cf to 0x15f8fe (0x1530 bytes) 000089[0] 1788823002.391007 mapread 0x1f5667 thru 0x2036ae (0xe048 bytes) 000090[1] 1788823002.417706 mapwrite 0x43a344 thru 0x445157 (0xae14 bytes) 000092[1] 1788823002.575169 punch from 0x159ed6 to 0x15e1c7 (0x42f2 bytes) 000093[0] 1788823002.599782 punch from 0x274886 to 0x278b5f (0x42da bytes) 000094[0] 1788823002.626521 trunc from 0x578b0b to 0x2b76a1 (0x2c1469 bytes) 000096[1] 1788823002.750598 punch from 0x23897e to 0x23c689 (0x3d0c bytes) 000097[1] 1788823002.826793 punch from 0x2ac6dd to 0x2b76a1 (0xafc5 bytes) 000098[1] 1788823002.856855 mapwrite 0x7fc67a thru 0x800d07 (0x468e bytes) 000099[0] 1788823003.138319 write 0x263594 thru 0x26c416 (0x8e83 bytes) 000100[0] 1788823003.188840 trunc from 0x800d08 to 0x6dd874 (0x123493 bytes) 000102[1] 1788823003.227142 read 0x528152 thru 0x52d99b (0x584a bytes) 000103[0] 1788823003.267316 mapread 0x691e7e thru 0x696991 (0x4b14 bytes) 000104[0] 1788823003.326122 trunc from 0x6dd875 to 0x3530a3 (0x38a7d1 bytes) 000105[0] 1788823003.382790 read 0x02908b thru 0x030ecd (0x7e43 bytes) 000107[1] 1788823003.533222 write 0x10c3df thru 0x11445a (0x807c bytes) 000108[0] 1788823003.591941 punch from 0x268461 to 0x276649 (0xe1e9 bytes) 000109[0] 1788823003.614207 mapwrite 0x195e8a thru 0x199530 (0x36a7 bytes) 000110[1] 1788823003.780203 punch from 0x1272fe to 0x129d4c (0x2a4f bytes) 000111[0] 1788823003.814941 mapwrite 0x320a8a thru 0x3268ee (0x5e65 bytes) 000112[1] 1788823003.959459 write 0x062193 thru 0x065109 (0x2f77 bytes) 000113[0] 1788823004.009744 punch from 0x29506c to 0x29ed1d (0x9cb2 bytes) 000115[0] 1788823004.078168 mapwrite 0x42e0bc thru 0x43cd4c (0xec91 bytes) 000117[1] 1788823004.180546 read 0x163a19 thru 0x166b8d (0x3175 bytes) 000118[1] 1788823004.276993 write 0x10a771 thru 0x10b00b (0x89b bytes) 000119[1] 1788823004.320812 write 0x33b9ae thru 0x33f2d0 (0x3923 bytes) 000120[0] 1788823004.342719 mapwrite 0x33a42a thru 0x33e99b (0x4572 bytes) 000121[1] 1788823004.503893 mapread 0x26f08b thru 0x270bdb (0x1b51 bytes) 000123[1] 1788823004.560830 read 0x18f6a8 thru 0x193ee7 (0x4840 bytes) 000124[1] 1788823004.568865 write 0x8a6a5c thru 0x8b69a7 (0xff4c bytes) 000126[0] 1788823004.629411 write 0x122679 thru 0x131b7c (0xf504 bytes) 000127[1] 1788823004.667150 punch from 0x2af4af to 0x2b2b88 (0x36da bytes) 000128[1] 1788823004.750004 write 0x927a9f thru 0x9299c4 (0x1f26 bytes) 000130[0] 1788823004.804083 mapwrite 0x955849 thru 0x96374e (0xdf06 bytes) 000131[1] 1788823005.144686 mapread 0x42c2aa thru 0x42d3ce (0x1125 bytes) 000133[0] 1788823005.237815 trunc from 0x96374f to 0x0b8f38 (0x8aa816 bytes) 000139[0] 1788823005.368430 read 0x0b0f3d thru 0x0b8f38 (0x7ffc bytes) 000140[0] 1788823005.442131 trunc from 0x0b8f39 to 0x99f047 (0x8e610f bytes) 000141[0] 1788823005.487390 mapread 0x3f5b7a thru 0x4028a9 (0xcd30 bytes) 000143[0] 1788823005.626223 punch from 0x9593d5 to 0x95e032 (0x4c5e bytes) 000144[0] 1788823005.631423 write 0x2de2c7 thru 0x2e705f (0x8d99 bytes) 000145[0] 1788823005.632665 write 0x214f88 thru 0x21f98c (0xaa05 bytes) 000147[0] 1788823005.635129 write 0x73c8f2 thru 0x7460f0 (0x97ff bytes) 000148[0] 1788823005.731043 write 0x8611a0 thru 0x86d07e (0xbedf bytes) 000150[1] 1788823005.774850 write 0x1625b1 thru 0x16d882 (0xb2d2 bytes) 000153[0] 1788823005.861705 read 0x30ad29 thru 0x30fac0 (0x4d98 bytes) 000155[0] 1788823005.931349 trunc from 0x99f048 to 0x946e81 (0x581c6 bytes) 000156[0] 1788823005.974375 read 0x6b890a thru 0x6c23f2 (0x9ae9 bytes) 000158[0] 1788823006.014603 write 0x2acb77 thru 0x2b4526 (0x79b0 bytes) 000159[0] 1788823006.062154 mapread 0x6ffca5 thru 0x707a96 (0x7df2 bytes) 000160[0] 1788823006.063761 punch from 0x461182 to 0x462091 (0xf10 bytes) 000162[0] 1788823006.076837 trunc from 0x946e82 to 0x17de02 (0x7c907f bytes) 000164[1] 1788823006.152660 mapwrite 0x530553 thru 0x53593e (0x53ec bytes) 000165[1] 1788823006.379464 punch from 0x32b331 to 0x32e2f5 (0x2fc5 bytes) 000166[0] 1788823006.489941 read 0x3813d2 thru 0x38afed (0x9c1c bytes) 000168[0] 1788823006.520525 trunc from 0x53593f to 0x3abab7 (0x189e87 bytes) 000169[0] 1788823006.621592 mapwrite 0x363b35 thru 0x36c2fd (0x87c9 bytes) 000170[1] 1788823006.773551 mapread 0x1fbd20 thru 0x20499d (0x8c7e bytes) 000171[1] 1788823006.837240 trunc from 0x3abab8 to 0x6a049a (0x2f49e3 bytes) 000172[1] 1788823006.927414 read 0x2c6018 thru 0x2d1b88 (0xbb71 bytes) 000173[1] 1788823006.963500 write 0x2fa023 thru 0x304d84 (0xad62 bytes) 000174[1] 1788823007.011855 trunc from 0x6a049b to 0x013d6e (0x68c72c bytes) 000180[1] 1788823007.198008 mapread 0x000f72 thru 0x005ebb (0x4f4a bytes) 000185[0] 1788823007.337520 read 0x000d89 thru 0x004905 (0x3b7d bytes) 000187[0] 1788823007.470916 mapwrite 0x5e528b thru 0x5eca1d (0x7793 bytes) 000188[1] 1788823007.757743 mapwrite 0x20f625 thru 0x213086 (0x3a62 bytes) 000189[1] 1788823008.033307 mapwrite 0x425403 thru 0x42aa76 (0x5674 bytes) 000190[0] 1788823008.218231 mapread 0x16cb76 thru 0x175535 (0x89c0 bytes) 000191[1] 1788823008.417130 punch from 0x439617 to 0x43ec3e (0x5628 bytes) 000192[1] 1788823008.444450 write 0x38527f thru 0x39131e (0xc0a0 bytes) 000193[0] 1788823008.586074 read 0x29c69b thru 0x29ffb0 (0x3916 bytes) 000194[0] 1788823008.658240 mapread 0x48f370 thru 0x497f2a (0x8bbb bytes) 000195[1] 1788823008.727301 mapread 0x21db48 thru 0x224aa6 (0x6f5f bytes) 000196[0] 1788823008.752579 write 0x74610a thru 0x749d48 (0x3c3f bytes) 000198[1] 1788823008.776010 read 0x1dbd26 thru 0x1e3821 (0x7afc bytes) 000199[1] 1788823008.795991 mapread 0x06873f thru 0x06daef (0x53b1 bytes) 000200[0] 1788823008.831704 trunc from 0x749d49 to 0x961a8d (0x217d45 bytes) 000203[0] 1788823008.884688 mapread 0x208099 thru 0x20e94b (0x68b3 bytes) 000204[1] 1788823008.901276 mapread 0x5a5121 thru 0x5ac3c2 (0x72a2 bytes) 000205[0] 1788823008.941978 mapread 0x6b0f2b thru 0x6be004 (0xd0da bytes) 000208[0] 1788823008.971710 read 0x661806 thru 0x66e5a7 (0xcda2 bytes) 000209[1] 1788823008.981728 trunc from 0x961a8e to 0x541080 (0x420a0d bytes) 000210[0] 1788823009.022458 write 0x7636f9 thru 0x76ec1d (0xb525 bytes) 000211[0] 1788823009.051334 punch from 0x6857f7 to 0x693292 (0xda9c bytes) 000213[0] 1788823009.074768 punch from 0x556ad8 to 0x56439b (0xd8c4 bytes) 000215[0] 1788823009.101958 mapread 0x11dc98 thru 0x1286d4 (0xaa3d bytes) 000216[1] 1788823009.142512 write 0x401285 thru 0x40809d (0x6e19 bytes) 000217[1] 1788823009.172369 mapwrite 0x151fd5 thru 0x158dee (0x6e1a bytes) 000220[1] 1788823009.267979 mapwrite 0x39a4b8 thru 0x39f1f8 (0x4d41 bytes) 000221[0] 1788823009.352625 punch from 0x6b8080 to 0x6c2020 (0x9fa1 bytes) 000223[1] 1788823009.398634 mapread 0x3b28ad thru 0x3b7fe9 (0x573d bytes) 000225[0] 1788823009.447760 trunc from 0x76ec1e to 0x2abac7 (0x4c3156 bytes) 000228[0] 1788823009.496129 mapread 0x234909 thru 0x23da15 (0x910d bytes) 000231[0] 1788823009.590040 mapread 0x190b95 thru 0x197506 (0x6972 bytes) 000232[0] 1788823009.660662 punch from 0x17d2af to 0x186090 (0x8de2 bytes) 000234[0] 1788823009.756408 read 0x0cc151 thru 0x0cdf3e (0x1dee bytes) 000236[0] 1788823009.804017 punch from 0x13c32b to 0x14a0fd (0xddd3 bytes) 000238[0] 1788823009.831549 punch from 0x25f9f7 to 0x26774c (0x7d56 bytes) 000239[0] 1788823009.860368 trunc from 0x2abac8 to 0x747d6c (0x49c2a5 bytes) 000241[0] 1788823009.926842 write 0x1e5b89 thru 0x1f249b (0xc913 bytes) 000242[0] 1788823009.940607 punch from 0x64759d to 0x64df1d (0x6981 bytes) 000245[0] 1788823009.986775 punch from 0x2caf78 to 0x2cd483 (0x250c bytes) 000246[1] 1788823010.032231 write 0x1a792d thru 0x1b633c (0xea10 bytes) 000247[0] 1788823010.134976 trunc from 0x747d6d to 0x47d22a (0x2cab42 bytes) 000248[0] 1788823010.231547 mapwrite 0x7c5fa0 thru 0x7c7432 (0x1493 bytes) 000250[1] 1788823010.470642 read 0x3bce7b thru 0x3c1bfa (0x4d80 bytes) 000252[0] 1788823010.515826 punch from 0x1dc0a2 to 0x1dfe07 (0x3d66 bytes) 000253[0] 1788823010.553468 mapread 0x4bb273 thru 0x4c97c5 (0xe553 bytes) 000254[1] 1788823010.612121 mapwrite 0x567008 thru 0x573564 (0xc55d bytes) 000257[1] 1788823010.758923 mapwrite 0x8e2a68 thru 0x8f0d71 (0xe30a bytes) 000262[1] 1788823011.137188 write 0x5a4b5e thru 0x5abde5 (0x7288 bytes) 000265[1] 1788823011.233927 mapread 0x2687ec thru 0x26fb5b (0x7370 bytes) 000267[1] 1788823011.304358 punch from 0x79b18c to 0x7a51d5 (0xa04a bytes) 000268[0] 1788823011.309695 mapwrite 0x2fdf07 thru 0x306133 (0x822d bytes) 000270[0] 1788823011.504815 trunc from 0x8f0d72 to 0x2e92f4 (0x607a7d bytes) 000271[0] 1788823011.569172 mapwrite 0x8e5d6f thru 0x8e66f8 (0x98a bytes) 000272[1] 1788823011.719765 mapwrite 0x2e8e28 thru 0x2f01e9 (0x73c2 bytes) 000273[0] 1788823011.847938 read 0x54c494 thru 0x54e57a (0x20e7 bytes) 000274[0] 1788823011.894001 mapread 0x1a90a4 thru 0x1ae3e6 (0x5343 bytes) 000276[0] 1788823011.967768 mapwrite 0x8c50b4 thru 0x8c6b6a (0x1ab7 bytes) 000277[1] 1788823012.080009 write 0x9b1265 thru 0x9bd334 (0xc0d0 bytes) 000278[1] 1788823012.114736 mapread 0x072e01 thru 0x07dae9 (0xace9 bytes) 000279[0] 1788823012.177230 write 0x056ffa thru 0x057617 (0x61e bytes) 000282[0] 1788823012.314486 mapwrite 0x430370 thru 0x43f154 (0xede5 bytes) 000284[1] 1788823012.595905 mapwrite 0x847454 thru 0x85034a (0x8ef7 bytes) 000286[0] 1788823012.974899 punch from 0x619fad to 0x61b31c (0x1370 bytes) 000287[0] 1788823013.002775 write 0x50e1e0 thru 0x51be33 (0xdc54 bytes) 000288[1] 1788823013.035659 mapwrite 0x38d6c9 thru 0x38e137 (0xa6f bytes) 000289[1] 1788823013.197351 punch from 0x7c6a78 to 0x7cb365 (0x48ee bytes) 000293[0] 1788823013.254919 punch from 0x07ad44 to 0x07fae0 (0x4d9d bytes) 000294[0] 1788823013.297754 trunc from 0x9bd335 to 0x13e86a (0x87eaca bytes) 000295[0] 1788823013.415048 punch from 0x12ac4b to 0x13677f (0xbb35 bytes) 000296[0] 1788823013.461507 write 0x94b47e thru 0x95315a (0x7cdd bytes) 000297[0] 1788823013.476833 trunc from 0x95315b to 0x1adad6 (0x7a5684 bytes) 000298[0] 1788823013.522647 mapwrite 0x978fa1 thru 0x979304 (0x364 bytes) 000299[1] 1788823013.635501 punch from 0x73b947 to 0x73dadc (0x2196 bytes) 000300[0] 1788823013.676656 mapwrite 0x26ccee thru 0x279738 (0xca4b bytes) 000301[0] 1788823013.875152 trunc from 0x979305 to 0x43603f (0x5432c5 bytes) 000305[1] 1788823014.002748 write 0x08f760 thru 0x09a769 (0xb00a bytes) 000306[1] 1788823014.045645 punch from 0x0b5914 to 0x0c352f (0xdc1c bytes) 000307[0] 1788823014.072595 mapwrite 0x0adea4 thru 0x0bafbc (0xd119 bytes) 000308[1] 1788823014.169460 write 0x30ee32 thru 0x31cc74 (0xde43 bytes) 000310[0] 1788823014.222419 read 0x0feafe thru 0x10e328 (0xf82b bytes) 000311[1] 1788823014.260441 trunc from 0x436040 to 0x02f631 (0x406a0e bytes) 000312[0] 1788823014.314216 mapwrite 0x686802 thru 0x6935d1 (0xcdd0 bytes) 000313[1] 1788823014.583523 trunc from 0x6935d2 to 0x7b3249 (0x11fc78 bytes) 000316[0] 1788823014.763488 write 0x00ea74 thru 0x015501 (0x6a8e bytes) 000317[1] 1788823014.891301 mapread 0x07f810 thru 0x084849 (0x503a bytes) 000318[0] 1788823015.029619 trunc from 0x7b324a to 0x04c011 (0x767238 bytes) 000319[1] 1788823015.116768 punch from 0x01458d to 0x01a87d (0x62f1 bytes) 000322[1] 1788823015.233306 mapread 0x025504 thru 0x02b4d5 (0x5fd2 bytes) 000323[0] 1788823015.303967 mapwrite 0x7243c5 thru 0x7254dd (0x1119 bytes) 000325[0] 1788823015.545568 mapread 0x4cdfd3 thru 0x4dd73e (0xf76c bytes) 000326[1] 1788823015.696473 trunc from 0x7254de to 0x49e3fc (0x2870e1 bytes) 000327[1] 1788823015.861847 mapwrite 0x08987a thru 0x08bd73 (0x24fa bytes) 000328[0] 1788823016.044327 trunc from 0x49e3fd to 0x898658 (0x3fa25c bytes) 000329[1] 1788823016.111241 trunc from 0x898659 to 0x995a53 (0xfd3fb bytes) 000330[0] 1788823016.176588 mapread 0x3c7f84 thru 0x3cd9bf (0x5a3c bytes) 000331[1] 1788823016.279994 read 0x62c324 thru 0x62d181 (0xe5e bytes) 000333[0] 1788823016.344898 mapread 0x71f31d thru 0x72d97d (0xe661 bytes) 000336[1] 1788823016.497739 mapread 0x51e947 thru 0x527369 (0x8a23 bytes) 000337[1] 1788823016.560845 write 0x823961 thru 0x8326e9 (0xed89 bytes) 000338[1] 1788823016.673358 trunc from 0x995a54 to 0x4a9a3f (0x4ec014 bytes) 000339[0] 1788823016.884129 write 0x679006 thru 0x686181 (0xd17c bytes) 000340[0] 1788823016.962668 mapread 0x575119 thru 0x5840be (0xefa6 bytes) 000341[1] 1788823017.096313 mapread 0x36a3b1 thru 0x36af5e (0xbae bytes) 000342[1] 1788823017.224025 trunc from 0x686182 to 0x007485 (0x67ecfc bytes) 000343[0] 1788823017.359335 write 0x4f78c1 thru 0x506cb5 (0xf3f5 bytes) 000345[1] 1788823017.528808 mapread 0x023ecd thru 0x02c237 (0x836b bytes) 000346[1] 1788823017.980194 mapwrite 0x255bab thru 0x258471 (0x28c7 bytes) 000347[0] 1788823018.270370 read 0x0af8d4 thru 0x0be71a (0xee47 bytes) 000350[0] 1788823018.339507 punch from 0x428473 to 0x43555a (0xd0e8 bytes) 000351[1] 1788823018.363300 mapwrite 0x5dfa0b thru 0x5e612e (0x6724 bytes) 000352[0] 1788823018.646357 read 0x2132ab thru 0x2133f8 (0x14e bytes) 000354[1] 1788823018.694736 trunc from 0x5e612f to 0x7fe355 (0x218227 bytes) 000357[1] 1788823018.895664 mapread 0x0b61e2 thru 0x0c374c (0xd56b bytes) 000358[1] 1788823018.978963 mapread 0x3ef7f4 thru 0x3fbab4 (0xc2c1 bytes) 000359[1] 1788823019.026713 trunc from 0x7fe356 to 0x64851a (0x1b5e3b bytes) 000362[0] 1788823019.095464 mapwrite 0x70f90d thru 0x7108ae (0xfa2 bytes) 000363[1] 1788823019.257945 mapread 0x01b5bb thru 0x02124b (0x5c91 bytes) 000365[1] 1788823019.334101 punch from 0x1bff13 to 0x1c2286 (0x2374 bytes) 000366[0] 1788823019.371153 punch from 0x122785 to 0x129282 (0x6afe bytes) 000367[1] 1788823019.420170 read 0x12f8d9 thru 0x134ebd (0x55e5 bytes) 000368[0] 1788823019.469232 punch from 0x6300c0 to 0x630852 (0x793 bytes) 000370[1] 1788823019.483277 punch from 0x2ee4c2 to 0x2f9619 (0xb158 bytes) 000371[1] 1788823019.529096 punch from 0x6e2817 to 0x6e2c09 (0x3f3 bytes) 000372[1] 1788823019.578802 mapwrite 0x68ae45 thru 0x69436d (0x9529 bytes) 000373[0] 1788823019.666148 read 0x4a9ce3 thru 0x4ac968 (0x2c86 bytes) 000375[1] 1788823019.737946 write 0x1a035a thru 0x1aaf37 (0xabde bytes) 000377[0] 1788823019.754247 punch from 0x4234dd to 0x42b4e2 (0x8006 bytes) 000379[1] 1788823019.833668 read 0x69fa12 thru 0x6abdfd (0xc3ec bytes) 000382[1] 1788823019.900073 punch from 0x31c94a to 0x32687e (0x9f35 bytes) 000383[1] 1788823019.914998 punch from 0x52bd47 to 0x539b41 (0xddfb bytes) 000384[0] 1788823019.997422 trunc from 0x7108af to 0x38a6fb (0x3861b3 bytes) 000385[0] 1788823020.234561 trunc from 0x38a6fc to 0x487966 (0xfd26b bytes) 000387[1] 1788823020.292739 mapread 0x385c28 thru 0x38f844 (0x9c1d bytes) 000388[0] 1788823020.407134 mapread 0x27e119 thru 0x28919c (0xb084 bytes) 000389[1] 1788823020.560400 read 0x041c09 thru 0x04ee96 (0xd28e bytes) 000390[1] 1788823020.596801 write 0x52e7e2 thru 0x532e56 (0x4675 bytes) 000391[1] 1788823020.702384 write 0x89148e thru 0x89b7a3 (0xa316 bytes) 000394[1] 1788823020.846502 mapread 0x4d3f38 thru 0x4e24f5 (0xe5be bytes) 000395[1] 1788823020.897894 read 0x544441 thru 0x552f7f (0xeb3f bytes) 000396[1] 1788823021.060513 mapread 0x49996f thru 0x4a000a (0x669c bytes) 000399[0] 1788823021.102416 mapread 0x2f0747 thru 0x2f8e9c (0x8756 bytes) 000400[1] 1788823021.180814 punch from 0x1f47c2 to 0x1fd9b3 (0x91f2 bytes) 000401[1] 1788823021.253503 write 0x872999 thru 0x878de7 (0x644f bytes) 000402[1] 1788823021.271965 write 0x2aaf1c thru 0x2b5cd8 (0xadbd bytes) 000405[0] 1788823021.342995 write 0x193de7 thru 0x1a05ce (0xc7e8 bytes) 000406[1] 1788823021.396788 mapwrite 0x186bca thru 0x18d92b (0x6d62 bytes) 000407[1] 1788823021.584297 mapread 0x11a978 thru 0x1234bf (0x8b48 bytes) 000408[0] 1788823021.609599 mapread 0x475d78 thru 0x484cdf (0xef68 bytes) 000409[1] 1788823021.766522 read 0x84ef0b thru 0x850c7f (0x1d75 bytes) 000411[0] 1788823021.851793 read 0x842d75 thru 0x848b97 (0x5e23 bytes) 000414[1] 1788823021.925097 mapread 0x1e6aa4 thru 0x1ebb80 (0x50dd bytes) 000415[1] 1788823021.985148 mapread 0x7c8eba thru 0x7d6331 (0xd478 bytes) 000416[1] 1788823022.164796 punch from 0x05b605 to 0x0656ff (0xa0fb bytes) 000417[1] 1788823022.193239 punch from 0x7500d3 to 0x751241 (0x116f bytes) 000418[0] 1788823022.215459 mapwrite 0x858d70 thru 0x85f1a9 (0x643a bytes) 000419[0] 1788823022.400985 mapread 0x44393d thru 0x450a35 (0xd0f9 bytes) 000421[0] 1788823022.493153 mapwrite 0x98d8b6 thru 0x994eec (0x7637 bytes) 000424[0] 1788823022.806723 write 0x266f9d thru 0x276a95 (0xfaf9 bytes) 000425[0] 1788823022.827797 read 0x7ca815 thru 0x7d3372 (0x8b5e bytes) 000426[1] 1788823022.913160 write 0x197538 thru 0x1a3db3 (0xc87c bytes) 000431[1] 1788823023.214530 punch from 0x6e0b5b to 0x6e5931 (0x4dd7 bytes) 000432[1] 1788823023.249873 punch from 0x9745e7 to 0x983bb0 (0xf5ca bytes) 000434[1] 1788823023.310306 mapread 0x381e5d thru 0x38eab7 (0xcc5b bytes) 000435[1] 1788823023.441158 mapread 0x6abddf thru 0x6b15d4 (0x57f6 bytes) 000437[0] 1788823023.531087 mapread 0x7fd493 thru 0x8049e1 (0x754f bytes) 000438[0] 1788823023.712247 mapwrite 0x594172 thru 0x5a1137 (0xcfc6 bytes) 000440[1] 1788823024.057120 trunc from 0x994eed to 0x219ae3 (0x77b409 bytes) 000444[1] 1788823024.316574 mapwrite 0x66732a thru 0x674c81 (0xd958 bytes) 000445[0] 1788823024.620849 punch from 0x340397 to 0x34bdf8 (0xba62 bytes) 000446[1] 1788823024.719211 write 0x665890 thru 0x66a4ae (0x4c1f bytes) 000447[0] 1788823024.781204 mapwrite 0x8edda5 thru 0x8f587f (0x7adb bytes) 000448[0] 1788823025.261960 read 0x0bcecf thru 0x0c5059 (0x818b bytes) 000449[0] 1788823025.372369 write 0x9e8707 thru 0x9e8dbe (0x6b8 bytes) 000451[0] 1788823025.454549 trunc from 0x9e8dbf to 0x125f12 (0x8c2eac bytes) 000453[1] 1788823025.500389 write 0x853ae6 thru 0x8563ef (0x290a bytes) 000455[0] 1788823025.627375 write 0x4a40df thru 0x4acdec (0x8d0e bytes) 000456[1] 1788823025.656855 punch from 0x451cf8 to 0x4592b6 (0x75bf bytes) 000457[1] 1788823025.715334 trunc from 0x8563f0 to 0x6e0e4c (0x1755a3 bytes) 000459[0] 1788823025.805996 punch from 0x2d6ede to 0x2de9ed (0x7b10 bytes) 000461[1] 1788823025.874982 mapwrite 0x44eadf thru 0x455ba2 (0x70c4 bytes) 000463[1] 1788823026.083319 mapread 0x1e1f80 thru 0x1e3e43 (0x1ec4 bytes) 000464[1] 1788823026.178036 trunc from 0x6e0e4d to 0x2ed94d (0x3f34ff bytes) 000465[0] 1788823026.255912 trunc from 0x2ed94e to 0x07b3c3 (0x27258a bytes) 000466[0] 1788823026.338766 mapread 0x026301 thru 0x030bc2 (0xa8c2 bytes) 000469[0] 1788823026.474423 trunc from 0x07b3c4 to 0x0f61a4 (0x7ade1 bytes) 000470[0] 1788823026.579617 punch from 0x0e87df to 0x0f3ac2 (0xb2e4 bytes) 000471[1] 1788823026.634608 read 0x0e15b5 thru 0x0e1d74 (0x7c0 bytes) 000472[0] 1788823026.772090 write 0x038c94 thru 0x03af39 (0x22a6 bytes) 000473[1] 1788823026.831137 trunc from 0x0f61a5 to 0x1052c6 (0xf122 bytes) 000474[1] 1788823026.958345 write 0x507e6a thru 0x5158d9 (0xda70 bytes) 000476[1] 1788823027.005967 write 0x78c2f8 thru 0x79a448 (0xe151 bytes) 000477[1] 1788823027.029989 read 0x402b7c thru 0x406377 (0x37fc bytes) 000480[1] 1788823027.064047 punch from 0x5cf53d to 0x5d6042 (0x6b06 bytes) 000481[1] 1788823027.102856 punch from 0x69e38a to 0x69eb5b (0x7d2 bytes) 000482[0] 1788823027.107985 trunc from 0x79a449 to 0x21019c (0x58a2ac bytes) 000483[0] 1788823027.156942 read 0x0dd857 thru 0x0e92cd (0xba77 bytes) 000487[1] 1788823027.254745 mapwrite 0x123e7e thru 0x130fff (0xd182 bytes) 000490[1] 1788823027.455568 read 0x15a97e thru 0x15f8e0 (0x4f63 bytes) 000491[0] 1788823027.557539 mapread 0x0e0490 thru 0x0e71de (0x6d4f bytes) 000492[1] 1788823027.639781 trunc from 0x21019d to 0x6971ce (0x487032 bytes) 000493[1] 1788823027.820485 write 0x0e25d5 thru 0x0ee0dd (0xbb09 bytes) 000496[0] 1788823028.015871 write 0x96bd97 thru 0x97bca3 (0xff0d bytes) 000497[1] 1788823028.116988 read 0x67571d thru 0x6849de (0xf2c2 bytes) 000499[0] 1788823028.244539 mapwrite 0x1aec46 thru 0x1b5c76 (0x7031 bytes) All operations completed A-OK! Chance of close/open is 1 in 50 Seed set to 3028 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported truncating to largest ever: 0x7b6fc 000004[0] 1788823029.360310 trunc from 00000000 to 0x07b6fb (0x7b6fc bytes) 000008[0] 1788823029.482020 mapread 0x03f000 thru 0x04b4bc (0xc4bd bytes) truncating to largest ever: 0x281077 000010[1] 1788823029.544418 trunc from 0x07b6fc to 0x281076 (0x20597b bytes) 000015[0] 1788823029.702996 write_OD 0x105000 thru 0x10afff (0x6000 bytes) 000016[1] 1788823029.756446 mapwrite 0x5e7000 thru 0x5eebaa (0x7bab bytes) truncating to largest ever: 0x74e3af 000018[0] 1788823029.878526 trunc from 0x5eebab to 0x74e3ae (0x15f804 bytes) 000019[1] 1788823029.935187 write 0x13b000 thru 0x13e02e (0x302f bytes) 000022[1] 1788823029.964239 mapread 0x2f1000 thru 0x2f3a11 (0x2a12 bytes) 000024[0] 1788823030.034939 trunc from 0x74e3af to 0x54013e (0x20e270 bytes) 000026[1] 1788823030.064935 read 0x42e000 thru 0x439c14 (0xbc15 bytes) 000030[0] 1788823030.091310 read_OD 0x0ee000 thru 0x0f1fff (0x4000 bytes) truncating to largest ever: 0x8e321b 000031[0] 1788823030.098495 trunc from 0x54013f to 0x8e321a (0x3a30dc bytes) 000034[1] 1788823030.176901 mapwrite 0x035000 thru 0x03f480 (0xa481 bytes) 000036[0] 1788823030.332862 trunc from 0x8e321b to 0x792bdc (0x15063e bytes) 000037[0] 1788823030.411582 read_OD 0x452000 thru 0x460fff (0xf000 bytes) 000038[0] 1788823030.453741 trunc from 0x792bdd to 0x78469b (0xe541 bytes) 000041[1] 1788823030.522182 trunc from 0x78469c to 0x315d55 (0x46e946 bytes) 000043[0] 1788823030.604700 mapread 0x252000 thru 0x256615 (0x4616 bytes) 000047[0] 1788823030.657165 read_OD 0x0a7000 thru 0x0b1fff (0xb000 bytes) 000048[0] 1788823030.672960 punch from 0x220fec to 0x225f44 (0x4f59 bytes) 000049[1] 1788823030.688246 punch from 0x046b4c to 0x04775f (0xc14 bytes) 000050[1] 1788823030.714860 mapwrite 0x621000 thru 0x62d17b (0xc17c bytes) 000051[1] 1788823030.887236 punch from 0x06e763 to 0x0769f9 (0x8297 bytes) truncating to largest ever: 0x8f5f8b 000052[1] 1788823030.944300 trunc from 0x62d17c to 0x8f5f8a (0x2c8e0f bytes) 000053[0] 1788823031.000825 punch from 0x2373a5 to 0x240af1 (0x974d bytes) 000054[0] 1788823031.031011 read_OD 0x18a000 thru 0x18bfff (0x2000 bytes) 000055[0] 1788823031.053299 trunc from 0x8f5f8b to 0x66bd4f (0x28a23b bytes) 000059[1] 1788823031.145889 write 0x323000 thru 0x32ab45 (0x7b46 bytes) 000063[0] 1788823031.219870 read_OD 0x2fa000 thru 0x306fff (0xd000 bytes) 000064[0] 1788823031.258110 punch from 0x0ac481 to 0x0bb9a2 (0xf522 bytes) 000068[0] 1788823031.325084 punch from 0x116592 to 0x12340d (0xce7c bytes) 000070[1] 1788823031.333853 punch from 0x293501 to 0x298eb5 (0x59b5 bytes) 000071[0] 1788823031.373468 mapread 0x104000 thru 0x1046cc (0x6cd bytes) 000072[0] 1788823031.419285 read_OD 0x364000 thru 0x364fff (0x1000 bytes) 000073[0] 1788823031.425788 read_OD 0x669000 thru 0x669fff (0x1000 bytes) 000075[1] 1788823031.430041 write 0x6be000 thru 0x6c8fe3 (0xafe4 bytes) 000076[0] 1788823031.456951 write_OD 0x1e2000 thru 0x1f0fff (0xf000 bytes) 000077[1] 1788823031.578659 mapwrite 0x74d000 thru 0x74e873 (0x1874 bytes) 000078[1] 1788823031.678126 read 0x685000 thru 0x690979 (0xb97a bytes) 000080[0] 1788823031.723358 mapwrite 0x863000 thru 0x86c718 (0x9719 bytes) 000081[0] 1788823031.846925 trunc from 0x86c719 to 0x4d4d44 (0x3979d4 bytes) 000082[0] 1788823031.859779 trunc from 0x4d4d45 to 0x32d600 (0x1a7744 bytes) 000086[0] 1788823031.989934 trunc from 0x32d601 to 0x3c173b (0x9413b bytes) 000087[1] 1788823032.011552 read 0x266000 thru 0x270d83 (0xad84 bytes) 000088[1] 1788823032.071748 mapread 0x00d000 thru 0x015f52 (0x8f53 bytes) 000089[1] 1788823032.076320 punch from 0x1010f6 to 0x10a6d5 (0x95e0 bytes) 000090[1] 1788823032.120620 mapread 0x3bf000 thru 0x3bf00e (0xf bytes) 000091[1] 1788823032.128495 mapread 0x378000 thru 0x382aed (0xaaee bytes) 000092[0] 1788823032.221992 read_OD 0x0d2000 thru 0x0dbfff (0xa000 bytes) 000097[1] 1788823032.263854 trunc from 0x3c173c to 0x7bbb1c (0x3fa3e1 bytes) 000098[1] 1788823032.302442 mapread 0x4dd000 thru 0x4e2cb5 (0x5cb6 bytes) 000099[0] 1788823032.390819 punch from 0x68a80d to 0x698e26 (0xe61a bytes) 000102[0] 1788823032.461760 read_OD 0x611000 thru 0x619fff (0x9000 bytes) truncating to largest ever: 0x94ed48 000103[1] 1788823032.478994 trunc from 0x7bbb1d to 0x94ed47 (0x19322b bytes) 000104[0] 1788823032.517887 trunc from 0x94ed48 to 0x7fc0e2 (0x152c65 bytes) 000106[0] 1788823032.557078 mapread 0x11a000 thru 0x122fb1 (0x8fb2 bytes) 000107[1] 1788823032.630744 punch from 0x6d268c to 0x6de515 (0xbe8a bytes) 000108[0] 1788823032.696994 trunc from 0x7fc0e3 to 0x15a455 (0x6a1c8d bytes) 000109[0] 1788823032.775815 read_OD 0x14f000 thru 0x154fff (0x6000 bytes) 000110[0] 1788823032.811809 mapwrite 0x7fc000 thru 0x7fcee0 (0xee1 bytes) 000111[0] 1788823032.964960 punch from 0x08d911 to 0x09d69a (0xfd8a bytes) 000113[0] 1788823032.978146 mapwrite 0x309000 thru 0x3165d9 (0xd5da bytes) 000115[1] 1788823033.138899 write 0x46a000 thru 0x46b21b (0x121c bytes) 000117[0] 1788823033.237996 mapwrite 0x68b000 thru 0x68bfca (0xfcb bytes) 000121[0] 1788823033.488663 punch from 0x50bb2c to 0x50c5a8 (0xa7d bytes) 000122[0] 1788823033.520760 trunc from 0x7fcee1 to 0x7d0485 (0x2ca5b bytes) 000124[0] 1788823033.633279 mapread 0x50c000 thru 0x512af5 (0x6af6 bytes) 000126[1] 1788823033.676376 mapwrite 0x2f3000 thru 0x2f4a76 (0x1a77 bytes) 000129[0] 1788823033.861772 mapwrite 0x477000 thru 0x47920a (0x220b bytes) 000130[0] 1788823033.986709 mapread 0x26b000 thru 0x2789af (0xd9b0 bytes) 000131[1] 1788823034.172759 read 0x3ec000 thru 0x3f8747 (0xc748 bytes) 000133[0] 1788823034.237953 mapwrite 0x555000 thru 0x55c0f1 (0x70f2 bytes) 000134[0] 1788823034.405856 write_OD 0x88a000 thru 0x893fff (0xa000 bytes) 000135[1] 1788823034.476391 punch from 0x815d58 to 0x820c7f (0xaf28 bytes) 000136[1] 1788823034.524578 punch from 0x202860 to 0x20ed07 (0xc4a8 bytes) 000137[1] 1788823034.547987 trunc from 0x894000 to 0x4f27aa (0x3a1855 bytes) 000140[1] 1788823034.612756 write 0x1d6000 thru 0x1daffa (0x4ffb bytes) 000141[0] 1788823034.648817 mapread 0x024000 thru 0x031706 (0xd707 bytes) 000142[1] 1788823034.854543 read 0x2ea000 thru 0x2efe10 (0x5e11 bytes) 000144[1] 1788823034.865665 mapwrite 0x474000 thru 0x47abcd (0x6bce bytes) 000145[1] 1788823034.991372 trunc from 0x4f27ab to 0x69e8e9 (0x1ac13f bytes) 000148[1] 1788823035.071613 mapread 0x317000 thru 0x31dc06 (0x6c07 bytes) 000149[1] 1788823035.109914 read 0x508000 thru 0x50dba4 (0x5ba5 bytes) 000150[1] 1788823035.151587 read 0x03c000 thru 0x049825 (0xd826 bytes) 000152[0] 1788823035.199420 read_OD 0x203000 thru 0x204fff (0x2000 bytes) 000153[0] 1788823035.230771 punch from 0x15d5bd to 0x1670ef (0x9b33 bytes) 000155[0] 1788823035.297145 mapread 0x563000 thru 0x571fd3 (0xefd4 bytes) 000156[0] 1788823035.347342 read_OD 0x669000 thru 0x676fff (0xe000 bytes) 000158[0] 1788823035.374027 mapwrite 0x26b000 thru 0x27a9d9 (0xf9da bytes) 000159[1] 1788823035.474929 write 0x4ef000 thru 0x4f6e5a (0x7e5b bytes) 000160[1] 1788823035.515916 mapwrite 0x56f000 thru 0x577129 (0x812a bytes) 000161[0] 1788823035.688682 trunc from 0x69e8ea to 0x6a0984 (0x209b bytes) 000162[0] 1788823035.774605 mapwrite 0x33c000 thru 0x340002 (0x4003 bytes) 000163[0] 1788823035.891917 trunc from 0x6a0985 to 0x0cdd54 (0x5d2c30 bytes) 000164[1] 1788823035.951355 punch from 0x0a2426 to 0x0af00e (0xcbe9 bytes) 000165[1] 1788823036.001975 mapwrite 0x31e000 thru 0x32d434 (0xf435 bytes) 000166[0] 1788823036.167125 trunc from 0x32d435 to 0x78b644 (0x45e210 bytes) 000169[1] 1788823036.266558 write 0x503000 thru 0x50a804 (0x7805 bytes) 000170[1] 1788823036.306059 write 0x686000 thru 0x6895d4 (0x35d5 bytes) 000171[0] 1788823036.344302 mapread 0x0bc000 thru 0x0c2f56 (0x6f57 bytes) 000173[0] 1788823036.491522 read_OD 0x246000 thru 0x248fff (0x3000 bytes) 000176[0] 1788823036.506283 mapread 0x15e000 thru 0x16ddb7 (0xfdb8 bytes) 000178[0] 1788823036.525994 mapread 0x3dd000 thru 0x3eacd6 (0xdcd7 bytes) 000180[1] 1788823036.567331 read 0x532000 thru 0x53d61f (0xb620 bytes) 000182[0] 1788823036.590238 write_OD 0x5fb000 thru 0x607fff (0xd000 bytes) 000183[1] 1788823036.736673 read 0x0c4000 thru 0x0c9681 (0x5682 bytes) 000184[1] 1788823036.770173 mapwrite 0x124000 thru 0x12aeee (0x6eef bytes) 000185[0] 1788823036.916644 punch from 0x5c19d2 to 0x5cf9e9 (0xe018 bytes) 000187[0] 1788823036.963659 trunc from 0x78b645 to 0x3d6c98 (0x3b49ac bytes) 000190[0] 1788823036.992020 mapwrite 0x92f000 thru 0x935663 (0x6664 bytes) 000191[1] 1788823037.087041 mapwrite 0x1d5000 thru 0x1e3c49 (0xec4a bytes) 000192[1] 1788823037.207548 mapread 0x314000 thru 0x3217bc (0xd7bd bytes) 000193[1] 1788823037.256684 punch from 0x797f1c to 0x79e501 (0x65e6 bytes) 000194[0] 1788823037.262714 trunc from 0x935664 to 0x7ddc8a (0x1579d9 bytes) 000198[1] 1788823037.338536 read 0x5bf000 thru 0x5c1510 (0x2511 bytes) 000199[1] 1788823037.367598 mapread 0x286000 thru 0x2872cf (0x12d0 bytes) 000206[0] 1788823037.384900 mapread 0x69c000 thru 0x69c8ec (0x8ed bytes) 000208[0] 1788823037.393287 mapread 0x090000 thru 0x09ef1d (0xef1e bytes) 000209[0] 1788823037.434794 mapwrite 0x251000 thru 0x25c2fb (0xb2fc bytes) 000210[0] 1788823037.567897 mapread 0x685000 thru 0x68e3b5 (0x93b6 bytes) 000211[1] 1788823037.609197 trunc from 0x7ddc8b to 0x0d2557 (0x70b733 bytes) 000212[0] 1788823037.653991 write_OD 0x50d000 thru 0x512fff (0x6000 bytes) 000213[0] 1788823037.745916 write_OD 0x9b0000 thru 0x9bbfff (0xc000 bytes) 000214[1] 1788823037.820312 trunc from 0x9bc000 to 0x373ab4 (0x64854b bytes) 000215[1] 1788823037.913182 read 0x063000 thru 0x06d63a (0xa63b bytes) 000216[0] 1788823038.065438 trunc from 0x373ab5 to 0x6cba03 (0x357f4f bytes) 000219[1] 1788823038.144562 mapread 0x29b000 thru 0x29db2d (0x2b2e bytes) 000220[1] 1788823038.237579 mapwrite 0x462000 thru 0x46fd20 (0xdd21 bytes) 000222[1] 1788823038.477852 mapwrite 0x7ee000 thru 0x7ef0e2 (0x10e3 bytes) 000225[0] 1788823038.664146 punch from 0x61f9de to 0x62b69f (0xbcc2 bytes) 000226[1] 1788823038.713126 read 0x435000 thru 0x435892 (0x893 bytes) 000229[1] 1788823038.765018 punch from 0x677954 to 0x6832ba (0xb967 bytes) 000230[0] 1788823038.786703 write_OD 0x9d4000 thru 0x9defff (0xb000 bytes) 000231[0] 1788823038.924654 punch from 0x78bc92 to 0x79359c (0x790b bytes) 000232[0] 1788823038.972888 mapwrite 0x9c7000 thru 0x9d3e0e (0xce0f bytes) 000233[1] 1788823039.127075 read 0x79d000 thru 0x79e585 (0x1586 bytes) 000234[0] 1788823039.206941 read_OD 0x21a000 thru 0x21afff (0x1000 bytes) 000235[1] 1788823039.217002 write 0x8e6000 thru 0x8f0c17 (0xac18 bytes) 000236[1] 1788823039.280466 mapwrite 0x2ce000 thru 0x2ce562 (0x563 bytes) 000237[1] 1788823039.425568 trunc from 0x9df000 to 0x12bb39 (0x8b34c6 bytes) 000238[1] 1788823039.461941 punch from 0x074616 to 0x07b42e (0x6e19 bytes) 000239[1] 1788823039.494065 punch from 0x01dac9 to 0x02abf4 (0xd12c bytes) 000241[0] 1788823039.534969 write_OD 0x878000 thru 0x886fff (0xf000 bytes) 000242[1] 1788823039.623687 read 0x44a000 thru 0x4548c7 (0xa8c8 bytes) 000243[0] 1788823039.788588 trunc from 0x887000 to 0x568889 (0x31e776 bytes) 000245[0] 1788823039.872748 mapread 0x44d000 thru 0x45bdac (0xedad bytes) 000246[1] 1788823039.925983 mapwrite 0x8d8000 thru 0x8e4e3a (0xce3b bytes) 000251[0] 1788823040.173582 write_OD 0x85a000 thru 0x861fff (0x8000 bytes) 000252[0] 1788823040.259150 trunc from 0x8e4e3b to 0x0fdf46 (0x7e6ef4 bytes) 000253[1] 1788823040.309023 punch from 0x05a012 to 0x05a49e (0x48d bytes) 000254[0] 1788823040.366342 read_OD 0x065000 thru 0x066fff (0x2000 bytes) 000259[1] 1788823040.424396 mapread 0x034000 thru 0x036050 (0x2051 bytes) 000261[1] 1788823040.441777 punch from 0x0c2e2b to 0x0cd06b (0xa241 bytes) 000263[0] 1788823040.502665 mapwrite 0x319000 thru 0x31ce64 (0x3e65 bytes) 000264[0] 1788823040.751653 trunc from 0x31ce65 to 0x318298 (0x4bcc bytes) 000266[1] 1788823040.800701 mapwrite 0x490000 thru 0x496710 (0x6711 bytes) 000267[1] 1788823040.952078 trunc from 0x496711 to 0x68a04c (0x1f393c bytes) 000272[1] 1788823041.026950 write 0x4be000 thru 0x4c247c (0x447d bytes) 000276[0] 1788823041.096000 punch from 0x605e77 to 0x60a8db (0x4a65 bytes) 000280[0] 1788823041.294817 trunc from 0x68a04d to 0x1ab8b6 (0x4de796 bytes) 000281[0] 1788823041.328025 mapwrite 0x27e000 thru 0x28be39 (0xde3a bytes) 000282[0] 1788823041.807942 punch from 0x19dae9 to 0x1a5870 (0x7d88 bytes) 000283[1] 1788823041.821560 trunc from 0x28be3a to 0x5962ed (0x30a4b4 bytes) truncating to largest ever: 0x9f98c3 000284[1] 1788823041.926638 trunc from 0x5962ee to 0x9f98c2 (0x4635d5 bytes) 000288[0] 1788823042.081569 read_OD 0x07a000 thru 0x07dfff (0x4000 bytes) 000289[1] 1788823042.152429 punch from 0x04e274 to 0x056a80 (0x880d bytes) 000290[1] 1788823042.205047 read 0x75f000 thru 0x767c4e (0x8c4f bytes) 000291[1] 1788823042.260914 read 0x887000 thru 0x8950e2 (0xe0e3 bytes) 000294[1] 1788823042.384737 punch from 0x048486 to 0x051f8f (0x9b0a bytes) 000296[0] 1788823042.401830 write_OD 0x239000 thru 0x23afff (0x2000 bytes) 000297[1] 1788823042.492005 mapread 0x8c7000 thru 0x8cbae4 (0x4ae5 bytes) 000299[1] 1788823042.547970 mapwrite 0x2e5000 thru 0x2e7614 (0x2615 bytes) 000300[1] 1788823042.671633 mapwrite 0x76d000 thru 0x771e77 (0x4e78 bytes) 000302[1] 1788823042.803227 mapread 0x9b7000 thru 0x9c12c4 (0xa2c5 bytes) 000304[1] 1788823042.857025 trunc from 0x9f98c3 to 0x8e0e0c (0x118ab6 bytes) 000307[1] 1788823042.922754 mapread 0x468000 thru 0x46d622 (0x5623 bytes) 000308[1] 1788823042.958087 trunc from 0x8e0e0d to 0x28c44d (0x6549bf bytes) 000309[1] 1788823043.003339 trunc from 0x28c44e to 0x59b27d (0x30ee30 bytes) 000311[1] 1788823043.050650 mapread 0x064000 thru 0x07396a (0xf96b bytes) 000312[0] 1788823043.111847 read_OD 0x1c7000 thru 0x1d4fff (0xe000 bytes) 000314[0] 1788823043.181715 trunc from 0x59b27e to 0x9dba6f (0x4407f2 bytes) 000315[1] 1788823043.292411 mapread 0x572000 thru 0x57a1d1 (0x81d2 bytes) 000318[1] 1788823043.366069 trunc from 0x9dba70 to 0x00d985 (0x9ce0ea bytes) 000320[1] 1788823043.400337 write 0x1a7000 thru 0x1a7a6b (0xa6c bytes) 000325[0] 1788823043.492145 mapread 0x047000 thru 0x04f535 (0x8536 bytes) 000328[1] 1788823043.551102 read 0x0f2000 thru 0x0fe065 (0xc066 bytes) 000330[1] 1788823043.583866 write 0x92e000 thru 0x937f8e (0x9f8f bytes) 000333[0] 1788823043.632554 punch from 0x535578 to 0x54310f (0xdb98 bytes) 000334[0] 1788823043.680585 read_OD 0x378000 thru 0x381fff (0xa000 bytes) 000335[1] 1788823043.716888 read 0x631000 thru 0x635b58 (0x4b59 bytes) 000336[0] 1788823043.731706 mapwrite 0x95b000 thru 0x96157b (0x657c bytes) 000337[1] 1788823043.956920 mapread 0x562000 thru 0x56f4f0 (0xd4f1 bytes) 000338[0] 1788823044.089546 mapwrite 0x1d2000 thru 0x1d2ebc (0xebd bytes) 000339[1] 1788823044.204521 mapwrite 0x521000 thru 0x52d333 (0xc334 bytes) 000340[0] 1788823044.418165 punch from 0x54de84 to 0x55b4d8 (0xd655 bytes) 000341[1] 1788823044.448082 mapread 0x8d3000 thru 0x8d4149 (0x114a bytes) 000344[0] 1788823044.519151 mapwrite 0x229000 thru 0x2384f7 (0xf4f8 bytes) 000346[0] 1788823044.762932 mapread 0x5c2000 thru 0x5c23c3 (0x3c4 bytes) 000348[1] 1788823044.772803 trunc from 0x96157c to 0x5f136c (0x37020f bytes) 000350[1] 1788823044.898837 mapread 0x1a6000 thru 0x1addb9 (0x7dba bytes) 000351[0] 1788823044.928781 trunc from 0x5f136d to 0x8e2155 (0x2f0de9 bytes) 000352[1] 1788823044.973701 read 0x0e2000 thru 0x0e251a (0x51b bytes) 000353[0] 1788823044.994402 trunc from 0x8e2156 to 0x6d3ae2 (0x20e673 bytes) 000354[1] 1788823045.038737 write 0x792000 thru 0x796bfa (0x4bfb bytes) 000356[1] 1788823045.070861 trunc from 0x796bfb to 0x216b2c (0x5800ce bytes) 000358[0] 1788823045.158941 mapread 0x0a4000 thru 0x0b0639 (0xc63a bytes) 000359[0] 1788823045.287990 mapwrite 0x74f000 thru 0x755bf8 (0x6bf9 bytes) 000361[0] 1788823045.512485 write_OD 0x374000 thru 0x37cfff (0x9000 bytes) 000362[0] 1788823045.624883 read_OD 0x17e000 thru 0x185fff (0x8000 bytes) 000363[0] 1788823045.648455 write_OD 0x817000 thru 0x825fff (0xf000 bytes) 000365[0] 1788823045.849526 punch from 0x3253bc to 0x326f2d (0x1b72 bytes) 000366[0] 1788823045.905518 punch from 0x423158 to 0x42d5a5 (0xa44e bytes) 000367[1] 1788823045.912026 write 0x604000 thru 0x605b65 (0x1b66 bytes) 000370[1] 1788823045.986742 punch from 0x0559e1 to 0x061a34 (0xc054 bytes) 000373[1] 1788823046.069043 punch from 0x39daf8 to 0x3a8c01 (0xb10a bytes) 000374[0] 1788823046.082373 read_OD 0x239000 thru 0x239fff (0x1000 bytes) 000375[1] 1788823046.181116 trunc from 0x826000 to 0x4f60cd (0x32ff32 bytes) 000378[1] 1788823046.426506 write 0x687000 thru 0x68e74b (0x774c bytes) 000379[1] 1788823046.436654 trunc from 0x68e74c to 0x8c270b (0x233fc0 bytes) 000381[0] 1788823046.557523 write_OD 0x57c000 thru 0x57dfff (0x2000 bytes) 000382[1] 1788823046.710663 mapread 0x496000 thru 0x49976e (0x376f bytes) 000384[0] 1788823046.783581 mapwrite 0x16f000 thru 0x17c98a (0xd98b bytes) 000385[1] 1788823047.060289 write 0x652000 thru 0x6591e6 (0x71e7 bytes) 000386[1] 1788823047.106890 mapread 0x363000 thru 0x36dfd3 (0xafd4 bytes) 000387[0] 1788823047.157775 write_OD 0x488000 thru 0x492fff (0xb000 bytes) 000389[1] 1788823047.282521 write 0x884000 thru 0x88a39c (0x639d bytes) 000390[1] 1788823047.342666 mapwrite 0x698000 thru 0x69d01b (0x501c bytes) 000392[0] 1788823047.553875 mapread 0x6fc000 thru 0x707568 (0xb569 bytes) 000394[1] 1788823047.663567 read 0x701000 thru 0x70c2a5 (0xb2a6 bytes) 000395[0] 1788823047.700932 mapwrite 0x57e000 thru 0x587315 (0x9316 bytes) 000397[0] 1788823047.943091 trunc from 0x8c270c to 0x603b67 (0x2beba4 bytes) 000398[0] 1788823047.995835 write_OD 0x61c000 thru 0x625fff (0xa000 bytes) 000400[0] 1788823048.175805 write_OD 0x8e9000 thru 0x8f5fff (0xd000 bytes) 000401[0] 1788823048.307555 mapwrite 0x68c000 thru 0x694fb6 (0x8fb7 bytes) 000403[1] 1788823048.552729 punch from 0x6e7ca1 to 0x6e9edb (0x223b bytes) 000407[0] 1788823048.679994 write_OD 0x816000 thru 0x81dfff (0x8000 bytes) 000409[0] 1788823048.741773 read_OD 0x368000 thru 0x36efff (0x7000 bytes) 000410[1] 1788823048.772433 read 0x4f1000 thru 0x4ff9e8 (0xe9e9 bytes) 000412[1] 1788823048.783281 write 0x3df000 thru 0x3e8562 (0x9563 bytes) 000413[1] 1788823048.857218 mapwrite 0x6de000 thru 0x6eca22 (0xea23 bytes) 000414[1] 1788823048.980891 write 0x6ea000 thru 0x6f4ba7 (0xaba8 bytes) 000415[1] 1788823049.014878 write 0x19a000 thru 0x19dd61 (0x3d62 bytes) 000416[0] 1788823049.065614 write_OD 0x411000 thru 0x41bfff (0xb000 bytes) 000417[0] 1788823049.120843 mapread 0x3ab000 thru 0x3ac738 (0x1739 bytes) 000418[0] 1788823049.131809 mapread 0x2aa000 thru 0x2aa5a5 (0x5a6 bytes) 000420[1] 1788823049.137105 read 0x5eb000 thru 0x5f6294 (0xb295 bytes) 000422[1] 1788823049.146446 write 0x9df000 thru 0x9e518a (0x618b bytes) 000425[0] 1788823049.197839 punch from 0x9163ee to 0x91d24d (0x6e60 bytes) 000426[0] 1788823049.228666 write_OD 0x439000 thru 0x441fff (0x9000 bytes) 000428[0] 1788823049.266163 mapread 0x889000 thru 0x88ee39 (0x5e3a bytes) 000429[0] 1788823049.315839 write_OD 0x466000 thru 0x471fff (0xc000 bytes) 000430[1] 1788823049.403768 punch from 0x9b9f3b to 0x9bdd55 (0x3e1b bytes) 000431[1] 1788823049.432500 write 0x966000 thru 0x97250a (0xc50b bytes) 000433[1] 1788823049.462647 mapwrite 0x19f000 thru 0x1a1395 (0x2396 bytes) 000434[0] 1788823049.573736 punch from 0x50ebed to 0x516fb2 (0x83c6 bytes) 000435[1] 1788823049.603816 read 0x42b000 thru 0x430366 (0x5367 bytes) 000437[1] 1788823049.645239 trunc from 0x9e518b to 0x78ba7b (0x25970f bytes) 000438[0] 1788823049.713899 mapread 0x6f3000 thru 0x6fc8fe (0x98ff bytes) 000439[0] 1788823049.786874 mapread 0x5ff000 thru 0x5ffd6d (0xd6e bytes) 000440[1] 1788823049.798962 read 0x609000 thru 0x614437 (0xb438 bytes) 000441[0] 1788823049.810447 mapread 0x245000 thru 0x249bcb (0x4bcc bytes) 000443[1] 1788823049.852897 read 0x0f5000 thru 0x0fef6c (0x9f6d bytes) 000447[1] 1788823049.869701 mapwrite 0x4b0000 thru 0x4b3c2c (0x3c2d bytes) 000448[1] 1788823049.998345 mapread 0x080000 thru 0x0803fe (0x3ff bytes) 000449[1] 1788823050.004920 punch from 0x4d70ca to 0x4d8f01 (0x1e38 bytes) 000450[0] 1788823050.023505 trunc from 0x78ba7c to 0x7e23e4 (0x56969 bytes) 000453[0] 1788823050.121542 trunc from 0x7e23e5 to 0x3f722b (0x3eb1b9 bytes) 000454[1] 1788823050.143800 read 0x3c6000 thru 0x3d4c4f (0xec50 bytes) 000455[1] 1788823050.166726 mapwrite 0x024000 thru 0x03004f (0xc050 bytes) 000456[1] 1788823050.373788 read 0x129000 thru 0x13309e (0xa09f bytes) 000457[1] 1788823050.494500 write 0x0ca000 thru 0x0d9bd8 (0xfbd9 bytes) 000458[0] 1788823050.495576 punch from 0x0f7908 to 0x0fe625 (0x6d1e bytes) 000460[1] 1788823050.622753 read 0x36c000 thru 0x379c94 (0xdc95 bytes) 000461[1] 1788823050.792052 write 0x77f000 thru 0x78ad65 (0xbd66 bytes) 000463[0] 1788823050.822438 trunc from 0x78ad66 to 0x02c8b1 (0x75e4b4 bytes) 000464[0] 1788823050.868389 mapwrite 0x3b0000 thru 0x3beb79 (0xeb7a bytes) 000466[0] 1788823051.132503 punch from 0x01542e to 0x01b051 (0x5c24 bytes) 000467[0] 1788823051.159944 mapwrite 0x7b1000 thru 0x7b1c65 (0xc66 bytes) 000469[0] 1788823051.510620 write_OD 0x34d000 thru 0x351fff (0x5000 bytes) 000470[1] 1788823051.620436 mapread 0x310000 thru 0x315733 (0x5734 bytes) 000471[1] 1788823051.706592 punch from 0x3a5b56 to 0x3b1f1d (0xc3c8 bytes) 000472[0] 1788823051.740218 trunc from 0x7b1c66 to 0x3fca8c (0x3b51d9 bytes) 000473[0] 1788823051.828159 mapwrite 0x308000 thru 0x30ee38 (0x6e39 bytes) 000474[0] 1788823052.063982 mapread 0x2f9000 thru 0x302870 (0x9871 bytes) 000475[1] 1788823052.135847 punch from 0x2dbbb4 to 0x2e3ad4 (0x7f21 bytes) 000476[1] 1788823052.158001 read 0x11a000 thru 0x12369c (0x969d bytes) 000479[0] 1788823052.345940 read_OD 0x397000 thru 0x3a0fff (0xa000 bytes) 000481[1] 1788823052.420238 write 0x777000 thru 0x782714 (0xb715 bytes) 000484[1] 1788823052.465904 punch from 0x6a9cc4 to 0x6b1856 (0x7b93 bytes) 000488[0] 1788823052.535500 mapread 0x1bb000 thru 0x1c60f9 (0xb0fa bytes) 000489[1] 1788823052.688714 read 0x362000 thru 0x366b74 (0x4b75 bytes) 000490[0] 1788823052.714899 write_OD 0x110000 thru 0x11bfff (0xc000 bytes) 000491[1] 1788823052.914220 write 0x17d000 thru 0x186f0c (0x9f0d bytes) 000492[0] 1788823052.939104 punch from 0x385aac to 0x388ca9 (0x31fe bytes) 000493[0] 1788823052.997530 read_OD 0x706000 thru 0x706fff (0x1000 bytes) 000495[0] 1788823053.041879 punch from 0x7468e3 to 0x7568a1 (0xffbf bytes) 000496[0] 1788823053.092405 mapread 0x734000 thru 0x73ab0a (0x6b0b bytes) 000499[1] 1788823053.199904 trunc from 0x782715 to 0x9c6d0e (0x2445fa bytes) All operations completed A-OK! PASS 16a (68s) == sanityn test 16b: 500 iterations of dual-mount fsx at small size ========================================================== 19:17:41 (1788823061) keep default fallocate mode: 2 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.774735 s, 10.8 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.301687 s, 27.8 MB/s Chance of close/open is 1 in 50 Seed set to 3067 fd 0: /mnt/lustre/f16b.sanityn fd 1: /mnt/lustre2/f16b.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000001[0] 1788823068.152390 write 0x25eea9 thru 0x25f89d (0x9f5 bytes) 000002[0] 1788823068.196862 write 0x95e1d4 thru 0x95f3e9 (0x1216 bytes) 000006[1] 1788823068.277698 mapread 0x44eb68 thru 0x44f2da (0x773 bytes) 000007[1] 1788823068.327227 mapread 0x187540 thru 0x187c27 (0x6e8 bytes) 000008[0] 1788823068.371657 mapread 0x56ccbb thru 0x56d4c4 (0x80a bytes) 000009[0] 1788823068.403460 read 0x0c85ac thru 0x0c9a10 (0x1465 bytes) 000011[0] 1788823068.413030 write 0x754af7 thru 0x7565ce (0x1ad8 bytes) truncating to largest ever: 0x24a1ed 000012[0] 1788823068.443807 trunc from 0x95f3ea to 0x24a1ec (0x7151fd bytes) 000013[0] 1788823068.512860 trunc from 0x24a1ed to 0x04a1d9 (0x200013 bytes) 000016[1] 1788823068.625937 write 0x136d2f thru 0x1373d8 (0x6aa bytes) 000017[1] 1788823068.660584 read 0x06609c thru 0x06640e (0x373 bytes) 000018[0] 1788823068.682877 read 0x095a40 thru 0x0962a4 (0x865 bytes) 000021[0] 1788823068.811109 punch from 0x0288cf to 0x029a53 (0x1185 bytes) 000022[1] 1788823068.834505 write 0x02d9b5 thru 0x02eae0 (0x112c bytes) 000023[0] 1788823068.868862 mapread 0x04d2e2 thru 0x04dd5e (0xa7d bytes) 000024[1] 1788823068.901939 mapread 0x0247e8 thru 0x0247f2 (0xb bytes) 000025[0] 1788823068.912539 write 0x5fb3ee thru 0x5fb433 (0x46 bytes) 000026[1] 1788823068.914618 punch from 0x05317f to 0x053372 (0x1f4 bytes) 000027[0] 1788823068.939312 punch from 0x58cf44 to 0x58dd44 (0xe01 bytes) 000029[1] 1788823068.971543 read 0x4162a9 thru 0x41792a (0x1682 bytes) truncating to largest ever: 0x4b5a4c 000033[1] 1788823069.019962 trunc from 0x5fb434 to 0x4b5a4b (0x1459e8 bytes) 000034[1] 1788823069.131825 write 0x8cb3f5 thru 0x8cc846 (0x1452 bytes) 000037[0] 1788823069.206525 mapread 0x47b00c thru 0x47c77b (0x1770 bytes) 000038[0] 1788823069.286198 trunc from 0x8cc847 to 0x483605 (0x449241 bytes) 000039[0] 1788823069.470314 read 0x045dc2 thru 0x047d3b (0x1f7a bytes) 000041[0] 1788823069.526954 write 0x12a4aa thru 0x12a783 (0x2da bytes) 000043[0] 1788823069.542061 mapread 0x3a97c8 thru 0x3aa93c (0x1175 bytes) 000046[0] 1788823069.643843 mapwrite 0x2f2c08 thru 0x2f4057 (0x1450 bytes) 000047[1] 1788823069.731454 write 0x28eb98 thru 0x290994 (0x1dfd bytes) 000051[0] 1788823069.851752 read 0x0d0bd3 thru 0x0d10ac (0x4da bytes) 000052[1] 1788823069.908734 write 0x697a9b thru 0x6996d0 (0x1c36 bytes) 000053[1] 1788823069.947333 mapread 0x403b32 thru 0x405950 (0x1e1f bytes) 000055[0] 1788823070.043877 write 0x212a6f thru 0x214018 (0x15aa bytes) 000060[0] 1788823070.345424 mapwrite 0x6ccdf6 thru 0x6ce2c8 (0x14d3 bytes) 000061[1] 1788823070.600221 mapread 0x2263a3 thru 0x22726d (0xecb bytes) 000063[1] 1788823070.731700 mapwrite 0x5efe76 thru 0x5f0d0e (0xe99 bytes) 000065[1] 1788823070.948246 mapwrite 0x8e5f5b thru 0x8e6838 (0x8de bytes) 000066[1] 1788823071.122438 read 0x34c1f9 thru 0x34d809 (0x1611 bytes) 000068[1] 1788823071.163409 mapread 0x866a3f thru 0x866c0b (0x1cd bytes) 000070[1] 1788823071.201938 write 0x979e86 thru 0x97ad58 (0xed3 bytes) 000074[0] 1788823071.281398 punch from 0x1967a7 to 0x197c0e (0x1468 bytes) 000075[0] 1788823071.328207 punch from 0x1112e4 to 0x1117fc (0x519 bytes) 000076[0] 1788823071.371931 trunc from 0x97ad59 to 0x062e8a (0x917ece bytes) 000079[0] 1788823071.486049 mapwrite 0x64973e thru 0x64b0b9 (0x197c bytes) 000080[0] 1788823071.708464 write 0x39da20 thru 0x39de7c (0x45d bytes) 000081[0] 1788823071.778151 punch from 0x5f77a6 to 0x5f8c09 (0x1464 bytes) 000082[1] 1788823071.826852 write 0x8d1d20 thru 0x8d2bc0 (0xea1 bytes) 000083[1] 1788823071.946503 trunc from 0x8d2bc1 to 0x02ca08 (0x8a61b8 bytes) 000084[1] 1788823072.053866 mapread 0x008a7c thru 0x008b32 (0xb7 bytes) 000085[0] 1788823072.084629 mapwrite 0x4e5baf thru 0x4e787a (0x1ccc bytes) 000086[1] 1788823072.297618 mapread 0x2f105d thru 0x2f2b05 (0x1aa9 bytes) truncating to largest ever: 0x755da6 000087[0] 1788823072.345614 trunc from 0x4e787b to 0x755da5 (0x26e52b bytes) truncating to largest ever: 0x7d1208 000089[1] 1788823072.445672 trunc from 0x755da6 to 0x7d1207 (0x7b462 bytes) 000090[0] 1788823072.494518 read 0x2649c5 thru 0x266709 (0x1d45 bytes) 000091[1] 1788823072.548369 mapread 0x4b3f42 thru 0x4b480f (0x8ce bytes) 000092[1] 1788823072.589982 trunc from 0x7d1208 to 0x5250c6 (0x2ac141 bytes) 000093[1] 1788823072.643673 write 0x326509 thru 0x3265ed (0xe5 bytes) 000094[0] 1788823072.691132 punch from 0x302ef2 to 0x30305d (0x16c bytes) 000096[0] 1788823072.753535 punch from 0x1db03e to 0x1db57a (0x53d bytes) 000097[1] 1788823072.776395 mapread 0x20b542 thru 0x20b884 (0x343 bytes) 000098[1] 1788823072.863755 punch from 0x27a3d2 to 0x27ac97 (0x8c6 bytes) 000099[0] 1788823072.953534 trunc from 0x5250c7 to 0x5f01f1 (0xcb12b bytes) 000101[1] 1788823073.078912 punch from 0x0394dc to 0x03ab31 (0x1656 bytes) 000103[1] 1788823073.152813 write 0x1f8ab5 thru 0x1f9994 (0xee0 bytes) 000106[1] 1788823073.256539 mapread 0x5288cf thru 0x528e31 (0x563 bytes) 000107[0] 1788823073.312631 trunc from 0x5f01f2 to 0x044ab4 (0x5ab73d bytes) 000112[1] 1788823073.471559 punch from 0x00d167 to 0x00e17b (0x1015 bytes) 000114[1] 1788823073.571818 read 0x0262c3 thru 0x027f5d (0x1c9b bytes) 000117[0] 1788823073.652312 mapwrite 0x45e3f0 thru 0x45e56c (0x17d bytes) 000119[1] 1788823073.848358 trunc from 0x45e56d to 0x3143cd (0x14a19f bytes) 000121[0] 1788823074.060056 mapread 0x2bdf3c thru 0x2beccd (0xd92 bytes) 000122[0] 1788823074.156165 read 0x038c33 thru 0x0391c5 (0x593 bytes) 000125[1] 1788823074.328818 trunc from 0x3143ce to 0x0fc2c6 (0x218107 bytes) 000126[0] 1788823074.368319 read 0x00950b thru 0x0097a5 (0x29b bytes) 000128[1] 1788823074.490043 read 0x041ad6 thru 0x042e1f (0x134a bytes) 000132[0] 1788823074.568962 write 0x914d0a thru 0x916286 (0x157d bytes) 000134[0] 1788823074.662176 mapread 0x56ff5c thru 0x57152f (0x15d4 bytes) 000135[1] 1788823074.841944 mapread 0x1d751f thru 0x1d7626 (0x108 bytes) 000136[1] 1788823074.973747 mapwrite 0x000973 thru 0x001dfd (0x148b bytes) 000137[0] 1788823075.202606 read 0x8fb205 thru 0x8fcd51 (0x1b4d bytes) 000138[1] 1788823075.221205 mapread 0x31c9a3 thru 0x31e08a (0x16e8 bytes) 000139[1] 1788823075.246072 mapwrite 0x2071fe thru 0x207a54 (0x857 bytes) 000140[1] 1788823075.352957 read 0x0e6226 thru 0x0e8178 (0x1f53 bytes) 000141[0] 1788823075.360619 write 0x2da5cf thru 0x2dc4f4 (0x1f26 bytes) 000143[0] 1788823075.400238 punch from 0x5276ea to 0x529344 (0x1c5b bytes) 000145[1] 1788823075.434202 mapread 0x84213b thru 0x842a77 (0x93d bytes) 000147[1] 1788823075.554670 mapread 0x37cd68 thru 0x37ce1c (0xb5 bytes) 000148[0] 1788823075.586975 write 0x485281 thru 0x4870ae (0x1e2e bytes) 000150[0] 1788823075.608346 punch from 0x49c036 to 0x49d116 (0x10e1 bytes) 000152[0] 1788823075.635764 punch from 0x297d93 to 0x298fb5 (0x1223 bytes) 000153[0] 1788823075.654948 trunc from 0x916287 to 0x537ebf (0x3de3c7 bytes) 000154[0] 1788823075.699807 mapread 0x0985e4 thru 0x09a15a (0x1b77 bytes) 000155[1] 1788823075.763317 write 0x3af455 thru 0x3b040f (0xfbb bytes) 000156[0] 1788823075.827106 read 0x04c0f7 thru 0x04da91 (0x199b bytes) 000157[0] 1788823075.869043 mapread 0x3d5e70 thru 0x3d660a (0x79b bytes) 000159[1] 1788823075.909689 write 0x83adef thru 0x83c70a (0x191c bytes) 000160[0] 1788823075.963433 trunc from 0x83c70b to 0x627dea (0x214920 bytes) 000161[0] 1788823076.028326 trunc from 0x627deb to 0x60da77 (0x1a373 bytes) 000162[0] 1788823076.091844 read 0x21a485 thru 0x21a4fd (0x79 bytes) 000164[0] 1788823076.128119 write 0x2052e7 thru 0x2069b2 (0x16cc bytes) 000165[0] 1788823076.143060 punch from 0x25053f to 0x251d11 (0x17d3 bytes) 000167[1] 1788823076.221100 write 0x1de0cf thru 0x1dee86 (0xdb8 bytes) 000168[1] 1788823076.268992 read 0x08ba82 thru 0x08c04f (0x5ce bytes) 000170[0] 1788823076.356814 mapread 0x2adf32 thru 0x2adfdb (0xaa bytes) 000172[0] 1788823076.480934 trunc from 0x60da78 to 0x79d2b3 (0x18f83c bytes) 000173[1] 1788823076.671686 punch from 0x2cffd0 to 0x2d082b (0x85c bytes) 000175[1] 1788823076.819625 trunc from 0x79d2b4 to 0x4add68 (0x2ef54b bytes) 000176[0] 1788823076.878099 read 0x49dd71 thru 0x49f48e (0x171e bytes) 000178[0] 1788823076.913608 mapread 0x49e40e thru 0x49fac4 (0x16b7 bytes) truncating to largest ever: 0x9cef46 000180[1] 1788823076.961061 trunc from 0x4add69 to 0x9cef45 (0x5211dd bytes) 000181[1] 1788823077.008995 write 0x93b120 thru 0x93bf5d (0xe3e bytes) 000182[0] 1788823077.022810 punch from 0x0ae67d to 0x0ae786 (0x10a bytes) 000183[0] 1788823077.094180 punch from 0x8ecd5a to 0x8edfa2 (0x1249 bytes) 000185[1] 1788823077.177325 read 0x973893 thru 0x975749 (0x1eb7 bytes) 000187[1] 1788823077.253990 read 0x54ff37 thru 0x551cc4 (0x1d8e bytes) 000189[1] 1788823077.265356 read 0x2826b5 thru 0x283288 (0xbd4 bytes) 000190[0] 1788823077.275121 mapread 0x0dd9ef thru 0x0ddcf7 (0x309 bytes) 000192[0] 1788823077.321495 punch from 0x29fcf3 to 0x2a08dc (0xbea bytes) 000193[0] 1788823077.404501 read 0x7119b0 thru 0x7136df (0x1d30 bytes) 000195[0] 1788823077.441756 read 0x43707a thru 0x43749c (0x423 bytes) 000196[0] 1788823077.472543 read 0x1a9332 thru 0x1aacf8 (0x19c7 bytes) 000197[0] 1788823077.491441 read 0x2502ba thru 0x250aff (0x846 bytes) 000198[1] 1788823077.533759 mapread 0x152dd2 thru 0x15441c (0x164b bytes) 000201[1] 1788823077.600024 mapread 0x2c2dc9 thru 0x2c431d (0x1555 bytes) 000203[0] 1788823077.645415 trunc from 0x9cef46 to 0x5f8036 (0x3d6f0f bytes) 000205[1] 1788823077.852374 read 0x0c035e thru 0x0c22d7 (0x1f7a bytes) 000206[1] 1788823077.888767 mapread 0x1383c1 thru 0x1390bf (0xcff bytes) 000207[1] 1788823077.927015 read 0x4b6811 thru 0x4b789d (0x108d bytes) 000208[0] 1788823077.951772 mapread 0x5b8624 thru 0x5b9b5b (0x1538 bytes) 000209[1] 1788823077.967760 write 0x3cc2bd thru 0x3cd5dc (0x1320 bytes) 000210[1] 1788823077.995277 write 0x24e19e thru 0x25017f (0x1fe2 bytes) 000211[1] 1788823078.007673 punch from 0x4541ce to 0x4549db (0x80e bytes) 000213[1] 1788823078.027961 read 0x28b92d thru 0x28c8ee (0xfc2 bytes) 000214[0] 1788823078.069407 mapread 0x09089d thru 0x090b32 (0x296 bytes) 000216[1] 1788823078.147649 mapread 0x48b66e thru 0x48bf45 (0x8d8 bytes) 000217[1] 1788823078.154230 trunc from 0x5f8037 to 0x3baa39 (0x23d5fd bytes) 000218[0] 1788823078.197066 mapwrite 0x6c14af thru 0x6c194d (0x49f bytes) 000221[1] 1788823078.507393 trunc from 0x6c194e to 0x4e5bf8 (0x1dbd55 bytes) 000223[1] 1788823078.592957 write 0x332ca4 thru 0x333b3d (0xe9a bytes) 000224[0] 1788823078.642007 trunc from 0x4e5bf9 to 0x8c7879 (0x3e1c81 bytes) 000225[1] 1788823078.746713 read 0x666e6e thru 0x667627 (0x7ba bytes) 000228[1] 1788823078.870406 mapread 0x55ca4d thru 0x55e768 (0x1d1c bytes) 000230[0] 1788823079.014920 punch from 0x12aeb8 to 0x12b57a (0x6c3 bytes) 000234[0] 1788823079.115462 write 0x3e61f7 thru 0x3e750f (0x1319 bytes) 000236[1] 1788823079.188703 write 0x654a0e thru 0x654e78 (0x46b bytes) 000237[1] 1788823079.249746 write 0x1136ff thru 0x114bdc (0x14de bytes) 000238[1] 1788823079.294748 mapwrite 0x6321c5 thru 0x632790 (0x5cc bytes) 000241[1] 1788823079.425004 trunc from 0x8c787a to 0x1a2fa5 (0x7248d4 bytes) 000248[1] 1788823079.714038 mapwrite 0x76e476 thru 0x76fc3f (0x17ca bytes) 000250[1] 1788823079.889432 punch from 0x477c07 to 0x4796ac (0x1aa6 bytes) 000253[0] 1788823079.933870 trunc from 0x76fc40 to 0x959c3a (0x1e9ffb bytes) 000255[1] 1788823080.045884 mapwrite 0x67e21a thru 0x67f5a5 (0x138c bytes) 000257[0] 1788823080.203891 mapread 0x0251aa thru 0x02544b (0x2a2 bytes) 000258[1] 1788823080.244002 write 0x461958 thru 0x46209a (0x743 bytes) 000259[0] 1788823080.272806 trunc from 0x959c3b to 0x10f10f (0x84ab2b bytes) 000260[0] 1788823080.359139 punch from 0x0dd56e to 0x0deff3 (0x1a86 bytes) 000264[1] 1788823080.432965 write 0x87048d thru 0x870f70 (0xae4 bytes) 000265[0] 1788823080.461728 mapread 0x82bde3 thru 0x82dbd6 (0x1df4 bytes) 000267[0] 1788823080.506311 trunc from 0x870f71 to 0x399f04 (0x4d706c bytes) 000268[1] 1788823080.545839 read 0x18b5d8 thru 0x18cb5d (0x1586 bytes) 000270[1] 1788823080.571671 write 0x8cb7f7 thru 0x8cd44a (0x1c54 bytes) 000272[1] 1788823080.626326 trunc from 0x8cd44b to 0x33b3ca (0x592080 bytes) 000273[1] 1788823080.694038 trunc from 0x33b3cb to 0x8b47ef (0x579425 bytes) 000274[0] 1788823080.720453 mapread 0x508d9d thru 0x509565 (0x7c9 bytes) 000278[0] 1788823080.831212 read 0x6c196a thru 0x6c3369 (0x1a00 bytes) 000280[0] 1788823080.852839 mapwrite 0x7ed639 thru 0x7ee2d0 (0xc98 bytes) 000281[1] 1788823080.972227 read 0x257cf4 thru 0x259363 (0x1670 bytes) 000282[0] 1788823080.999732 read 0x69de94 thru 0x69e747 (0x8b4 bytes) 000283[0] 1788823081.060537 write 0x63889a thru 0x6399ba (0x1121 bytes) 000286[1] 1788823081.157764 trunc from 0x8b47f0 to 0x45ad59 (0x459a96 bytes) 000287[1] 1788823081.233301 mapwrite 0x6b92e0 thru 0x6baaa2 (0x17c3 bytes) 000288[0] 1788823081.434616 mapwrite 0x3fe251 thru 0x3fe517 (0x2c7 bytes) 000289[0] 1788823081.569082 mapwrite 0x26bb59 thru 0x26c643 (0xaeb bytes) 000290[0] 1788823081.684082 punch from 0x0a2551 to 0x0a2eb6 (0x966 bytes) 000291[1] 1788823081.723473 read 0x2814f9 thru 0x28187f (0x387 bytes) 000292[1] 1788823081.768346 mapread 0x6510ae thru 0x651171 (0xc4 bytes) 000293[1] 1788823081.791826 punch from 0x18af6a to 0x18ca70 (0x1b07 bytes) 000294[0] 1788823081.817545 mapwrite 0x3a7d37 thru 0x3a92ac (0x1576 bytes) 000295[0] 1788823081.920044 trunc from 0x6baaa3 to 0x37bd28 (0x33ed7a bytes) 000297[0] 1788823082.007385 punch from 0x0bac2a to 0x0bcb9c (0x1f73 bytes) 000299[1] 1788823082.103567 trunc from 0x37bd29 to 0x87aeda (0x4ff1b2 bytes) 000300[1] 1788823082.152356 write 0x265400 thru 0x265d53 (0x954 bytes) 000302[0] 1788823082.218823 trunc from 0x87aedb to 0x5106ab (0x36a82f bytes) 000303[0] 1788823082.274294 trunc from 0x5106ac to 0x8b4ee4 (0x3a4839 bytes) 000304[0] 1788823082.322886 punch from 0x8349fc to 0x8367aa (0x1daf bytes) 000306[0] 1788823082.361998 mapwrite 0x6f198b thru 0x6f334a (0x19c0 bytes) 000307[0] 1788823082.497844 read 0x62ae41 thru 0x62c753 (0x1913 bytes) 000308[0] 1788823082.510354 mapwrite 0x0ec626 thru 0x0eda75 (0x1450 bytes) 000309[1] 1788823082.701291 mapread 0x5fe3ae thru 0x5ff044 (0xc97 bytes) 000314[1] 1788823082.864240 mapwrite 0x2da46d thru 0x2da7ef (0x383 bytes) 000317[0] 1788823083.066601 mapwrite 0x58d486 thru 0x58d5df (0x15a bytes) 000320[0] 1788823083.237097 write 0x74cf2e thru 0x74e3cc (0x149f bytes) 000323[0] 1788823083.334446 punch from 0x2ee7e0 to 0x2efa94 (0x12b5 bytes) 000324[0] 1788823083.419281 write 0x260a1c thru 0x261870 (0xe55 bytes) 000326[0] 1788823083.451879 mapread 0x6af9fa thru 0x6b10a8 (0x16af bytes) 000327[0] 1788823083.549752 punch from 0x758c9e to 0x758ee3 (0x246 bytes) 000330[1] 1788823083.569180 punch from 0x32cc68 to 0x32eb1a (0x1eb3 bytes) 000331[0] 1788823083.667978 write 0x2cf572 thru 0x2d09dd (0x146c bytes) 000332[0] 1788823083.796158 read 0x283cf5 thru 0x2844a0 (0x7ac bytes) 000333[0] 1788823083.859943 read 0x1f6d0d thru 0x1f76fa (0x9ee bytes) 000334[0] 1788823083.956860 trunc from 0x8b4ee5 to 0x4dfb1f (0x3d53c5 bytes) 000336[1] 1788823084.102687 mapwrite 0x8a2941 thru 0x8a3b53 (0x1213 bytes) 000338[1] 1788823084.445942 mapread 0x57bba4 thru 0x57ce0c (0x1269 bytes) 000339[0] 1788823084.509212 mapread 0x0bdc3a thru 0x0bdfd2 (0x399 bytes) 000340[1] 1788823084.585836 punch from 0x22ed66 to 0x22f9e3 (0xc7e bytes) 000342[1] 1788823084.651919 trunc from 0x8a3b54 to 0x484221 (0x41f932 bytes) 000343[1] 1788823084.681521 punch from 0x397a3e to 0x398e51 (0x1414 bytes) 000344[0] 1788823084.688031 mapwrite 0x9e4331 thru 0x9e612d (0x1dfd bytes) 000345[1] 1788823084.850939 mapwrite 0x186bc2 thru 0x18832a (0x1769 bytes) 000346[0] 1788823085.158999 write 0x4f6013 thru 0x4f744a (0x1438 bytes) 000347[0] 1788823085.224877 mapwrite 0x0882a8 thru 0x0892c6 (0x101f bytes) 000348[1] 1788823085.397714 punch from 0x9a6049 to 0x9a6c34 (0xbec bytes) 000350[0] 1788823085.462004 write 0x7b151d thru 0x7b1695 (0x179 bytes) 000352[1] 1788823085.519147 trunc from 0x9e612e to 0x8acffd (0x139130 bytes) 000353[0] 1788823085.563974 mapwrite 0x0044d5 thru 0x005c67 (0x1793 bytes) 000354[1] 1788823085.661416 trunc from 0x8acffe to 0x7d60db (0xd6f22 bytes) 000355[0] 1788823085.695245 write 0x7f4d7a thru 0x7f56c2 (0x949 bytes) 000356[1] 1788823085.721361 trunc from 0x7f56c3 to 0x5b48a9 (0x240e19 bytes) 000357[0] 1788823085.801816 trunc from 0x5b48aa to 0x28036f (0x33453a bytes) 000358[1] 1788823085.852360 mapwrite 0x14c18a thru 0x14c31d (0x194 bytes) 000359[0] 1788823085.921048 read 0x04d642 thru 0x04eace (0x148d bytes) 000361[1] 1788823086.002870 mapread 0x22156c thru 0x221781 (0x216 bytes) 000362[1] 1788823086.024539 mapread 0x097294 thru 0x0978b3 (0x620 bytes) 000364[0] 1788823086.065510 punch from 0x27c448 to 0x27c7f2 (0x3ab bytes) 000365[1] 1788823086.114216 mapwrite 0x924020 thru 0x925a1e (0x19ff bytes) 000366[1] 1788823086.317031 write 0x31373d thru 0x31398f (0x253 bytes) 000367[0] 1788823086.343625 punch from 0x167b27 to 0x167bc8 (0xa2 bytes) 000368[1] 1788823086.385120 write 0x77264b thru 0x77415a (0x1b10 bytes) 000373[1] 1788823086.472960 mapwrite 0x6ddb31 thru 0x6defa4 (0x1474 bytes) 000374[0] 1788823086.555729 mapread 0x3ee18e thru 0x3efe50 (0x1cc3 bytes) 000375[1] 1788823086.627922 punch from 0x6ff383 to 0x6ff7d2 (0x450 bytes) 000378[0] 1788823086.670737 mapread 0x4cdb72 thru 0x4cdf7b (0x40a bytes) 000379[0] 1788823086.783673 read 0x44266c thru 0x4438b4 (0x1249 bytes) 000380[1] 1788823086.796925 punch from 0x3e2c4f to 0x3e470d (0x1abf bytes) 000385[1] 1788823086.854588 trunc from 0x925a1f to 0x9aab58 (0x8513a bytes) 000386[1] 1788823086.905647 read 0x694022 thru 0x6946cb (0x6aa bytes) 000389[0] 1788823086.976170 mapread 0x1b687c thru 0x1b74ff (0xc84 bytes) 000390[1] 1788823087.012637 read 0x51b3d0 thru 0x51cb09 (0x173a bytes) 000392[1] 1788823087.018371 trunc from 0x9aab59 to 0x907434 (0xa3724 bytes) 000395[1] 1788823087.124137 mapread 0x0b3d76 thru 0x0b499d (0xc28 bytes) 000397[0] 1788823087.156473 mapread 0x139e42 thru 0x13aa2a (0xbe9 bytes) 000399[0] 1788823087.227661 mapread 0x89b522 thru 0x89ba47 (0x526 bytes) 000400[0] 1788823087.275630 write 0x9ed9c7 thru 0x9ee38d (0x9c7 bytes) 000403[1] 1788823087.488026 mapwrite 0x757dd4 thru 0x759c7e (0x1eab bytes) 000404[0] 1788823087.707813 mapwrite 0x82b0cc thru 0x82c4d5 (0x140a bytes) 000405[1] 1788823087.903358 write 0x0a8761 thru 0x0aa187 (0x1a27 bytes) 000409[0] 1788823088.005394 read 0x723347 thru 0x723a1c (0x6d6 bytes) 000411[0] 1788823088.131943 mapread 0x8e6eb9 thru 0x8e840b (0x1553 bytes) 000412[0] 1788823088.248514 trunc from 0x9ee38e to 0x1801a7 (0x86e1e6 bytes) 000414[0] 1788823088.361985 mapwrite 0x625550 thru 0x6267f1 (0x12a2 bytes) 000415[0] 1788823088.478579 trunc from 0x6267f2 to 0x846d7e (0x22058d bytes) 000416[1] 1788823088.513823 mapread 0x2def88 thru 0x2e0412 (0x148b bytes) 000417[0] 1788823088.605975 write 0x1feb76 thru 0x1ff6bb (0xb46 bytes) 000418[1] 1788823088.640131 punch from 0x230408 to 0x2323ec (0x1fe5 bytes) 000419[0] 1788823088.696936 mapwrite 0x3d6fda thru 0x3d86e4 (0x170b bytes) 000420[1] 1788823088.908104 read 0x11a77a thru 0x11adc4 (0x64b bytes) 000421[0] 1788823088.940533 write 0x438ef1 thru 0x439558 (0x668 bytes) 000423[1] 1788823089.011642 mapwrite 0x553150 thru 0x553393 (0x244 bytes) 000424[0] 1788823089.117189 write 0x9374ea thru 0x93833b (0xe52 bytes) 000425[1] 1788823089.148027 read 0x27a59b thru 0x27ab0f (0x575 bytes) 000429[0] 1788823089.189725 write 0x6db9b3 thru 0x6dc49c (0xaea bytes) 000431[0] 1788823089.232353 trunc from 0x93833c to 0x009d11 (0x92e62a bytes) 000432[0] 1788823089.274953 mapwrite 0x4f980d thru 0x4fa242 (0xa36 bytes) 000434[1] 1788823089.414442 mapwrite 0x9f2076 thru 0x9f276e (0x6f9 bytes) 000435[0] 1788823089.555651 punch from 0x9417cb to 0x94269d (0xed3 bytes) 000436[1] 1788823089.581920 read 0x42a8d4 thru 0x42a9ef (0x11c bytes) 000438[1] 1788823089.617716 trunc from 0x9f276f to 0x4730a7 (0x57f6c7 bytes) 000439[1] 1788823089.679441 punch from 0x16e729 to 0x16f8c1 (0x1199 bytes) 000440[1] 1788823089.685248 trunc from 0x4730a8 to 0x327261 (0x14be46 bytes) 000441[1] 1788823089.741136 mapwrite 0x1dc134 thru 0x1dde82 (0x1d4f bytes) truncating to largest ever: 0x9e3197 000443[1] 1788823089.838508 trunc from 0x327262 to 0x9e3196 (0x6bbf35 bytes) 000444[0] 1788823089.864554 mapread 0x3845d3 thru 0x3861fe (0x1c2c bytes) 000445[0] 1788823089.914534 trunc from 0x9e3197 to 0x88ad46 (0x158450 bytes) 000446[0] 1788823089.959964 trunc from 0x88ad47 to 0x549bcc (0x34117a bytes) 000448[1] 1788823090.005037 trunc from 0x549bcd to 0x5532d7 (0x970b bytes) 000450[1] 1788823090.059023 punch from 0x0c40b9 to 0x0c454d (0x495 bytes) 000451[1] 1788823090.065645 read 0x479628 thru 0x47ad27 (0x1700 bytes) 000452[1] 1788823090.113027 write 0x0c4bc1 thru 0x0c52f5 (0x735 bytes) 000453[1] 1788823090.124957 read 0x25a0fe thru 0x25a99c (0x89f bytes) 000462[0] 1788823090.232408 mapwrite 0x43e9d9 thru 0x44070c (0x1d34 bytes) 000463[0] 1788823090.446709 read 0x24264d thru 0x243c82 (0x1636 bytes) 000465[0] 1788823090.519877 trunc from 0x5532d8 to 0x0a9712 (0x4a9bc5 bytes) 000466[1] 1788823090.589794 read 0x0a30c1 thru 0x0a375e (0x69e bytes) 000469[0] 1788823090.652520 mapread 0x01e37e thru 0x01efdb (0xc5e bytes) 000471[1] 1788823090.692026 read 0x081326 thru 0x082635 (0x1310 bytes) 000472[1] 1788823090.708930 read 0x01d6dc thru 0x01dba1 (0x4c6 bytes) 000473[0] 1788823090.723337 punch from 0x033df7 to 0x0358c9 (0x1ad3 bytes) 000475[0] 1788823090.742245 mapwrite 0x6acaff thru 0x6acbf1 (0xf3 bytes) 000476[0] 1788823090.823088 mapread 0x240bfc thru 0x241bbe (0xfc3 bytes) 000478[1] 1788823090.834591 read 0x23d44b thru 0x23e107 (0xcbd bytes) 000479[0] 1788823090.967815 punch from 0x56176a to 0x562285 (0xb1c bytes) 000480[0] 1788823090.993114 punch from 0x03b150 to 0x03c558 (0x1409 bytes) 000481[1] 1788823091.034536 write 0x361e89 thru 0x363519 (0x1691 bytes) 000482[0] 1788823091.072601 read 0x29b0a2 thru 0x29b316 (0x275 bytes) 000486[0] 1788823091.157975 mapwrite 0x9572b1 thru 0x958e4a (0x1b9a bytes) 000487[1] 1788823091.307835 trunc from 0x958e4b to 0x0ad07a (0x8abdd0 bytes) 000489[0] 1788823091.374167 read 0x0463d3 thru 0x047ff1 (0x1c1f bytes) 000490[0] 1788823091.426525 write 0x89094d thru 0x8913d8 (0xa8c bytes) 000492[1] 1788823091.472890 trunc from 0x8913d9 to 0x19e734 (0x6f2ca4 bytes) 000493[0] 1788823091.502678 write 0x7b4fc5 thru 0x7b54aa (0x4e6 bytes) 000496[0] 1788823091.559333 read 0x507f0e thru 0x508579 (0x66c bytes) 000498[0] 1788823091.590535 mapwrite 0x23c451 thru 0x23c914 (0x4c4 bytes) 000499[0] 1788823091.658950 write 0x2d25f4 thru 0x2d2792 (0x19f bytes) 000500[1] 1788823091.664981 trunc from 0x7b54ab to 0x5646bd (0x250ded bytes) All operations completed A-OK! PASS 16b (36s) == sanityn test 16c: verify data consistency on ldiskfs with cache disabled (b=17397) ========================================================== 19:18:17 (1788823097) SKIP: sanityn test_16c dio on ldiskfs only SKIP 16c (5s) == sanityn test 16d: Verify DIO and buffer IO with two clients ========================================================== 19:18:22 (1788823102) 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.369097 s, 22.7 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.353514 s, 23.7 MB/s ldlm.namespaces.MGC192.168.204.107@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff92ca0a087000.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff92ca100da000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff92ca0a087000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff92ca100da000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff92ca0a087000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff92ca100da000.lru_size=clear 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 12.6931 s, 8.3 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 3.72537 s, 28.1 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 3.02329 s, 34.7 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.412 s, 16.4 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.48572 s, 11.1 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.92168 s, 21.3 MB/s PASS 16d (56s) == sanityn test 16e: Verify size consistency for O_DIRECT write ========================================================== 19:19:18 (1788823158) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.24678 s, 42.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164953 s, 6.4 MB/s PASS 16e (9s) == sanityn test 16f: rw sequential consistency vs drop_caches ========================================================== 19:19:27 (1788823167) TIMEOUT PASS 16f (29s) == sanityn test 16g: mmap rw sequential consistency vs drop_caches ========================================================== 19:19:56 (1788823196) TIMEOUT PASS 16g (29s) == sanityn test 16h: mmap read after truncate file ======= 19:20:25 (1788823225) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16h (9s) == sanityn test 16i: read after truncate file ============ 19:20:34 (1788823234) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16i (8s) == sanityn test 16j: race dio with buffered i/o ========== 19:20:43 (1788823243) 1+0 records in 1+0 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.40599 s, 10.9 MB/s bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.6011 s, 16.4 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.27602 s, 8.0 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.78121 s, 14.7 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.44763 s, 10.7 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.84566 s, 9.2 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.77262 s, 6.9 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.40173 s, 10.9 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.07703 s, 8.5 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 5.45018 s, 4.8 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 5.98367 s, 4.4 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.40393 s, 7.7 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.65258 s, 7.2 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 4.14099 s, 6.3 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 4.20274 s, 6.2 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 4.88938 s, 5.4 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 5.30994 s, 4.9 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 5.2189 s, 5.0 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 5.55661 s, 4.7 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK PASS 16j (50s) == sanityn test 16k: Parallel FSX and drop caches should not panic ========================================================== 19:21:33 (1788823293) + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.1 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.2 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.3 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.4 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.5 -l 67108864 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read skipping zero size read truncating to largest ever: 0x1a7062c skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported truncating to largest ever: 0x232e1d3 fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read truncating to largest ever: 0x1a7062c truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x3c569db truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! + fsx 1 done: rc=0 Mon Sep 7 19:22:35 EDT 2026 + fsx 2 done: rc=0 Mon Sep 7 19:22:35 EDT 2026 All operations completed A-OK! All operations completed A-OK! + fsx 3 done: rc=0 Mon Sep 7 19:22:35 EDT 2026 + fsx 4 done: rc=0 Mon Sep 7 19:22:35 EDT 2026 + fsx 5 done: rc=0 Mon Sep 7 19:22:35 EDT 2026 PASS 16k (74s) == sanityn test 17: resource creation/LVB creation race ========================================================================= 19:22:47 (1788823367) fail_loc=0x8000030a PASS 17 (12s) == sanityn test 18: mmap sanity check =========================================================================================== 19:22:59 (1788823379) mmap test1: basic mmap operation (PASS, 0.12969s) mmap test2: MAP_PRIVATE not write back (PASS, 0.13254s) mmap test3: concurrent mmap ops on two nodes (PASS, 5.8627s) mmap test4: c1 write to f1 from mmapped f2, c2 write to f1 from mmapped f1 (PASS, 4.6858s) mmap test5: read/write file to/from the buffer which mmapped to just this file (PASS, 0.19409s) mmap test6: check mmap write/read content on two nodes (PASS, 0.27104s) mmap test7: file i/o with an unmapped buffer (SKIPPED, 0s) mmap test8: SIGBUS for beyond file size (SKIPPED, 0s) mmap test9: SIGBUS for negative file offset (SKIPPED, 0s) mmap test10: multi-thread mmap access (PASS, 1.456s) mmap test11: multi-thread shared mmap (PASS, 0.91258s) mmap test10: mtime not change for readonly mmap access (PASS, 1.127s) PASS 18 (24s) == sanityn test 19: test concurrent uncached read races ========================================================================= 19:23:23 (1788823403) oleg407-server: error: get_param: param_path 'osd-*/lustre-OST*/read_cache_enable': No such file or directory pdsh@oleg407-client: oleg407-server: ssh exited with exit code 2 SKIP: sanityn test_19 not cache-capable obdfilter SKIP 19 (5s) == sanityn test 20: test extra readahead page left in cache ============================================================== 19:23:28 (1788823408) PASS 20 (7s) == sanityn test 21: Try to remove mountpoint on another dir ============================================================== 19:23:36 (1788823416) rmdir: removing directory, '/mnt/lustre/d21.sanityn' rmdir: failed to remove '/mnt/lustre/d21.sanityn': Device or resource busy rmdir: removing directory, '/mnt/lustre2/d21.sanityn' Removed mounted directory from another mountpoint, needs to be fixed PASS 21 (8s) == sanityn test 23: others should see updated atime while another read============================================================== 19:23:44 (1788823424) now is 1788823427 starting reads multiop /mnt/lustre/f23.sanityn vor20_c TMPPIPE=/tmp/multiop_open_wait_pipe.7612 new atime is 1788823488 PASS 23 (73s) == sanityn test 24a: lfs df [-ih] [path] test =================================================================================== 19:24:57 (1788823497) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 5120 3764224 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 5120 3764224 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre2 UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 115.3K 403 114.9K 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 114.5K 399 114.1K 1% /mnt/lustre[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.3K 403 114.9K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 399 114.1K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre2 UUID bytes Used Available Use% Mounted on lustre-MDT0000_UUID 2.1G 2.1M 2.1G 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3.6G 5.0M 3.6G 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3.6G 32.0M 3.6G 1% /mnt/lustre[OST:1] filesystem_summary: 7.2G 37.0M 7.2G 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118099 403 117696 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117231 399 116832 1% /mnt/lustre2[OST:1] filesystem_summary: 234914 386 234528 1% /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 5120 3764224 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.3K 403 114.9K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 399 114.1K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 118099 403 117696 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 117231 399 116832 1% /mnt/lustre[OST:1] filesystem_summary: 234914 386 234528 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118099 403 117696 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117231 399 116832 1% /mnt/lustre2[OST:1] filesystem_summary: 234914 386 234528 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 5120 3764224 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 5120 3764224 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre2 PASS 24a (9s) == sanityn test 24b: lfs df should show both filesystems ========================================================================= 19:25:07 (1788823507) PASS 24b (9s) == sanityn test 25a: change ACL on one mountpoint be seen on another ============================================================= 19:25:16 (1788823516) running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] PASS 25a (10s) == sanityn test 25b: change ACL under remote dir on one mountpoint be seen on another ========================================================== 19:25:25 (1788823525) SKIP: sanityn test_25b needs >= 2 MDTs SKIP 25b (4s) == sanityn test 26a: allow mtime to get older ============ 19:25:30 (1788823530) utime: good mknod times 1788823530 <= 1788823531 <= 1788823531 for /mnt/lustre/f26a utime: good mknod times 1788823530 <= 1788823531 <= 1788823531 for /mnt/lustre2/f26a utime: good utime mtimes 100000, atime 200000 utime: updated times for /mnt/lustre2/f26a PASS 26a (9s) == sanityn test 26b: sync mtime between ost and mds ====== 19:25:38 (1788823538) PASS 26b (10s) == sanityn test 26c: set-in-past on open file is not lost on close ========================================================== 19:25:48 (1788823548) multiop /mnt/lustre/f26c.sanityn vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7612 File: /mnt/lustre/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502395 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-09-07 19:25:50.000000000 -0400 Birth: 2026-09-07 19:25:50.000000000 -0400 File: /mnt/lustre2/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502395 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-09-07 19:25:50.000000000 -0400 Birth: 2026-09-07 19:25:50.000000000 -0400 PASS 26c (9s) SKIP: sanityn test_27 skipping excluded test 27 == sanityn test 30: recreate file race =================== 19:25:59 (1788823559) 0 PASS 30 (10s) == sanityn test 31a: voluntary cancel / blocking ast race======================================================================== 19:26:09 (1788823569) fail_loc=0x314 PASS 31a (9s) == sanityn test 31b: voluntary OST cancel / blocking ast race======================================================================== 19:26:18 (1788823578) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg407-server mds-ost sync done. sleep 5 for ZFS MDS Waiting for MDT destroys to complete fail_loc=0x314 fail_loc=0x316 fail_loc=0 fail_loc=0 Filesystem 1K-blocks Used Available Use% Mounted on 192.168.204.107@tcp:/lustre 7542784 39936 7498752 1% /mnt/lustre2 PASS 31b (25s) == sanityn test 31r: open-rename(replace) race =========== 19:26:43 (1788823603) -rw-r--r-- 1 root root 0 Sep 7 19:26 /mnt/lustre/f31r.sanityn.target fail_loc=0x1419 fail_val=3 PASS 31r (11s) == sanityn test 31s: open should not revalidate invalid dentry ========================================================== 19:26:54 (1788823614) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 1205: 43158 Terminated tail -f $tfile PASS 31s (9s) == sanityn test 31t: getattr should not revalidate invalid dentry ========================================================== 19:27:03 (1788823623) PASS 31t (9s) == sanityn test 32b: lockless i/o ======================== 19:27:12 (1788823632) SKIP: sanityn test_32b max_nolock_bytes is removed >= 2.17.53 SKIP 32b (4s) == sanityn test 32c: contention detection ================ 19:27:16 (1788823636) ldlm.namespaces.filter-lustre-OST0000_UUID.contended_locks=2 ldlm.namespaces.filter-lustre-OST0001_UUID.contended_locks=2 ldlm.namespaces.filter-lustre-OST0000_UUID.contention_events=0 ldlm.namespaces.filter-lustre-OST0001_UUID.contention_events=0 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.85853 s, 28.2 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.27904 s, 41.0 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.37185 s, 22.1 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.65605 s, 19.7 MB/s contention_seconds 4, contention_events 21 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.95667 s, 26.8 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.25042 s, 23.3 MB/s ldlm.namespaces.filter-lustre-OST0000_UUID.contention_events=0 ldlm.namespaces.filter-lustre-OST0001_UUID.contention_events=0 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.35267 s, 38.8 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.42977 s, 36.7 MB/s PASS 32c (32s) SKIP: sanityn test_33a skipping SLOW test 33a == sanityn test 33b: COS: cross create/delete, 2 clients, benchmark under remote dir ========================================================== 19:27:51 (1788823671) SKIP: sanityn test_33b Need two or more clients, have 1 SKIP 33b (5s) == sanityn test 33c: Cancel cross-MDT lock should trigger Sync-on-Lock-Cancel ========================================================== 19:27:55 (1788823675) SKIP: sanityn test_33c needs >= 2 MDTs SKIP 33c (5s) == sanityn test 33d: dependent transactions should trigger COS ========================================================== 19:28:00 (1788823680) SKIP: sanityn test_33d needs >= 2 MDTs SKIP 33d (4s) == sanityn test 33e: independent transactions shouldn't trigger COS ========================================================== 19:28:04 (1788823684) SKIP: sanityn test_33e needs >= 2 MDTs SKIP 33e (4s) == sanityn test 34: no lock timeout under IO ============= 19:28:08 (1788823688) fail_loc=0x512 fail_loc=0x512 lock should not expire writing on client1 reading on client2 success fail_loc=0x511 fail_loc=0x511 lock should expire writing on client1 reading on client2 success oleg407-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff92ca0a087000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff92ca0a087000.ost_server_uuid in FULL state after 0 sec oleg407-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff92ca0a087000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff92ca0a087000.ost_server_uuid in IDLE state after 0 sec oleg407-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff92ca0a087000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff92ca0a087000.ost_server_uuid in FULL state after 0 sec oleg407-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff92ca0a087000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff92ca0a087000.ost_server_uuid in IDLE state after 0 sec oleg407-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff92ca0a087000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff92ca0a087000.ost_server_uuid in FULL state after 0 sec oleg407-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff92ca0a087000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff92ca0a087000.ost_server_uuid in IDLE state after 0 sec PASS 34 (149s) == sanityn test 35: -EINTR cp_ast vs. bl_ast race does not evict client ========================================================== 19:30:37 (1788823837) fail_loc=0x80000317 Race attempt 0 Wait for 50467 50502 for 60 sec... - open/close 858 (time 1788823855.06 total 10.00 last 85.76) - open/close 1632 (time 1788823865.07 total 20.01 last 77.32) - open/close 2386 (time 1788823875.07 total 30.02 last 75.38) - open/close 3204 (time 1788823885.07 total 40.02 last 81.77) - open/close 3947 (time 1788823895.11 total 50.05 last 74.07) total: 4000 open/close in 50.86 seconds: 78.65 ops/second fail_loc=0x0 Filesystem Size Used Avail Use% Mounted on 192.168.204.107@tcp:/lustre 7.2G 40M 7.2G 1% /mnt/lustre 192.168.204.107@tcp:/lustre 7.2G 40M 7.2G 1% /mnt/lustre2 PASS 35 (74s) == sanityn test 36: handle ESTALE/open-unlink correctly == 19:31:51 (1788823911) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg407-server mds-ost sync done. Waiting for MDT destroys to complete 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.61365 s, 32.5 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7612 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(0) *** before(7497728) after_dd(7439360) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.58364 s, 33.1 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7612 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(1) *** before(7497728) after_dd(7441408) after(7492608) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.49926 s, 35.0 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7612 sleep 5 for ZFS MDS Waiting for MDT destroys to complete *** cycle(2) *** before(7492608) after_dd(7446528) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.58797 s, 33.0 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7612 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(3) *** before(7497728) after_dd(7440384) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.5177 s, 34.5 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7612 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(4) *** before(7497728) after_dd(7444480) after(7495680) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.75911 s, 29.8 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7612 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(5) *** before(7495680) after_dd(7446528) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.66808 s, 31.4 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7612 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(6) *** before(7497728) after_dd(7440384) after(7491584) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.37303 s, 22.1 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7612 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(7) *** before(7491584) after_dd(7444480) after(7495680) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.37818 s, 38.0 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7612 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(8) *** before(7495680) after_dd(7442432) after(7493632) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.24984 s, 41.9 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7612 sleep 5 for ZFS MDS Waiting for MDT destroys to complete *** cycle(9) *** before(7493632) after_dd(7439360) after(7490560) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.18796 s, 44.1 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7612 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(10) *** before(7490560) after_dd(7442432) after(7493632) PASS 36 (315s) == sanityn test 37: check i_size is not updated for directory on close (bug 18695) ======================================================================== 19:37:06 (1788824226) multiop /mnt/lustre/d37.sanityn vD_c TMPPIPE=/tmp/multiop_open_wait_pipe.7612 - create 1044 (time 1788824242.19 total 10.00 last 104.37) - create 2135 (time 1788824252.19 total 20.00 last 109.08) - create 3644 (time 1788824262.19 total 30.01 last 150.80) - create 4628 (time 1788824272.20 total 40.02 last 98.35) - create 5780 (time 1788824282.21 total 50.03 last 115.09) - create 6970 (time 1788824292.21 total 60.03 last 118.95) - create 8310 (time 1788824302.22 total 70.03 last 133.94) - create 9563 (time 1788824312.22 total 80.03 last 125.28) total: 10000 create in 83.34 seconds: 119.99 ops/second PASS 37 (96s) == sanityn test 39a: file mtime does not change after rename ========================================================== 19:38:42 (1788824322) repeat after cancel_lru_locks PASS 39a (9s) == sanityn test 39b: file mtime the same on clients with/out lock ========================================================== 19:38:52 (1788824332) repeat after cancel_lru_locks PASS 39b (9s) == sanityn test 39c: check truncate mtime update ================================================================================ 19:39:01 (1788824341) repeat after cancel_lru_locks PASS 39c (8s) == sanityn test 39d: sync write should update mtime ====== 19:39:09 (1788824349) fail_loc=0x411 fail_loc=0 PASS 39d (7s) SKIP: sanityn test_40a skipping ALWAYS excluded test 40a == sanityn test 40b: pdirops: open|create and others ======================================================================== 19:39:18 (1788824358) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40b (17s) == sanityn test 40c: pdirops: link and others ======================================================================== 19:39:35 (1788824375) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40c (17s) == sanityn test 40d: pdirops: unlink and others ======================================================================== 19:39:53 (1788824393) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40d (15s) == sanityn test 40e: pdirops: rename and others ======================================================================== 19:40:08 (1788824408) No conflict No conflict No conflict No conflict No conflict No conflict PASS 40e (16s) == sanityn test 41a: pdirops: create vs mkdir ======================================================================== 19:40:24 (1788824424) mkdir: cannot create directory '/mnt/lustre2/f41a.sanityn': File exists Conflict PASS 41a (11s) == sanityn test 41b: pdirops: create vs create ======================================================================== 19:40:35 (1788824435) open: File exists Conflict PASS 41b (13s) == sanityn test 41c: pdirops: create vs link ======================================================================== 19:40:48 (1788824448) link: cannot create link '/mnt/lustre2/f41c.sanityn' to '/mnt/lustre2/f41c.sanityn-2': File exists Conflict PASS 41c (12s) == sanityn test 41d: pdirops: create vs unlink ======================================================================== 19:41:00 (1788824460) Conflict PASS 41d (12s) == sanityn test 41e: pdirops: create and rename (tgt) ======================================================================== 19:41:12 (1788824472) Conflict PASS 41e (13s) == sanityn test 41f: pdirops: create and rename (src) ======================================================================== 19:41:25 (1788824485) Conflict PASS 41f (12s) == sanityn test 41g: pdirops: create vs getattr ======================================================================== 19:41:37 (1788824497) Conflict PASS 41g (12s) == sanityn test 41h: pdirops: create vs readdir ======================================================================== 19:41:49 (1788824509) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 2427: lctl set_param -n fail_loc=0 2>/dev/null || true: No such file or directory Conflict PASS 41h (11s) == sanityn test 41i: reint_open: create vs create ======== 19:42:00 (1788824520) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 41i (948s) == sanityn test 42a: pdirops: mkdir vs mkdir ======================================================================== 19:57:48 (1788825468) mkdir: cannot create directory '/mnt/lustre2/f42a.sanityn': File exists Conflict PASS 42a (11s) == sanityn test 42b: pdirops: mkdir vs create ======================================================================== 19:57:59 (1788825479) open: File exists Conflict PASS 42b (9s) == sanityn test 42c: pdirops: mkdir vs link ======================================================================== 19:58:08 (1788825488) link: cannot create link '/mnt/lustre2/f42c.sanityn' to '/mnt/lustre2/f42c.sanityn-2': File exists Conflict PASS 42c (10s) == sanityn test 42d: pdirops: mkdir vs unlink ======================================================================== 19:58:18 (1788825498) Conflict PASS 42d (10s) == sanityn test 42e: pdirops: mkdir and rename (tgt) ======================================================================== 19:58:28 (1788825508) mv: cannot overwrite directory '/mnt/lustre2/f42e.sanityn' with non-directory Conflict PASS 42e (10s) == sanityn test 42f: pdirops: mkdir and rename (src) ======================================================================== 19:58:38 (1788825518) Conflict PASS 42f (10s) == sanityn test 42g: pdirops: mkdir vs getattr ======================================================================== 19:58:48 (1788825528) Conflict PASS 42g (10s) == sanityn test 42h: pdirops: mkdir vs readdir ======================================================================== 19:58:58 (1788825538) Conflict PASS 42h (10s) == sanityn test 43a: rmdir,mkdir doesn't return -EEXIST ======================================================================== 19:59:08 (1788825548) PASS 43a (56s) == sanityn test 43b: pdirops: unlink vs create ======================================================================== 20:00:04 (1788825604) Conflict PASS 43b (11s) == sanityn test 43c: pdirops: unlink vs link ======================================================================== 20:00:15 (1788825615) Conflict PASS 43c (10s) == sanityn test 43d: pdirops: unlink vs unlink ======================================================================== 20:00:26 (1788825626) rm: cannot remove '/mnt/lustre2/f43d.sanityn': No such file or directory Conflict PASS 43d (11s) == sanityn test 43e: pdirops: unlink and rename (tgt) ======================================================================== 20:00:36 (1788825636) Conflict PASS 43e (9s) == sanityn test 43f: pdirops: unlink and rename (src) ======================================================================== 20:00:46 (1788825646) mv: cannot stat '/mnt/lustre2/f43f.sanityn': No such file or directory Conflict PASS 43f (10s) == sanityn test 43g: pdirops: unlink vs getattr ======================================================================== 20:00:55 (1788825655) stat: cannot statx '/mnt/lustre2/f43g.sanityn': No such file or directory Conflict PASS 43g (10s) == sanityn test 43h: pdirops: unlink vs readdir ======================================================================== 20:01:06 (1788825666) Conflict PASS 43h (10s) == sanityn test 43i: pdirops: unlink vs remote mkdir ===== 20:01:15 (1788825675) SKIP: sanityn test_43i needs >= 2 MDTs SKIP 43i (3s) == sanityn test 43j: racy mkdir return EEXIST ======================================================================== 20:01:18 (1788825678) mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists PASS 43j (79s) == sanityn test 43k: unlink vs create ==================== 20:02:37 (1788825757) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 43k (845s) == sanityn test 44a: pdirops: rename tgt vs mkdir ======================================================================== 20:16:42 (1788826602) mkdir: cannot create directory '/mnt/lustre2/f44a.sanityn': File exists Conflict Mon Sep 7 20:16:48 EDT 2026 PASS 44a (9s) == sanityn test 44b: pdirops: rename tgt vs create ======================================================================== 20:16:51 (1788826611) open: File exists Conflict PASS 44b (11s) == sanityn test 44c: pdirops: rename tgt vs link ======================================================================== 20:17:02 (1788826622) link: cannot create link '/mnt/lustre2/f44c.sanityn' to '/mnt/lustre2/f44c.sanityn-3': File exists Conflict PASS 44c (10s) == sanityn test 44d: pdirops: rename tgt vs unlink ======================================================================== 20:17:12 (1788826632) Conflict PASS 44d (10s) == sanityn test 44e: pdirops: rename tgt and rename (tgt) ======================================================================== 20:17:22 (1788826642) Conflict PASS 44e (10s) == sanityn test 44f: pdirops: rename tgt and rename (src) ======================================================================== 20:17:32 (1788826652) Conflict PASS 44f (9s) == sanityn test 44g: pdirops: rename tgt vs getattr ======================================================================== 20:17:42 (1788826662) Conflict PASS 44g (10s) == sanityn test 44h: pdirops: rename tgt vs readdir ======================================================================== 20:17:52 (1788826672) Conflict PASS 44h (11s) == sanityn test 44i: pdirops: rename tgt vs remote mkdir ========================================================== 20:18:03 (1788826683) SKIP: sanityn test_44i needs >= 2 MDTs SKIP 44i (2s) == sanityn test 45a: rename,mkdir doesn't return -EEXIST ======================================================================== 20:18:06 (1788826686) PASS 45a (107s) == sanityn test 45b: pdirops: rename src vs create ======================================================================== 20:19:52 (1788826792) Conflict PASS 45b (10s) == sanityn test 45c: pdirops: rename src vs link ======================================================================== 20:20:03 (1788826803) Conflict PASS 45c (10s) == sanityn test 45d: pdirops: rename src vs unlink ======================================================================== 20:20:13 (1788826813) rm: cannot remove '/mnt/lustre2/f45d.sanityn': No such file or directory Conflict PASS 45d (10s) == sanityn test 45e: pdirops: rename src and rename (tgt) ======================================================================== 20:20:22 (1788826822) Conflict PASS 45e (9s) == sanityn test 45f: pdirops: rename src and rename (src) ======================================================================== 20:20:31 (1788826831) mv: cannot stat '/mnt/lustre2/f45f.sanityn': No such file or directory Conflict PASS 45f (10s) == sanityn test 45g: pdirops: rename src vs getattr ======================================================================== 20:20:41 (1788826841) stat: cannot statx '/mnt/lustre2/f45g.sanityn': No such file or directory Conflict PASS 45g (9s) == sanityn test 45h: pdirops: unlink vs readdir ======================================================================== 20:20:50 (1788826850) Conflict PASS 45h (7s) == sanityn test 45i: pdirops: rename src vs remote mkdir ========================================================== 20:20:57 (1788826857) SKIP: sanityn test_45i needs >= 2 MDTs SKIP 45i (2s) == sanityn test 45j: read vs rename ====================== 20:20:59 (1788826859) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 45j (660s) == sanityn test 46a: pdirops: link vs mkdir ======================================================================== 20:31:59 (1788827519) mkdir: cannot create directory '/mnt/lustre2/f46a.sanityn': File exists Conflict PASS 46a (8s) == sanityn test 46b: pdirops: link vs create ======================================================================== 20:32:07 (1788827527) open: File exists Conflict PASS 46b (8s) == sanityn test 46c: pdirops: link vs link ======================================================================== 20:32:15 (1788827535) link: cannot create link '/mnt/lustre2/f46c.sanityn' to '/mnt/lustre2/f46c.sanityn': File exists Conflict PASS 46c (7s) == sanityn test 46d: pdirops: link vs unlink ======================================================================== 20:32:22 (1788827542) Conflict PASS 46d (8s) == sanityn test 46e: pdirops: link and rename (tgt) ======================================================================== 20:32:30 (1788827550) Conflict PASS 46e (8s) == sanityn test 46f: pdirops: link and rename (src) ======================================================================== 20:32:38 (1788827558) Conflict PASS 46f (7s) == sanityn test 46g: pdirops: link vs getattr ======================================================================== 20:32:45 (1788827565) Conflict PASS 46g (8s) == sanityn test 46h: pdirops: link vs readdir ======================================================================== 20:32:53 (1788827573) Conflict PASS 46h (7s) == sanityn test 46i: pdirops: link vs remote mkdir ======= 20:33:00 (1788827580) SKIP: sanityn test_46i needs >= 2 MDTs SKIP 46i (2s) == sanityn test 47a: pdirops: remote mkdir vs mkdir ====== 20:33:02 (1788827582) SKIP: sanityn test_47a needs >= 2 MDTs SKIP 47a (2s) == sanityn test 47b: pdirops: remote mkdir vs create ===== 20:33:04 (1788827584) SKIP: sanityn test_47b needs >= 2 MDTs SKIP 47b (2s) == sanityn test 47c: pdirops: remote mkdir vs link ======= 20:33:06 (1788827586) SKIP: sanityn test_47c needs >= 2 MDTs SKIP 47c (1s) == sanityn test 47d: pdirops: remote mkdir vs unlink ===== 20:33:08 (1788827588) SKIP: sanityn test_47d needs >= 2 MDTs SKIP 47d (2s) == sanityn test 47e: pdirops: remote mkdir and rename (tgt) ========================================================== 20:33:09 (1788827589) SKIP: sanityn test_47e needs >= 2 MDTs SKIP 47e (2s) == sanityn test 47f: pdirops: remote mkdir and rename (src) ========================================================== 20:33:11 (1788827591) SKIP: sanityn test_47f needs >= 2 MDTs SKIP 47f (2s) == sanityn test 47g: pdirops: remote mkdir vs getattr ==== 20:33:13 (1788827593) SKIP: sanityn test_47g needs >= 2 MDTs SKIP 47g (1s) == sanityn test 50: osc lvb attrs: enqueue vs. CP AST ======================================================================== 20:33:15 (1788827595) 10+0 records in 10+0 records out 10240 bytes (10 kB, 10 KiB) copied, 0.0170831 s, 599 kB/s fail_loc=0x410 fail_loc=0x0 PASS 50 (9s) == sanityn test 51a: layout lock: refresh layout should work ========================================================== 20:33:23 (1788827603) 0+1 records in 0+1 records out 158 bytes copied, 0.0175887 s, 9.0 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 3690: kill: (218409) - No such process PASS 51a (6s) == sanityn test 51b: layout lock: glimpse should be able to restart if layout changed ========================================================== 20:33:29 (1788827609) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0136205 s, 75.2 kB/s fail_loc=0x1404 fail_val=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0396517 s, 26.4 MB/s 3145728 PASS 51b (15s) == sanityn test 51c: layout lock: IT_LAYOUT blocked and correct layout can be returned ========================================================== 20:33:44 (1788827624) fail_loc=0x172 Setting layout to have 2 stripes ... 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0161094 s, 63.6 kB/s PASS 51c (8s) == sanityn test 51d: layout lock: losing layout lock should clean up memory map region ========================================================== 20:33:52 (1788827632) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0293597 s, 35.7 MB/s Before revoking layout lock: 1024 KB mapped PASS 51d (5s) == sanityn test 51e: lfs getstripe does not break leases, part 2 ========================================================== 20:33:57 (1788827637) /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 2448 0x990 0x280000400 write lease(2) has applied. /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 2448 0x990 0x280000400 read lease(1) has applied. PASS 51e (5s) == sanityn test 54: rename locking ======================= 20:34:02 (1788827642) llite.lustre-ffff92ca0a087000.xattr_cache=1 llite.lustre-ffff92ca100da000.xattr_cache=1 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505012 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-07 20:34:03.000000000 -0400 Modify: 2026-09-07 20:34:03.000000000 -0400 Change: 2026-09-07 20:34:03.000000000 -0400 Birth: 2026-09-07 20:34:03.000000000 -0400 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000154 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505015 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-07 20:34:09.000000000 -0400 Modify: 2026-09-07 20:34:09.000000000 -0400 Change: 2026-09-07 20:34:09.000000000 -0400 Birth: 2026-09-07 20:34:09.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505018 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-07 20:34:15.000000000 -0400 Modify: 2026-09-07 20:34:15.000000000 -0400 Change: 2026-09-07 20:34:15.000000000 -0400 Birth: 2026-09-07 20:34:15.000000000 -0400 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505017 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-07 20:34:14.000000000 -0400 Modify: 2026-09-07 20:34:15.000000000 -0400 Change: 2026-09-07 20:34:15.000000000 -0400 Birth: 2026-09-07 20:34:15.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000154 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505021 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-07 20:34:20.000000000 -0400 Modify: 2026-09-07 20:34:21.000000000 -0400 Change: 2026-09-07 20:34:21.000000000 -0400 Birth: 2026-09-07 20:34:20.000000000 -0400 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505020 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-07 20:34:20.000000000 -0400 Modify: 2026-09-07 20:34:21.000000000 -0400 Change: 2026-09-07 20:34:21.000000000 -0400 Birth: 2026-09-07 20:34:20.000000000 -0400 PASS 54 (26s) == sanityn test 55a: rename vs unlink target dir ========= 20:34:28 (1788827668) fail_loc=0x80000156 mv: cannot move '/mnt/lustre/d55a.sanityn/d1/d2' to '/mnt/lustre/d55a.sanityn/d3/d2': No such file or directory PASS 55a (10s) == sanityn test 55b: rename vs unlink source dir ========= 20:34:38 (1788827678) fail_loc=0x80000156 mv: cannot stat '/mnt/lustre/d55b.sanityn/d1/d2': No such file or directory PASS 55b (13s) == sanityn test 55c: rename vs unlink orphan target dir == 20:34:51 (1788827691) fail_loc=0x156 mv: cannot move '/mnt/lustre/d55c.sanityn/d1/d2' to '/mnt/lustre/d55c.sanityn/d3/d2': No such file or directory PASS 55c (23s) == sanityn test 55d: rename file vs link ================= 20:35:14 (1788827714) fail_loc=0x155 mv: '/mnt/lustre/d55d.sanityn/f1' and '/mnt/lustre/d55d.sanityn/d55d.sanityn/f1' are the same file PASS 55d (20s) == sanityn test 55e: rename race AB/BA under the same parent dir ========================================================== 20:35:34 (1788827734) SKIP: sanityn test_55e needs >= 2 MDTs SKIP 55e (4s) == sanityn test 55f: rename: (P1/A -> P2/B) race with (P2/B -> P1/A) ========================================================== 20:35:38 (1788827738) fail_loc=0x155 We survived after P1A_P2B/P2B_P1A race test fail_loc=0x0 PASS 55f (21s) == sanityn test 55g: rename: race with trylock =========== 20:35:59 (1788827759) mdt.lustre-MDT0000.enable_rename_trylock=1 fail_loc=0x155 mv: cannot move '/mnt/lustre/d55g.sanityn/a1/a2' to a subdirectory of itself, '/mnt/lustre/d55g.sanityn/b1/b2/b3/b4/a2' We survived race with trylock test fail_loc=0x0 PASS 55g (21s) == sanityn test 56a: test llverdev with single large stripe ========================================================== 20:36:20 (1788827780) run_llverdev /mnt/lustre/f56a.sanityn -p -s 16000000 -c 4k llverdev: /mnt/lustre/f56a.sanityn is 16777216000000 bytes (15625 GB) in size timestamp: 1788827782 chunksize: 4096 size: 16777216000000 write offset: 16384000000kB 0.709757 MB/s read offset: 16384000000kB inf MB/s llverdev: data verified successfully PASS 56a (93s) == sanityn test 56b: test llverdev and partial verify of wide stripe file ========================================================== 20:37:53 (1788827873) run_llverdev /mnt/lustre/f56b.sanityn -p -s 32000000 -c 4k llverdev: /mnt/lustre/f56b.sanityn is 33554432000000 bytes (31250 GB) in size timestamp: 1788827876 chunksize: 4096 size: 33554432000000 write offset: 32768000000kB 1.47077 MB/s read offset: 32768000000kB inf MB/s llverdev: data verified successfully PASS 56b (92s) == sanityn test 60: Verify data_version behaviour ======== 20:39:25 (1788827965) PASS 60 (8s) == sanityn test 70a: cd directory && rm directory ======== 20:39:33 (1788827973) PASS 70a (5s) == sanityn test 70b: remove files after calling rm_entry ========================================================== 20:39:39 (1788827979) PASS 70b (7s) == sanityn test 71a: correct file map just after write operation is finished ========================================================== 20:39:45 (1788827985) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0101036 s, 6.5 MB/s SKIP: sanityn test_71a ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71a (3s) == sanityn test 71b: check fiemap support for stripecount > 1 ========================================================== 20:39:48 (1788827988) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0171213 s, 3.8 MB/s SKIP: sanityn test_71b ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71b (3s) == sanityn test 71c: check FIEMAP_EXTENT_LAST flag with different extents number ========================================================== 20:39:51 (1788827991) SKIP: sanityn test_71c support only ldiskfs ost SKIP 71c (2s) == sanityn test 71d: fiemap corruption test with fm_extent_count=0 ========================================================== 20:39:54 (1788827994) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.00979659 s, 6.7 MB/s SKIP: sanityn test_71d ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71d (2s) == sanityn test 72: getxattr/setxattr cache should be consistent between nodes ========================================================== 20:39:56 (1788827996) llite.lustre-ffff92ca0a087000.xattr_cache=1 llite.lustre-ffff92ca100da000.xattr_cache=1 getfattr: Removing leading '/' from absolute path names user.attr1="value1" getfattr: Removing leading '/' from absolute path names user.attr1="value2" getfattr: Removing leading '/' from absolute path names getfattr: Removing leading '/' from absolute path names b362740ad371eae580e7f85452e8830b - 870e072da1ebfc20ea8c0cd3d6bdf3b6 - PASS 72 (5s) == sanityn test 73: getxattr should not cause xattr lock cancellation ========================================================== 20:40:01 (1788828001) llite.lustre-ffff92ca0a087000.xattr_cache=1 llite.lustre-ffff92ca100da000.xattr_cache=1 getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" PASS 73 (6s) == sanityn test 74: flock deadlock: different mounts ======================================================================== 20:40:07 (1788828007) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0134997 s, 75.9 kB/s 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00805737 s, 127 kB/s 231213: get lock2 231214: get lock1 231213: done 231213: get lock1 231214: done 231214: get lock2 fcntl cmd 7 failed: Resource deadlock avoided 231213: cannot get lock1: Resource deadlock avoided 231214: done 231214: exit rc=0 231213: done 231213: exit rc=0 PASS 74 (9s) == sanityn test 75: osc: upcall after unuse lock============================================================================= 20:40:16 (1788828016) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0775766 s, 27.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0388139 s, 27.0 MB/s fail_loc=0x31d /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 4373: 231795 Killed stat -c %s $DIR1/$tfile fail_loc=0 PASS 75 (11s) == sanityn test 76: Verify MDT open_files listing ======== 20:40:27 (1788828027) opening files: ........ closing files: ........ PASS 76 (60s) == sanityn test 77a: check FIFO NRS policy =============== 20:41:28 (1788828088) ost.OSS.ost.nrs_policies=fifo ost.OSS.ost_create.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_out.nrs_policies=fifo ost.OSS.ost_seq.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.309511 s, 54.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0453408 s, 23.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0479353 s, 21.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148029 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155338 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177798 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17775 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176835 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147922 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201794 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.233296 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190994 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242252 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23454 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205575 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193682 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213875 s, 4.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff92ca100da000.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff92ca100da000.lock_unused_count=7 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.03509 s, 29.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0394709 s, 26.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0409677 s, 25.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0615436 s, 17.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0467977 s, 22.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0625853 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.052503 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.055089 s, 19.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0574548 s, 18.3 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0490928 s, 21.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0512844 s, 20.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0516022 s, 20.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0549453 s, 19.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0559847 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.070432 s, 14.9 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0619953 s, 16.9 MB/s removed '/mnt/lustre/d77a.sanityn/nrs_w_oleg407-client.virtnet' removed '/mnt/lustre/d77a.sanityn/nrs_r_oleg407-client.virtnet' removed directory '/mnt/lustre/d77a.sanityn' PASS 77a (8s) == sanityn test 77b: check CRR-N NRS policy ============== 20:41:36 (1788828096) ost.OSS.ost.nrs_policies=crrn ost.OSS.ost_create.nrs_policies=crrn ost.OSS.ost_io.nrs_policies=crrn ost.OSS.ost_out.nrs_policies=crrn ost.OSS.ost_seq.nrs_policies=crrn ost.OSS.ost.nrs_crrn_quantum=1 ost.OSS.ost_create.nrs_crrn_quantum=1 ost.OSS.ost_io.nrs_crrn_quantum=1 ost.OSS.ost_out.nrs_crrn_quantum=1 ost.OSS.ost_seq.nrs_crrn_quantum=1 policy: crr-n, crrn_quantum 1 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.447077 s, 37.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0469098 s, 22.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115267 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143249 s, 7.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149819 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165645 s, 6.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181066 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198228 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164391 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168624 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211201 s, 5.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162684 s, 6.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169554 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172675 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.219714 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185511 s, 5.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff92ca100da000.lock_unused_count=5 ldlm.namespaces.lustre-OST0001-osc-ffff92ca100da000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13426 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0577364 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0778882 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0341047 s, 30.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0770231 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0452621 s, 23.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0472785 s, 22.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0368771 s, 28.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.05777 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0631398 s, 16.6 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0647586 s, 16.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0830747 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0653321 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0717002 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0699363 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0648385 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0846856 s, 12.4 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg407-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg407-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost.nrs_crrn_quantum=64 ost.OSS.ost_create.nrs_crrn_quantum=64 ost.OSS.ost_io.nrs_crrn_quantum=64 ost.OSS.ost_out.nrs_crrn_quantum=64 ost.OSS.ost_seq.nrs_crrn_quantum=64 policy: crr-n, crrn_quantum 64 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.333636 s, 50.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0665403 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0853584 s, 12.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0690466 s, 15.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0628558 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0974992 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.079709 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0977372 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0984271 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100155 s, 10.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff92ca100da000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171174 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206023 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.230594 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.257111 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23687 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.238533 s, 4.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0233491 s, 44.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0348962 s, 30.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0767336 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0530882 s, 19.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0407192 s, 25.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0178532 s, 58.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.066185 s, 15.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0403244 s, 26.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.312338 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0359685 s, 29.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0530906 s, 19.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00767695 s, 137 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0288086 s, 36.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0409591 s, 25.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0104478 s, 100 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00549909 s, 191 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00925547 s, 113 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff92ca100da000.lock_unused_count=8 removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg407-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg407-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77b (10s) == sanityn test 77c: check ORR NRS policy ================ 20:41:46 (1788828106) ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_orr_quantum=1 ost.OSS.ost_io.nrs_orr_offset_type=physical ost.OSS.ost_io.nrs_orr_supported=reads policy: orr, orr_quantum 1, orr_offset_type physical, orr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.365648 s, 45.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0835227 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111887 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102509 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0987605 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134679 s, 7.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103942 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122517 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122385 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145654 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167784 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148274 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159136 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198528 s, 5.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15552 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192183 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.245236 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0399552 s, 26.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0465195 s, 22.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0616006 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0889126 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0568077 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.057216 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0525089 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0520072 s, 20.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106814 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0587106 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0781184 s, 13.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0928283 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0618499 s, 17.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0592921 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0888562 s, 11.8 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff92ca100da000.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0801322 s, 13.1 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg407-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg407-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=writes ost.OSS.ost_io.nrs_orr_quantum=64 policy: orr, orr_quantum 64, orr_offset_type physical, orr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.40858 s, 41.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0473412 s, 22.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154891 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133905 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148081 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124031 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205605 s, 5.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174488 s, 6.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15232 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15542 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196938 s, 5.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff92ca100da000.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167807 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.248182 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0868748 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.249346 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.231897 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.226094 s, 4.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff92ca100da000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff92ca100da000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0592223 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0598217 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0596851 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0535979 s, 19.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.060734 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0612723 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0424954 s, 24.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.054693 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0549592 s, 19.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0469336 s, 22.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0427489 s, 24.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0567091 s, 18.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0585077 s, 17.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0617197 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0561961 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0632824 s, 16.6 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg407-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg407-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=reads_and_writes ost.OSS.ost_io.nrs_orr_offset_type=logical policy: orr, orr_quantum 64, orr_offset_type logical, orr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.414448 s, 40.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0822739 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0760302 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137293 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120178 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163987 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200209 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195068 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.224881 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.224943 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16979 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197792 s, 5.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff92ca100da000.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.230046 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.245733 s, 4.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215258 s, 4.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221355 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.273222 s, 3.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165674 s, 63.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.029554 s, 35.5 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0448201 s, 23.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0211408 s, 49.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165633 s, 63.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00574766 s, 182 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00736431 s, 142 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00834845 s, 126 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00728298 s, 144 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00460967 s, 227 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.031124 s, 33.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00267853 s, 391 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.005846 s, 179 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0316806 s, 33.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.038034 s, 27.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0196631 s, 53.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff92ca100da000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff92ca100da000.lock_unused_count=2 removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg407-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg407-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77c (13s) == sanityn test 77d: check TRR nrs policy ================ 20:41:59 (1788828119) ost.OSS.ost_io.nrs_policies=trr ost.OSS.ost_io.nrs_trr_quantum=1 ost.OSS.ost_io.nrs_trr_offset_type=physical ost.OSS.ost_io.nrs_trr_supported=reads policy: trr, trr_quantum 1, trr_offset_type physical, trr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.271645 s, 61.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0827605 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0940407 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0912143 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.085166 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111532 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125448 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104431 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103731 s, 10.1 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12691 s, 8.3 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff92ca100da000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.233109 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.245306 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.224619 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.246783 s, 4.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.264248 s, 4.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242937 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.298928 s, 3.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff92ca100da000.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0175551 s, 59.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0810927 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00394498 s, 266 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00413198 s, 254 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00257312 s, 408 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00228383 s, 459 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123168 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00625919 s, 168 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00606928 s, 173 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00411356 s, 255 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0690572 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100135 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112049 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0507831 s, 20.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0265084 s, 39.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0547519 s, 19.2 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg407-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg407-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=writes ost.OSS.ost_io.nrs_trr_quantum=64 policy: trr, trr_quantum 64, trr_offset_type physical, trr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.256726 s, 65.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0611172 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0649886 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137021 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175017 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184717 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183543 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197174 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178356 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186289 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19745 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.214728 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200291 s, 5.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21731 s, 4.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202575 s, 5.2 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.217314 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.243709 s, 4.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff92ca100da000.lock_unused_count=3 ldlm.namespaces.lustre-OST0001-osc-ffff92ca100da000.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0376537 s, 27.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0416553 s, 25.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0044876 s, 234 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00433765 s, 242 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0352123 s, 29.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0352118 s, 29.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0432 s, 24.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0378001 s, 27.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0504742 s, 20.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00322809 s, 325 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0407205 s, 25.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0287677 s, 36.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0201486 s, 52.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0174661 s, 60.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0123903 s, 84.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170504 s, 61.5 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg407-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg407-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=reads_and_writes ost.OSS.ost_io.nrs_trr_offset_type=logical policy: trr, trr_quantum 64, trr_offset_type logical, trr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.335279 s, 50.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0673376 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0840877 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0870074 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0877391 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0839247 s, 12.5 MB/s 1+0 records in 1+0 records out ldlm.namespaces.lustre-OST0001-osc-ffff92ca100da000.lock_unused_count=1 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180125 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157196 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100039 s, 10.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163631 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107069 s, 9.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159111 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201838 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199491 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207487 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177136 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208674 s, 5.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff92ca100da000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff92ca100da000.lock_unused_count=4 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0323167 s, 32.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0338369 s, 31.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0328326 s, 31.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0457499 s, 22.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0347399 s, 30.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0351603 s, 29.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0410639 s, 25.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0406332 s, 25.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0401754 s, 26.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.062808 s, 16.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0456008 s, 23.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0616318 s, 17.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.051179 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0702852 s, 14.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0512855 s, 20.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0640929 s, 16.4 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg407-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg407-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77d (12s) == sanityn test 77e: check TBF NID nrs policy ============ 20:42:11 (1788828131) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_tbf_rule=start localhost nid={0@lo} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start clients nid={192.168.204.7@tcp} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start others nid={*.*.*.*@tcp} rate=50 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.406102 s, 41.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0670423 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0921897 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0782441 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173329 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1594 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163653 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.219377 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192123 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209135 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190061 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.264205 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.284792 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205501 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.293498 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.281825 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.365965 s, 2.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff92ca100da000.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163203 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11632 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148159 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153823 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154117 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157609 s, 6.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159515 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177874 s, 5.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162744 s, 6.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166626 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162067 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162838 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15205 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171235 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171582 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189411 s, 5.5 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg407-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg407-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change localhost rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change clients rate=101 ost.OSS.ost_io.nrs_tbf_rule=change others rate=51 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.337102 s, 49.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0589582 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0832588 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129119 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16937 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191033 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216117 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222454 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.2136 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.252134 s, 4.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.225588 s, 4.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.241578 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.281014 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.2317 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.283491 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.28696 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106789 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107083 s, 9.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0357567 s, 29.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0677678 s, 15.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0900321 s, 11.6 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0900023 s, 11.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0968985 s, 10.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0523114 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0550323 s, 19.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0425895 s, 24.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119599 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101063 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103908 s, 10.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0840368 s, 12.5 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0744334 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.311257 s, 3.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff92ca100da000.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff92ca100da000.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11013 s, 9.5 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg407-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg407-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop localhost ost.OSS.ost_io.nrs_tbf_rule=stop clients ost.OSS.ost_io.nrs_tbf_rule=stop others 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.329549 s, 50.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0431615 s, 24.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0486358 s, 21.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162312 s, 6.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147126 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168019 s, 6.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166132 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194115 s, 5.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178476 s, 5.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191513 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192717 s, 5.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201661 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166692 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190539 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161194 s, 6.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209811 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221751 s, 4.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff92ca100da000.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00379289 s, 276 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00417473 s, 251 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0888084 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.092752 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0829551 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0736377 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110476 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0107624 s, 97.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0598443 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0556466 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0481396 s, 21.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137722 s, 76.1 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0330182 s, 31.8 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0234302 s, 44.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0420607 s, 24.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.03062 s, 34.2 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg407-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg407-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.315102 s, 53.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0469855 s, 22.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0608204 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0632731 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123899 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119341 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107511 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151835 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144136 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184094 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141914 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143869 s, 7.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff92ca100da000.lock_unused_count=5 ldlm.namespaces.lustre-OST0001-osc-ffff92ca100da000.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143459 s, 7.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198701 s, 5.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162958 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.217646 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178944 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0226016 s, 46.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.05471 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0562847 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.035199 s, 29.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.044103 s, 23.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0789043 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0746573 s, 14.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0810151 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0516143 s, 20.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0808242 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0766798 s, 13.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0625785 s, 16.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0509128 s, 20.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0789758 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0527851 s, 19.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0536305 s, 19.6 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg407-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg407-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' PASS 77e (17s) == sanityn test 77f: check TBF JobID nrs policy ========== 20:42:28 (1788828148) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start runas jobid={iozone.500 dd.500 tiotest.500} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start iozone_runas jobid={iozone.500} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg407-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.401991 s, 41.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0498472 s, 21.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0671071 s, 15.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0601956 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151564 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191961 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.273056 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.278745 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.290039 s, 3.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.275399 s, 3.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.24401 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.297744 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.28338 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.254565 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.26235 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.294244 s, 3.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff92ca100da000.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff92ca100da000.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.34035 s, 3.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00251885 s, 416 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0459421 s, 22.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0102184 s, 103 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00266046 s, 394 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152929 s, 68.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.072278 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00631042 s, 166 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0841525 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0670333 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.057204 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00758706 s, 138 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119318 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0717228 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0614867 s, 17.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0746014 s, 14.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0759802 s, 13.8 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg407-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg407-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change runas rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change iozone_runas rate=101 ost.OSS.ost_io.nrs_tbf_rule=change dd_runas rate=51 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg407-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.354945 s, 47.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103575 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0789942 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.058621 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158167 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166235 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.219944 s, 4.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.241217 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19895 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.260149 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213285 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.269055 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227392 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.271644 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.255469 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.258044 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.291608 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00673973 s, 156 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0529074 s, 19.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0646461 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0333647 s, 31.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0375088 s, 28.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0735791 s, 14.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0897277 s, 11.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0373985 s, 28.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0941754 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109195 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137883 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0804042 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11424 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141502 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0192639 s, 54.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00952622 s, 110 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg407-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg407-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop runas ost.OSS.ost_io.nrs_tbf_rule=stop iozone_runas ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg407-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.378607 s, 44.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0879608 s, 11.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0978636 s, 10.7 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0851837 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0725507 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127721 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0850996 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168334 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116774 s, 9.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112376 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127509 s, 8.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124824 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13526 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165323 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112644 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186929 s, 5.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff92ca100da000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff92ca100da000.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117005 s, 9.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0300228 s, 34.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00679131 s, 154 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00839375 s, 125 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00290464 s, 361 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0486438 s, 21.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00583705 s, 180 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00602965 s, 174 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0454084 s, 23.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0605044 s, 17.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0703141 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0570412 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0704119 s, 14.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149791 s, 70.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0290301 s, 36.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0828753 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0106728 s, 98.2 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg407-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg407-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_policies=fifo running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg407-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.362612 s, 46.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101625 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123103 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.228267 s, 4.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205018 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.252212 s, 4.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.285207 s, 3.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.263475 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.288423 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.288375 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.295448 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.324972 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.283446 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.357338 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.360139 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.28985 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.402115 s, 2.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0107145 s, 97.9 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0669277 s, 15.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0315155 s, 33.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0557388 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0426355 s, 24.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0838728 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0448267 s, 23.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0490047 s, 21.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0842386 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0485772 s, 21.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0544729 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0583946 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0569868 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0618639 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00493179 s, 213 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff92ca100da000.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff92ca100da000.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015662 s, 67.0 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg407-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg407-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' PASS 77f (19s) == sanityn test 77g: Change TBF type directly ============ 20:42:47 (1788828167) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 ost.OSS.ost_io.nrs_policies=fifo PASS 77g (8s) == sanityn test 77h: Wrong policy name should report error, not LBUG ========================================================== 20:42:55 (1788828175) oleg407-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=abc: No such file or directory oleg407-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='abc': No such file or directory pdsh@oleg407-client: oleg407-server: ssh exited with exit code 2 oleg407-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc: Invalid argument oleg407-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc': Invalid argument pdsh@oleg407-client: oleg407-server: ssh exited with exit code 22 oleg407-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf reg abc: Invalid argument oleg407-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf reg abc': Invalid argument pdsh@oleg407-client: oleg407-server: ssh exited with exit code 22 oleg407-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc efg: Invalid argument oleg407-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc efg': Invalid argument pdsh@oleg407-client: oleg407-server: ssh exited with exit code 22 PASS 77h (8s) == sanityn test 77i: Change rank of TBF rule ============= 20:43:03 (1788828183) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start before jobid={jobid} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start after jobid={jobid} rate=1000 rank=default ost.OSS.ost_io.nrs_tbf_rule=start target jobid={jobid} rate=1000 rank=after Move before itself ost.OSS.ost_io.nrs_tbf_rule=change target rank=target Move to higher rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=before Move to lower rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=after Move before default ost.OSS.ost_io.nrs_tbf_rule=change target rank=default ost.OSS.ost_io.nrs_policies=fifo PASS 77i (18s) == sanityn test 77j: check TBF-OPCode NRS policy ========= 20:43:21 (1788828201) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf opcode ost.OSS.ost_io.nrs_tbf_rule=start ost_r opcode={ost_read} rate=5 ost.OSS.ost_io.nrs_tbf_rule=start ost_w opcode={ost_write} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.463286 s, 36.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0486603 s, 21.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0578325 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0591971 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0611886 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0602565 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172137 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211574 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209072 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195511 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.246848 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.247566 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.255333 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.271656 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.229766 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.271873 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.287283 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0407868 s, 25.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0326001 s, 32.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0483741 s, 21.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0551836 s, 19.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0632089 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0398491 s, 26.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0415345 s, 25.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0414705 s, 25.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0493379 s, 21.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0544423 s, 19.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0715835 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.064747 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0661939 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0606614 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0685126 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0905912 s, 11.6 MB/s removed '/mnt/lustre/d77j.sanityn/nrs_r_oleg407-client.virtnet' removed '/mnt/lustre/d77j.sanityn/nrs_w_oleg407-client.virtnet' removed directory '/mnt/lustre/d77j.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 5, bs: 1048576 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.14509 s, 14.7 MB/s Write runtime is 8 s, speed is 12.500000 vs. 20 Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.6001 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 vs. 5 cleanup_tbf /mnt/lustre/d77j.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_r ost.OSS.ost_io.nrs_tbf_rule=stop ost_w ost.OSS.ost_io.nrs_policies=fifo PASS 77j (60s) == sanityn test 77ja: check TBF-UID/GID NRS policy ======= 20:44:21 (1788828261) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg407-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg407-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg407-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg407-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg407-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.27355 s, 13.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0550382 s, 19.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.051387 s, 20.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117696 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11489 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12086 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113475 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0789353 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0604816 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148318 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143368 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135941 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123997 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16068 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209905 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197228 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150215 s, 7.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0644164 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0639944 s, 16.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0725581 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0713549 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0789083 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0741539 s, 14.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0861961 s, 12.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0887252 s, 11.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0948677 s, 11.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0981051 s, 10.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0886491 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0879861 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0867157 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0869729 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10231 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112825 s, 9.3 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg407-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg407-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5, bs: 1048576 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1048576] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9256 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 vs. 5 Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1048576] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9532 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 vs. 5 cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg407-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg407-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg407-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg407-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg407-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.27516 s, 13.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0869206 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104691 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131017 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133471 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102487 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144207 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162424 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138693 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148991 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138383 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148216 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136098 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.204979 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154319 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207717 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206982 s, 5.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0715832 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0399871 s, 26.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0533666 s, 19.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0690433 s, 15.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0561776 s, 18.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0908385 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0627633 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0557578 s, 18.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0690037 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0695233 s, 15.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0845877 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0532066 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0968755 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0505396 s, 20.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0895314 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0646972 s, 16.2 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg407-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg407-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5, bs: 1048576 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1048576] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.925 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 vs. 5 Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1048576] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9208 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 vs. 5 cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77ja (148s) == sanityn test 77jb: check TBF-UID/GID NRS policy on files that don't belong to us ========================================================== 20:46:49 (1788828409) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg407-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg407-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg407-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg407-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg407-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.26821 s, 13.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135424 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126624 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148214 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170138 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142248 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193172 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198524 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200671 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223767 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.218815 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.226576 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200729 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152111 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.266188 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221694 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216311 s, 4.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0585484 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0643605 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0823826 s, 12.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0779603 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.065182 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0427025 s, 24.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0870028 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.063199 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0913225 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0734485 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0874389 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0711073 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0679764 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117532 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0845357 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0874835 s, 12.0 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg407-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg407-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5, bs: 1048576 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1048576] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.848 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 vs. 5 Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1048576] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9075 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 vs. 5 cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg407-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg407-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg407-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg407-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg407-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.2824 s, 13.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0412147 s, 25.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0623584 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102494 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.068147 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0679123 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0660731 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0560394 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0955805 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110655 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164926 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179443 s, 5.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.231337 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186375 s, 5.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213444 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.240172 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.247775 s, 4.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0747663 s, 14.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.04467 s, 23.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0797061 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0507225 s, 20.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0490893 s, 21.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0849942 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0841493 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0835422 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0549881 s, 19.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0910468 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0600273 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0644264 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0665061 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0892243 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0739966 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0864982 s, 12.1 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg407-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg407-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5, bs: 1048576 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1048576] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9152 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 vs. 5 Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1048576] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8993 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 vs. 5 cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77jb (148s) == sanityn test 77k: check TBF policy with UID/GID/JobID/OPCode expression ========================================================== 20:49:17 (1788828557) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode+nid+uid+gid ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg407-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.444778 s, 37.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0400911 s, 26.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0676022 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0580192 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0517431 s, 20.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0659317 s, 15.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183469 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188755 s, 5.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162282 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148387 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193056 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192655 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203008 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15709 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202558 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165309 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150598 s, 7.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0417372 s, 25.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0534205 s, 19.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0458581 s, 22.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0387416 s, 27.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0300408 s, 34.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.052093 s, 20.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0454943 s, 23.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0384716 s, 27.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0518407 s, 20.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0487923 s, 21.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0364024 s, 28.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0594791 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0464594 s, 22.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0534847 s, 19.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0454093 s, 23.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0529511 s, 19.8 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg407-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg407-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10, bs: 1048576 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1048576] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.88838 s, 17.8 MB/s Write runtime is 7 s, speed is 14.285714 vs. 20 Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1048576] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.86388 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 vs. 10 cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.204.7@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.204.7@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.371703 s, 45.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0474504 s, 22.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0538001 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10727 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0988327 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149453 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160136 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11183 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172273 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208782 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203983 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207255 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189195 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202302 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202436 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.218945 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213978 s, 4.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0538874 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0836013 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0872527 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0794842 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0939013 s, 11.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0881495 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0939844 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0797818 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0960574 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0895515 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0924367 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0788838 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0881143 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0946956 s, 11.1 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105212 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108861 s, 9.6 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg407-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg407-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10, bs: 1048576 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.47729 s, 16.2 MB/s Write runtime is 7 s, speed is 14.285714 vs. 20 Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.85996 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 vs. 10 cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.204.7@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg407-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.388532 s, 43.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0669945 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0677546 s, 15.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151926 s, 6.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175364 s, 6.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16604 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141594 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195158 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176688 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198913 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159692 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177132 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203353 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213838 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179368 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1728 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198496 s, 5.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.043769 s, 24.0 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0360283 s, 29.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.02642 s, 39.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0322736 s, 32.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0286794 s, 36.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0349861 s, 30.0 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0374505 s, 28.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.039098 s, 26.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0400631 s, 26.2 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0434642 s, 24.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0676175 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.043871 s, 23.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0743405 s, 14.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0487966 s, 21.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.054096 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0550211 s, 19.1 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg407-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg407-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20, bs: 1048576 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1048576] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.46083 s, 16.2 MB/s Write runtime is 7 s, speed is 14.285714 vs. 20 Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1048576] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.02938 s, 20.8 MB/s Read runtime is 6 s, speed is 16.666666 vs. 20 cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg407-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.350067 s, 47.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0363186 s, 28.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0476133 s, 22.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122735 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124543 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141108 s, 7.4 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126229 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146597 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153763 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145058 s, 7.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169015 s, 6.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168133 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142077 s, 7.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187328 s, 5.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171017 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194359 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193925 s, 5.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0543536 s, 19.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.021095 s, 49.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0321384 s, 32.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0509864 s, 20.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.026826 s, 39.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0542904 s, 19.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0312801 s, 33.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0339321 s, 30.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0316997 s, 33.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.060329 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0540787 s, 19.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0378886 s, 27.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0393287 s, 26.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0377492 s, 27.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0591298 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0454645 s, 23.1 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg407-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg407-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10, bs: 1048576 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1048576] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0343 s, 10.4 MB/s Write runtime is 12 s, speed is 8.333333 vs. 10 Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1048576] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.85177 s, 10.6 MB/s Read runtime is 10 s, speed is 10.000000 vs. 10 cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10, bs: 1048576 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.391 s, 16.4 MB/s Write runtime is 8 s, speed is 12.500000 vs. 20 Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.85641 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 vs. 10 cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg407-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.308457 s, 54.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0420361 s, 24.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0436447 s, 24.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0468962 s, 22.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120819 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113779 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146447 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126433 s, 8.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137824 s, 7.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155102 s, 6.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169568 s, 6.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174349 s, 6.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156018 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162618 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160377 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165436 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154413 s, 6.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0353152 s, 29.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164281 s, 63.8 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0265886 s, 39.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0348261 s, 30.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0428183 s, 24.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0246402 s, 42.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0259773 s, 40.4 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0266525 s, 39.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0310932 s, 33.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0369732 s, 28.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0576367 s, 18.2 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0583398 s, 18.0 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.034476 s, 30.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0424824 s, 24.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0400385 s, 26.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0403437 s, 26.0 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg407-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg407-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5, bs: 1048576 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1048576] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.6745 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 vs. 5 Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1048576] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.5278 s, 5.4 MB/s Read runtime is 21 s, speed is 4.761904 vs. 5 cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_ug ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg407-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.410422 s, 40.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.025024 s, 41.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0577305 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0522347 s, 20.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0571868 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0642911 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150108 s, 7.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156003 s, 6.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132649 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152304 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171485 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116254 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171406 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175904 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183209 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158556 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172523 s, 6.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014273 s, 73.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0199098 s, 52.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139333 s, 75.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015933 s, 65.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0223425 s, 46.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0264768 s, 39.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0233359 s, 44.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0242045 s, 43.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0271258 s, 38.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.026315 s, 39.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0566703 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0287428 s, 36.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0330596 s, 31.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0334618 s, 31.3 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0332718 s, 31.5 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.033366 s, 31.4 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg407-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg407-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10, bs: 1048576 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1048576] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.88383 s, 17.8 MB/s Write runtime is 7 s, speed is 14.285714 vs. 20 Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1048576] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.82951 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 vs. 10 cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg407-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.563171 s, 29.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0403342 s, 26.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0371916 s, 28.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0510399 s, 20.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0490628 s, 21.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0667709 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0985698 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0969974 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128442 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125313 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123749 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130339 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194409 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209899 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211992 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.226609 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.25481 s, 4.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.046606 s, 22.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0455394 s, 23.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0299759 s, 35.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0421609 s, 24.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0247181 s, 42.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0371656 s, 28.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0350214 s, 29.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0608174 s, 17.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0761188 s, 13.8 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0404782 s, 25.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0863206 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0479484 s, 21.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0513364 s, 20.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0877928 s, 11.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0534296 s, 19.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0645035 s, 16.3 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg407-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg407-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10, bs: 1048576 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1048576] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.95501 s, 10.5 MB/s Write runtime is 11 s, speed is 9.090909 vs. 10 Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1048576] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.81867 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 vs. 10 cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10, bs: 1048576 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1048576] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0462 s, 10.4 MB/s Write runtime is 11 s, speed is 9.090909 vs. 20 Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1048576] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.87006 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 vs. 10 cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Cleanup TBF ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_policies=fifo PASS 77k (394s) == sanityn test 77kb: Check different granular TBF type combination: jobid+opcode ========================================================== 20:55:51 (1788828951) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/nrs_r_oleg407-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.355189 s, 47.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0450006 s, 23.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0463397 s, 22.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0529591 s, 19.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0598249 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113968 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113339 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112709 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117493 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0949798 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130255 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125272 s, 8.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114546 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147555 s, 7.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135533 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133911 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132245 s, 7.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0178572 s, 58.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170437 s, 61.5 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0247057 s, 42.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.024098 s, 43.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.02664 s, 39.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.028876 s, 36.3 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.028993 s, 36.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0397249 s, 26.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0455713 s, 23.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0313909 s, 33.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.031681 s, 33.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0327609 s, 32.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0334552 s, 31.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0345827 s, 30.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0357613 s, 29.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0352153 s, 29.8 MB/s removed '/mnt/lustre/d77kb.sanityn/nrs_w_oleg407-client.virtnet' removed '/mnt/lustre/d77kb.sanityn/nrs_r_oleg407-client.virtnet' removed directory '/mnt/lustre/d77kb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10, bs: 1048576 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/tbf] [bs=1048576] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.06922 s, 20.7 MB/s Write runtime is 6 s, speed is 16.666666 vs. 20 Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kb.sanityn/tbf] [of=/dev/null] [bs=1048576] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.82481 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 vs. 10 cleanup_tbf /mnt/lustre/d77kb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Cleanup TBF ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77kb (46s) == sanityn test 77kc: Check different granular TBF type combination: nid+opcode ========================================================== 20:56:37 (1788828997) ost.OSS.ost_io.nrs_policies=tbf nid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.204.7@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.204.7@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.33289 s, 50.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0486834 s, 21.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0565397 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.061039 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0572728 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0722582 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0730492 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0655434 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.068356 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118576 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131343 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128104 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127572 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155584 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160744 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142736 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154035 s, 6.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0294021 s, 35.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0266748 s, 39.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0391581 s, 26.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0242324 s, 43.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0305133 s, 34.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0366557 s, 28.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0271814 s, 38.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0285804 s, 36.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0294341 s, 35.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.037109 s, 28.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0350434 s, 29.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0334071 s, 31.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0385228 s, 27.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0302065 s, 34.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0332374 s, 31.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.057331 s, 18.3 MB/s removed '/mnt/lustre/d77kc.sanityn/nrs_r_oleg407-client.virtnet' removed '/mnt/lustre/d77kc.sanityn/nrs_w_oleg407-client.virtnet' removed directory '/mnt/lustre/d77kc.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10, bs: 1048576 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.14078 s, 20.4 MB/s Write runtime is 6 s, speed is 16.666666 vs. 20 Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.8059 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 vs. 10 cleanup_tbf /mnt/lustre/d77kc.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Cleanup TBF ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77kc (54s) == sanityn test 77kd: Check different granular TBF type combination: nid+jobid ========================================================== 20:57:31 (1788829051) ost.OSS.ost_io.nrs_policies=tbf nid+jobid ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.204.7@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/nrs_r_oleg407-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.478713 s, 35.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0249837 s, 42.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0358692 s, 29.2 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0333519 s, 31.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0529051 s, 19.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0660875 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122335 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167616 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138508 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183061 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143266 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195116 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170593 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171474 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20854 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16721 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179321 s, 5.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.065274 s, 16.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0689841 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0361701 s, 29.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0288012 s, 36.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0268263 s, 39.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0478175 s, 21.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0392326 s, 26.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0541198 s, 19.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0397003 s, 26.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0557589 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.055679 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0915394 s, 11.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0445336 s, 23.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.044226 s, 23.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0462535 s, 22.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0550889 s, 19.0 MB/s removed '/mnt/lustre/d77kd.sanityn/nrs_w_oleg407-client.virtnet' removed '/mnt/lustre/d77kd.sanityn/nrs_r_oleg407-client.virtnet' removed directory '/mnt/lustre/d77kd.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20, bs: 1048576 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/tbf] [bs=1048576] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.05964 s, 14.9 MB/s Write runtime is 8 s, speed is 12.500000 vs. 20 Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kd.sanityn/tbf] [of=/dev/null] [bs=1048576] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.19756 s, 20.2 MB/s Read runtime is 7 s, speed is 14.285714 vs. 20 cleanup_tbf /mnt/lustre/d77kd.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Cleanup TBF ost.OSS.ost_io.nrs_tbf_rule=stop ext ost.OSS.ost_io.nrs_policies=fifo PASS 77kd (50s) == sanityn test 77ke: Check different granular TBF type combination: jobid+opcode ========================================================== 20:58:21 (1788829101) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/nrs_r_oleg407-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.561082 s, 29.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0741816 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0703337 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0660026 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0636255 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0742445 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124419 s, 8.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148195 s, 7.1 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165532 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146714 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112875 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152602 s, 6.9 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140212 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.26902 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.28187 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.283452 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.348388 s, 3.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0264454 s, 39.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0314321 s, 33.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0434253 s, 24.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0366728 s, 28.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0341178 s, 30.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0285509 s, 36.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0376358 s, 27.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0342815 s, 30.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0341807 s, 30.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0398922 s, 26.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0383013 s, 27.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0416416 s, 25.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0403349 s, 26.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0383207 s, 27.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0429857 s, 24.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0467346 s, 22.4 MB/s removed '/mnt/lustre/d77ke.sanityn/nrs_r_oleg407-client.virtnet' removed '/mnt/lustre/d77ke.sanityn/nrs_w_oleg407-client.virtnet' removed directory '/mnt/lustre/d77ke.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10, bs: 1048576 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/tbf] [bs=1048576] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.95064 s, 10.5 MB/s Write runtime is 11 s, speed is 9.090909 vs. 10 Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ke.sanityn/tbf] [of=/dev/null] [bs=1048576] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.84665 s, 10.6 MB/s Read runtime is 10 s, speed is 10.000000 vs. 10 cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10, bs: 1048576 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.4188 s, 19.4 MB/s Write runtime is 7 s, speed is 14.285714 vs. 20 Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.8248 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 vs. 10 cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Cleanup TBF ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_policies=fifo PASS 77ke (91s) == sanityn test 77kf: Check different granular TBF type combination: uid+gid ========================================================== 20:59:52 (1788829192) ost.OSS.ost_io.nrs_policies=tbf uid+gid ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/nrs_r_oleg407-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.943551 s, 17.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0559759 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0864764 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0537619 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.074137 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112337 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11221 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123462 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175887 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136945 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199462 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158572 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184302 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196668 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159096 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206213 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1932 s, 5.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168902 s, 62.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0195307 s, 53.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0373713 s, 28.1 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0254344 s, 41.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0384917 s, 27.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0288926 s, 36.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0306783 s, 34.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0234515 s, 44.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0253552 s, 41.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0270291 s, 38.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0269378 s, 38.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0281793 s, 37.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0528796 s, 19.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0324277 s, 32.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0372394 s, 28.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0375257 s, 27.9 MB/s removed '/mnt/lustre/d77kf.sanityn/nrs_r_oleg407-client.virtnet' removed '/mnt/lustre/d77kf.sanityn/nrs_w_oleg407-client.virtnet' removed directory '/mnt/lustre/d77kf.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5, bs: 1048576 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/tbf] [bs=1048576] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.6989 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 vs. 5 Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kf.sanityn/tbf] [of=/dev/null] [bs=1048576] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9554 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 vs. 5 cleanup_tbf /mnt/lustre/d77kf.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Cleanup TBF ost.OSS.ost_io.nrs_tbf_rule=stop ext_ug ost.OSS.ost_io.nrs_policies=fifo PASS 77kf (71s) == sanityn test 77kg: check different granular TBF type combination: uid+opcode ========================================================== 21:01:03 (1788829263) ost.OSS.ost_io.nrs_policies=tbf uid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg407-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.379343 s, 44.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0538971 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0520855 s, 20.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118251 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0869441 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0990019 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135118 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149681 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129137 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155594 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129426 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164271 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122079 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17571 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147275 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140469 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141791 s, 7.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0343873 s, 30.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0216754 s, 48.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0272487 s, 38.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0378122 s, 27.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0176103 s, 59.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0223782 s, 46.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0457028 s, 22.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0466528 s, 22.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0253688 s, 41.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0434461 s, 24.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0515271 s, 20.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0372847 s, 28.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0301014 s, 34.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0312414 s, 33.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0332661 s, 31.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0342092 s, 30.7 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg407-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg407-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10, bs: 1048576 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1048576] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.58506 s, 18.8 MB/s Write runtime is 6 s, speed is 16.666666 vs. 20 Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1048576] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.81767 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 vs. 10 cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg407-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.306557 s, 54.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0361264 s, 29.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0582196 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119638 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0997112 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0950888 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136478 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144895 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101528 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128637 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122129 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136156 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102448 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139233 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164932 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133852 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142173 s, 7.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0216625 s, 48.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.039096 s, 26.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0269544 s, 38.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0274204 s, 38.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0261185 s, 40.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0309913 s, 33.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0343904 s, 30.5 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0353053 s, 29.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0366217 s, 28.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0279261 s, 37.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0393152 s, 26.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00391811 s, 268 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0419006 s, 25.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0317266 s, 33.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0450916 s, 23.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0372749 s, 28.1 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg407-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg407-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10, bs: 1048576 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1048576] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.99659 s, 10.5 MB/s Write runtime is 11 s, speed is 9.090909 vs. 10 Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1048576] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.87403 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 vs. 10 cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10, bs: 1048576 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1048576] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.9752 s, 10.5 MB/s Write runtime is 11 s, speed is 9.090909 vs. 20 Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1048576] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.87945 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 vs. 10 cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Cleanup TBF ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_policies=fifo PASS 77kg (135s) == sanityn test 77kh: Verify that Project ID support for NRS TBF rule ========================================================== 21:03:18 (1788829398) using SAVE_PROJECT_SUPPORTED=0 Stopping clients: oleg407-client.virtnet /mnt/lustre (opts:) Stopping client oleg407-client.virtnet /mnt/lustre opts: Stopping clients: oleg407-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg407-client.virtnet /mnt/lustre2 opts: Stopping /mnt/lustre-mds1 (opts:-f) on oleg407-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg407-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg407-server sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime) proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) devtmpfs on /dev type devtmpfs (rw,nosuid,mode=755) securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime) tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev) devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000) tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755) tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755) cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd) pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime) bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700) cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio) cgroup on /sys/fs/cgroup/misc type cgroup (rw,nosuid,nodev,noexec,relatime,misc) cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory) cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer) cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio) cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb) cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices) cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids) cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event) cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset) cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma) cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct) none on /sys/kernel/tracing type tracefs (rw,relatime) configfs on /sys/kernel/config type configfs (rw,relatime) /dev/nbd0 on / type ext4 (ro,relatime,stripe=32) rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) mqueue on /dev/mqueue type mqueue (rw,relatime) hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M) debugfs on /sys/kernel/debug type debugfs (rw,relatime) none on /mnt type ramfs (rw,relatime) /dev/vda on /home/green/git/lustre-release type squashfs (ro,relatime) none on /var/lib/stateless/writable type tmpfs (rw,relatime) none on /var/cache/man type tmpfs (rw,relatime) none on /var/log type tmpfs (rw,relatime) none on /var/lib/dbus type tmpfs (rw,relatime) none on /tmp type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) none on /var/tmp type tmpfs (rw,relatime) none on /var/lib/NetworkManager type tmpfs (rw,relatime) none on /var/lib/systemd/random-seed type tmpfs (rw,relatime) none on /var/spool type tmpfs (rw,relatime) none on /var/lib/nfs type tmpfs (rw,relatime) none on /var/lib/gssproxy type tmpfs (rw,relatime) none on /var/lib/logrotate type tmpfs (rw,relatime) none on /etc type tmpfs (rw,relatime) none on /var/lib/rsyslog type tmpfs (rw,relatime) none on /root/.ssh type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) 192.168.200.253:/exports/state/oleg407-client.virtnet on /var/lib/stateless/state type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.204.7,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg407-client.virtnet/boot on /boot type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.204.7,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg407-client.virtnet/etc/kdump.conf on /etc/kdump.conf type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.204.7,local_lock=none,addr=192.168.200.253) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=378180k,mode=700) 192.168.200.253:/exports/testreports/69785/testresults/sanityn-zfs-rocky8.10_x86_64-rocky8.10_x86_64 on /tmp/testlogs type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.204.7,local_lock=none,addr=192.168.200.253) /dev/vda on /usr/sbin/mount.lustre type squashfs (ro,relatime) Checking servers environments Checking clients oleg407-client.virtnet environments Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg407-server' oleg407-server: oleg407-server.virtnet: executing load_modules_local oleg407-server: kmemleak disabled oleg407-server: Loading modules from /home/green/git/lustre-release/lustre oleg407-server: detected 4 online CPUs by sysfs oleg407-server: MODOPTS_LIBCFS= oleg407-server: Force libcfs to create 2 CPU partitions oleg407-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Setup mgs, mdt, osts Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg407-server: oleg407-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg407-client: oleg407-server: ssh exited with exit code 1 Started lustre-MDT0000 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg407-server: oleg407-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg407-client: oleg407-server: ssh exited with exit code 1 Started lustre-OST0000 Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg407-server: oleg407-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg407-client: oleg407-server: ssh exited with exit code 1 Started lustre-OST0001 Starting client: oleg407-client.virtnet: -o user_xattr,flock 192.168.204.107@tcp:/lustre /mnt/lustre Starting client oleg407-client.virtnet: -o user_xattr,flock 192.168.204.107@tcp:/lustre /mnt/lustre Started clients oleg407-client.virtnet: 192.168.204.107@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Starting client: oleg407-client.virtnet: -o user_xattr,flock 192.168.204.107@tcp:/lustre /mnt/lustre2 Starting client oleg407-client.virtnet: -o user_xattr,flock 192.168.204.107@tcp:/lustre /mnt/lustre2 Started clients oleg407-client.virtnet: 192.168.204.107@tcp:/lustre on /mnt/lustre2 type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff92ca056f8800.idle_timeout=debug osc.lustre-OST0000-osc-ffff92ca0a97c800.idle_timeout=debug osc.lustre-OST0001-osc-ffff92ca056f8800.idle_timeout=debug osc.lustre-OST0001-osc-ffff92ca0a97c800.idle_timeout=debug disable quota as required ost.OSS.ost_io.nrs_policies=tbf projid ost.OSS.ost_io.nrs_tbf_rule=start ext_proj_rw projid={100} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.347639 s, 48.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0456086 s, 23.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0408267 s, 25.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0865651 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137282 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147895 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143958 s, 7.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166964 s, 6.3 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172243 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146618 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163937 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154852 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180044 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178151 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176203 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191277 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213641 s, 4.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149353 s, 70.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0499934 s, 21.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.033129 s, 31.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00477374 s, 220 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00602793 s, 174 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00716103 s, 146 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0039065 s, 268 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0112816 s, 92.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0785865 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00770393 s, 136 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0766154 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0421972 s, 24.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00339286 s, 309 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01427 s, 73.5 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0574575 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0271078 s, 38.7 MB/s removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg407-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg407-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20, bs: 1048576 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.38272 s, 19.5 MB/s Write runtime is 6 s, speed is 16.666666 vs. 20 Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.97169 s, 21.1 MB/s Read runtime is 6 s, speed is 16.666666 vs. 20 cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Cleanup TBF ost.OSS.ost_io.nrs_tbf_rule=stop ext_proj_rw ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf projid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w projid={100}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r projid={100}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.328004 s, 51.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0389844 s, 26.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0430912 s, 24.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0209657 s, 50.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0756207 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0677518 s, 15.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0922366 s, 11.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113984 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0816814 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117033 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.069514 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0976725 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10264 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184046 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187881 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212914 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223151 s, 4.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156281 s, 67.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169647 s, 61.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.02238 s, 46.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0250054 s, 41.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163636 s, 64.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158012 s, 66.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0190959 s, 54.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165711 s, 63.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0184419 s, 56.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0215202 s, 48.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0240759 s, 43.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.024305 s, 43.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0216642 s, 48.4 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.024306 s, 43.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.02652 s, 39.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0321025 s, 32.7 MB/s removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg407-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg407-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10, bs: 1048576 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.24987 s, 20.0 MB/s Write runtime is 6 s, speed is 16.666666 vs. 20 Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.7947 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 vs. 10 cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Cleanup TBF ost.OSS.ost_io.nrs_tbf_rule=stop ext_proj_rw ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77kh (161s) == sanityn test 77ki: Add rule with unsupported TBF type should fail for generic TBF ========================================================== 21:05:59 (1788829559) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 oleg407-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_fail projid={100}&opcode={ost_write} rate=20: Invalid argument oleg407-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_fail projid={100}&opcode={ost_write} rate=20': Invalid argument pdsh@oleg407-client: oleg407-server: ssh exited with exit code 22 Cleanup TBF ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77ki (11s) == sanityn test 77kj: Verify nodemap support for NRS TBF rule ========================================================== 21:06:10 (1788829570) On MGS 192.168.204.107, active = nodemap.active=1 On MGS 192.168.204.107, TBF.id = nodemap.TBF.id=1 ost.OSS.ost_io.nrs_policies=tbf nodemap ost.OSS.ost_io.nrs_tbf_rule=start ext_nm_rw nodemap={TBF} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.361519 s, 46.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0466914 s, 22.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0471497 s, 22.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0931448 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12988 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145423 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138743 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161928 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166027 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140292 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131605 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146106 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171851 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178327 s, 5.9 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189673 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.229393 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.233648 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00486598 s, 215 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0195905 s, 53.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0210022 s, 49.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0330565 s, 31.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0512455 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0458992 s, 22.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0478756 s, 21.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0179239 s, 58.5 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0227715 s, 46.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00335725 s, 312 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0066249 s, 158 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00860353 s, 122 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0220783 s, 47.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0450265 s, 23.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138485 s, 75.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00818424 s, 128 MB/s removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg407-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg407-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20, bs: 1048576 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.18632 s, 20.2 MB/s Write runtime is 6 s, speed is 16.666666 vs. 20 Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.97605 s, 21.1 MB/s Read runtime is 6 s, speed is 16.666666 vs. 20 cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete On MGS 192.168.204.107, TBF.id = On MGS 192.168.204.107, active = nodemap.active=0 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.191974 s, 87.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.03184 s, 32.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0376569 s, 27.8 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff92ca056f8800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.085751 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0945309 s, 11.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118443 s, 8.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134399 s, 7.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111854 s, 9.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137494 s, 7.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127098 s, 8.3 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137634 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138565 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144742 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134618 s, 7.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137095 s, 7.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124593 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144078 s, 7.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0222472 s, 47.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128009 s, 81.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.010928 s, 96.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159097 s, 65.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00603746 s, 174 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0177582 s, 59.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0246318 s, 42.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0248799 s, 42.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0223947 s, 46.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00572383 s, 183 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0279136 s, 37.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00985031 s, 106 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0250491 s, 41.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0290415 s, 36.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00433708 s, 242 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff92ca056f8800.lock_unused_count=5 ldlm.namespaces.lustre-OST0001-osc-ffff92ca056f8800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0107653 s, 97.4 MB/s removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg407-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg407-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 1 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20, bs: 1048576 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 3.82243 s, 27.4 MB/s Write runtime is 5 s, speed is 20.000000 vs. 20 Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 1.31715 s, 79.6 MB/s Read runtime is 2 s, speed is 50.000000 vs. 20 read rate (50.000000) should exceed limit (20 * 1) cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Cleanup TBF ost.OSS.ost_io.nrs_tbf_rule=stop ext_nm_rw ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf nodemap+opcode oleg407-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_w nodemap={TBF}&opcode={ost_write} rate=20: Invalid argument oleg407-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_w nodemap={TBF}&opcode={ost_write} rate=20': Invalid argument oleg407-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_r nodemap={TBF}&opcode={ost_read} rate=10: Invalid argument oleg407-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_r nodemap={TBF}&opcode={ost_read} rate=10': Invalid argument pdsh@oleg407-client: oleg407-server: ssh exited with exit code 22 On MGS 192.168.204.107, active = nodemap.active=1 On MGS 192.168.204.107, TBF.id = nodemap.TBF.id=2 ost.OSS.ost_io.nrs_tbf_rule=start ext_w nodemap={TBF}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nodemap={TBF}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.308345 s, 54.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0665405 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12131 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132196 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104028 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143675 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147405 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16785 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152222 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150059 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133189 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216206 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208926 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.279463 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242023 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.283262 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.307634 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0296623 s, 35.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0280281 s, 37.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0301757 s, 34.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0329327 s, 31.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0317675 s, 33.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0334478 s, 31.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0330138 s, 31.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0342605 s, 30.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.039665 s, 26.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0387335 s, 27.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0573147 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0427236 s, 24.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.040665 s, 25.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0443443 s, 23.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0455738 s, 23.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0425675 s, 24.6 MB/s removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg407-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg407-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10, bs: 1048576 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.18509 s, 20.2 MB/s Write runtime is 6 s, speed is 16.666666 vs. 20 Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.81447 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 vs. 10 cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Cleanup TBF ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf nid classify=dynamic ost.OSS.ost_io.nrs_tbf_rule=start nmdyn nodemap={TBF} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.302002 s, 55.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0287129 s, 36.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0306231 s, 34.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0815405 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0790413 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0999907 s, 10.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0783774 s, 13.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10369 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0783757 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106846 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0946416 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102121 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0829027 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196517 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227162 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.26091 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.269317 s, 3.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016481 s, 63.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0223405 s, 46.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0237071 s, 44.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0301762 s, 34.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0343318 s, 30.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0224609 s, 46.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0227432 s, 46.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0239542 s, 43.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0287011 s, 36.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0299715 s, 35.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0276104 s, 38.0 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.027645 s, 37.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0321903 s, 32.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0287657 s, 36.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0303554 s, 34.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0311579 s, 33.7 MB/s removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg407-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg407-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20, bs: 1048576 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.01055 s, 20.9 MB/s Write runtime is 6 s, speed is 16.666666 vs. 20 Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.00064 s, 21.0 MB/s Read runtime is 6 s, speed is 16.666666 vs. 20 cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Cleanup TBF ost.OSS.ost_io.nrs_tbf_rule=stop nmdyn ost.OSS.ost_io.nrs_policies=fifo On MGS 192.168.204.107, TBF.id = On MGS 192.168.204.107, active = nodemap.active=0 Cleanup TBF ost.OSS.ost_io.nrs_policies=fifo Cleanup TBF ost.OSS.ost_io.nrs_policies=fifo Cleanup TBF ost.OSS.ost_io.nrs_policies=fifo On MGS 192.168.204.107, active = nodemap.active=0 PASS 77kj (180s) == sanityn test 77l: check the output of NRS policies for generic TBF ========================================================== 21:09:10 (1788829750) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_policies=fifo PASS 77l (5s) == sanityn test 77m: check NRS Delay slows write RPC processing ========================================================== 21:09:15 (1788829755) ost.OSS.ost_io.nrs_policies=delay ost.OSS.ost_io.nrs_delay_min=4 ost.OSS.ost_io.nrs_delay_max=4 ost.OSS.ost_io.nrs_delay_pct=100 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg407-client.virtnet] [bs=1M] [count=1] [oflag=direct] [conv=fdatasync] 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 3.34736 s, 313 kB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg407-client.virtnet] [bs=1M] [count=10] [oflag=direct] [conv=fdatasync] 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 40.0347 s, 262 kB/s ost.OSS.ost_io.nrs_policies=fifo PASS 77m (51s) == sanityn test 77n: check wildcard support for TBF JobID NRS policy ========================================================== 21:10:06 (1788829806) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={*.500} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.165749 s, 101 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0460459 s, 22.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108947 s, 9.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124487 s, 8.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125498 s, 8.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122356 s, 8.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135557 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109417 s, 9.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126906 s, 8.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117887 s, 8.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108274 s, 9.7 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120838 s, 8.7 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134083 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126756 s, 8.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120024 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103193 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101847 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01899 s, 55.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0240134 s, 43.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0189066 s, 55.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169641 s, 61.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0210587 s, 49.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0302298 s, 34.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.022001 s, 47.7 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0247181 s, 42.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.02221 s, 47.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0232587 s, 45.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0254867 s, 41.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.027819 s, 37.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0338576 s, 31.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0288486 s, 36.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0429096 s, 24.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0298083 s, 35.2 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff92ca056f8800.lock_unused_count=1 removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg407-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg407-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20, bs: 1048576 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77n.sanityn/tbf] [bs=1048576] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.96835 s, 21.1 MB/s Write runtime is 6 s, speed is 16.666666 vs. 20 Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77n.sanityn/tbf] [of=/dev/null] [bs=1048576] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.95935 s, 21.1 MB/s Read runtime is 6 s, speed is 16.666666 vs. 20 cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.*} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.199953 s, 83.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0311181 s, 33.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0324242 s, 32.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0910056 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104484 s, 10.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111044 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10322 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125715 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130926 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132199 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120733 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114492 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134338 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115235 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124317 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165679 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171885 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144503 s, 72.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0196847 s, 53.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0337757 s, 31.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0287243 s, 36.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0230331 s, 45.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0262391 s, 40.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0386367 s, 27.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0301865 s, 34.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.036026 s, 29.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0243297 s, 43.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0248499 s, 42.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0336009 s, 31.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0307048 s, 34.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0308257 s, 34.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0297965 s, 35.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0287825 s, 36.4 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg407-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg407-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20, bs: 1048576 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.0293 s, 20.8 MB/s Write runtime is 6 s, speed is 16.666666 vs. 20 Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.95898 s, 21.1 MB/s Read runtime is 6 s, speed is 16.666666 vs. 20 cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_policies=fifo PASS 77n (70s) == sanityn test 77o: Changing rank should not panic ====== 21:11:16 (1788829876) mds.MDS.mdt.nrs_policies=tbf nid mds.MDS.mdt.nrs_tbf_rule=start name nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=start name1 nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=change name1 rank=name mds.MDS.mdt.nrs_tbf_rule=stop name mds.MDS.mdt.nrs_policies=fifo PASS 77o (5s) == sanityn test 77q: Parallel TBF rule definitions should not panic ========================================================== 21:11:22 (1788829882) mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_policies=fifo PASS 77q (47s) == sanityn test 77p: Check validity of rule names for TBF policies ========================================================== 21:12:08 (1788829928) mds.MDS.mdt.nrs_policies=tbf oleg407-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_overflo uid={500} rate=500: File name too long oleg407-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_overflo uid={500} rate=500': File name too long pdsh@oleg407-client: oleg407-server: ssh exited with exit code 36 oleg407-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start uid={500} rate=500: Invalid argument oleg407-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start uid={500} rate=500': Invalid argument pdsh@oleg407-client: oleg407-server: ssh exited with exit code 22 oleg407-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_empty: Invalid argument oleg407-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_empty': Invalid argument pdsh@oleg407-client: oleg407-server: ssh exited with exit code 22 oleg407-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p@spec uid={500} rate=500: Invalid argument oleg407-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p@spec uid={500} rate=500': Invalid argument pdsh@oleg407-client: oleg407-server: ssh exited with exit code 22 oleg407-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p.spec uid={500} rate=500: Invalid argument oleg407-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p.spec uid={500} rate=500': Invalid argument pdsh@oleg407-client: oleg407-server: ssh exited with exit code 22 oleg407-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p~spec uid={500} rate=500: Invalid argument oleg407-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p~spec uid={500} rate=500': Invalid argument pdsh@oleg407-client: oleg407-server: ssh exited with exit code 22 oleg407-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p#spec uid={500} rate=500: Invalid argument oleg407-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p#spec uid={500} rate=500': Invalid argument pdsh@oleg407-client: oleg407-server: ssh exited with exit code 22 oleg407-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p/spec uid={500} rate=500: Invalid argument oleg407-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p/spec uid={500} rate=500': Invalid argument pdsh@oleg407-client: oleg407-server: ssh exited with exit code 22 oleg407-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p^spec uid={500} rate=500: Invalid argument oleg407-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p^spec uid={500} rate=500': Invalid argument pdsh@oleg407-client: oleg407-server: ssh exited with exit code 22 oleg407-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p%spec uid={500} rate=500: Invalid argument oleg407-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p%spec uid={500} rate=500': Invalid argument pdsh@oleg407-client: oleg407-server: ssh exited with exit code 22 oleg407-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p*spec uid={500} rate=500: Invalid argument oleg407-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p*spec uid={500} rate=500': Invalid argument pdsh@oleg407-client: oleg407-server: ssh exited with exit code 22 oleg407-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p;spec uid={500} rate=500: Invalid argument oleg407-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p;spec uid={500} rate=500': Invalid argument pdsh@oleg407-client: oleg407-server: ssh exited with exit code 22 oleg407-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p,spec uid={500} rate=500: Invalid argument oleg407-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p,spec uid={500} rate=500': Invalid argument pdsh@oleg407-client: oleg407-server: ssh exited with exit code 22 oleg407-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p?spec uid={500} rate=500: Invalid argument oleg407-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p?spec uid={500} rate=500': Invalid argument pdsh@oleg407-client: oleg407-server: ssh exited with exit code 22 oleg407-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77pspec uid={500} rate=500: Invalid argument oleg407-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p>spec uid={500} rate=500': Invalid argument pdsh@oleg407-client: oleg407-server: ssh exited with exit code 22 oleg407-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p:spec uid={500} rate=500: Invalid argument oleg407-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p:spec uid={500} rate=500': Invalid argument pdsh@oleg407-client: oleg407-server: ssh exited with exit code 22 oleg407-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p+spec uid={500} rate=500: Invalid argument oleg407-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p+spec uid={500} rate=500': Invalid argument pdsh@oleg407-client: oleg407-server: ssh exited with exit code 22 oleg407-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p=spec uid={500} rate=500: Invalid argument oleg407-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p=spec uid={500} rate=500': Invalid argument pdsh@oleg407-client: oleg407-server: ssh exited with exit code 22 oleg407-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p)spec uid={500} rate=500: Invalid argument oleg407-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p)spec uid={500} rate=500': Invalid argument pdsh@oleg407-client: oleg407-server: ssh exited with exit code 22 oleg407-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p(spec uid={500} rate=500: Invalid argument oleg407-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p(spec uid={500} rate=500': Invalid argument pdsh@oleg407-client: oleg407-server: ssh exited with exit code 22 oleg407-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p{spec uid={500} rate=500: Invalid argument oleg407-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p{spec uid={500} rate=500': Invalid argument pdsh@oleg407-client: oleg407-server: ssh exited with exit code 22 oleg407-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p}spec uid={500} rate=500: Invalid argument oleg407-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p}spec uid={500} rate=500': Invalid argument pdsh@oleg407-client: oleg407-server: ssh exited with exit code 22 oleg407-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p|spec uid={500} rate=500: Invalid argument oleg407-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p|spec uid={500} rate=500': Invalid argument pdsh@oleg407-client: oleg407-server: ssh exited with exit code 22 oleg407-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p[spec uid={500} rate=500: Invalid argument oleg407-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p[spec uid={500} rate=500': Invalid argument pdsh@oleg407-client: oleg407-server: ssh exited with exit code 22 oleg407-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p]spec uid={500} rate=500: Invalid argument oleg407-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p]spec uid={500} rate=500': Invalid argument pdsh@oleg407-client: oleg407-server: ssh exited with exit code 22 oleg407-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p!spec uid={500} rate=500: Invalid argument oleg407-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p!spec uid={500} rate=500': Invalid argument pdsh@oleg407-client: oleg407-server: ssh exited with exit code 22 oleg407-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p&spec uid={500} rate=500: Invalid argument oleg407-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p&spec uid={500} rate=500': Invalid argument pdsh@oleg407-client: oleg407-server: ssh exited with exit code 22 oleg407-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p$spec uid={500} rate=500: Invalid argument oleg407-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p$spec uid={500} rate=500': Invalid argument pdsh@oleg407-client: oleg407-server: ssh exited with exit code 22 oleg407-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p`spec uid={500} rate=500: Invalid argument oleg407-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p`spec uid={500} rate=500': Invalid argument pdsh@oleg407-client: oleg407-server: ssh exited with exit code 22 oleg407-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p\spec uid={500} rate=500: Invalid argument oleg407-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p\spec uid={500} rate=500': Invalid argument pdsh@oleg407-client: oleg407-server: ssh exited with exit code 22 mds.MDS.mdt.nrs_policies=fifo PASS 77p (15s) == sanityn test 77r: Change type of tbf policy at run time ========================================================== 21:12:23 (1788829943) mds.MDS.mdt.nrs_policies=tbf uid mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf gid mds.MDS.mdt.nrs_tbf_rule=start md_rule gid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start md_rule jobid={*.500} rate=1 mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=fifo check the number of requests in queue: fifo: 0 crrn: 0 tbf: 0 mclock: 0 delay: 0 fifo: 0 crrn: 0 tbf: 0 mclock: 0 delay: 0 cleanup 77r 277027 kill md_thread xargs PASS 77r (43s) == sanityn test 77s: Check TBF LRU shrinker ============== 21:13:06 (1788829986) jobid_name=%e.%p jobid_var=TEST_77s_JOBVAR mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start touch jobid={touch.*} rate=1000 mds.MDS.mdt.nrs_tbf_rule=start unlink jobid={unlink.*} rate=1000 touch jobid={touch.*} 1000, seq 2, type=jobid, ref 9 touch jobid={touch.*} 1000, seq 2, type=jobid, ref 0 unlink jobid={unlink.*} 1000, seq 3, type=jobid, ref 9 unlink jobid={unlink.*} 1000, seq 3, type=jobid, ref 0 mds.MDS.mdt.nrs_policies=fifo jobid_var=procname_uid jobid_name=%e.%u PASS 77s (24s) == sanityn test 77ta: Testing for dynamic|static classify parameter setting for TBF ========================================================== 21:13:31 (1788830011) ost.OSS.ost_io.nrs_policies=fifo oleg407-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf nid+uid classify=foo: Invalid argument oleg407-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf nid+uid classify=foo': Invalid argument pdsh@oleg407-client: oleg407-server: ssh exited with exit code 22 oleg407-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf nid+uid foo=static: Invalid argument oleg407-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf nid+uid foo=static': Invalid argument pdsh@oleg407-client: oleg407-server: ssh exited with exit code 22 oleg407-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf nid+uid classify=: Invalid argument oleg407-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf nid+uid classify=': Invalid argument pdsh@oleg407-client: oleg407-server: ssh exited with exit code 22 ost.OSS.ost_io.nrs_policies=tbf nid classify=dynamic ost.OSS.ost_io.nrs_policies=tbf nid classify=static ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf reg nid classify=dynamic ost.OSS.ost_io.nrs_policies=tbf hp nid ost.OSS.ost_io.nrs_tbf_rule=start mixed1 nid={0@lo} rate=100 ost.OSS.ost_io.nrs_tbf_rule=stop mixed1 ost.OSS.ost_io.nrs_policies=tbf reg nid classify=static ost.OSS.ost_io.nrs_policies=tbf hp nid classify=dynamic ost.OSS.ost_io.nrs_tbf_rule=start mixed2 nid={0@lo} rate=100 ost.OSS.ost_io.nrs_tbf_rule=stop mixed2 ost.OSS.ost_io.nrs_policies=tbf reg nid+uid ost.OSS.ost_io.nrs_policies=tbf hp nid oleg407-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start mixed3 nid={0@lo}&uid={500} rate=100: Invalid argument oleg407-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start mixed3 nid={0@lo}&uid={500} rate=100': Invalid argument pdsh@oleg407-client: oleg407-server: ssh exited with exit code 22 ost.OSS.ost_io.nrs_policies=tbf nid classify=static classify=dynamic ost.OSS.ost_io.nrs_policies=tbf reg nid classify=dynamic ost.OSS.ost_io.nrs_policies= regular_requests: - name: fifo state: started fallback: yes queued: 0 active: 0 - name: crrn state: stopped fallback: no queued: 0 active: 0 - name: orr state: stopped fallback: no queued: 0 active: 0 - name: trr state: stopped fallback: no queued: 0 active: 0 - name: tbf nid classify=dynamic state: started fallback: no queued: 0 active: 0 - name: mclock state: stopped fallback: no queued: 0 active: 0 - name: delay state: stopped fallback: no queued: 0 active: 0 high_priority_requests: - name: fifo state: started fallback: yes queued: 0 active: 0 - name: crrn state: stopped fallback: no queued: 0 active: 0 - name: orr state: stopped fallback: no queued: 0 active: 0 - name: trr state: stopped fallback: no queued: 0 active: 0 - name: tbf nid classify=static classify=dynamic state: started fallback: no queued: 0 active: 0 - name: mclock state: stopped fallback: no queued: 0 active: 0 - name: delay state: stopped fallback: no queued: 0 active: 0 ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf hp nid classify=static ost.OSS.ost_io.nrs_policies= regular_requests: - name: fifo state: started fallback: yes queued: 0 active: 0 - name: crrn state: stopped fallback: no queued: 0 active: 0 - name: orr state: stopped fallback: no queued: 0 active: 0 - name: trr state: stopped fallback: no queued: 0 active: 0 - name: tbf state: stopped fallback: no queued: 0 active: 0 - name: mclock state: stopped fallback: no queued: 0 active: 0 - name: delay state: stopped fallback: no queued: 0 active: 0 high_priority_requests: - name: fifo state: started fallback: yes queued: 0 active: 0 - name: crrn state: stopped fallback: no queued: 0 active: 0 - name: orr state: stopped fallback: no queued: 0 active: 0 - name: trr state: stopped fallback: no queued: 0 active: 0 - name: tbf nid classify=static state: started fallback: no queued: 0 active: 0 - name: mclock state: stopped fallback: no queued: 0 active: 0 - name: delay state: stopped fallback: no queued: 0 active: 0 ost.OSS.ost_io.nrs_policies=fifo PASS 77ta (22s) == sanityn test 77tB: Request falling to default rule should not panic for dynamic TBF ========================================================== 21:13:52 (1788830032) ost.OSS.ost_io.nrs_policies=tbf nid classify=dynamic not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5, bs: 1048576 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.6245 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 vs. 5 Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9134 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 vs. 5 cleanup_tbf /mnt/lustre/d77tB.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=start unmatchrule nid={192.0.2.1@tcp} rate=100 not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5, bs: 1048576 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.561 s, 5.4 MB/s Write runtime is 21 s, speed is 4.761904 vs. 5 Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9436 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 vs. 5 cleanup_tbf /mnt/lustre/d77tB.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop unmatchrule ost.OSS.ost_io.nrs_policies=fifo PASS 77tB (124s) == sanityn test 77th: TBF: a mixed classify head pair keeps each head type ========================================================== 21:15:56 (1788830156) ost.OSS.ost_io.nrs_policies=tbf reg nid+opcode ost.OSS.ost_io.nrs_policies=tbf hp nid classify=dynamic ost.OSS.ost_io.nrs_tbf_rule=start mixedhp nid={0@lo} rate=100 regular_requests: CPT 0: mixedhp nid={0@lo} 100, seq 2, type=nid+opcode, ref 0 default * 10000, seq 1, type=nid+opcode, ref 0 high_priority_requests: CPT 0: mixedhp nid={0@lo} 100, seq 2, type=nid, ref 0 default * 10000, seq 1, type=nid, ref 0 ost.OSS.ost_io.nrs_tbf_rule=stop mixedhp ost.OSS.ost_io.nrs_policies=fifo PASS 77th (6s) == sanityn test 77tb: Testing for dynamic classify TBF Scheduler ========================================================== 21:16:02 (1788830162) ost.OSS.ost_io.nrs_policies=tbf nid classify=dynamic ost.OSS.ost_io.nrs_policies= regular_requests: - name: fifo state: started fallback: yes queued: 0 active: 0 - name: crrn state: stopped fallback: no queued: 0 active: 0 - name: orr state: stopped fallback: no queued: 0 active: 0 - name: trr state: stopped fallback: no queued: 0 active: 0 - name: tbf nid classify=dynamic state: started fallback: no queued: 0 active: 0 - name: mclock state: stopped fallback: no queued: 0 active: 0 - name: delay state: stopped fallback: no queued: 0 active: 0 high_priority_requests: - name: fifo state: started fallback: yes queued: 0 active: 0 - name: crrn state: stopped fallback: no queued: 0 active: 0 - name: orr state: stopped fallback: no queued: 0 active: 0 - name: trr state: stopped fallback: no queued: 0 active: 0 - name: tbf nid classify=dynamic state: started fallback: no queued: 0 active: 0 - name: mclock state: stopped fallback: no queued: 0 active: 0 - name: delay state: stopped fallback: no queued: 0 active: 0 ost.OSS.ost_io.nrs_tbf_classify= reg_classify:dynamic hp_classify:dynamic ost.OSS.ost_io.nrs_tbf_rule=start nidall nid={0@lo 192.168.204.7@tcp} rate=10 ost.OSS.ost_io.nrs_tbf_rule=start nidrd nid={0@lo 192.168.204.7@tcp}&opcode={ost_read} rate=5 ost.OSS.ost_io.nrs_tbf_rule=start nidwr nid={0@lo 192.168.204.7@tcp}&opcode={ost_write} rate=15 ost.OSS.ost_io.nrs_tbf_rule= regular_requests: CPT 0: nidwr nid={0@lo 192.168.204.7@tcp}&opcode={ost_write} 15, seq 4, type=nid+opcode, ref 0 nidrd nid={0@lo 192.168.204.7@tcp}&opcode={ost_read} 5, seq 3, type=nid+opcode, ref 0 nidall nid={0@lo 192.168.204.7@tcp} 10, seq 2, type=nid, ref 0 default * 10000, seq 1, type=nid, ref 0 high_priority_requests: CPT 0: nidwr nid={0@lo 192.168.204.7@tcp}&opcode={ost_write} 15, seq 4, type=nid+opcode, ref 0 nidrd nid={0@lo 192.168.204.7@tcp}&opcode={ost_read} 5, seq 3, type=nid+opcode, ref 0 nidall nid={0@lo 192.168.204.7@tcp} 10, seq 2, type=nid, ref 0 default * 10000, seq 1, type=nid, ref 0 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.374294 s, 44.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0247239 s, 42.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0324786 s, 32.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0835118 s, 12.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0905025 s, 11.6 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0909663 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0847493 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0942977 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0901892 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101828 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123043 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120313 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135915 s, 7.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.336283 s, 3.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.330302 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.408399 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.422952 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0321032 s, 32.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0330434 s, 31.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0288924 s, 36.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0370413 s, 28.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.038346 s, 27.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0343844 s, 30.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.036295 s, 28.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0477767 s, 21.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0307871 s, 34.1 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0310582 s, 33.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0421908 s, 24.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0334894 s, 31.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0473571 s, 22.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.035175 s, 29.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0392544 s, 26.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0502193 s, 20.9 MB/s removed '/mnt/lustre/d77tb.sanityn/nrs_r_oleg407-client.virtnet' removed '/mnt/lustre/d77tb.sanityn/nrs_w_oleg407-client.virtnet' removed directory '/mnt/lustre/d77tb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 15, read rate: 5, bs: 1048576 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.62509 s, 15.8 MB/s Write runtime is 8 s, speed is 12.500000 vs. 15 Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.488 s, 5.4 MB/s Read runtime is 20 s, speed is 5.000000 vs. 5 cleanup_tbf /mnt/lustre/d77tb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Cleanup TBF ost.OSS.ost_io.nrs_tbf_rule=stop nidall ost.OSS.ost_io.nrs_tbf_rule=stop nidrd ost.OSS.ost_io.nrs_tbf_rule=stop nidwr ost.OSS.ost_io.nrs_policies=fifo PASS 77tb (59s) == sanityn test 77tc: Testing for the setting of allowed TBF classification type ========================================================== 21:17:02 (1788830222) oleg407-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf nid classify=dynamic allow=nid allow=nid: File exists oleg407-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf nid classify=dynamic allow=nid allow=nid': File exists pdsh@oleg407-client: oleg407-server: ssh exited with exit code 17 oleg407-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf nid allow=nid allow=uid: Invalid argument oleg407-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf nid allow=nid allow=uid': Invalid argument pdsh@oleg407-client: oleg407-server: ssh exited with exit code 22 ost.OSS.ost_io.nrs_policies=tbf nid classify=dynamic allow=nid+opcode allow=nid allow=uid+opcode oleg407-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start another uid={500} rate=10: Operation not permitted oleg407-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start another uid={500} rate=10': Operation not permitted pdsh@oleg407-client: oleg407-server: ssh exited with exit code 1 oleg407-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start another uid={500}&opcode={ost_read}&gid={500} rate=10: Operation not permitted oleg407-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start another uid={500}&opcode={ost_read}&gid={500} rate=10': Operation not permitted pdsh@oleg407-client: oleg407-server: ssh exited with exit code 1 ost.OSS.ost_io.nrs_tbf_rule=start nidall nid={0@lo} rate=10 ost.OSS.ost_io.nrs_tbf_rule=start nidrd nid={0@lo}&opcode={ost_read} rate=5 Cleanup TBF ost.OSS.ost_io.nrs_tbf_rule=stop nidall ost.OSS.ost_io.nrs_tbf_rule=stop nidrd ost.OSS.ost_io.nrs_policies=fifo oleg407-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf nodemap classify=dynamic allow=nid allow=uid allow=gid allow=jobid allow=opcode allow=projid allow=nid+uid allow=nid+gid allow=uid+gid allow=nid+jobid allow=uid+jobid allow=gid+jobid allow=nid+opcode allow=uid+opcode allow=gid+opcode allow=nid+projid: Argument list too long oleg407-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf nodemap classify=dynamic allow=nid allow=uid allow=gid allow=jobid allow=opcode allow=projid allow=nid+uid allow=nid+gid allow=uid+gid allow=nid+jobid allow=uid+jobid allow=gid+jobid allow=nid+opcode allow=uid+opcode allow=gid+opcode allow=nid+projid': Argument list too long pdsh@oleg407-client: oleg407-server: ssh exited with exit code 7 Cleanup TBF ost.OSS.ost_io.nrs_policies=fifo PASS 77tc (18s) == sanityn test 77td: Testing Number of allowed rules larger or smaller than threshold ========================================================== 21:17:20 (1788830240) using SAVE_PROJECT_SUPPORTED=0 Stopping clients: oleg407-client.virtnet /mnt/lustre (opts:) Stopping client oleg407-client.virtnet /mnt/lustre opts: Stopping clients: oleg407-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg407-client.virtnet /mnt/lustre2 opts: Stopping /mnt/lustre-mds1 (opts:-f) on oleg407-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg407-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg407-server sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime) proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) devtmpfs on /dev type devtmpfs (rw,nosuid,mode=755) securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime) tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev) devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000) tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755) tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755) cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd) pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime) bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700) cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio) cgroup on /sys/fs/cgroup/misc type cgroup (rw,nosuid,nodev,noexec,relatime,misc) cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory) cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer) cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio) cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb) cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices) cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids) cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event) cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset) cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma) cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct) none on /sys/kernel/tracing type tracefs (rw,relatime) configfs on /sys/kernel/config type configfs (rw,relatime) /dev/nbd0 on / type ext4 (ro,relatime,stripe=32) rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) mqueue on /dev/mqueue type mqueue (rw,relatime) hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M) debugfs on /sys/kernel/debug type debugfs (rw,relatime) none on /mnt type ramfs (rw,relatime) /dev/vda on /home/green/git/lustre-release type squashfs (ro,relatime) none on /var/lib/stateless/writable type tmpfs (rw,relatime) none on /var/cache/man type tmpfs (rw,relatime) none on /var/log type tmpfs (rw,relatime) none on /var/lib/dbus type tmpfs (rw,relatime) none on /tmp type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) none on /var/tmp type tmpfs (rw,relatime) none on /var/lib/NetworkManager type tmpfs (rw,relatime) none on /var/lib/systemd/random-seed type tmpfs (rw,relatime) none on /var/spool type tmpfs (rw,relatime) none on /var/lib/nfs type tmpfs (rw,relatime) none on /var/lib/gssproxy type tmpfs (rw,relatime) none on /var/lib/logrotate type tmpfs (rw,relatime) none on /etc type tmpfs (rw,relatime) none on /var/lib/rsyslog type tmpfs (rw,relatime) none on /root/.ssh type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) 192.168.200.253:/exports/state/oleg407-client.virtnet on /var/lib/stateless/state type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.204.7,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg407-client.virtnet/boot on /boot type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.204.7,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg407-client.virtnet/etc/kdump.conf on /etc/kdump.conf type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.204.7,local_lock=none,addr=192.168.200.253) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=378180k,mode=700) 192.168.200.253:/exports/testreports/69785/testresults/sanityn-zfs-rocky8.10_x86_64-rocky8.10_x86_64 on /tmp/testlogs type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.204.7,local_lock=none,addr=192.168.200.253) /dev/vda on /usr/sbin/mount.lustre type squashfs (ro,relatime) Checking servers environments Checking clients oleg407-client.virtnet environments Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg407-server' oleg407-server: oleg407-server.virtnet: executing load_modules_local oleg407-server: kmemleak disabled oleg407-server: Loading modules from /home/green/git/lustre-release/lustre oleg407-server: detected 4 online CPUs by sysfs oleg407-server: MODOPTS_LIBCFS= oleg407-server: Force libcfs to create 2 CPU partitions oleg407-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Setup mgs, mdt, osts Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg407-server: oleg407-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg407-client: oleg407-server: ssh exited with exit code 1 Started lustre-MDT0000 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg407-server: oleg407-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg407-client: oleg407-server: ssh exited with exit code 1 Started lustre-OST0000 Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg407-server: oleg407-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg407-client: oleg407-server: ssh exited with exit code 1 Started lustre-OST0001 Starting client: oleg407-client.virtnet: -o user_xattr,flock 192.168.204.107@tcp:/lustre /mnt/lustre Starting client oleg407-client.virtnet: -o user_xattr,flock 192.168.204.107@tcp:/lustre /mnt/lustre Started clients oleg407-client.virtnet: 192.168.204.107@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Starting client: oleg407-client.virtnet: -o user_xattr,flock 192.168.204.107@tcp:/lustre /mnt/lustre2 Starting client oleg407-client.virtnet: -o user_xattr,flock 192.168.204.107@tcp:/lustre /mnt/lustre2 Started clients oleg407-client.virtnet: 192.168.204.107@tcp:/lustre on /mnt/lustre2 type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff92ca103df800.idle_timeout=debug osc.lustre-OST0000-osc-ffff92ca385af000.idle_timeout=debug osc.lustre-OST0001-osc-ffff92ca103df800.idle_timeout=debug osc.lustre-OST0001-osc-ffff92ca385af000.idle_timeout=debug disable quota as required ost.OSS.ost_io.nrs_policies=tbf nid classify=dynamic allow=projid+opcode allow=projid ost.OSS.ost_io.nrs_tbf_allowed_types= regular_allowed_types: CPT 0: 0: 0x00000024 opcode+projid 1: 0x00000020 projid 2: 0x00000002 nid high_priority_allowed_types: CPT 0: 0: 0x00000024 opcode+projid 1: 0x00000020 projid 2: 0x00000002 nid 8 ost.OSS.ost_io.nrs_tbf_rule=start rulerd0 projid={100}&opcode={ost_read} rate=5 ost.OSS.ost_io.nrs_tbf_rule=start rulewr0 projid={100}&opcode={ost_write} rate=10 ost.OSS.ost_io.nrs_tbf_rule=start ruleall0 projid={200} rate=15 ost.OSS.ost_io.nrs_tbf_rule= regular_requests: CPT 0: ruleall0 projid={200} 15, seq 4, type=projid, ref 0 rulewr0 projid={100}&opcode={ost_write} 10, seq 3, type=opcode+projid, ref 0 rulerd0 projid={100}&opcode={ost_read} 5, seq 2, type=opcode+projid, ref 0 default * 10000, seq 1, type=nid+opcode+projid, ref 0 high_priority_requests: CPT 0: ruleall0 projid={200} 15, seq 4, type=projid, ref 0 rulewr0 projid={100}&opcode={ost_write} 10, seq 3, type=opcode+projid, ref 0 rulerd0 projid={100}&opcode={ost_read} 5, seq 2, type=opcode+projid, ref 0 default * 10000, seq 1, type=nid+opcode+projid, ref 0 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.242112 s, 69.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0341757 s, 30.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0495715 s, 21.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0514327 s, 20.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0562472 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145046 s, 7.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148798 s, 7.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189819 s, 5.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188418 s, 5.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183849 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1686 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196984 s, 5.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173708 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162873 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202886 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179268 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207214 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0474576 s, 22.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00663559 s, 158 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0352231 s, 29.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00240323 s, 436 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0754181 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00830266 s, 126 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0741068 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0667508 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0647254 s, 16.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0625986 s, 16.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0593804 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0563333 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00663861 s, 158 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0530484 s, 19.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0471606 s, 22.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0355997 s, 29.5 MB/s removed '/mnt/lustre/d77td.sanityn/nrs_w_oleg407-client.virtnet' removed '/mnt/lustre/d77td.sanityn/nrs_r_oleg407-client.virtnet' removed directory '/mnt/lustre/d77td.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 5, bs: 1048576 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0315 s, 10.5 MB/s Write runtime is 11 s, speed is 9.090909 vs. 10 Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.5633 s, 5.4 MB/s Read runtime is 21 s, speed is 4.761904 vs. 5 cleanup_tbf /mnt/lustre/d77td.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 15, read rate: 15, bs: 1048576 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.86671 s, 13.3 MB/s Write runtime is 9 s, speed is 11.111111 vs. 15 Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.72694 s, 15.6 MB/s Read runtime is 8 s, speed is 12.500000 vs. 15 cleanup_tbf /mnt/lustre/d77td.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=change default rate=15 ost.OSS.ost_io.nrs_tbf_rule= regular_requests: CPT 0: ruleall0 projid={200} 15, seq 4, type=projid, ref 1 rulewr0 projid={100}&opcode={ost_write} 10, seq 3, type=opcode+projid, ref 1 rulerd0 projid={100}&opcode={ost_read} 5, seq 2, type=opcode+projid, ref 1 default * 15, seq 1, type=nid+opcode+projid, ref 2 high_priority_requests: CPT 0: ruleall0 projid={200} 15, seq 4, type=projid, ref 0 rulewr0 projid={100}&opcode={ost_write} 10, seq 3, type=opcode+projid, ref 0 rulerd0 projid={100}&opcode={ost_read} 5, seq 2, type=opcode+projid, ref 0 default * 15, seq 1, type=nid+opcode+projid, ref 1 not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 15, read rate: 15, bs: 1048576 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.23208 s, 14.5 MB/s Write runtime is 8 s, speed is 12.500000 vs. 15 Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.62032 s, 15.8 MB/s Read runtime is 7 s, speed is 14.285714 vs. 15 cleanup_tbf /mnt/lustre/d77td.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf nid classify=dynamic allow=projid+opcode allow=projid ost.OSS.ost_io.nrs_tbf_allowed_types= regular_allowed_types: CPT 0: 0: 0x00000024 opcode+projid 1: 0x00000020 projid 2: 0x00000002 nid high_priority_allowed_types: CPT 0: 0: 0x00000024 opcode+projid 1: 0x00000020 projid 2: 0x00000002 nid 8 ost.OSS.ost_io.nrs_tbf_rule=start rulerd0 projid={100}&opcode={ost_read} rate=5 ost.OSS.ost_io.nrs_tbf_rule=start rulewr0 projid={100}&opcode={ost_write} rate=10 ost.OSS.ost_io.nrs_tbf_rule=start rulerd1 projid={101}&opcode={ost_read} rate=5 ost.OSS.ost_io.nrs_tbf_rule=start rulewr1 projid={101}&opcode={ost_write} rate=10 ost.OSS.ost_io.nrs_tbf_rule=start rulerd2 projid={102}&opcode={ost_read} rate=5 ost.OSS.ost_io.nrs_tbf_rule=start rulewr2 projid={102}&opcode={ost_write} rate=10 ost.OSS.ost_io.nrs_tbf_rule=start rulerd3 projid={103}&opcode={ost_read} rate=5 ost.OSS.ost_io.nrs_tbf_rule=start rulewr3 projid={103}&opcode={ost_write} rate=10 ost.OSS.ost_io.nrs_tbf_rule=start rulerd4 projid={104}&opcode={ost_read} rate=5 ost.OSS.ost_io.nrs_tbf_rule=start rulewr4 projid={104}&opcode={ost_write} rate=10 ost.OSS.ost_io.nrs_tbf_rule=start ruleall0 projid={200} rate=15 ost.OSS.ost_io.nrs_tbf_rule=start ruleall1 projid={201} rate=15 ost.OSS.ost_io.nrs_tbf_rule=start ruleall2 projid={202} rate=15 ost.OSS.ost_io.nrs_tbf_rule=start ruleall3 projid={203} rate=15 ost.OSS.ost_io.nrs_tbf_rule=start ruleall4 projid={204} rate=15 ost.OSS.ost_io.nrs_tbf_rule= regular_requests: CPT 0: ruleall4 projid={204} 15, seq 16, type=projid, ref 0 ruleall3 projid={203} 15, seq 15, type=projid, ref 0 ruleall2 projid={202} 15, seq 14, type=projid, ref 0 ruleall1 projid={201} 15, seq 13, type=projid, ref 0 ruleall0 projid={200} 15, seq 12, type=projid, ref 0 rulewr4 projid={104}&opcode={ost_write} 10, seq 11, type=opcode+projid, ref 0 rulerd4 projid={104}&opcode={ost_read} 5, seq 10, type=opcode+projid, ref 0 rulewr3 projid={103}&opcode={ost_write} 10, seq 9, type=opcode+projid, ref 0 rulerd3 projid={103}&opcode={ost_read} 5, seq 8, type=opcode+projid, ref 0 rulewr2 projid={102}&opcode={ost_write} 10, seq 7, type=opcode+projid, ref 0 rulerd2 projid={102}&opcode={ost_read} 5, seq 6, type=opcode+projid, ref 0 rulewr1 projid={101}&opcode={ost_write} 10, seq 5, type=opcode+projid, ref 0 rulerd1 projid={101}&opcode={ost_read} 5, seq 4, type=opcode+projid, ref 0 rulewr0 projid={100}&opcode={ost_write} 10, seq 3, type=opcode+projid, ref 0 rulerd0 projid={100}&opcode={ost_read} 5, seq 2, type=opcode+projid, ref 0 default * 10000, seq 1, type=nid+opcode+projid, ref 0 high_priority_requests: CPT 0: ruleall4 projid={204} 15, seq 16, type=projid, ref 0 ruleall3 projid={203} 15, seq 15, type=projid, ref 0 ruleall2 projid={202} 15, seq 14, type=projid, ref 0 ruleall1 projid={201} 15, seq 13, type=projid, ref 0 ruleall0 projid={200} 15, seq 12, type=projid, ref 0 rulewr4 projid={104}&opcode={ost_write} 10, seq 11, type=opcode+projid, ref 0 rulerd4 projid={104}&opcode={ost_read} 5, seq 10, type=opcode+projid, ref 0 rulewr3 projid={103}&opcode={ost_write} 10, seq 9, type=opcode+projid, ref 0 rulerd3 projid={103}&opcode={ost_read} 5, seq 8, type=opcode+projid, ref 0 rulewr2 projid={102}&opcode={ost_write} 10, seq 7, type=opcode+projid, ref 0 rulerd2 projid={102}&opcode={ost_read} 5, seq 6, type=opcode+projid, ref 0 rulewr1 projid={101}&opcode={ost_write} 10, seq 5, type=opcode+projid, ref 0 rulerd1 projid={101}&opcode={ost_read} 5, seq 4, type=opcode+projid, ref 0 rulewr0 projid={100}&opcode={ost_write} 10, seq 3, type=opcode+projid, ref 0 rulerd0 projid={100}&opcode={ost_read} 5, seq 2, type=opcode+projid, ref 0 default * 10000, seq 1, type=nid+opcode+projid, ref 0 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.396637 s, 42.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0742804 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0549144 s, 19.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109514 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127912 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119322 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143293 s, 7.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153768 s, 6.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167983 s, 6.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159988 s, 6.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16246 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0520956 s, 20.1 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff92ca385af000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.268536 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.229173 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22398 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.307625 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.326649 s, 3.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff92ca385af000.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0558108 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0915598 s, 11.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0965559 s, 10.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0624254 s, 16.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0501658 s, 20.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0524733 s, 20.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.074657 s, 14.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10399 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0741244 s, 14.1 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0700629 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0813838 s, 12.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0803429 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0849964 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0742343 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0818973 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113308 s, 9.3 MB/s removed '/mnt/lustre/d77td.sanityn/nrs_r_oleg407-client.virtnet' removed '/mnt/lustre/d77td.sanityn/nrs_w_oleg407-client.virtnet' removed directory '/mnt/lustre/d77td.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 5, bs: 1048576 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.1885 s, 10.3 MB/s Write runtime is 12 s, speed is 8.333333 vs. 10 Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.5538 s, 5.4 MB/s Read runtime is 20 s, speed is 5.000000 vs. 5 cleanup_tbf /mnt/lustre/d77td.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 15, read rate: 15, bs: 1048576 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.01203 s, 13.1 MB/s Write runtime is 9 s, speed is 11.111111 vs. 15 Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.70789 s, 15.6 MB/s Read runtime is 7 s, speed is 14.285714 vs. 15 cleanup_tbf /mnt/lustre/d77td.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=change default rate=15 ost.OSS.ost_io.nrs_tbf_rule= regular_requests: CPT 0: ruleall4 projid={204} 15, seq 16, type=projid, ref 0 ruleall3 projid={203} 15, seq 15, type=projid, ref 0 ruleall2 projid={202} 15, seq 14, type=projid, ref 0 ruleall1 projid={201} 15, seq 13, type=projid, ref 0 ruleall0 projid={200} 15, seq 12, type=projid, ref 1 rulewr4 projid={104}&opcode={ost_write} 10, seq 11, type=opcode+projid, ref 0 rulerd4 projid={104}&opcode={ost_read} 5, seq 10, type=opcode+projid, ref 0 rulewr3 projid={103}&opcode={ost_write} 10, seq 9, type=opcode+projid, ref 0 rulerd3 projid={103}&opcode={ost_read} 5, seq 8, type=opcode+projid, ref 0 rulewr2 projid={102}&opcode={ost_write} 10, seq 7, type=opcode+projid, ref 0 rulerd2 projid={102}&opcode={ost_read} 5, seq 6, type=opcode+projid, ref 0 rulewr1 projid={101}&opcode={ost_write} 10, seq 5, type=opcode+projid, ref 0 rulerd1 projid={101}&opcode={ost_read} 5, seq 4, type=opcode+projid, ref 0 rulewr0 projid={100}&opcode={ost_write} 10, seq 3, type=opcode+projid, ref 1 rulerd0 projid={100}&opcode={ost_read} 5, seq 2, type=opcode+projid, ref 1 default * 15, seq 1, type=nid+opcode+projid, ref 2 high_priority_requests: CPT 0: ruleall4 projid={204} 15, seq 16, type=projid, ref 0 ruleall3 projid={203} 15, seq 15, type=projid, ref 0 ruleall2 projid={202} 15, seq 14, type=projid, ref 0 ruleall1 projid={201} 15, seq 13, type=projid, ref 0 ruleall0 projid={200} 15, seq 12, type=projid, ref 0 rulewr4 projid={104}&opcode={ost_write} 10, seq 11, type=opcode+projid, ref 0 rulerd4 projid={104}&opcode={ost_read} 5, seq 10, type=opcode+projid, ref 0 rulewr3 projid={103}&opcode={ost_write} 10, seq 9, type=opcode+projid, ref 0 rulerd3 projid={103}&opcode={ost_read} 5, seq 8, type=opcode+projid, ref 0 rulewr2 projid={102}&opcode={ost_write} 10, seq 7, type=opcode+projid, ref 0 rulerd2 projid={102}&opcode={ost_read} 5, seq 6, type=opcode+projid, ref 0 rulewr1 projid={101}&opcode={ost_write} 10, seq 5, type=opcode+projid, ref 0 rulerd1 projid={101}&opcode={ost_read} 5, seq 4, type=opcode+projid, ref 0 rulewr0 projid={100}&opcode={ost_write} 10, seq 3, type=opcode+projid, ref 0 rulerd0 projid={100}&opcode={ost_read} 5, seq 2, type=opcode+projid, ref 0 default * 15, seq 1, type=nid+opcode+projid, ref 1 not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 15, read rate: 15, bs: 1048576 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.32483 s, 14.3 MB/s Write runtime is 8 s, speed is 12.500000 vs. 15 Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.63052 s, 15.8 MB/s Read runtime is 8 s, speed is 12.500000 vs. 15 cleanup_tbf /mnt/lustre/d77td.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=fifo PASS 77td (380s) == sanityn test 77tf: TBF: rule churn under load on a dynamic classify policy ========================================================== 21:23:40 (1788830620) jobid_name=%e.%p jobid_var=TEST_77TF_JOBVAR ost.OSS.ost_io.nrs_policies=tbf jobid+opcode classify=dynamic ost.OSS.ost_io.nrs_tbf_rule=start churn1 jobid={dd.*} rate=2000 ost.OSS.ost_io.nrs_tbf_rule=start churn2 jobid={dd.*}&opcode={ost_write} rate=2000 ost.OSS.ost_io.nrs_tbf_rule=stop churn2 ost.OSS.ost_io.nrs_tbf_rule=stop churn1 ost.OSS.ost_io.nrs_tbf_rule=start churn1 jobid={dd.*} rate=2000 ost.OSS.ost_io.nrs_tbf_rule=start churn2 jobid={dd.*}&opcode={ost_write} rate=2000 ost.OSS.ost_io.nrs_tbf_rule=stop churn2 ost.OSS.ost_io.nrs_tbf_rule=stop churn1 ost.OSS.ost_io.nrs_tbf_rule=start churn1 jobid={dd.*} rate=2000 ost.OSS.ost_io.nrs_tbf_rule=start churn2 jobid={dd.*}&opcode={ost_write} rate=2000 ost.OSS.ost_io.nrs_tbf_rule=stop churn2 ost.OSS.ost_io.nrs_tbf_rule=stop churn1 ost.OSS.ost_io.nrs_tbf_rule=start churn1 jobid={dd.*} rate=2000 ost.OSS.ost_io.nrs_tbf_rule=start churn2 jobid={dd.*}&opcode={ost_write} rate=2000 ost.OSS.ost_io.nrs_tbf_rule=stop churn2 ost.OSS.ost_io.nrs_tbf_rule=stop churn1 ost.OSS.ost_io.nrs_tbf_rule=start churn1 jobid={dd.*} rate=2000 ost.OSS.ost_io.nrs_tbf_rule=start churn2 jobid={dd.*}&opcode={ost_write} rate=2000 ost.OSS.ost_io.nrs_tbf_rule=stop churn2 ost.OSS.ost_io.nrs_tbf_rule=stop churn1 ost.OSS.ost_io.nrs_tbf_rule=start churn1 jobid={dd.*} rate=2000 ost.OSS.ost_io.nrs_tbf_rule=start churn2 jobid={dd.*}&opcode={ost_write} rate=2000 ost.OSS.ost_io.nrs_tbf_rule=stop churn2 ost.OSS.ost_io.nrs_tbf_rule=stop churn1 ost.OSS.ost_io.nrs_tbf_rule=start churn1 jobid={dd.*} rate=2000 ost.OSS.ost_io.nrs_tbf_rule=start churn2 jobid={dd.*}&opcode={ost_write} rate=2000 ost.OSS.ost_io.nrs_tbf_rule=stop churn2 ost.OSS.ost_io.nrs_tbf_rule=stop churn1 ost.OSS.ost_io.nrs_tbf_rule=start churn1 jobid={dd.*} rate=2000 ost.OSS.ost_io.nrs_tbf_rule=start churn2 jobid={dd.*}&opcode={ost_write} rate=2000 ost.OSS.ost_io.nrs_tbf_rule=stop churn2 ost.OSS.ost_io.nrs_tbf_rule=stop churn1 ost.OSS.ost_io.nrs_tbf_rule=start churn1 jobid={dd.*} rate=2000 ost.OSS.ost_io.nrs_tbf_rule=start churn2 jobid={dd.*}&opcode={ost_write} rate=2000 ost.OSS.ost_io.nrs_tbf_rule=stop churn2 ost.OSS.ost_io.nrs_tbf_rule=stop churn1 ost.OSS.ost_io.nrs_tbf_rule=start churn1 jobid={dd.*} rate=2000 ost.OSS.ost_io.nrs_tbf_rule=start churn2 jobid={dd.*}&opcode={ost_write} rate=2000 ost.OSS.ost_io.nrs_tbf_rule=stop churn2 ost.OSS.ost_io.nrs_tbf_rule=stop churn1 Cleanup TBF ost.OSS.ost_io.nrs_tbf_rule=stop churn1 ost.OSS.ost_io.nrs_tbf_rule=stop churn2 ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid jobid_name=%e.%u PASS 77tf (46s) == sanityn test 77te: Dynamic type lookup should obey the rule order ========================================================== 21:24:26 (1788830666) using SAVE_PROJECT_SUPPORTED=0 ost.OSS.ost_io.nrs_policies=tbf projid classify=dynamic allow=projid+opcode allow=uid ost.OSS.ost_io.nrs_tbf_allowed_types= regular_allowed_types: CPT 0: 0: 0x00000024 opcode+projid 1: 0x00000020 projid 2: 0x00000008 uid high_priority_allowed_types: CPT 0: 0: 0x00000024 opcode+projid 1: 0x00000020 projid 2: 0x00000008 uid ost.OSS.ost_io.nrs_tbf_rule=start rule1 projid={100}&opcode={ost_read} rate=10 ost.OSS.ost_io.nrs_tbf_rule=start rule2 projid={100}&opcode={ost_write} rate=10 ost.OSS.ost_io.nrs_tbf_rule=start rule3 uid={500} rate=5 regular_requests: CPT 0: rule3 uid={500} 5, seq 4, type=uid, ref 0 rule2 projid={100}&opcode={ost_write} 10, seq 3, type=opcode+projid, ref 0 rule1 projid={100}&opcode={ost_read} 10, seq 2, type=opcode+projid, ref 0 default * 10000, seq 1, type=opcode+uid+projid, ref 0 high_priority_requests: CPT 0: rule3 uid={500} 5, seq 4, type=uid, ref 0 rule2 projid={100}&opcode={ost_write} 10, seq 3, type=opcode+projid, ref 0 rule1 projid={100}&opcode={ost_read} 10, seq 2, type=opcode+projid, ref 0 default * 10000, seq 1, type=opcode+uid+projid, ref 0 not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10, bs: 1048576 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0256 s, 10.5 MB/s Write runtime is 11 s, speed is 9.090909 vs. 10 Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.87184 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 vs. 10 cleanup_tbf /mnt/lustre/d77te.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5, bs: 1048576 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77te.sanityn/tbf] [bs=1048576] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.7639 s, 5.3 MB/s Write runtime is 20 s, speed is 5.000000 vs. 5 Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77te.sanityn/tbf] [of=/dev/null] [bs=1048576] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.916 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 vs. 5 cleanup_tbf /mnt/lustre/d77te.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_policies=fifo PASS 77te (117s) == sanityn test 77tg: TBF: the default rule holds its rate with many rules ========================================================== 21:26:24 (1788830784) ost.OSS.ost_io.nrs_policies=tbf uid classify=dynamic allow=uid ost.OSS.ost_io.nrs_tbf_rule=start r5000 uid={5000} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5001 uid={5001} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5002 uid={5002} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5003 uid={5003} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5004 uid={5004} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5005 uid={5005} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5006 uid={5006} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5007 uid={5007} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5008 uid={5008} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5009 uid={5009} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5010 uid={5010} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5011 uid={5011} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5012 uid={5012} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5013 uid={5013} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5014 uid={5014} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5015 uid={5015} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5016 uid={5016} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5017 uid={5017} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5018 uid={5018} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5019 uid={5019} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5020 uid={5020} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5021 uid={5021} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5022 uid={5022} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5023 uid={5023} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5024 uid={5024} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5025 uid={5025} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5026 uid={5026} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5027 uid={5027} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5028 uid={5028} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5029 uid={5029} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5030 uid={5030} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5031 uid={5031} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5032 uid={5032} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5033 uid={5033} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5034 uid={5034} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5035 uid={5035} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5036 uid={5036} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5037 uid={5037} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5038 uid={5038} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5039 uid={5039} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5040 uid={5040} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5041 uid={5041} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5042 uid={5042} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5043 uid={5043} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5044 uid={5044} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5045 uid={5045} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5046 uid={5046} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5047 uid={5047} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5048 uid={5048} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5049 uid={5049} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5050 uid={5050} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5051 uid={5051} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5052 uid={5052} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5053 uid={5053} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5054 uid={5054} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5055 uid={5055} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5056 uid={5056} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5057 uid={5057} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5058 uid={5058} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5059 uid={5059} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5060 uid={5060} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5061 uid={5061} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5062 uid={5062} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5063 uid={5063} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5064 uid={5064} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5065 uid={5065} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5066 uid={5066} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5067 uid={5067} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5068 uid={5068} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5069 uid={5069} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5070 uid={5070} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5071 uid={5071} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5072 uid={5072} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5073 uid={5073} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5074 uid={5074} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5075 uid={5075} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5076 uid={5076} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5077 uid={5077} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5078 uid={5078} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5079 uid={5079} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5080 uid={5080} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5081 uid={5081} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5082 uid={5082} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5083 uid={5083} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5084 uid={5084} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5085 uid={5085} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5086 uid={5086} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5087 uid={5087} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5088 uid={5088} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5089 uid={5089} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5090 uid={5090} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5091 uid={5091} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5092 uid={5092} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5093 uid={5093} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5094 uid={5094} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5095 uid={5095} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5096 uid={5096} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5097 uid={5097} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5098 uid={5098} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5099 uid={5099} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5100 uid={5100} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5101 uid={5101} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5102 uid={5102} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5103 uid={5103} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5104 uid={5104} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5105 uid={5105} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5106 uid={5106} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5107 uid={5107} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5108 uid={5108} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5109 uid={5109} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5110 uid={5110} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5111 uid={5111} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5112 uid={5112} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5113 uid={5113} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5114 uid={5114} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5115 uid={5115} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5116 uid={5116} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5117 uid={5117} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5118 uid={5118} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5119 uid={5119} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5120 uid={5120} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5121 uid={5121} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5122 uid={5122} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5123 uid={5123} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5124 uid={5124} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5125 uid={5125} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5126 uid={5126} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5127 uid={5127} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5128 uid={5128} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5129 uid={5129} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5130 uid={5130} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5131 uid={5131} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5132 uid={5132} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5133 uid={5133} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5134 uid={5134} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5135 uid={5135} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5136 uid={5136} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5137 uid={5137} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5138 uid={5138} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5139 uid={5139} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5140 uid={5140} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5141 uid={5141} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5142 uid={5142} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5143 uid={5143} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5144 uid={5144} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5145 uid={5145} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5146 uid={5146} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5147 uid={5147} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5148 uid={5148} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5149 uid={5149} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5150 uid={5150} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5151 uid={5151} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5152 uid={5152} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5153 uid={5153} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5154 uid={5154} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5155 uid={5155} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5156 uid={5156} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5157 uid={5157} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5158 uid={5158} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5159 uid={5159} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5160 uid={5160} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5161 uid={5161} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5162 uid={5162} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5163 uid={5163} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5164 uid={5164} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5165 uid={5165} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5166 uid={5166} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5167 uid={5167} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5168 uid={5168} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5169 uid={5169} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5170 uid={5170} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5171 uid={5171} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5172 uid={5172} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5173 uid={5173} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5174 uid={5174} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5175 uid={5175} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5176 uid={5176} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5177 uid={5177} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5178 uid={5178} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5179 uid={5179} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5180 uid={5180} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5181 uid={5181} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5182 uid={5182} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5183 uid={5183} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5184 uid={5184} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5185 uid={5185} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5186 uid={5186} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5187 uid={5187} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5188 uid={5188} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5189 uid={5189} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5190 uid={5190} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5191 uid={5191} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5192 uid={5192} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5193 uid={5193} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5194 uid={5194} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5195 uid={5195} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5196 uid={5196} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5197 uid={5197} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5198 uid={5198} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5199 uid={5199} rate=1000 not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10, bs: 1048576 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0036 s, 10.5 MB/s Write runtime is 11 s, speed is 9.090909 vs. 10 Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.92774 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 vs. 10 cleanup_tbf /mnt/lustre/d77tg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Cleanup TBF ost.OSS.ost_io.nrs_policies=fifo PASS 77tg (64s) == sanityn test 77ua: Setting TBF rate performance mode: BPS and IOPS ========================================================== 21:27:28 (1788830848) ost.OSS.ost_io.nrs_policies=tbf nid+opcode ost.OSS.ost_io.nrs_policies= regular_requests: - name: fifo state: started fallback: yes queued: 0 active: 0 - name: crrn state: stopped fallback: no queued: 0 active: 0 - name: orr state: stopped fallback: no queued: 0 active: 0 - name: trr state: stopped fallback: no queued: 0 active: 0 - name: tbf nid+opcode state: started fallback: no queued: 0 active: 0 - name: mclock state: stopped fallback: no queued: 0 active: 0 - name: delay state: stopped fallback: no queued: 0 active: 0 high_priority_requests: - name: fifo state: started fallback: yes queued: 0 active: 0 - name: crrn state: stopped fallback: no queued: 0 active: 0 - name: orr state: stopped fallback: no queued: 0 active: 0 - name: trr state: stopped fallback: no queued: 0 active: 0 - name: tbf nid+opcode state: started fallback: no queued: 0 active: 0 - name: mclock state: stopped fallback: no queued: 0 active: 0 - name: delay state: stopped fallback: no queued: 0 active: 0 ost.OSS.ost_io.nrs_tbf_rule=start nidrd nid={0@lo 192.168.204.7@tcp}&opcode={ost_read} bps=5MiB/s ost.OSS.ost_io.nrs_tbf_rule=start nidwr nid={0@lo 192.168.204.7@tcp}&opcode={ost_write} bps=15MiB/s regular_requests: CPT 0: nidwr nid={0@lo 192.168.204.7@tcp}&opcode={ost_write} 15, seq 3, type=nid+opcode, ref 0 bps=15728640 nidrd nid={0@lo 192.168.204.7@tcp}&opcode={ost_read} 5, seq 2, type=nid+opcode, ref 0 bps=5242880 default * 10000, seq 1, type=nid+opcode, ref 0 high_priority_requests: CPT 0: nidwr nid={0@lo 192.168.204.7@tcp}&opcode={ost_write} 15, seq 3, type=nid+opcode, ref 0 bps=15728640 nidrd nid={0@lo 192.168.204.7@tcp}&opcode={ost_read} 5, seq 2, type=nid+opcode, ref 0 bps=5242880 default * 10000, seq 1, type=nid+opcode, ref 0 ost.OSS.ost_io.nrs_tbf_rule=stop nidrd ost.OSS.ost_io.nrs_tbf_rule=stop nidwr ost.OSS.ost_io.nrs_tbf_rule=start nidrd nid={0@lo 192.168.204.7@tcp}&opcode={ost_read} bps=500K ost.OSS.ost_io.nrs_tbf_rule=start nidwr nid={0@lo 192.168.204.7@tcp}&opcode={ost_write} bps=750K regular_requests: CPT 0: nidwr nid={0@lo 192.168.204.7@tcp}&opcode={ost_write} 1, seq 5, type=nid+opcode, ref 0 bps=768000 nidrd nid={0@lo 192.168.204.7@tcp}&opcode={ost_read} 1, seq 4, type=nid+opcode, ref 0 bps=512000 default * 10000, seq 1, type=nid+opcode, ref 0 high_priority_requests: CPT 0: nidwr nid={0@lo 192.168.204.7@tcp}&opcode={ost_write} 1, seq 5, type=nid+opcode, ref 0 bps=768000 nidrd nid={0@lo 192.168.204.7@tcp}&opcode={ost_read} 1, seq 4, type=nid+opcode, ref 0 bps=512000 default * 10000, seq 1, type=nid+opcode, ref 0 ost.OSS.ost_io.nrs_tbf_rule=stop nidrd ost.OSS.ost_io.nrs_tbf_rule=stop nidwr ost.OSS.ost_io.nrs_tbf_rule=start nidrd nid={0@lo 192.168.204.7@tcp}&opcode={ost_read} rate=5MB/s ost.OSS.ost_io.nrs_tbf_rule=start nidwr nid={0@lo 192.168.204.7@tcp}&opcode={ost_write} rate=15MB/s regular_requests: CPT 0: nidwr nid={0@lo 192.168.204.7@tcp}&opcode={ost_write} 14, seq 7, type=nid+opcode, ref 0 bps=15000000 nidrd nid={0@lo 192.168.204.7@tcp}&opcode={ost_read} 4, seq 6, type=nid+opcode, ref 0 bps=5000000 default * 10000, seq 1, type=nid+opcode, ref 0 high_priority_requests: CPT 0: nidwr nid={0@lo 192.168.204.7@tcp}&opcode={ost_write} 14, seq 7, type=nid+opcode, ref 0 bps=15000000 nidrd nid={0@lo 192.168.204.7@tcp}&opcode={ost_read} 4, seq 6, type=nid+opcode, ref 0 bps=5000000 default * 10000, seq 1, type=nid+opcode, ref 0 ost.OSS.ost_io.nrs_tbf_rule=stop nidrd ost.OSS.ost_io.nrs_tbf_rule=stop nidwr ost.OSS.ost_io.nrs_tbf_rule=start nidrd nid={0@lo 192.168.204.7@tcp}&opcode={ost_read} rate=5MB ost.OSS.ost_io.nrs_tbf_rule=start nidwr nid={0@lo 192.168.204.7@tcp}&opcode={ost_write} rate=15MB regular_requests: CPT 0: nidwr nid={0@lo 192.168.204.7@tcp}&opcode={ost_write} 14, seq 9, type=nid+opcode, ref 0 bps=15000000 nidrd nid={0@lo 192.168.204.7@tcp}&opcode={ost_read} 4, seq 8, type=nid+opcode, ref 0 bps=5000000 default * 10000, seq 1, type=nid+opcode, ref 0 high_priority_requests: CPT 0: nidwr nid={0@lo 192.168.204.7@tcp}&opcode={ost_write} 14, seq 9, type=nid+opcode, ref 0 bps=15000000 nidrd nid={0@lo 192.168.204.7@tcp}&opcode={ost_read} 4, seq 8, type=nid+opcode, ref 0 bps=5000000 default * 10000, seq 1, type=nid+opcode, ref 0 ost.OSS.ost_io.nrs_tbf_rule=stop nidrd ost.OSS.ost_io.nrs_tbf_rule=stop nidwr ost.OSS.ost_io.nrs_tbf_rule=start nidrd nid={0@lo 192.168.204.7@tcp}&opcode={ost_read} bps=5 ost.OSS.ost_io.nrs_tbf_rule=start nidwr nid={0@lo 192.168.204.7@tcp}&opcode={ost_write} bps=15 regular_requests: CPT 0: nidwr nid={0@lo 192.168.204.7@tcp}&opcode={ost_write} 15, seq 11, type=nid+opcode, ref 0 bps=15728640 nidrd nid={0@lo 192.168.204.7@tcp}&opcode={ost_read} 5, seq 10, type=nid+opcode, ref 0 bps=5242880 default * 10000, seq 1, type=nid+opcode, ref 0 high_priority_requests: CPT 0: nidwr nid={0@lo 192.168.204.7@tcp}&opcode={ost_write} 15, seq 11, type=nid+opcode, ref 0 bps=15728640 nidrd nid={0@lo 192.168.204.7@tcp}&opcode={ost_read} 5, seq 10, type=nid+opcode, ref 0 bps=5242880 default * 10000, seq 1, type=nid+opcode, ref 0 ost.OSS.ost_io.nrs_tbf_rule=stop nidrd ost.OSS.ost_io.nrs_tbf_rule=stop nidwr ost.OSS.ost_io.nrs_tbf_rule=start nidrd nid={0@lo 192.168.204.7@tcp}&opcode={ost_read} bps=5 iops=1000 ost.OSS.ost_io.nrs_tbf_rule=start nidwr nid={0@lo 192.168.204.7@tcp}&opcode={ost_write} bps=15 iops=1200 regular_requests: CPT 0: nidwr nid={0@lo 192.168.204.7@tcp}&opcode={ost_write} 1200, seq 13, type=nid+opcode, ref 0 bps=15728640 nidrd nid={0@lo 192.168.204.7@tcp}&opcode={ost_read} 1000, seq 12, type=nid+opcode, ref 0 bps=5242880 default * 10000, seq 1, type=nid+opcode, ref 0 high_priority_requests: CPT 0: nidwr nid={0@lo 192.168.204.7@tcp}&opcode={ost_write} 1200, seq 13, type=nid+opcode, ref 0 bps=15728640 nidrd nid={0@lo 192.168.204.7@tcp}&opcode={ost_read} 1000, seq 12, type=nid+opcode, ref 0 bps=5242880 default * 10000, seq 1, type=nid+opcode, ref 0 ost.OSS.ost_io.nrs_tbf_rule=stop nidrd ost.OSS.ost_io.nrs_tbf_rule=stop nidwr ost.OSS.ost_io.nrs_policies=fifo PASS 77ua (43s) == sanityn test 77ub: Testing for TBF rate control in BPS performance mode ========================================================== 21:28:11 (1788830891) ost.OSS.ost_io.nrs_policies=tbf nid+opcode ost.OSS.ost_io.nrs_tbf_rule=start nidrd nid={0@lo 192.168.204.7@tcp}&opcode={ost_read} bps=5 ost.OSS.ost_io.nrs_tbf_rule=start nidwr nid={0@lo 192.168.204.7@tcp}&opcode={ost_write} bps=15 ost.OSS.ost_io.nrs_tbf_rule= regular_requests: CPT 0: nidwr nid={0@lo 192.168.204.7@tcp}&opcode={ost_write} 15, seq 3, type=nid+opcode, ref 0 bps=15728640 nidrd nid={0@lo 192.168.204.7@tcp}&opcode={ost_read} 5, seq 2, type=nid+opcode, ref 0 bps=5242880 default * 10000, seq 1, type=nid+opcode, ref 0 high_priority_requests: CPT 0: nidwr nid={0@lo 192.168.204.7@tcp}&opcode={ost_write} 15, seq 3, type=nid+opcode, ref 0 bps=15728640 nidrd nid={0@lo 192.168.204.7@tcp}&opcode={ost_read} 5, seq 2, type=nid+opcode, ref 0 bps=5242880 default * 10000, seq 1, type=nid+opcode, ref 0 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.836304 s, 20.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0716674 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0807902 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0894827 s, 11.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.064627 s, 16.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0730667 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.083028 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0253669 s, 41.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13842 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0433023 s, 24.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0732298 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0926 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242847 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.329258 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.381295 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.300912 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.379863 s, 2.8 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff92ca385af000.lock_unused_count=2 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.271357 s, 3.9 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.301329 s, 3.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.273208 s, 3.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.245835 s, 4.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.251663 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.262256 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.257719 s, 4.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.244117 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.272658 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.333425 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.299015 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.282874 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.289442 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.299265 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.305267 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.402678 s, 2.6 MB/s removed '/mnt/lustre/d77ub.sanityn/nrs_w_oleg407-client.virtnet' removed '/mnt/lustre/d77ub.sanityn/nrs_r_oleg407-client.virtnet' removed directory '/mnt/lustre/d77ub.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 15, read rate: 5, bs: 4096 Verify the write rate is under TBF control 25600+0 records in 25600+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 1684.22 s, 62.3 kB/s Write runtime is 1686 s, speed is .059311 vs. 15 Verify the read rate is under TBF control 25600+0 records in 25600+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 193.836 s, 541 kB/s Read runtime is 195 s, speed is .512820 vs. 5 cleanup_tbf /mnt/lustre/d77ub.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 15, read rate: 5, bs: 65536 Verify the write rate is under TBF control 1600+0 records in 1600+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 107.801 s, 973 kB/s Write runtime is 108 s, speed is .925925 vs. 15 Verify the read rate is under TBF control 1600+0 records in 1600+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 30.6948 s, 3.4 MB/s Read runtime is 31 s, speed is 3.225806 vs. 5 cleanup_tbf /mnt/lustre/d77ub.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 15, read rate: 5, bs: 131072 Verify the write rate is under TBF control 800+0 records in 800+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 57.9166 s, 1.8 MB/s Write runtime is 59 s, speed is 1.694915 vs. 15 Verify the read rate is under TBF control 800+0 records in 800+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 24.456 s, 4.3 MB/s Read runtime is 25 s, speed is 4.000000 vs. 5 cleanup_tbf /mnt/lustre/d77ub.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 15, read rate: 5, bs: 262144 Verify the write rate is under TBF control 400+0 records in 400+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 30.1961 s, 3.5 MB/s Write runtime is 31 s, speed is 3.225806 vs. 15 Verify the read rate is under TBF control 400+0 records in 400+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.8315 s, 5.0 MB/s Read runtime is 22 s, speed is 4.545454 vs. 5 cleanup_tbf /mnt/lustre/d77ub.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 15, read rate: 5, bs: 1048576 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.6646 s, 9.8 MB/s Write runtime is 11 s, speed is 9.090909 vs. 15 Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.0015 s, 5.2 MB/s Read runtime is 21 s, speed is 4.761904 vs. 5 cleanup_tbf /mnt/lustre/d77ub.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Cleanup TBF ost.OSS.ost_io.nrs_tbf_rule=stop nidrd ost.OSS.ost_io.nrs_tbf_rule=stop nidwr ost.OSS.ost_io.nrs_policies=fifo PASS 77ub (2329s) == sanityn test 77uc: TBF: the realtime key does not work with a byte rate ========================================================== 22:07:00 (1788833220) ost.OSS.ost_io.nrs_policies=tbf nid+opcode oleg407-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start rtwr nid={0@lo 192.168.204.7@tcp}&opcode={ost_write} realtime=1 bps=5MiB/s: Operation not supported oleg407-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start rtwr nid={0@lo 192.168.204.7@tcp}&opcode={ost_write} realtime=1 bps=5MiB/s': Operation not supported pdsh@oleg407-client: oleg407-server: ssh exited with exit code 95 ost.OSS.ost_io.nrs_tbf_rule=start rtwr nid={0@lo 192.168.204.7@tcp}&opcode={ost_write} realtime=1 rate=100 ost.OSS.ost_io.nrs_tbf_rule= regular_requests: CPT 0: rtwr nid={0@lo 192.168.204.7@tcp}&opcode={ost_write} 100, seq 2, type=nid+opcode, ref 0 default * 10000, seq 1, type=nid+opcode, ref 0 high_priority_requests: CPT 0: rtwr nid={0@lo 192.168.204.7@tcp}&opcode={ost_write} 100, seq 2, type=nid+opcode, ref 0 default * 10000, seq 1, type=nid+opcode, ref 0 Cleanup TBF ost.OSS.ost_io.nrs_tbf_rule=stop rtwr ost.OSS.ost_io.nrs_policies=fifo PASS 77uc (17s) == sanityn test 77ud: TBF: a rate change on a BPS rule leaves an IOPS rate ========================================================== 22:07:18 (1788833238) ost.OSS.ost_io.nrs_policies=tbf nid+opcode ost.OSS.ost_io.nrs_tbf_rule=start nidrd nid={0@lo 192.168.204.7@tcp}&opcode={ost_read} bps=5MiB/s ost.OSS.ost_io.nrs_tbf_rule=start nidwr nid={0@lo 192.168.204.7@tcp}&opcode={ost_write} bps=5MiB/s ost.OSS.ost_io.nrs_tbf_rule= regular_requests: CPT 0: nidwr nid={0@lo 192.168.204.7@tcp}&opcode={ost_write} 5, seq 3, type=nid+opcode, ref 0 bps=5242880 nidrd nid={0@lo 192.168.204.7@tcp}&opcode={ost_read} 5, seq 2, type=nid+opcode, ref 0 bps=5242880 default * 10000, seq 1, type=nid+opcode, ref 0 high_priority_requests: CPT 0: nidwr nid={0@lo 192.168.204.7@tcp}&opcode={ost_write} 5, seq 3, type=nid+opcode, ref 0 bps=5242880 nidrd nid={0@lo 192.168.204.7@tcp}&opcode={ost_read} 5, seq 2, type=nid+opcode, ref 0 bps=5242880 default * 10000, seq 1, type=nid+opcode, ref 0 ost.OSS.ost_io.nrs_tbf_rule=change nidrd rate=100 ost.OSS.ost_io.nrs_tbf_rule=change nidwr rate=100 ost.OSS.ost_io.nrs_tbf_rule= regular_requests: CPT 0: nidwr nid={0@lo 192.168.204.7@tcp}&opcode={ost_write} 100, seq 3, type=nid+opcode, ref 0 nidrd nid={0@lo 192.168.204.7@tcp}&opcode={ost_read} 100, seq 2, type=nid+opcode, ref 0 default * 10000, seq 1, type=nid+opcode, ref 0 high_priority_requests: CPT 0: nidwr nid={0@lo 192.168.204.7@tcp}&opcode={ost_write} 100, seq 3, type=nid+opcode, ref 0 nidrd nid={0@lo 192.168.204.7@tcp}&opcode={ost_read} 100, seq 2, type=nid+opcode, ref 0 default * 10000, seq 1, type=nid+opcode, ref 0 ost.OSS.ost_io.nrs_tbf_rule=change nidrd rate=20 ost.OSS.ost_io.nrs_tbf_rule=change nidwr rate=20 not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20, bs: 1048576 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.86192 s, 11.8 MB/s Write runtime is 10 s, speed is 10.000000 vs. 20 Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.26075 s, 19.9 MB/s Read runtime is 7 s, speed is 14.285714 vs. 20 cleanup_tbf /mnt/lustre/d77ud.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Cleanup TBF ost.OSS.ost_io.nrs_tbf_rule=stop nidrd ost.OSS.ost_io.nrs_tbf_rule=stop nidwr ost.OSS.ost_io.nrs_policies=fifo PASS 77ud (62s) == sanityn test 77va: Reserved rate mClock: check OPCode NRS policy ========================================================== 22:08:21 (1788833301) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=mclock opcode ost.OSS.ost_io.nrs_mclock_rule=start ost_r opcode={ost_read} resvd_rate=5 rate=5 ost.OSS.ost_io.nrs_mclock_rule=start ost_w opcode={ost_write} resvd_rate=20 rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.544957 s, 30.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0760291 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.062168 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173539 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162486 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213771 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192719 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211763 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208515 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20096 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.244814 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191574 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237458 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.292323 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.303271 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.262734 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.277819 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0271822 s, 38.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0635389 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0455664 s, 23.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0384303 s, 27.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108528 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134813 s, 77.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0573845 s, 18.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0741703 s, 14.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0981454 s, 10.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0803813 s, 13.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0807825 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0966271 s, 10.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0812566 s, 12.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139309 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00671765 s, 156 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0191187 s, 54.8 MB/s removed '/mnt/lustre/d77va.sanityn/nrs_w_oleg407-client.virtnet' removed '/mnt/lustre/d77va.sanityn/nrs_r_oleg407-client.virtnet' removed directory '/mnt/lustre/d77va.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 5, bs: 1048576 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.5661 s, 9.9 MB/s Write runtime is 12 s, speed is 8.333333 vs. 20 Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8439 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 vs. 5 cleanup_tbf /mnt/lustre/d77va.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_mclock_rule=stop ost_r ost.OSS.ost_io.nrs_mclock_rule=stop ost_w ost.OSS.ost_io.nrs_policies=fifo PASS 77va (68s) == sanityn test 77vb: Reserved rate mClock: check UID/GID/JobID/OPCode expression ========================================================== 22:09:29 (1788833369) ost.OSS.ost_io.nrs_policies=mclock jobid+opcode+nid+uid+gid ost.OSS.ost_io.nrs_mclock_rule=start ext_w jobid={dd.500}&opcode={ost_write} resvd_rate=20 rate=20 ost.OSS.ost_io.nrs_mclock_rule=start ext_r jobid={dd.500}&opcode={ost_read} resvd_rate=10 rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77vb.sanityn/nrs_r_oleg407-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.525284 s, 31.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0904578 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115686 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19109 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20756 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.256944 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.277391 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.243229 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.316638 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.271097 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.337694 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.307988 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.356233 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.34856 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.349442 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.337856 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.34844 s, 3.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff92ca385af000.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff92ca385af000.lock_unused_count=1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0371405 s, 28.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0782994 s, 13.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0371173 s, 28.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0592187 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0472344 s, 22.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.090668 s, 11.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0578826 s, 18.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0563885 s, 18.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0660018 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0673498 s, 15.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0643971 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0706257 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0702395 s, 14.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0828009 s, 12.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0865211 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0933198 s, 11.2 MB/s removed '/mnt/lustre/d77vb.sanityn/nrs_w_oleg407-client.virtnet' removed '/mnt/lustre/d77vb.sanityn/nrs_r_oleg407-client.virtnet' removed directory '/mnt/lustre/d77vb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10, bs: 1048576 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77vb.sanityn/tbf] [bs=1048576] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.45567 s, 12.4 MB/s Write runtime is 9 s, speed is 11.111111 vs. 20 Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77vb.sanityn/tbf] [of=/dev/null] [bs=1048576] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0426 s, 10.4 MB/s Read runtime is 11 s, speed is 9.090909 vs. 10 cleanup_tbf /mnt/lustre/d77vb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_mclock_rule=stop ext_w ost.OSS.ost_io.nrs_mclock_rule=stop ext_r ost.OSS.ost_io.nrs_mclock_rule=start ext_w nid={0@lo 192.168.204.7@tcp}&opcode={ost_write} resvd_rate=20 rate=20 ost.OSS.ost_io.nrs_mclock_rule=start ext_r nid={0@lo 192.168.204.7@tcp}&opcode={ost_read} resvd_rate=10 rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.673836 s, 24.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0919374 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108481 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119682 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194006 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.255482 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.286526 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.291527 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.289074 s, 3.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.260117 s, 4.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.315112 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.351291 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.302296 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.336736 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.30896 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.324564 s, 3.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff92ca385af000.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff92ca385af000.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.373989 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0486648 s, 21.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0357405 s, 29.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0288866 s, 36.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0400127 s, 26.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0754897 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0429159 s, 24.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0795566 s, 13.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0819917 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0587787 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0813974 s, 12.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.07761 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0773399 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0717447 s, 14.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0685861 s, 15.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0712489 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111722 s, 9.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff92ca385af000.lock_unused_count=4 removed '/mnt/lustre/d77vb.sanityn/nrs_r_oleg407-client.virtnet' removed '/mnt/lustre/d77vb.sanityn/nrs_w_oleg407-client.virtnet' removed directory '/mnt/lustre/d77vb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10, bs: 1048576 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.68404 s, 10.8 MB/s Write runtime is 11 s, speed is 9.090909 vs. 20 Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.97956 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 vs. 10 cleanup_tbf /mnt/lustre/d77vb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_mclock_rule=stop ext_w ost.OSS.ost_io.nrs_mclock_rule=stop ext_r ost.OSS.ost_io.nrs_mclock_rule=start ext nid={0@lo 192.168.204.7@tcp}&jobid={dd.500} resvd_rate=20 rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77vb.sanityn/nrs_r_oleg407-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.806698 s, 20.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0657735 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110067 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164873 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137736 s, 7.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.239382 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.260933 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.255949 s, 4.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.357258 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.325022 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.296823 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.328038 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.389632 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.335331 s, 3.1 MB/s1+0 records in 1+0 records out ldlm.namespaces.lustre-OST0000-osc-ffff92ca385af000.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff92ca385af000.lock_unused_count=5 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.330486 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.443818 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.384998 s, 2.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104806 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107525 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0609672 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0576412 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114475 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0526888 s, 19.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0884173 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0945291 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114062 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0722045 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102507 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0607234 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0726338 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115988 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0917866 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0979042 s, 10.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff92ca385af000.lock_unused_count=4 removed '/mnt/lustre/d77vb.sanityn/nrs_w_oleg407-client.virtnet' removed '/mnt/lustre/d77vb.sanityn/nrs_r_oleg407-client.virtnet' removed directory '/mnt/lustre/d77vb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20, bs: 1048576 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77vb.sanityn/tbf] [bs=1048576] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.2194 s, 9.3 MB/s Write runtime is 12 s, speed is 8.333333 vs. 20 Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77vb.sanityn/tbf] [of=/dev/null] [bs=1048576] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.4027 s, 19.4 MB/s Read runtime is 7 s, speed is 14.285714 vs. 20 cleanup_tbf /mnt/lustre/d77vb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_mclock_rule=stop ext ost.OSS.ost_io.nrs_mclock_rule=start ext_a jobid={dd.500},opcode={ost_write} resvd_rate=20 rate=20 ost.OSS.ost_io.nrs_mclock_rule=start ext_b jobid={dd.500},opcode={ost_read} resvd_rate=10 rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77vb.sanityn/nrs_r_oleg407-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.715099 s, 23.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0790223 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0943143 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110322 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180469 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169292 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208989 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.231968 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207405 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.243716 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198154 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.248744 s, 4.2 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.281603 s, 3.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.302773 s, 3.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.226309 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.281147 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.313763 s, 3.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00740858 s, 142 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0329365 s, 31.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132538 s, 79.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0754104 s, 13.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0366129 s, 28.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0517375 s, 20.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.034386 s, 30.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0444276 s, 23.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0767347 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0639031 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0588521 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.057879 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0489239 s, 21.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0492485 s, 21.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0897684 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0659844 s, 15.9 MB/s removed '/mnt/lustre/d77vb.sanityn/nrs_w_oleg407-client.virtnet' removed '/mnt/lustre/d77vb.sanityn/nrs_r_oleg407-client.virtnet' removed directory '/mnt/lustre/d77vb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10, bs: 1048576 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77vb.sanityn/tbf] [bs=1048576] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.714 s, 9.8 MB/s Write runtime is 12 s, speed is 8.333333 vs. 10 Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77vb.sanityn/tbf] [of=/dev/null] [bs=1048576] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.97839 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 vs. 10 cleanup_tbf /mnt/lustre/d77vb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10, bs: 1048576 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.61989 s, 13.8 MB/s Write runtime is 8 s, speed is 12.500000 vs. 20 Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.9699 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 vs. 10 cleanup_tbf /mnt/lustre/d77vb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_mclock_rule=stop ext_a ost.OSS.ost_io.nrs_mclock_rule=stop ext_b ost.OSS.ost_io.nrs_mclock_rule=start ext_ug uid={500}&gid={500} resvd_rate=5 rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77vb.sanityn/nrs_r_oleg407-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.31995 s, 12.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0597318 s, 17.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0674987 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182707 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.249011 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.243891 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.273638 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.279859 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.254606 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.28175 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.262205 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242678 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.284334 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.299915 s, 3.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.272004 s, 3.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.329295 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.30464 s, 3.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0524642 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00917472 s, 114 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00583855 s, 180 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0415798 s, 25.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.094997 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0667707 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0507992 s, 20.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00805648 s, 130 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0524103 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00953367 s, 110 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0488877 s, 21.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0215237 s, 48.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0955932 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0543085 s, 19.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0793453 s, 13.2 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120905 s, 8.7 MB/s removed '/mnt/lustre/d77vb.sanityn/nrs_r_oleg407-client.virtnet' removed '/mnt/lustre/d77vb.sanityn/nrs_w_oleg407-client.virtnet' removed directory '/mnt/lustre/d77vb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5, bs: 1048576 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77vb.sanityn/tbf] [bs=1048576] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9241 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 vs. 5 Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77vb.sanityn/tbf] [of=/dev/null] [bs=1048576] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8669 s, 5.3 MB/s Read runtime is 20 s, speed is 5.000000 vs. 5 cleanup_tbf /mnt/lustre/d77vb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_mclock_rule=stop ext_ug ost.OSS.ost_io.nrs_mclock_rule=start ext_uw uid={500}&opcode={ost_write} resvd_rate=20 rate=20 ost.OSS.ost_io.nrs_mclock_rule=start ext_ur uid={500}&opcode={ost_read} resvd_rate=10 rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77vb.sanityn/nrs_r_oleg407-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.727115 s, 23.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110756 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0667265 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128684 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0650404 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0721004 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0272689 s, 38.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151475 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127267 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151073 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0382044 s, 27.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff92ca385af000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.280071 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.268062 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.330425 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.232657 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.294764 s, 3.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.364859 s, 2.9 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff92ca385af000.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0811047 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.076523 s, 13.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0293231 s, 35.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0967927 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.072837 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0356329 s, 29.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0443973 s, 23.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113804 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0558126 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0577613 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0724501 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0610126 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0825652 s, 12.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0609503 s, 17.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0620623 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00729412 s, 144 MB/s removed '/mnt/lustre/d77vb.sanityn/nrs_r_oleg407-client.virtnet' removed '/mnt/lustre/d77vb.sanityn/nrs_w_oleg407-client.virtnet' removed directory '/mnt/lustre/d77vb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10, bs: 1048576 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77vb.sanityn/tbf] [bs=1048576] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.94723 s, 10.5 MB/s Write runtime is 11 s, speed is 9.090909 vs. 20 Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77vb.sanityn/tbf] [of=/dev/null] [bs=1048576] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.95324 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 vs. 10 cleanup_tbf /mnt/lustre/d77vb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_mclock_rule=stop ext_uw ost.OSS.ost_io.nrs_mclock_rule=stop ext_ur ost.OSS.ost_io.nrs_mclock_rule=start ext_a uid={500},opcode={ost_write} resvd_rate=20 rate=20 ost.OSS.ost_io.nrs_mclock_rule=start ext_b uid={500},opcode={ost_read} resvd_rate=10 rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77vb.sanityn/nrs_r_oleg407-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.873069 s, 19.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0513382 s, 20.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0939446 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106872 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114348 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179986 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.265712 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.275735 s, 3.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.299929 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.296136 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.344532 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.312475 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.322327 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.401904 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.426143 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.433159 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.372567 s, 2.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0416406 s, 25.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0182302 s, 57.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0660007 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0388163 s, 27.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0277904 s, 37.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0327334 s, 32.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0525578 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.046854 s, 22.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0779208 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0824417 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0565851 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0108308 s, 96.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0584083 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0649557 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.059099 s, 17.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0631633 s, 16.6 MB/s removed '/mnt/lustre/d77vb.sanityn/nrs_w_oleg407-client.virtnet' removed '/mnt/lustre/d77vb.sanityn/nrs_r_oleg407-client.virtnet' removed directory '/mnt/lustre/d77vb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10, bs: 1048576 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77vb.sanityn/tbf] [bs=1048576] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.9877 s, 9.5 MB/s Write runtime is 12 s, speed is 8.333333 vs. 10 Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77vb.sanityn/tbf] [of=/dev/null] [bs=1048576] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.93775 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 vs. 10 cleanup_tbf /mnt/lustre/d77vb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10, bs: 1048576 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77vb.sanityn/tbf] [bs=1048576] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.7568 s, 9.7 MB/s Write runtime is 12 s, speed is 8.333333 vs. 20 Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77vb.sanityn/tbf] [of=/dev/null] [bs=1048576] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0543 s, 10.4 MB/s Read runtime is 11 s, speed is 9.090909 vs. 10 cleanup_tbf /mnt/lustre/d77vb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Cleanup TBF ost.OSS.ost_io.nrs_mclock_rule=stop ext_a ost.OSS.ost_io.nrs_mclock_rule=stop ext_b ost.OSS.ost_io.nrs_policies=fifo PASS 77vb (449s) == sanityn test 77vc: Reserved rate mClock: type granular combination: nid+opcode ========================================================== 22:16:58 (1788833818) ost.OSS.ost_io.nrs_policies=mclock nid+opcode ost.OSS.ost_io.nrs_mclock_rule=start ext_w nid={0@lo 192.168.204.7@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_mclock_rule=start ext_r nid={0@lo 192.168.204.7@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.563391 s, 29.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0755699 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0715693 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.218766 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.272935 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196394 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.283856 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.263782 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.306331 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.256022 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.316189 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.27187 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.263098 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.32667 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.297212 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.317273 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.318635 s, 3.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff92ca385af000.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff92ca385af000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0419046 s, 25.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0799328 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0827957 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0657924 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0899425 s, 11.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110904 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0931692 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126165 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0668967 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0999134 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126912 s, 8.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132575 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.088318 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0763264 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10977 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0968497 s, 10.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff92ca385af000.lock_unused_count=3 removed '/mnt/lustre/d77vc.sanityn/nrs_r_oleg407-client.virtnet' removed '/mnt/lustre/d77vc.sanityn/nrs_w_oleg407-client.virtnet' removed directory '/mnt/lustre/d77vc.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10, bs: 1048576 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.151 s, 12.9 MB/s Write runtime is 9 s, speed is 11.111111 vs. 20 Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.9482 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 vs. 10 cleanup_tbf /mnt/lustre/d77vc.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Cleanup TBF ost.OSS.ost_io.nrs_mclock_rule=stop ext_w ost.OSS.ost_io.nrs_mclock_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77vc (59s) == sanityn test 77vd: Reserved rate mClock: rate control in BPS performance mode ========================================================== 22:17:57 (1788833877) ost.OSS.ost_io.nrs_policies=mclock nid+opcode ost.OSS.ost_io.nrs_mclock_rule=start nidrd nid={0@lo 192.168.204.7@tcp}&opcode={ost_read} resvd_bps=5 bps=5 ost.OSS.ost_io.nrs_mclock_rule=start nidwr nid={0@lo 192.168.204.7@tcp}&opcode={ost_write} resvd_bps=15 bps=15 ost.OSS.ost_io.nrs_mclock_rule= regular_requests: CPT 0: nidwr nid={0@lo 192.168.204.7@tcp}&opcode={ost_write}, bps 15-15, seq 3, type=nid+opcode, ref 0 nidrd nid={0@lo 192.168.204.7@tcp}&opcode={ost_read}, bps 5-5, seq 2, type=nid+opcode, ref 0 default *, limit_iops 10000, seq 1, type=nid+opcode, ref 0 high_priority_requests: CPT 0: nidwr nid={0@lo 192.168.204.7@tcp}&opcode={ost_write}, bps 15-15, seq 3, type=nid+opcode, ref 0 nidrd nid={0@lo 192.168.204.7@tcp}&opcode={ost_read}, bps 5-5, seq 2, type=nid+opcode, ref 0 default *, limit_iops 10000, seq 1, type=nid+opcode, ref 0 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.674419 s, 24.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0522583 s, 20.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100301 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0941908 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0819365 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144886 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0771749 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18373 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161988 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14349 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.233118 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.278767 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195705 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.29561 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.320183 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.530428 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.433761 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.073398 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0712378 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0841774 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0573216 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0506885 s, 20.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0575517 s, 18.2 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.061421 s, 17.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0722496 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0610385 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0781635 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.06928 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0635888 s, 16.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0705085 s, 14.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0832183 s, 12.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0754233 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0790892 s, 13.3 MB/s removed '/mnt/lustre/d77vd.sanityn/nrs_r_oleg407-client.virtnet' removed '/mnt/lustre/d77vd.sanityn/nrs_w_oleg407-client.virtnet' removed directory '/mnt/lustre/d77vd.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 15, read rate: 5, bs: 4096 Verify the write rate is under TBF control 25600+0 records in 25600+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 1676.76 s, 62.5 kB/s Write runtime is 1677 s, speed is .059630 vs. 15 Verify the read rate is under TBF control 25600+0 records in 25600+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 174.117 s, 602 kB/s Read runtime is 175 s, speed is .571428 vs. 5 cleanup_tbf /mnt/lustre/d77vd.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 15, read rate: 5, bs: 65536 Verify the write rate is under TBF control 1600+0 records in 1600+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 121.727 s, 861 kB/s Write runtime is 123 s, speed is .813008 vs. 15 Verify the read rate is under TBF control 1600+0 records in 1600+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 27.3463 s, 3.8 MB/s Read runtime is 28 s, speed is 3.571428 vs. 5 cleanup_tbf /mnt/lustre/d77vd.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 15, read rate: 5, bs: 131072 Verify the write rate is under TBF control 800+0 records in 800+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 50.7459 s, 2.1 MB/s Write runtime is 52 s, speed is 1.923076 vs. 15 Verify the read rate is under TBF control 800+0 records in 800+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 21.4028 s, 4.9 MB/s Read runtime is 22 s, speed is 4.545454 vs. 5 cleanup_tbf /mnt/lustre/d77vd.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 15, read rate: 5, bs: 262144 Verify the write rate is under TBF control 400+0 records in 400+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 24.8648 s, 4.2 MB/s Write runtime is 26 s, speed is 3.846153 vs. 15 Verify the read rate is under TBF control 400+0 records in 400+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.0902 s, 5.2 MB/s Read runtime is 21 s, speed is 4.761904 vs. 5 cleanup_tbf /mnt/lustre/d77vd.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 15, read rate: 5, bs: 1048576 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.2972 s, 10.2 MB/s Write runtime is 11 s, speed is 9.090909 vs. 15 Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8696 s, 5.3 MB/s Read runtime is 20 s, speed is 5.000000 vs. 5 cleanup_tbf /mnt/lustre/d77vd.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Cleanup TBF ost.OSS.ost_io.nrs_mclock_rule=stop nidrd ost.OSS.ost_io.nrs_mclock_rule=stop nidwr ost.OSS.ost_io.nrs_policies=fifo PASS 77vd (2281s) == sanityn test 77ve: mClock: rule Reserved and Limit rate setting ========================================================== 22:55:58 (1788836158) ost.OSS.ost_io.nrs_policies=mclock projid reserve=1 limit=1 oleg407-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_mclock_rule=start proj100 projid={100} resvd_rate=100 limit_rate=50: Numerical result out of range oleg407-server: error: set_param: setting 'ost/OSS/ost_io/nrs_mclock_rule'='start proj100 projid={100} resvd_rate=100 limit_rate=50': Numerical result out of range pdsh@oleg407-client: oleg407-server: ssh exited with exit code 34 ost.OSS.ost_io.nrs_mclock_rule=start proj100 projid={100} iops=100 Rule rate: limit_iops 100 ost.OSS.ost_io.nrs_mclock_rule=stop proj100 ost.OSS.ost_io.nrs_mclock_rule=start proj100 projid={100} rate=5MiB/s Rule rate: limit_bps 5 ost.OSS.ost_io.nrs_mclock_rule=stop proj100 ost.OSS.ost_io.nrs_mclock_rule=start proj100 projid={100} limit_rate=300MiB/s Rule rate: limit_bps 300 ost.OSS.ost_io.nrs_mclock_rule=stop proj100 ost.OSS.ost_io.nrs_mclock_rule=start proj100 projid={100} resvd_rate=100 limit_rate=300 Rule rate: iops 100-300 ost.OSS.ost_io.nrs_mclock_rule=stop proj100 ost.OSS.ost_io.nrs_mclock_rule=start proj100 projid={100} resvd_rate=10MiB limit_rate=30MiB Rule rate: bps 10-30 ost.OSS.ost_io.nrs_mclock_rule=stop proj100 ost.OSS.ost_io.nrs_mclock_rule=start proj100 projid={100} resvd_rate=10MiB limit_rate=30MiB resvd_iops=5 limit_iops=25 Rule rate: iops 5-25 bps 10-30 ost.OSS.ost_io.nrs_mclock_rule=stop proj100 ost.OSS.ost_io.nrs_mclock_rule=start proj100 projid={100} bps=5MiB/s Rule rate: limit_bps 5 ost.OSS.ost_io.nrs_mclock_rule=stop proj100 ost.OSS.ost_io.nrs_mclock_rule=start proj100 projid={100} limit_bps=10MiB/s ost.OSS.ost_io.nrs_mclock_rule=change proj100 limit_rate=100 ost.OSS.ost_io.nrs_mclock_rule= regular_requests: CPT 0: proj100 projid={100}, limit_iops 100, seq 9, type=projid, ref 0 default *, limit_iops 10000, seq 1, type=projid, ref 0 high_priority_requests: CPT 0: proj100 projid={100}, limit_iops 100, seq 9, type=projid, ref 0 default *, limit_iops 10000, seq 1, type=projid, ref 0 Rule rate: limit_iops 100 ost.OSS.ost_io.nrs_mclock_rule=stop proj100 ost.OSS.ost_io.nrs_mclock_rule=start proj100 projid={100} resvd_rate=10 limit_rate=300 ost.OSS.ost_io.nrs_mclock_rule=change proj100 limit_rate=200 Rule rate: iops 10-200 ost.OSS.ost_io.nrs_mclock_rule=stop proj100 ost.OSS.ost_io.nrs_mclock_rule=start proj100 projid={100} limit_bps=5000B/s Rule rate: limit_bps 5000B/s ost.OSS.ost_io.nrs_mclock_rule=stop proj100 ost.OSS.ost_io.nrs_mclock_rule=start proj100 projid={100} limit_bps=1536KiB/s Rule rate: limit_bps 1572864B/s Cleanup TBF ost.OSS.ost_io.nrs_mclock_rule=stop proj100 ost.OSS.ost_io.nrs_policies=fifo PASS 77ve (44s) == sanityn test 77vf: mClock: I/O testing with both Reserved and Limit rate control ========================================================== 22:56:43 (1788836203) using SAVE_PROJECT_SUPPORTED=0 ost.OSS.ost_io.nrs_policies=mclock projid reserve=1 limit=1 ost.OSS.ost_io.nrs_mclock_rule=start proj100 projid={100} resvd_rate=5 limit_rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.62402 s, 26.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0750814 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0648732 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137618 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173926 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183706 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202961 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.230064 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.25438 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.280019 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198393 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.219781 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.27038 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.253401 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.235587 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.299381 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.247052 s, 4.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff92ca385af000.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffff92ca385af000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0695397 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0897847 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112102 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159462 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0971546 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0955179 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117283 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0684284 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114018 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0784495 s, 13.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0857353 s, 12.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0827577 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0873664 s, 12.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0989331 s, 10.6 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128374 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10123 s, 10.4 MB/s removed '/mnt/lustre/d77vf.sanityn/nrs_r_oleg407-client.virtnet' removed '/mnt/lustre/d77vf.sanityn/nrs_w_oleg407-client.virtnet' removed directory '/mnt/lustre/d77vf.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20, bs: 1048576 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.49004 s, 11.0 MB/s Write runtime is 11 s, speed is 9.090909 vs. 20 Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.36804 s, 19.5 MB/s Read runtime is 7 s, speed is 14.285714 vs. 20 cleanup_tbf /mnt/lustre/d77vf.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Cleanup TBF ost.OSS.ost_io.nrs_mclock_rule=stop proj100 ost.OSS.ost_io.nrs_policies=fifo PASS 77vf (59s) == sanityn test 77vg: mClock: multiple flows with different reserved and limit rates ========================================================== 22:57:43 (1788836263) using SAVE_PROJECT_SUPPORTED=0 100 P /mnt/lustre/d77vg.sanityn/f77vg.sanityn.100 200 P /mnt/lustre2/d77vg.sanityn/f77vg.sanityn.200 ost.OSS.ost_io.nrs_policies=mclock projid reserve=1 limit=1 ost.OSS.ost_io.nrs_mclock_rule=start proj100 projid={100} resvd_rate=5 limit_rate=20 ost.OSS.ost_io.nrs_mclock_rule=start proj200 projid={200} resvd_rate=10 limit_rate=15 cpu_npartitions on ost1 is 1 mClock: Run 2 flows in parallel mClock - file: /mnt/lustre/d77vg.sanityn/f77vg.sanityn.100 mClock - rw: w, resvd: 5, limit: 20 WRITE: resvd=5 limit=20 file=/mnt/lustre/d77vg.sanityn/f77vg.sanityn.100 (np=1) mClock - file: /mnt/lustre2/d77vg.sanityn/f77vg.sanityn.200 mClock - rw: w, resvd: 10, limit: 15 WRITE: resvd=10 limit=15 file=/mnt/lustre2/d77vg.sanityn/f77vg.sanityn.200 (np=1) 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.3215 s, 11.2 MB/s Write runtime is 11 s, speed is 9.090909 IOPS === DONE: rate=9.090909 within 5-20 and np=1 === 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.98774 s, 10.5 MB/s Write runtime is 11 s, speed is 9.090909 IOPS === DONE: rate=9.090909 within 10-15 and np=1 === cpu_npartitions on ost1 is 1 mClock: Run 2 flows in parallel mClock - file: /mnt/lustre2/d77vg.sanityn/f77vg.sanityn.200 mClock - rw: r, resvd: 10, limit: 15 mClock - file: /mnt/lustre/d77vg.sanityn/f77vg.sanityn.100 mClock - rw: r, resvd: 5, limit: 20 READ: resvd=10 limit=15 file=/mnt/lustre2/d77vg.sanityn/f77vg.sanityn.200 (np=1) READ: resvd=5 limit=20 file=/mnt/lustre/d77vg.sanityn/f77vg.sanityn.100 (np=1) 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.27151 s, 19.9 MB/s Read runtime is 6 s, speed is 16.666666 IOPS === DONE: rate=16.666666 within 5-20 and np=1 === 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.82804 s, 15.4 MB/s Read runtime is 8 s, speed is 12.500000 IOPS === DONE: rate=12.500000 within 10-15 and np=1 === cpu_npartitions on ost1 is 1 mClock: Run 2 flows in parallel mClock - file: /mnt/lustre/d77vg.sanityn/f77vg.sanityn.100 mClock - rw: r, resvd: 5, limit: 20 READ: resvd=5 limit=20 file=/mnt/lustre/d77vg.sanityn/f77vg.sanityn.100 (np=1) mClock - file: /mnt/lustre2/d77vg.sanityn/f77vg.sanityn.200 mClock - rw: w, resvd: 10, limit: 15 WRITE: resvd=10 limit=15 file=/mnt/lustre2/d77vg.sanityn/f77vg.sanityn.200 (np=1) 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.20678 s, 20.1 MB/s Read runtime is 7 s, speed is 14.285714 IOPS === DONE: rate=14.285714 within 5-20 and np=1 === 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.8797 s, 11.8 MB/s Write runtime is 10 s, speed is 10.000000 IOPS === DONE: rate=10.000000 within 10-15 and np=1 === cpu_npartitions on ost1 is 1 mClock: Run 2 flows in parallel mClock - file: /mnt/lustre2/d77vg.sanityn/f77vg.sanityn.200 mClock - rw: r, resvd: 10, limit: 15 READ: resvd=10 limit=15 file=/mnt/lustre2/d77vg.sanityn/f77vg.sanityn.200 (np=1) mClock - file: /mnt/lustre/d77vg.sanityn/f77vg.sanityn.100 mClock - rw: w, resvd: 5, limit: 20 WRITE: resvd=5 limit=20 file=/mnt/lustre/d77vg.sanityn/f77vg.sanityn.100 (np=1) 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.7655 s, 15.5 MB/s Read runtime is 8 s, speed is 12.500000 IOPS === DONE: rate=12.500000 within 10-15 and np=1 === 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.77117 s, 13.5 MB/s Write runtime is 9 s, speed is 11.111111 IOPS === DONE: rate=11.111111 within 5-20 and np=1 === ost.OSS.ost_io.nrs_mclock_rule=stop proj100 ost.OSS.ost_io.nrs_mclock_rule=stop proj200 ost.OSS.ost_io.nrs_policies=fifo removed '/mnt/lustre/d77vg.sanityn/f77vg.sanityn.100' removed '/mnt/lustre2/d77vg.sanityn/f77vg.sanityn.200' PASS 77vg (58s) == sanityn test 77vm: mClock: reserve=0 turns the reserve dimension off ========================================================== 22:58:40 (1788836320) oleg407-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=mclock projid reserve=0: Invalid argument oleg407-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='mclock projid reserve=0': Invalid argument pdsh@oleg407-client: oleg407-server: ssh exited with exit code 22 ost.OSS.ost_io.nrs_policies=mclock projid reserve=0 limit=1 ost.OSS.ost_io.nrs_mclock_rule=start lim100 projid={100} limit_rate=100 Rule rate: limit_iops 100 oleg407-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_mclock_rule=start res100 projid={200} resvd_rate=100: Invalid argument oleg407-server: error: set_param: setting 'ost/OSS/ost_io/nrs_mclock_rule'='start res100 projid={200} resvd_rate=100': Invalid argument pdsh@oleg407-client: oleg407-server: ssh exited with exit code 22 ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=mclock projid reserve=0 limit=0 weight=1 Cleanup TBF ost.OSS.ost_io.nrs_mclock_rule=stop lim100 ost.OSS.ost_io.nrs_policies=fifo PASS 77vm (22s) == sanityn test 77vh: mClock: weight share fair queueing ========================================================== 22:59:02 (1788836342) using SAVE_PROJECT_SUPPORTED=0 /usr/bin/fio /usr/bin/which: no jq in (/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:) SKIP: sanityn test_77vh jq is not installed SKIP 77vh (6s) == sanityn test 77vr: mClock: a reserve only policy schedules the default class ========================================================== 22:59:08 (1788836348) using SAVE_PROJECT_SUPPORTED=0 ost.OSS.ost_io.nrs_policies=mclock projid reserve=1 limit=0 Default rule rate: resvd_iops 10000 ost.OSS.ost_io.nrs_mclock_rule=start proj100 projid={100} resvd_rate=100 8+0 records in 8+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.774422 s, 10.8 MB/s 8+0 records in 8+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.217146 s, 38.6 MB/s Cleanup TBF ost.OSS.ost_io.nrs_mclock_rule=stop proj100 ost.OSS.ost_io.nrs_policies=fifo PASS 77vr (20s) == sanityn test 78: Enable policy and specify tunings right away ========================================================== 22:59:29 (1788836369) oleg407-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_orr_quantum=1: No such device oleg407-server: error: set_param: setting 'ost/OSS/*/nrs_orr_quantum'='1': No such device pdsh@oleg407-client: oleg407-server: ssh exited with exit code 19 ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_policies=fifo PASS 78 (9s) == sanityn test 79: xattr: intent error ================== 22:59:38 (1788836378) File: /mnt/lustre/d79.sanityn Size: 16384 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115272381366532 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-07 22:59:40.000000000 -0400 Modify: 2026-09-07 22:59:40.000000000 -0400 Change: 2026-09-07 22:59:40.000000000 -0400 Birth: 2026-09-07 22:59:40.000000000 -0400 fail_val=0 fail_loc=0x80000160 fail_val=0 fail_loc=0x80000131 # file: mnt/lustre/d79.sanityn trusted.name1="value1" PASS 79 (19s) == sanityn test 80a: migrate directory when some children is being opened ========================================================== 22:59:57 (1788836397) SKIP: sanityn test_80a needs >= 2 MDTs SKIP 80a (3s) == sanityn test 80b: Accessing directory during migration ========================================================== 23:00:00 (1788836400) SKIP: sanityn test_80b needs >= 2 MDTs SKIP 80b (3s) == sanityn test 81a: rename and stat under striped directory ========================================================== 23:00:03 (1788836403) SKIP: sanityn test_81a needs >= 2 MDTs SKIP 81a (4s) == sanityn test 81b: rename under striped directory doesn't deadlock ========================================================== 23:00:07 (1788836407) SKIP: sanityn test_81b We need at least 2 MDTs for this test SKIP 81b (3s) == sanityn test 81c: rename revoke LOOKUP lock for remote object ========================================================== 23:00:10 (1788836410) SKIP: sanityn test_81c needs >= 4 MDTs SKIP 81c (3s) == sanityn test 81d: parallel rename file cross-dir on same MDT ========================================================== 23:00:14 (1788836414) - open/close 781 (time 1788836430.01 total 10.01 last 78.05) - open/close 1507 (time 1788836440.02 total 20.02 last 72.52) - open/close 2174 (time 1788836450.03 total 30.03 last 66.60) total: 2500 open/close in 34.82 seconds: 71.80 ops/second snapshot_time 1788836550.689722868 secs.nsecs start_time 1788836417.393997542 secs.nsecs elapsed_time 133.295725326 secs.nsecs open 2505 samples [usecs] 140 19947 3987057 10993527847 close 2504 samples [usecs] 156 8238 917776 573940618 mknod 2501 samples [usecs] 612 19389 3494445 8928805517 unlink 2500 samples [usecs] 852 83769 5288317 27751829259 mkdir 2 samples [usecs] 1259 3208 4467 11876345 rmdir 2 samples [usecs] 13580 18394 31974 522755636 rename 2500 samples [usecs] 505 54234 17331477 239809909357 getattr 3760 samples [usecs] 10 5719 242166 126298956 parallel_rename_file 2500 samples [usecs] 505 54234 17331477 239809909357 crossdir_rename 2500 samples [usecs] 505 54234 17331477 239809909357 readpage 6 samples [usecs] 6187 40545 119560 3223710848 PASS 81d (142s) == sanityn test 82: fsetxattr and fgetxattr on orphan files ========================================================== 23:02:36 (1788836556) multiop /mnt/lustre/f82.sanityn vO_ac TMPPIPE=/tmp/multiop_open_wait_pipe.7612 multiop /mnt/lustre2/f82.sanityn vo_Ac TMPPIPE=/tmp/multiop_open_wait_pipe.7612 PASS 82 (7s) == sanityn test 83: access striped directory while it is being created/unlinked ========================================================== 23:02:43 (1788836563) SKIP: sanityn test_83 needs >= 2 MDTs SKIP 83 (3s) == sanityn test 84: 0-nlink race in lu_object_find() ===== 23:02:46 (1788836566) fail_loc=0x8000060b PASS 84 (12s) == sanityn test 85: Lustre API root cache race =========== 23:02:58 (1788836578) Starting: llapi_root_test /mnt/lustre/f85.sanityn /mnt/lustre2/f85.sanityn Starting test test1 at 1788836580 Finishing test test1 at 1788836584 PASS 85 (11s) == sanityn test 90: open/create and unlink striped directory ========================================================== 23:03:09 (1788836589) SKIP: sanityn test_90 needs >= 2 MDTs SKIP 90 (3s) == sanityn test 91: chmod and unlink striped directory === 23:03:12 (1788836592) SKIP: sanityn test_91 needs >= 2 MDTs SKIP 91 (4s) == sanityn test 92: create remote directory under orphan directory ========================================================== 23:03:16 (1788836596) SKIP: sanityn test_92 needs >= 2 MDTs SKIP 92 (5s) == sanityn test 93: alloc_rr should not allocate on same ost ========================================================== 23:03:21 (1788836601) fail_loc=0x00000163 fail_loc=0x0 /mnt/lustre/f93.sanityn-1/file1 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 6243 0x1863 0x280000400 0 6359 0x18d7 0x240000400 /mnt/lustre2/f93.sanityn-2/file2 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 6360 0x18d8 0x240000400 1 6244 0x1864 0x280000400 rc1=0 and rc2=0 PASS 93 (19s) == sanityn test 94: signal vs CP callback race =========== 23:03:40 (1788836620) osc.lustre-OST0000-osc-ffff92ca103df800.idle_timeout=0 osc.lustre-OST0000-osc-ffff92ca385af000.idle_timeout=0 osc.lustre-OST0001-osc-ffff92ca103df800.idle_timeout=0 osc.lustre-OST0001-osc-ffff92ca385af000.idle_timeout=0 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.34018 s, 24.1 kB/s fail_val=5 fail_loc=0x80000312 fail_val=6 fail_loc=0x80000329 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 2.96286 s, 1.4 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 7821: 334687 Alarm clock dd if=/dev/zero of=$DIR/$tfile conv=notrunc oflag=append bs=4k count=1 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.163231 s, 25.1 kB/s osc.lustre-OST0000-osc-ffff92ca103df800.idle_timeout=debug osc.lustre-OST0000-osc-ffff92ca385af000.idle_timeout=debug osc.lustre-OST0001-osc-ffff92ca103df800.idle_timeout=debug osc.lustre-OST0001-osc-ffff92ca385af000.idle_timeout=debug PASS 94 (18s) == sanityn test 95a: Check readpage() on a page that was removed from page cache ========================================================== 23:03:59 (1788836639) llite.lustre-ffff92ca103df800.fast_read=0 llite.lustre-ffff92ca385af000.fast_read=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0922496 s, 22.7 MB/s fail_loc=0x80001421 fail_val=7 fail_loc=0x80001422 fail_val=10 llite.lustre-ffff92ca103df800.fast_read=1 llite.lustre-ffff92ca385af000.fast_read=1 PASS 95a (20s) == sanityn test 95b: Check readpage() on a page that is no longer uptodate ========================================================== 23:04:19 (1788836659) llite.lustre-ffff92ca103df800.fast_read=0 llite.lustre-ffff92ca385af000.fast_read=0 1+0 records in 1+0 records out 12288 bytes (12 kB, 12 KiB) copied, 0.0111895 s, 1.1 MB/s fail_loc=0x80001424 fail_val=5 fail_loc=0 llite.lustre-ffff92ca103df800.fast_read=1 llite.lustre-ffff92ca385af000.fast_read=1 PASS 95b (12s) == sanityn test 100a: DoM: glimpse RPCs for stat without IO lock (DoM only file) ========================================================== 23:04:31 (1788836671) SKIP: sanityn test_100a Reserved for glimpse-ahead SKIP 100a (4s) == sanityn test 100b: DoM: no glimpse RPC for stat with IO lock (DoM only file) ========================================================== 23:04:35 (1788836675) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00407393 s, 1.0 MB/s /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK PASS 100b (6s) == sanityn test 100c: DoM: write vs stat without IO lock (combined file) ========================================================== 23:04:42 (1788836682) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.168461 s, 12.4 MB/s /mnt/lustre/d100c.sanityn/dom has type file OK /mnt/lustre/d100c.sanityn/dom has size 2097152 OK PASS 100c (7s) == sanityn test 100d: DoM: write+truncate vs stat without IO lock (combined file) ========================================================== 23:04:48 (1788836688) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.14355 s, 14.6 MB/s /mnt/lustre/d100d.sanityn/dom has type file OK /mnt/lustre/d100d.sanityn/dom has size 4096 OK PASS 100d (7s) == sanityn test 100e: DoM: read on open and file size ==== 23:04:56 (1788836696) 1+0 records in 1+0 records out 12000 bytes (12 kB, 12 KiB) copied, 0.0107466 s, 1.1 MB/s PASS 100e (8s) == sanityn test 101a: Discard DoM data on unlink ========= 23:05:03 (1788836703) /mnt/lustre/f101a.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0126512 s, 324 kB/s PASS 101a (8s) == sanityn test 101b: Discard DoM data on rename ========= 23:05:12 (1788836712) /mnt/lustre/f101b.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0193968 s, 211 kB/s PASS 101b (7s) == sanityn test 101c: Discard DoM data on close-unlink === 23:05:20 (1788836720) /mnt/lustre/f101c.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0216767 s, 189 kB/s PASS 101c (10s) == sanityn test 102: Test open by handle of unlinked file ========================================================== 23:05:30 (1788836730) file: /mnt/lustre/f102.sanityn fh_bytes: 32 fh_type: 151 fh_data: a1 13 00 00 02 00 00 00 e5 0a 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 file's parent FID is [0:0x0:0x0] file FID is [0x2000013a1:0xae5:0x0] access via mount point '/mnt/lustre2' - OK access after unlink - OK check_fhandle_syscalls test Passed! PASS 102 (13s) == sanityn test 103: Test size correctness with lockahead ========================================================== 23:05:43 (1788836743) fail_loc=0x415 fail_loc=0x415 fail_val=2 Incorrect size expected (no glimpse fix): Starting test test23 at 1788836748 lockahead_test: lockahead_test.c:1171: test23: assertion 'sb.st_size == sb2.st_size' failed: size on /mnt/lustre/d103.sanityn/f103.sanityn and /mnt/lustre2/d103.sanityn/f103.sanityn differs: 0 vs 1048576 fail_loc=0 fail_loc=0x214 fail_val=2 Starting test test23 at 1788836752 Finishing test test23 at 1788836753 PASS 103 (18s) == sanityn test 104: Verify that MDS stores atime/mtime/ctime during close ========================================================== 23:06:02 (1788836762) SKIP: sanityn test_104 ldiskfs only test SKIP 104 (4s) == sanityn test 105: Glimpse and lock cancel race ======== 23:06:05 (1788836765) fail_loc=0x416 PASS 105 (25s) == sanityn test 106a: Verify the btime via statx() ======= 23:06:31 (1788836791) SKIP: sanityn test_106a Test only for ldiskfs and statx() supported SKIP 106a (4s) == sanityn test 106b: Glimpse RPCs test for statx ======== 23:06:34 (1788836794) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.023816 s, 43.0 kB/s File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115272381369068 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-07 23:06:36.000000000 -0400 Modify: 2026-09-07 23:06:36.000000000 -0400 Change: 2026-09-07 23:06:36.000000000 -0400 Birth: 2026-09-07 23:06:36.000000000 -0400 /mnt/lustre/f106b.sanityn 144115272381369068 -rw-r--r-- 1 0 0 1788836796 1788836796 1788836796 File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115272381369068 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-07 23:06:36.000000000 -0400 Modify: 2026-09-07 23:06:36.000000000 -0400 Change: 2026-09-07 23:06:36.000000000 -0400 Birth: 2026-09-07 23:06:36.000000000 -0400 1788836796 1024 1 PASS 106b (8s) == sanityn test 106c: Verify statx attributes mask ======= 23:06:43 (1788836803) PASS 106c (7s) == sanityn test 107a: Basic grouplock conflict =========== 23:06:50 (1788836810) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.386526 s, 27.1 MB/s /mnt/lustre/f107a.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 6369 0x18e1 0x240000400 multiop /mnt/lustre/f107a.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7612 multiop /mnt/lustre2/f107a.sanityn vO_G16022000r10g16022000c TMPPIPE=/tmp/multiop_open_wait_pipe.7612 First grouplock blocks second one PASS 107a (10s) == sanityn test 107b: Grouplock is added to the head of waiting list ========================================================== 23:07:00 (1788836820) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.286366 s, 36.6 MB/s /mnt/lustre/f107b.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 6251 0x186b 0x280000400 multiop /mnt/lustre/f107b.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7612 Grouplock blocks IO First grouplock blocks second one Second grouplock blocks IO PASS 107b (13s) == sanityn test 108a: lseek: parallel updates ============ 23:07:14 (1788836834) 1+0 records in 1+0 records out 10000 bytes (10 kB, 9.8 KiB) copied, 0.00692836 s, 1.4 MB/s llite.lustre-ffff92ca103df800.hybrid_io=0 llite.lustre-ffff92ca385af000.hybrid_io=0 fail_val=4 fail_loc=0x80000414 1+0 records in 1+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 4.69015 s, 1.8 MB/s PASS 108a (12s) == sanityn test 109: Race with several mount instances on 1 node ========================================================== 23:07:26 (1788836846) 192.168.204.107@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg407-client.virtnet /mnt/lustre (opts:) 192.168.204.107@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg407-client.virtnet /mnt/lustre2 (opts:) Starting race between client mounts (duration 30) Iteration 0 Starting client: oleg407-client.virtnet: -o user_xattr,flock 192.168.204.107@tcp:/lustre /mnt/lustre Starting client: oleg407-client.virtnet: -o user_xattr,flock 192.168.204.107@tcp:/lustre /mnt/lustre2 192.168.204.107@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.107@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg407-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg407-client.virtnet /mnt/lustre (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg407-server' oleg407-server: oleg407-server.virtnet: executing load_modules_local oleg407-server: kmemleak disabled oleg407-server: Loading modules from /home/green/git/lustre-release/lustre oleg407-server: detected 4 online CPUs by sysfs oleg407-server: MODOPTS_LIBCFS= oleg407-server: Force libcfs to create 2 CPU partitions oleg407-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Starting client: oleg407-client.virtnet: -o user_xattr,flock 192.168.204.107@tcp:/lustre /mnt/lustre Starting client: oleg407-client.virtnet: -o user_xattr,flock 192.168.204.107@tcp:/lustre /mnt/lustre2 PASS 109 (58s) == sanityn test 110: do not grant another lock on resend ========================================================== 23:08:24 (1788836904) fail_loc=0x525 fail_val=3 fail_loc=0x534 File: /mnt/lustre/f110.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115272414920707 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-07 23:08:27.000000000 -0400 Modify: 2026-09-07 23:08:27.000000000 -0400 Change: 2026-09-07 23:08:27.000000000 -0400 Birth: 2026-09-07 23:08:27.000000000 -0400 fail_loc=0 fail_val=0 Connected clients: oleg407-client.virtnet oleg407-client.virtnet PASS 110 (74s) == sanityn test 111: A racy rename/link an open file should not cause fs corruption ========================================================== 23:09:38 (1788836978) SKIP: sanityn test_111 needs >= 2 MDTs SKIP 111 (4s) == sanityn test 112: update max-inherit in default LMV === 23:09:42 (1788836982) SKIP: sanityn test_112 We need at least 2 MDTs for this test SKIP 112 (4s) == sanityn test 113: check servers of specified fs ======= 23:09:46 (1788836986) lustre-OST0000-osc-ffff92ca056fa000 active. lustre-OST0001-osc-ffff92ca056fa000 active. lustre-OST0000-osc-ffff92ca056fa000 active. lustre-OST0001-osc-ffff92ca056fa000 active. MGC192.168.204.107@tcp active. PASS 113 (7s) == sanityn test 114: implicit default LMV inherit ======== 23:09:53 (1788836993) SKIP: sanityn test_114 We need at least 2 MDTs for this test SKIP 114 (3s) == sanityn test 115: ldiskfs doesn't check direntry for uniqueness ========================================================== 23:09:57 (1788836997) SKIP: sanityn test_115 ldiskfs only test SKIP 115 (3s) == sanityn test 116: DNE: Set default LMV layout from a remote client ========================================================== 23:10:00 (1788837000) SKIP: sanityn test_116 needs >= 2 MDTs SKIP 116 (4s) == sanityn test 121: trunc append race =================== 23:10:04 (1788837004) 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0123618 s, 1.7 MB/s fail_loc=0x80001436 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0844248 s, 243 kB/s 0000000 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0005000 0000 0000 0000 0000 0000 0000 0000 0000 * 000c800 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0011800 PASS 121 (10s) == sanityn test 122: directory size is consistent across mounts ========================================================== 23:10:14 (1788837014) PASS 122 (12s) == sanityn test 200: service remains healthy while able to process request ========================================================== 23:10:26 (1788837026) at_max=10 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=12 fail_loc=0x1e1 fail_val=20 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.207 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.2238 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1607 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.3034 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.2465 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.3413 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.2738 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.2289 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.2821 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.2453 s, 0.1 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.288 s, 0.1 kB/s 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.2948 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.3048 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.2711 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.3493 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.3093 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.3527 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.3693 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2362 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.403 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.4891 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.5179 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.499 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.5353 s, 0.1 kB/s at_max=600 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=116 oleg407-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff92ca056fa000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff92ca056fa000.ost_server_uuid in FULL state after 0 sec PASS 200 (90s) cleanup: ====================================================== == sanityn test complete, duration 14589 sec ============= 23:12:00 (1788837120) === sanityn: start cleanup 23:12:03 (1788837123) === Stopping clients: oleg407-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg407-client.virtnet /mnt/lustre2 opts: === sanityn: finish cleanup 23:16:33 (1788837393) ===