-----============= acceptance-small: sanityn ============----- Fri Sep 18 16:30:25 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg442-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanityn.*ex': No such file or directory excepting tests: 27 40a skipping tests SLOW=no: 33a === sanityn: start setup 16:30:51 (1789763451) === Starting client oleg442-client.virtnet: -o user_xattr,flock 192.168.204.142@tcp:/lustre /mnt/lustre2 Mount client oleg442-client.virtnet: mount -t lustre -o user_xattr,flock 192.168.204.142@tcp:/lustre /mnt/lustre2 Started clients oleg442-client.virtnet: 192.168.204.142@tcp:/lustre on /mnt/lustre2 type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) oleg442-client.virtnet: executing check_config_client /mnt/lustre oleg442-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg442-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9a1004e5e800.idle_timeout=debug osc.lustre-OST0000-osc-ffff9a100aa27800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9a1004e5e800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9a100aa27800.idle_timeout=debug disable quota as required oleg442-server: oleg442-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanityn: finish setup 16:31:31 (1789763491) === 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0725277 s, 14.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7652] == sanityn test 0: do_node_vp() and do_facet_vp() do the right thing ========================================================== 16:31:35 (1789763495) PASS 0 (10s) == sanityn test 1: Check attribute updates on 2 mount points ========================================================== 16:31:46 (1789763506) /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0777 OK /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0666 OK /mnt/lustre/f1.sanityn: absent OK PASS 1 (8s) == sanityn test 2a: check cached attribute updates on 2 mtpt's ================================================================== 16:31:54 (1789763514) -rw-r--r-- 1 root root 0 Sep 18 16:31 /mnt/lustre2/f2a /mnt/lustre/f2a has type file OK /mnt/lustre/f2a has perms 0777 OK PASS 2a (8s) == sanityn test 2b: check cached attribute updates on 2 mtpt's ================================================================== 16:32:03 (1789763523) -rw-r--r-- 1 root root 0 Sep 18 16:32 /mnt/lustre2/f2b /mnt/lustre2/f2b has type file OK /mnt/lustre2/f2b has perms 0777 OK PASS 2b (6s) == sanityn test 2c: check cached attribute updates on 2 mtpt's root ============================================================= 16:32:10 (1789763530) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0777 OK PASS 2c (6s) == sanityn test 2d: check cached attribute updates on 2 mtpt's root ============================================================= 16:32:17 (1789763537) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0755 OK PASS 2d (6s) == sanityn test 2e: check chmod on root is propagated to others ========================================================== 16:32:24 (1789763544) total 1 -rwxrwxrwx 1 root root 0 Sep 18 16:31 f2a -rwxrwxrwx 1 root root 0 Sep 18 16:32 f2b total 1 -rwxrwxrwx 1 root root 0 Sep 18 16:31 f2a -rwxrwxrwx 1 root root 0 Sep 18 16:32 f2b running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre2/f2e.sanityn] [count=1] 1+0 records in 1+0 records out 512 bytes copied, 0.0147199 s, 34.8 kB/s PASS 2e (11s) == sanityn test 2f: check attr/owner updates on DNE with 2 mtpt's ========================================================== 16:32:34 (1789763554) SKIP: sanityn test_2f needs >= 2 MDTs SKIP 2f (4s) == sanityn test 2g: check blocks update on sync write ==== 16:32:38 (1789763558) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.566271 s, 3.7 MB/s /mnt/lustre/f2g.sanityn has 4103 blocks /mnt/lustre2/f2g.sanityn has 4103 blocks PASS 2g (10s) == sanityn test 3: symlink on one mtpt, readlink on another ===================================================================== 16:32:48 (1789763568) PASS 3 (7s) == sanityn test 4: fstat validation on multiple mount points ==================================================================== 16:32:56 (1789763576) PASS 4 (7s) == sanityn test 5: create a file on one mount, truncate it on the other ========================================================== 16:33:03 (1789763583) /mnt/lustre/f5 has type file OK /mnt/lustre/f5 has size 100 OK PASS 5 (8s) == sanityn test 6: remove of open file on other node ============================================================================ 16:33:11 (1789763591) opening writing unlinking /mnt/lustre2/f6.sanityn accessing (1) seeking (1) accessing (2) fstat... reading comparing data truncating seeking (2) writing again seeking (3) reading again comparing data again closing SUCCESS - goto beer PASS 6 (9s) == sanityn test 7: remove of open directory on other node ======================================================================= 16:33:20 (1789763600) creating directory /mnt/lustre/d7 opening directory unlinking /mnt/lustre/d7 Ok, everything goes well. PASS 7 (10s) == sanityn test 8: remove of open special file on other node ==================================================================== 16:33:31 (1789763611) creating special file /mnt/lustre/f8.sanityn opening file unlinking /mnt/lustre/f8.sanityn Ok, everything goes well. PASS 8 (10s) == sanityn test 9a: append of file with sub-page size on multiple mounts ========================================================== 16:33:40 (1789763620) PASS 9a (9s) == sanityn test 9b: append to striped sparse file ======== 16:33:49 (1789763629) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0762196 s, 13.8 MB/s 3+0 records in 3+0 records out 3 bytes copied, 0.0755666 s, 0.0 kB/s Data read (expecting 'foo'): foo PASS 9b (7s) == sanityn test 10a: write of file with sub-page size on multiple mounts ========================================================== 16:33:56 (1789763636) 1+0 records in 1+0 records out 1 byte copied, 0.0135051 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0106091 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0143352 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0105225 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.013112 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0122268 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0101197 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0100364 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0232527 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.014721 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00922584 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00946437 s, 0.1 kB/s PASS 10a (8s) == sanityn test 10b: write of file with sub-page size on multiple mounts ========================================================== 16:34:04 (1789763644) 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.0203975 s, 151 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0410014 s, 99.9 kB/s 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.00289401 s, 1.1 MB/s PASS 10b (9s) == sanityn test 11: execution of file opened for write should return error ============================================================== 16:34:13 (1789763653) multiop /mnt/lustre/d11/f vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7652 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 298: /mnt/lustre2/d11/f: Text file busy PASS 11 (8s) == sanityn test 12: test lock ordering (link, stat, unlink) ========================================================== 16:34:21 (1789763661) warning: '-runtime' deprecated, use '-t runtime' instead - link 241 (time 1789763674.36 total 10.00 last 24.10) - link 491 (time 1789763684.36 total 20.00 last 25.00) - link 780 (time 1789763694.42 total 30.06 last 28.73) - link 1064 (time 1789763704.46 total 40.09 last 28.30) - link 1356 (time 1789763714.47 total 50.11 last 29.15) - link 1643 (time 1789763724.51 total 60.15 last 28.59) - link 1909 (time 1789763734.52 total 70.16 last 26.57) - link 2155 (time 1789763744.53 total 80.17 last 24.59) - link 2425 (time 1789763754.55 total 90.19 last 26.94) total: 2667 link in 100.03 seconds: 26.66 ops/second using seed 1224432958 running for 100 seconds - stat 3634 (time 1789763676 ; total 11 ; last 11) - stat 8987 (time 1789763687 ; total 22 ; last 11) - stat 10000 (time 1789763689 ; total 24 ; last 2) - stat 15048 (time 1789763700 ; total 35 ; last 11) - stat 19833 (time 1789763711 ; total 46 ; last 11) - stat 20000 (time 1789763711 ; total 46 ; last 0) - stat 23597 (time 1789763722 ; total 57 ; last 11) - stat 25953 (time 1789763733 ; total 68 ; last 11) - stat 27936 (time 1789763744 ; total 79 ; last 11) - stat 30000 (time 1789763752 ; total 87 ; last 8) - stat 32043 (time 1789763763 ; total 98 ; last 11) total: 32589 stats in 100 seconds: 325.890015 stats/second - unlinked 0 (time 1789763715 ; total 0 ; last 0) unlink(/mnt/lustre2/lockdir/lockfile2667) error: No such file or directory total: 2666 unlinks in 86 seconds: 31.000000 unlinks/second /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 77: kill: (21458) - No such process /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 78: kill: (21460) - No such process PASS 12 (146s) == sanityn test 13: test directory page revocation ======= 16:36:48 (1789763808) f13 f13 PASS 13 (11s) == sanityn test 14aa: execution of file open for write returns -ETXTBSY ========================================================== 16:36:59 (1789763819) multiop /mnt/lustre/d14aa.sanityn/f14aa.sanityn vOw_c TMPPIPE=/tmp/multiop_open_wait_pipe.7652 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 333: /mnt/lustre2/d14aa.sanityn/f14aa.sanityn: Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 22644 Terminated $MULTIOP_PROG $FILE v$ARGS > $TMPPIPE (wd: ~) PASS 14aa (7s) == sanityn test 14ab: open(RDWR) of executing file returns -ETXTBSY ========================================================== 16:37:06 (1789763826) open(O_RDWR|O_CREAT): Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 23223 Terminated $DIR1/$tdir/sleep 60 (wd: ~) PASS 14ab (7s) == sanityn test 14b: truncate of executing file returns -ETXTBSY ================================================================ 16:37:13 (1789763833) truncate: cannot truncate '/mnt/lustre2/d14b.sanityn/sleep' to length 60: Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 348: 23803 Terminated $DIR1/$tdir/sleep 60 PASS 14b (7s) == sanityn test 14c: open(O_TRUNC) of executing file return -ETXTBSY ============================================================ 16:37:21 (1789763841) cp: cannot create regular file '/mnt/lustre2/d14c.sanityn/sleep': Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 360: 24386 Terminated $DIR1/$tdir/sleep 60 PASS 14c (8s) == sanityn test 14d: chmod of executing file is still possible ================================================================== 16:37:29 (1789763849) chmod /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 371: 24969 Terminated $DIR1/$tdir/sleep 60 PASS 14d (10s) == sanityn test 15: test out-of-space with multiple writers ===================================================================== 16:37:39 (1789763859) PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin: MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky sleep 5 for ZFS MDS Waiting for MDT destroys to complete fallocate on zfs doesn't consume space STRIPECOUNT=2 ORIGFREE=7531520 MAXFREE=800000 SKIP: oos2 test_15 oos2.sh: 7531520KB free > 800000KB, increase MAXFREE (or reduce fs size) sleep 5 for ZFS MDS Waiting for MDT destroys to complete SKIP 15 (56s) == sanityn test 16a: 500 iterations of dual-mount fsx ==== 16:38:36 (1789763916) keep default fallocate mode: 2 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.620257 s, 13.5 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.420374 s, 20.0 MB/s Chance of close/open is 1 in 50 Seed set to 3921 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000002[1] 1789763923.118145 mapwrite 0x9b3b26 thru 0x9bb7e3 (0x7cbe bytes) truncating to largest ever: 0x8f9050 000003[0] 1789763923.349237 trunc from 0x9bb7e4 to 0x8f904f (0xc2794 bytes) 000004[0] 1789763923.430915 mapwrite 0x90d00d thru 0x9168f7 (0x98eb bytes) 000005[1] 1789763923.618939 write 0x468624 thru 0x469a39 (0x1416 bytes) 000006[0] 1789763923.676138 write 0x032fc8 thru 0x036bb0 (0x3be9 bytes) 000008[1] 1789763923.760768 read 0x10e97b thru 0x11c4ba (0xdb40 bytes) 000009[1] 1789763923.909313 mapwrite 0x2744e5 thru 0x2839d6 (0xf4f2 bytes) 000016[1] 1789763924.288006 punch from 0x5de061 to 0x5ea26a (0xc20a bytes) 000018[0] 1789763924.325575 mapread 0x4bc501 thru 0x4be453 (0x1f53 bytes) 000019[1] 1789763924.492538 mapwrite 0x5c2974 thru 0x5c94c9 (0x6b56 bytes) 000020[1] 1789763924.727868 mapread 0x845d53 thru 0x854128 (0xe3d6 bytes) truncating to largest ever: 0x9b4a2a 000028[1] 1789763925.169114 trunc from 0x9168f8 to 0x9b4a29 (0x9e132 bytes) 000029[1] 1789763925.264941 punch from 0x25d5fa to 0x2654b3 (0x7eba bytes) 000030[1] 1789763925.309392 punch from 0x6085e8 to 0x60983a (0x1253 bytes) 000031[0] 1789763925.315586 mapwrite 0x8770a1 thru 0x885fc0 (0xef20 bytes) 000033[0] 1789763925.522734 punch from 0x36ac74 to 0x374254 (0x95e1 bytes) 000034[0] 1789763925.549405 trunc from 0x9b4a2a to 0x43796d (0x57d0bc bytes) 000035[0] 1789763925.602006 read 0x41e082 thru 0x425a0f (0x798e bytes) 000036[1] 1789763925.671908 read 0x14e7fe thru 0x15e354 (0xfb57 bytes) 000037[1] 1789763925.719536 read 0x39577a thru 0x398e40 (0x36c7 bytes) 000039[1] 1789763925.785451 write 0x9ba8b6 thru 0x9c459e (0x9ce9 bytes) 000040[1] 1789763925.835560 read 0x225de9 thru 0x22a532 (0x474a bytes) 000042[0] 1789763925.871345 mapwrite 0x3a2690 thru 0x3acdff (0xa770 bytes) 000043[0] 1789763926.018647 read 0x117493 thru 0x123004 (0xbb72 bytes) 000047[0] 1789763926.106103 write 0x7dacdc thru 0x7dff22 (0x5247 bytes) 000048[0] 1789763926.147710 write 0x8ee014 thru 0x8fbe9a (0xde87 bytes) 000049[0] 1789763926.190166 write 0x0ed5df thru 0x0f8c11 (0xb633 bytes) 000050[0] 1789763926.219709 read 0x4879d7 thru 0x495895 (0xdebf bytes) 000051[0] 1789763926.308116 mapwrite 0x54024c thru 0x544da0 (0x4b55 bytes) 000052[0] 1789763926.598892 punch from 0x1dac92 to 0x1e2014 (0x7383 bytes) 000053[1] 1789763926.625137 trunc from 0x9c459f to 0x867895 (0x15cd09 bytes) 000054[1] 1789763926.699011 read 0x3ad605 thru 0x3b4fc2 (0x79be bytes) 000056[0] 1789763926.761258 punch from 0x860bd8 to 0x867895 (0x6cbe bytes) 000057[1] 1789763926.799751 read 0x03c440 thru 0x0406e2 (0x42a3 bytes) 000059[1] 1789763926.854797 read 0x4530d1 thru 0x45e0a7 (0xafd7 bytes) 000061[0] 1789763926.904919 punch from 0x29d734 to 0x2aba49 (0xe316 bytes) 000063[0] 1789763926.973154 read 0x738e2e thru 0x745a87 (0xcc5a bytes) 000064[1] 1789763927.051964 trunc from 0x867896 to 0x2a0e31 (0x5c6a64 bytes) 000065[1] 1789763927.132896 punch from 0x03b504 to 0x044cb9 (0x97b6 bytes) 000067[0] 1789763927.210187 trunc from 0x2a0e32 to 0x78e0c1 (0x4ed290 bytes) 000069[1] 1789763927.396013 mapread 0x3bc2c0 thru 0x3ca511 (0xe252 bytes) 000070[0] 1789763927.584171 punch from 0x401f9c to 0x407928 (0x598d bytes) 000071[1] 1789763927.614477 mapwrite 0x48f847 thru 0x49510a (0x58c4 bytes) 000074[0] 1789763927.833800 read 0x2d2654 thru 0x2e1be5 (0xf592 bytes) 000078[0] 1789763927.949470 punch from 0x6ec799 to 0x6f013d (0x39a5 bytes) 000079[0] 1789763928.030167 write 0x1a4849 thru 0x1a668a (0x1e42 bytes) 000080[1] 1789763928.072147 mapread 0x4090ed thru 0x409baa (0xabe bytes) 000081[0] 1789763928.136500 trunc from 0x78e0c2 to 0x6791f4 (0x114ecd bytes) 000085[0] 1789763928.375595 read 0x3fe434 thru 0x3ff8ae (0x147b bytes) 000086[1] 1789763928.427559 read 0x01ec1f thru 0x02050e (0x18f0 bytes) 000087[0] 1789763928.463949 write 0x0cb205 thru 0x0ce721 (0x351d bytes) 000088[0] 1789763928.533099 punch from 0x20b850 to 0x20d50a (0x1cbb bytes) 000089[1] 1789763928.542151 mapread 0x18e5f9 thru 0x199fcc (0xb9d4 bytes) 000092[0] 1789763928.732609 mapwrite 0x5bf0b6 thru 0x5c0949 (0x1894 bytes) 000093[1] 1789763928.931763 mapwrite 0x91e618 thru 0x926017 (0x7a00 bytes) 000094[1] 1789763929.302069 mapread 0x194cfd thru 0x19cc66 (0x7f6a bytes) 000095[0] 1789763929.381254 mapwrite 0x211ed8 thru 0x21b643 (0x976c bytes) 000096[1] 1789763929.556554 read 0x2a905f thru 0x2a97aa (0x74c bytes) 000097[1] 1789763929.642014 read 0x366b73 thru 0x375abd (0xef4b bytes) 000101[1] 1789763929.813741 punch from 0x0aadd6 to 0x0b908e (0xe2b9 bytes) 000102[0] 1789763929.876324 write 0x5feec9 thru 0x60247a (0x35b2 bytes) 000105[0] 1789763930.001301 mapread 0x136dcf thru 0x13c3b7 (0x55e9 bytes) 000106[0] 1789763930.044734 write 0x415fce thru 0x4257e0 (0xf813 bytes) 000107[0] 1789763930.057135 mapwrite 0x83baae thru 0x843e8a (0x83dd bytes) 000111[1] 1789763930.489162 trunc from 0x926018 to 0x7b6f0a (0x16f10d bytes) 000112[1] 1789763930.615293 trunc from 0x7b6f0b to 0x819fbe (0x630b4 bytes) 000114[1] 1789763930.685090 mapwrite 0x1c5d33 thru 0x1d31f7 (0xd4c5 bytes) 000115[0] 1789763931.172836 mapread 0x398a95 thru 0x3a1e23 (0x938f bytes) 000116[1] 1789763931.353659 trunc from 0x819fbf to 0x842ce2 (0x28d24 bytes) 000117[0] 1789763931.521084 trunc from 0x842ce3 to 0x8a16e4 (0x5ea02 bytes) 000118[1] 1789763931.672786 mapread 0x897478 thru 0x8976ec (0x275 bytes) 000119[0] 1789763931.763407 read 0x0e3230 thru 0x0f1d76 (0xeb47 bytes) 000121[0] 1789763931.852892 punch from 0x7fb0bc to 0x801638 (0x657d bytes) 000122[0] 1789763931.893923 punch from 0x1b48c8 to 0x1bdd23 (0x945c bytes) 000123[0] 1789763931.901207 trunc from 0x8a16e5 to 0x06f8bd (0x831e27 bytes) 000124[0] 1789763931.954265 trunc from 0x06f8be to 0x675bf6 (0x606339 bytes) 000125[0] 1789763932.021567 punch from 0x47bda1 to 0x482be4 (0x6e44 bytes) 000127[0] 1789763932.120787 write 0x455aa9 thru 0x462815 (0xcd6d bytes) 000128[0] 1789763932.147104 read 0x599914 thru 0x5a40d0 (0xa7bd bytes) 000131[1] 1789763932.231953 read 0x040ed6 thru 0x043b03 (0x2c2e bytes) 000132[0] 1789763932.276663 mapread 0x32f2d9 thru 0x335dc1 (0x6ae9 bytes) 000133[1] 1789763932.303262 trunc from 0x675bf7 to 0x560393 (0x115863 bytes) 000136[0] 1789763932.435863 punch from 0x359852 to 0x35f8d1 (0x6080 bytes) 000139[1] 1789763932.542245 read 0x399cab thru 0x3a9a01 (0xfd57 bytes) 000140[0] 1789763932.638588 punch from 0x341b83 to 0x34e24b (0xc6c9 bytes) 000141[1] 1789763932.707516 punch from 0x06d63a to 0x0723a5 (0x4d6c bytes) 000145[0] 1789763932.835345 punch from 0x3216c0 to 0x32dbbb (0xc4fc bytes) 000146[0] 1789763932.921785 read 0x164bb8 thru 0x16ba50 (0x6e99 bytes) 000149[0] 1789763933.083013 mapwrite 0x15444e thru 0x15fbfe (0xb7b1 bytes) 000150[0] 1789763933.271375 punch from 0x53d7be to 0x53f917 (0x215a bytes) 000151[1] 1789763933.309818 write 0x972b82 thru 0x977cbd (0x513c bytes) 000155[0] 1789763933.419863 mapread 0x8e7d1a thru 0x8eccad (0x4f94 bytes) 000157[0] 1789763933.606808 mapread 0x0da3df thru 0x0e0aa3 (0x66c5 bytes) 000158[1] 1789763933.663544 write 0x7c36c3 thru 0x7c5e17 (0x2755 bytes) 000159[1] 1789763933.696454 write 0x26c60d thru 0x26e446 (0x1e3a bytes) 000161[1] 1789763933.824613 mapread 0x212737 thru 0x219564 (0x6e2e bytes) 000163[1] 1789763933.945354 mapwrite 0x312bda thru 0x31e3e0 (0xb807 bytes) 000166[0] 1789763934.117852 mapread 0x59632d thru 0x59b11c (0x4df0 bytes) 000167[0] 1789763934.224916 write 0x0ac77d thru 0x0b4d22 (0x85a6 bytes) 000168[1] 1789763934.242609 mapread 0x111300 thru 0x120518 (0xf219 bytes) 000169[1] 1789763934.326619 punch from 0x41c91a to 0x427cbf (0xb3a6 bytes) 000170[1] 1789763934.412199 punch from 0x4ed75b to 0x4edfe6 (0x88c bytes) 000171[1] 1789763934.494251 read 0x6724a0 thru 0x680a04 (0xe565 bytes) 000172[0] 1789763934.527701 trunc from 0x977cbe to 0x756a1f (0x22129e bytes) 000173[1] 1789763934.634640 mapread 0x26da79 thru 0x279160 (0xb6e8 bytes) 000174[1] 1789763934.769469 trunc from 0x756a20 to 0x305823 (0x4511fc bytes) 000175[1] 1789763934.855681 punch from 0x2b8483 to 0x2b9d9b (0x1919 bytes) 000177[1] 1789763934.962837 write 0x9b756f thru 0x9c270d (0xb19f bytes) 000178[1] 1789763935.040797 mapread 0x209cf9 thru 0x214cd9 (0xafe1 bytes) 000179[0] 1789763935.128564 write 0x6f0edd thru 0x6f1cc2 (0xde6 bytes) 000182[0] 1789763935.275881 mapwrite 0x2e1e8d thru 0x2e9aaf (0x7c23 bytes) 000183[1] 1789763935.431820 punch from 0x2cc0c8 to 0x2cf8cf (0x3808 bytes) 000184[0] 1789763935.484730 read 0x7fc384 thru 0x8085d0 (0xc24d bytes) 000186[1] 1789763935.611856 punch from 0x2ba0e4 to 0x2c647c (0xc399 bytes) 000189[0] 1789763935.734909 mapread 0x98952e thru 0x98cdee (0x38c1 bytes) 000190[1] 1789763935.839130 trunc from 0x9c270e to 0x11c003 (0x8a670a bytes) 000191[0] 1789763935.994743 write 0x86b57b thru 0x876587 (0xb00d bytes) 000192[1] 1789763936.058416 trunc from 0x876588 to 0x8af529 (0x38fa2 bytes) 000193[1] 1789763936.122911 trunc from 0x8af52a to 0x558d98 (0x356791 bytes) 000194[1] 1789763936.176643 mapwrite 0x0f969b thru 0x1055a2 (0xbf08 bytes) 000196[0] 1789763936.467700 write 0x9fb15f thru 0x9fec84 (0x3b26 bytes) 000197[1] 1789763936.572607 read 0x078365 thru 0x07c849 (0x44e5 bytes) 000198[1] 1789763936.739596 mapwrite 0x75808f thru 0x7585b9 (0x52b bytes) 000202[1] 1789763936.966381 punch from 0x8df3cd to 0x8edc63 (0xe897 bytes) 000206[0] 1789763937.095908 punch from 0x0964fb to 0x09c9a7 (0x64ad bytes) 000207[1] 1789763937.188839 read 0x3eae42 thru 0x3fa2cf (0xf48e bytes) 000208[0] 1789763937.292383 trunc from 0x9fec85 to 0x1875d3 (0x8776b1 bytes) 000209[1] 1789763937.470113 trunc from 0x1875d4 to 0x7cb55d (0x643f8a bytes) 000210[1] 1789763937.610746 read 0x665f68 thru 0x672578 (0xc611 bytes) 000211[1] 1789763937.676390 write 0x350cf7 thru 0x35db85 (0xce8f bytes) 000212[0] 1789763937.741455 write 0x96ffb6 thru 0x978f98 (0x8fe3 bytes) 000213[1] 1789763937.801248 trunc from 0x978f99 to 0x3f10cb (0x587ecd bytes) 000214[0] 1789763937.895509 write 0x97947d thru 0x984e6e (0xb9f2 bytes) 000215[1] 1789763937.995689 punch from 0x4677b4 to 0x4753fc (0xdc49 bytes) 000216[0] 1789763938.067701 punch from 0x96375b to 0x96ef45 (0xb7eb bytes) 000217[0] 1789763938.144943 punch from 0x70c8fb to 0x710478 (0x3b7e bytes) 000219[1] 1789763938.268104 mapread 0x6867a3 thru 0x69240c (0xbc6a bytes) 000223[0] 1789763938.641678 mapread 0x74771c thru 0x7517f4 (0xa0d9 bytes) 000226[0] 1789763938.781282 punch from 0x6da40b to 0x6de3d3 (0x3fc9 bytes) 000230[1] 1789763938.982788 punch from 0x8f57c6 to 0x8fb90d (0x6148 bytes) 000231[1] 1789763939.046079 read 0x3ad264 thru 0x3b75f3 (0xa390 bytes) 000233[1] 1789763939.132698 mapread 0x823450 thru 0x826fd3 (0x3b84 bytes) 000236[0] 1789763939.248756 read 0x4f1582 thru 0x4f4f94 (0x3a13 bytes) 000239[1] 1789763939.374590 punch from 0x72999d to 0x73496a (0xafce bytes) 000241[0] 1789763939.459806 read 0x39fe97 thru 0x3a13bb (0x1525 bytes) 000242[0] 1789763939.559104 mapread 0x761055 thru 0x76df6a (0xcf16 bytes) 000244[0] 1789763939.641215 read 0x8ce714 thru 0x8ddbea (0xf4d7 bytes) 000246[0] 1789763939.665572 trunc from 0x984e6f to 0x910d94 (0x740da bytes) 000247[1] 1789763939.764529 read 0x5dcf22 thru 0x5e069b (0x377a bytes) 000248[0] 1789763939.851304 mapwrite 0x61ab66 thru 0x62a9c2 (0xfe5d bytes) 000251[1] 1789763940.165886 punch from 0x0f42d2 to 0x0fc72e (0x845d bytes) 000252[0] 1789763940.215993 punch from 0x0c52ab to 0x0d0214 (0xaf6a bytes) 000253[0] 1789763940.230207 punch from 0x5ecaa1 to 0x5f7e47 (0xb3a7 bytes) 000254[0] 1789763940.235301 punch from 0x64f44d to 0x65740c (0x7fc0 bytes) 000255[0] 1789763940.242616 punch from 0x5c10f2 to 0x5cf82c (0xe73b bytes) 000257[0] 1789763940.252862 mapread 0x75abe0 thru 0x7686da (0xdafb bytes) 000259[0] 1789763940.295568 punch from 0x0b87b2 to 0x0c08af (0x80fe bytes) 000260[1] 1789763940.321581 trunc from 0x910d95 to 0x0dc799 (0x8345fb bytes) 000261[0] 1789763940.417844 write 0x677b32 thru 0x679d40 (0x220f bytes) 000263[1] 1789763940.475396 trunc from 0x679d41 to 0x3cc5c4 (0x2ad77c bytes) 000264[1] 1789763940.505718 trunc from 0x3cc5c5 to 0x9384d7 (0x56bf13 bytes) 000267[1] 1789763940.541775 trunc from 0x9384d8 to 0x4ed52d (0x44afaa bytes) 000268[1] 1789763940.563617 read 0x1f8f5d thru 0x202fb9 (0xa05d bytes) 000270[1] 1789763940.602995 read 0x206813 thru 0x20f67e (0x8e6c bytes) 000271[1] 1789763940.622189 mapread 0x4bb162 thru 0x4c91c0 (0xe05f bytes) 000274[0] 1789763940.686713 write 0x4e213b thru 0x4e6b12 (0x49d8 bytes) 000275[1] 1789763940.732428 mapwrite 0x666b09 thru 0x66c4dc (0x59d4 bytes) 000278[0] 1789763940.873419 read 0x4900aa thru 0x4989d9 (0x8930 bytes) 000279[0] 1789763940.907628 punch from 0x0ec02b to 0x0f9a6a (0xda40 bytes) 000280[0] 1789763940.939481 trunc from 0x66c4dd to 0x0a0e7d (0x5cb65f bytes) 000281[1] 1789763941.040597 punch from 0x010937 to 0x01e959 (0xe023 bytes) 000282[0] 1789763941.096766 trunc from 0x0a0e7e to 0x02be0c (0x75071 bytes) 000283[0] 1789763941.130844 write 0x9736be thru 0x979c50 (0x6593 bytes) 000286[1] 1789763941.193458 punch from 0x495973 to 0x4a35b0 (0xdc3e bytes) 000287[0] 1789763941.225143 punch from 0x8cea9b to 0x8d4c06 (0x616c bytes) 000288[1] 1789763941.241552 read 0x365bfd thru 0x369785 (0x3b89 bytes) 000289[0] 1789763941.296127 read 0x0089ac thru 0x01266e (0x9cc3 bytes) 000290[1] 1789763941.323285 mapread 0x7b0cf2 thru 0x7b65db (0x58ea bytes) 000291[1] 1789763941.382364 trunc from 0x979c51 to 0x10eed8 (0x86ad78 bytes) 000292[1] 1789763941.460624 mapread 0x055d59 thru 0x0606c9 (0xa971 bytes) 000293[0] 1789763941.622755 read 0x0baf26 thru 0x0c9396 (0xe471 bytes) 000297[0] 1789763941.790179 write 0x691711 thru 0x699b7d (0x846d bytes) 000300[1] 1789763941.841859 mapwrite 0x24d606 thru 0x252f0e (0x5909 bytes) 000301[1] 1789763942.304598 trunc from 0x699b7e to 0x34a5bc (0x34f5c1 bytes) 000302[0] 1789763942.446661 mapread 0x2f79c2 thru 0x30370c (0xbd4b bytes) 000303[1] 1789763942.604695 read 0x1329e4 thru 0x13a8c4 (0x7ee1 bytes) 000304[0] 1789763942.774466 punch from 0x0013e6 to 0x00d1a2 (0xbdbd bytes) 000305[1] 1789763942.852980 mapwrite 0x1b3c55 thru 0x1b891c (0x4cc8 bytes) 000306[1] 1789763943.076873 mapwrite 0x3491de thru 0x358b38 (0xf95b bytes) 000307[1] 1789763943.280564 mapwrite 0x824ddf thru 0x82ac7d (0x5e9f bytes) 000308[1] 1789763943.606662 read 0x6d93f6 thru 0x6ddf6e (0x4b79 bytes) 000309[0] 1789763943.636509 mapwrite 0x279f5d thru 0x28817c (0xe220 bytes) 000310[0] 1789763943.964185 read 0x3e9711 thru 0x3f59db (0xc2cb bytes) 000311[0] 1789763943.977376 trunc from 0x82ac7e to 0x00c618 (0x81e665 bytes) 000312[0] 1789763944.065080 write 0x4149d8 thru 0x423d70 (0xf399 bytes) 000313[1] 1789763944.095348 write 0x4a0cab thru 0x4ab44d (0xa7a3 bytes) 000314[1] 1789763944.154083 read 0x339b6e thru 0x3471b7 (0xd64a bytes) 000316[0] 1789763944.312772 mapread 0x2fc511 thru 0x3050a9 (0x8b99 bytes) 000317[1] 1789763944.594984 read 0x1841e0 thru 0x190765 (0xc586 bytes) truncating to largest ever: 0x9c81d7 000318[1] 1789763944.715291 trunc from 0x4ab44e to 0x9c81d6 (0x51cd89 bytes) 000320[0] 1789763944.767343 write 0x71325d thru 0x71b617 (0x83bb bytes) 000323[1] 1789763944.898421 punch from 0x2ec16b to 0x2ede02 (0x1c98 bytes) 000324[1] 1789763944.935933 mapread 0x5443e3 thru 0x553b29 (0xf747 bytes) 000325[0] 1789763945.034510 mapread 0x8a4d55 thru 0x8a8c4d (0x3ef9 bytes) 000326[0] 1789763945.144484 mapread 0x024d6a thru 0x0347bf (0xfa56 bytes) 000328[0] 1789763945.207545 trunc from 0x9c81d7 to 0x2fd323 (0x6caeb3 bytes) 000330[1] 1789763945.396467 trunc from 0x2fd324 to 0x7f2358 (0x4f5035 bytes) truncating to largest ever: 0x9f734b 000332[1] 1789763945.461807 trunc from 0x7f2359 to 0x9f734a (0x204ff2 bytes) 000333[1] 1789763945.513840 write 0x01e090 thru 0x021419 (0x338a bytes) 000336[0] 1789763945.623799 punch from 0x1ca6d0 to 0x1ced2b (0x465c bytes) 000339[1] 1789763945.721927 punch from 0x21b3c6 to 0x21d1a2 (0x1ddd bytes) 000340[1] 1789763945.767569 trunc from 0x9f734b to 0x62468f (0x3d2cbb bytes) 000341[0] 1789763945.792827 mapwrite 0x3a055a thru 0x3a55d4 (0x507b bytes) 000342[0] 1789763945.969698 read 0x232800 thru 0x23547d (0x2c7e bytes) 000343[0] 1789763946.013418 trunc from 0x624690 to 0x051068 (0x5d3627 bytes) 000345[1] 1789763946.136793 read 0x00f50a thru 0x0144f3 (0x4fea bytes) 000346[1] 1789763946.200535 mapread 0x04fc51 thru 0x051068 (0x1418 bytes) 000348[1] 1789763946.239959 trunc from 0x051069 to 0x0fbbfc (0xaab94 bytes) 000351[0] 1789763946.417795 mapread 0x0386e3 thru 0x041933 (0x9251 bytes) 000352[0] 1789763946.518406 punch from 0x056f6d to 0x05fc80 (0x8d14 bytes) 000353[0] 1789763946.574871 write 0x017834 thru 0x022db5 (0xb582 bytes) 000357[1] 1789763946.674010 mapread 0x04ace3 thru 0x05055e (0x587c bytes) 000358[0] 1789763946.844766 mapwrite 0x3e4941 thru 0x3f0d43 (0xc403 bytes) 000359[1] 1789763947.016707 read 0x01bcc2 thru 0x020ff6 (0x5335 bytes) 000360[1] 1789763947.265456 mapwrite 0x805225 thru 0x80ef60 (0x9d3c bytes) 000362[1] 1789763947.555476 trunc from 0x80ef61 to 0x1ec188 (0x622dd8 bytes) 000363[1] 1789763947.615823 mapread 0x01ce18 thru 0x01cef5 (0xde bytes) 000365[0] 1789763947.672933 read 0x096ae3 thru 0x0a1f8a (0xb4a8 bytes) 000366[1] 1789763947.782774 trunc from 0x1ec189 to 0x8d42cb (0x6e8143 bytes) 000367[1] 1789763947.843467 write 0x0e07d4 thru 0x0ea9d9 (0xa206 bytes) 000368[0] 1789763947.886764 write 0x446d13 thru 0x44fb7f (0x8e6d bytes) 000369[1] 1789763947.945376 trunc from 0x8d42cc to 0x483699 (0x450c32 bytes) 000370[0] 1789763947.986404 mapread 0x02316a thru 0x02dac0 (0xa957 bytes) 000371[0] 1789763948.202895 read 0x369b04 thru 0x372ee8 (0x93e5 bytes) 000372[1] 1789763948.209641 trunc from 0x48369a to 0x07fb00 (0x403b99 bytes) 000374[0] 1789763948.300474 mapread 0x00db44 thru 0x019d0d (0xc1ca bytes) 000375[0] 1789763948.437616 trunc from 0x07fb01 to 0x17953c (0xf9a3c bytes) 000377[1] 1789763948.650908 mapwrite 0x7a7a51 thru 0x7b0f29 (0x94d9 bytes) 000379[1] 1789763948.983989 mapwrite 0x9af04e thru 0x9b8345 (0x92f8 bytes) 000381[0] 1789763949.104536 mapwrite 0x830e42 thru 0x8353c9 (0x4588 bytes) 000383[0] 1789763949.320768 write 0x8dd871 thru 0x8e2b7a (0x530a bytes) 000385[0] 1789763949.422639 punch from 0x551cb1 to 0x553172 (0x14c2 bytes) 000386[0] 1789763949.456944 mapread 0x134eb5 thru 0x13a13d (0x5289 bytes) 000387[1] 1789763949.483911 read 0x019068 thru 0x028902 (0xf89b bytes) 000389[1] 1789763949.667904 write 0x827782 thru 0x82ac64 (0x34e3 bytes) 000390[1] 1789763949.779434 mapread 0x5b1901 thru 0x5c0bd0 (0xf2d0 bytes) 000391[1] 1789763949.987054 punch from 0x70aa82 to 0x70de35 (0x33b4 bytes) 000393[0] 1789763950.111935 punch from 0x746d89 to 0x7491de (0x2456 bytes) 000394[1] 1789763950.164635 read 0x486fa2 thru 0x48f314 (0x8373 bytes) 000395[1] 1789763950.212848 write 0x3feb6b thru 0x40a12d (0xb5c3 bytes) 000400[1] 1789763950.332600 trunc from 0x9b8346 to 0x02f85a (0x988aeb bytes) 000401[0] 1789763950.362896 trunc from 0x02f85b to 0x3f1d3b (0x3c24e1 bytes) 000402[0] 1789763950.486464 trunc from 0x3f1d3c to 0x51c37c (0x12a641 bytes) 000404[1] 1789763950.537491 mapwrite 0x7ed3b3 thru 0x7edef1 (0xb3f bytes) 000406[1] 1789763951.009008 mapread 0x0d4648 thru 0x0dae4b (0x6804 bytes) 000407[0] 1789763951.065392 mapread 0x1585d1 thru 0x167d4e (0xf77e bytes) 000409[1] 1789763951.210790 trunc from 0x7edef2 to 0x026b98 (0x7c7359 bytes) 000410[0] 1789763951.328601 trunc from 0x026b99 to 0x5af376 (0x5887de bytes) 000412[1] 1789763951.426446 mapwrite 0x976ff4 thru 0x985950 (0xe95d bytes) 000413[1] 1789763951.637423 read 0x91df3f thru 0x92b367 (0xd429 bytes) 000416[1] 1789763951.735502 punch from 0x7b7440 to 0x7bacfe (0x38bf bytes) 000417[1] 1789763951.805242 trunc from 0x985951 to 0x98bd53 (0x6403 bytes) 000418[0] 1789763951.853481 read 0x7a8d8f thru 0x7ae78e (0x5a00 bytes) 000419[0] 1789763951.895782 write 0x9d76b5 thru 0x9d8bc5 (0x1511 bytes) 000423[0] 1789763952.111592 read 0x0527b1 thru 0x05e159 (0xb9a9 bytes) 000424[0] 1789763952.145574 write 0x23b536 thru 0x23ec81 (0x374c bytes) 000426[1] 1789763952.212471 punch from 0x2a4365 to 0x2b3ab7 (0xf753 bytes) 000427[1] 1789763952.281448 read 0x52f86b thru 0x53c437 (0xcbcd bytes) 000430[1] 1789763952.366796 punch from 0x5a0947 to 0x5a3f6c (0x3626 bytes) 000433[1] 1789763952.430986 read 0x8ea00b thru 0x8f6ecc (0xcec2 bytes) 000434[0] 1789763952.454412 punch from 0x166cc0 to 0x173478 (0xc7b9 bytes) 000435[0] 1789763952.536237 write 0x4cdd01 thru 0x4d0152 (0x2452 bytes) 000436[0] 1789763952.683399 mapwrite 0x5f1005 thru 0x5f9975 (0x8971 bytes) 000437[1] 1789763952.926114 mapwrite 0x5506e0 thru 0x55634d (0x5c6e bytes) 000438[0] 1789763953.354440 trunc from 0x9d8bc6 to 0x72aba6 (0x2ae01f bytes) 000439[1] 1789763953.455847 punch from 0x1c621c to 0x1cf4cd (0x92b2 bytes) 000440[0] 1789763953.548839 mapread 0x12a3ce thru 0x13293e (0x8571 bytes) 000442[1] 1789763953.743591 write 0x82aeb8 thru 0x8376eb (0xc834 bytes) 000445[0] 1789763953.894156 mapwrite 0x3da1f9 thru 0x3e5668 (0xb470 bytes) 000450[0] 1789763954.229697 punch from 0x06f95e to 0x0780c7 (0x876a bytes) 000451[0] 1789763954.327821 write 0x8f283b thru 0x8ff9b4 (0xd17a bytes) 000452[1] 1789763954.365807 trunc from 0x8ff9b5 to 0x740e41 (0x1beb73 bytes) 000453[1] 1789763954.494664 read 0x358891 thru 0x366b4b (0xe2bb bytes) 000454[0] 1789763954.506788 mapwrite 0x28ba72 thru 0x293157 (0x76e6 bytes) 000455[0] 1789763954.834784 read 0x32bd90 thru 0x339e9f (0xe110 bytes) 000460[1] 1789763955.100475 mapwrite 0x0fcbee thru 0x0fe705 (0x1b18 bytes) 000461[1] 1789763955.333599 trunc from 0x740e42 to 0x22cc2a (0x514217 bytes) 000462[0] 1789763955.428813 mapread 0x18597d thru 0x18ede6 (0x946a bytes) 000463[1] 1789763955.557292 read 0x22b797 thru 0x22cc2a (0x1494 bytes) 000464[1] 1789763955.725314 read 0x1eaaeb thru 0x1f0fa5 (0x64bb bytes) 000465[1] 1789763955.762291 punch from 0x078637 to 0x07af80 (0x294a bytes) 000469[1] 1789763955.916994 mapread 0x10b338 thru 0x11a226 (0xeeef bytes) 000470[0] 1789763956.051777 mapread 0x1b9965 thru 0x1c16ea (0x7d86 bytes) 000471[1] 1789763956.238683 read 0x0cfd5a thru 0x0dabbc (0xae63 bytes) 000472[1] 1789763956.291554 trunc from 0x22cc2b to 0x9f0bc6 (0x7c3f9c bytes) 000474[1] 1789763956.366780 mapread 0x0d1a53 thru 0x0dcbbf (0xb16d bytes) 000475[1] 1789763956.570006 mapread 0x280b4e thru 0x283ad0 (0x2f83 bytes) 000477[1] 1789763956.619646 mapwrite 0x619e2a thru 0x61fbc8 (0x5d9f bytes) 000478[1] 1789763957.025521 write 0x9be59f thru 0x9c5bf7 (0x7659 bytes) 000479[0] 1789763957.163590 write 0x1f9e29 thru 0x208416 (0xe5ee bytes) 000480[0] 1789763957.255363 mapwrite 0x8c805e thru 0x8d3a6e (0xba11 bytes) 000481[0] 1789763957.389730 read 0x1042c8 thru 0x10f0bf (0xadf8 bytes) 000482[1] 1789763957.446897 write 0x7ce214 thru 0x7da6a6 (0xc493 bytes) 000483[0] 1789763957.472914 mapwrite 0x247312 thru 0x256078 (0xed67 bytes) 000484[1] 1789763957.592662 mapwrite 0x7638b1 thru 0x767cd4 (0x4424 bytes) 000487[1] 1789763957.833362 punch from 0x759ef4 to 0x75bdea (0x1ef7 bytes) 000489[0] 1789763957.878496 write 0x076262 thru 0x07800e (0x1dad bytes) 000490[0] 1789763957.926380 mapread 0x4f29ab thru 0x4ff94e (0xcfa4 bytes) 000492[0] 1789763958.057517 mapread 0x783b41 thru 0x78d15c (0x961c bytes) 000494[0] 1789763958.107658 trunc from 0x9f0bc7 to 0x092288 (0x95e93e bytes) 000495[0] 1789763958.182949 read 0x003813 thru 0x0065bd (0x2dab bytes) 000497[1] 1789763958.246469 mapwrite 0x9d2247 thru 0x9e0844 (0xe5fe bytes) 000498[0] 1789763958.645614 trunc from 0x9e0845 to 0x19ce50 (0x8439f4 bytes) All operations completed A-OK! Chance of close/open is 1 in 50 Seed set to 3958 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read truncating to largest ever: 0x603184 000002[0] 1789763959.979409 trunc from 00000000 to 0x603183 (0x603184 bytes) 000003[1] 1789763960.041226 punch from 0x35185f to 0x35b65d (0x9dff bytes) 000006[0] 1789763960.113287 read 0x0f8000 thru 0x103d24 (0xbd25 bytes) 000007[0] 1789763960.146365 punch from 0x41cbf1 to 0x422f2f (0x633f bytes) 000010[0] 1789763960.195227 mapread 0x299000 thru 0x29e897 (0x5898 bytes) 000011[1] 1789763960.250752 write 0x22c000 thru 0x23514a (0x914b bytes) 000013[0] 1789763960.305335 mapread 0x042000 thru 0x043b67 (0x1b68 bytes) 000015[0] 1789763960.403497 trunc from 0x603184 to 0x5bb0bf (0x480c4 bytes) 000016[1] 1789763960.481937 write 0x678000 thru 0x682e23 (0xae24 bytes) 000018[1] 1789763960.629532 trunc from 0x682e24 to 0x269928 (0x4194fb bytes) 000019[0] 1789763960.685645 read 0x22d000 thru 0x23198e (0x498f bytes) 000020[1] 1789763960.734347 write 0x069000 thru 0x074141 (0xb142 bytes) 000021[1] 1789763960.819627 read 0x067000 thru 0x070f6a (0x9f6b bytes) truncating to largest ever: 0x966bfa 000022[1] 1789763960.829212 trunc from 0x269929 to 0x966bf9 (0x6fd2d1 bytes) 000023[1] 1789763960.892746 write 0x797000 thru 0x7a6e07 (0xfe08 bytes) 000026[1] 1789763960.922697 mapwrite 0x4b5000 thru 0x4c2b5d (0xdb5e bytes) 000027[0] 1789763961.021665 mapwrite 0x039000 thru 0x048faa (0xffab bytes) 000028[0] 1789763961.208875 mapwrite 0x15d000 thru 0x16cf98 (0xff99 bytes) 000029[0] 1789763961.339913 mapread 0x3f3000 thru 0x3f5636 (0x2637 bytes) 000031[1] 1789763961.386929 write 0x1c1000 thru 0x1c5829 (0x482a bytes) 000033[1] 1789763961.444370 punch from 0x3b6821 to 0x3b928b (0x2a6b bytes) 000034[0] 1789763961.474442 mapread 0x8c1000 thru 0x8ccba1 (0xbba2 bytes) 000037[0] 1789763961.558176 trunc from 0x966bfa to 0x36b761 (0x5fb498 bytes) 000038[1] 1789763961.640891 punch from 0x2d0d11 to 0x2de92d (0xdc1d bytes) 000041[1] 1789763961.685481 mapwrite 0x34d000 thru 0x356d47 (0x9d48 bytes) 000042[0] 1789763961.784374 trunc from 0x36b762 to 0x335c54 (0x35b0d bytes) 000046[0] 1789763961.848331 trunc from 0x335c55 to 0x39c1df (0x6658b bytes) 000048[0] 1789763961.899669 punch from 0x13dfb8 to 0x13fbc5 (0x1c0e bytes) 000049[0] 1789763961.933108 mapwrite 0x670000 thru 0x6742f5 (0x42f6 bytes) 000050[0] 1789763962.013655 write 0x9c1000 thru 0x9cdde6 (0xcde7 bytes) 000052[0] 1789763962.046920 write 0x349000 thru 0x35721a (0xe21b bytes) 000055[1] 1789763962.077667 mapwrite 0x650000 thru 0x65f369 (0xf36a bytes) 000056[1] 1789763962.174278 read 0x312000 thru 0x313f13 (0x1f14 bytes) 000057[0] 1789763962.201892 mapwrite 0x3e7000 thru 0x3ec84d (0x584e bytes) 000058[0] 1789763962.302466 mapwrite 0x397000 thru 0x39acca (0x3ccb bytes) 000061[1] 1789763962.458921 mapwrite 0x9a5000 thru 0x9a85de (0x35df bytes) 000063[1] 1789763962.656202 punch from 0x1d1d5f to 0x1df503 (0xd7a5 bytes) 000064[1] 1789763962.667372 trunc from 0x9cdde7 to 0x3ba332 (0x613ab4 bytes) 000065[1] 1789763962.746634 write 0x5f7000 thru 0x5f8490 (0x1491 bytes) 000066[0] 1789763962.786657 punch from 0x52e803 to 0x536bbb (0x83b9 bytes) 000067[1] 1789763962.831055 write 0x7e5000 thru 0x7e7930 (0x2931 bytes) 000068[1] 1789763962.865567 trunc from 0x7e7931 to 0x01f63d (0x7c82f3 bytes) 000069[0] 1789763962.904516 trunc from 0x01f63e to 0x7e1e6e (0x7c2831 bytes) 000070[1] 1789763962.979608 punch from 0x0e524b to 0x0f3ad4 (0xe88a bytes) 000071[0] 1789763963.005649 mapread 0x6cf000 thru 0x6d2e34 (0x3e35 bytes) 000072[0] 1789763963.102587 trunc from 0x7e1e6f to 0x672321 (0x16fb4d bytes) 000073[0] 1789763963.148103 write 0x545000 thru 0x54dd89 (0x8d8a bytes) 000074[0] 1789763963.156914 punch from 0x1cbb00 to 0x1d0422 (0x4923 bytes) 000077[0] 1789763963.193425 punch from 0x25c2fe to 0x26aaee (0xe7f1 bytes) 000078[1] 1789763963.248753 read 0x04b000 thru 0x04b8c9 (0x8ca bytes) 000080[1] 1789763963.311106 mapwrite 0x32e000 thru 0x32ed1f (0xd20 bytes) 000082[0] 1789763963.563923 read 0x329000 thru 0x33418a (0xb18b bytes) 000084[0] 1789763963.627105 mapwrite 0x45a000 thru 0x4660c4 (0xc0c5 bytes) 000085[0] 1789763963.815651 mapread 0x081000 thru 0x08b371 (0xa372 bytes) 000089[1] 1789763963.966796 mapwrite 0x7ae000 thru 0x7bd86b (0xf86c bytes) 000091[0] 1789763964.207505 read 0x788000 thru 0x78edce (0x6dcf bytes) 000094[1] 1789763964.268453 write 0x763000 thru 0x77079b (0xd79c bytes) 000095[0] 1789763964.312427 mapwrite 0x586000 thru 0x59044e (0xa44f bytes) 000096[0] 1789763964.555559 mapwrite 0x192000 thru 0x19e3d9 (0xc3da bytes) 000097[0] 1789763964.865821 trunc from 0x7bd86c to 0x7f6ca8 (0x3943d bytes) 000102[0] 1789763965.028968 write 0x65b000 thru 0x661999 (0x699a bytes) 000104[1] 1789763965.055664 write 0x102000 thru 0x111745 (0xf746 bytes) 000105[1] 1789763965.156592 mapread 0x778000 thru 0x77a7e6 (0x27e7 bytes) 000108[0] 1789763965.282145 mapread 0x564000 thru 0x565e3c (0x1e3d bytes) 000114[0] 1789763965.442906 write 0x976000 thru 0x97c4bd (0x64be bytes) 000116[0] 1789763965.602565 trunc from 0x97c4be to 0x238f04 (0x7435b9 bytes) 000117[0] 1789763965.657395 mapwrite 0x302000 thru 0x30d5be (0xb5bf bytes) 000118[0] 1789763965.867105 trunc from 0x30d5bf to 0x6b11d7 (0x3a3c19 bytes) 000121[0] 1789763965.973941 trunc from 0x6b11d8 to 0x840c6f (0x18fa98 bytes) 000123[0] 1789763966.026953 read 0x4b2000 thru 0x4b89e9 (0x69ea bytes) 000124[0] 1789763966.042352 trunc from 0x840c70 to 0x63738d (0x2098e2 bytes) 000125[1] 1789763966.128404 read 0x176000 thru 0x17f440 (0x9441 bytes) 000126[1] 1789763966.178810 mapread 0x17b000 thru 0x17d6d2 (0x26d3 bytes) 000128[0] 1789763966.184626 mapread 0x522000 thru 0x525c1c (0x3c1d bytes) 000129[1] 1789763966.241252 trunc from 0x63738e to 0x011a59 (0x625934 bytes) 000132[1] 1789763966.443144 mapread 0x010000 thru 0x010afe (0xaff bytes) 000133[1] 1789763966.486591 mapwrite 0x53e000 thru 0x54dfa9 (0xffaa bytes) 000136[0] 1789763966.743839 trunc from 0x54dfaa to 0x1493e1 (0x404bc8 bytes) 000138[1] 1789763966.872561 write 0x03e000 thru 0x0475a2 (0x95a3 bytes) 000139[0] 1789763966.934962 trunc from 0x1493e2 to 0x6180bd (0x4cecdc bytes) 000141[0] 1789763967.061695 read 0x0d8000 thru 0x0e7b41 (0xfb42 bytes) 000146[0] 1789763967.182012 mapwrite 0x802000 thru 0x80b692 (0x9693 bytes) 000147[1] 1789763967.369508 trunc from 0x80b693 to 0x89c62d (0x90f9b bytes) 000148[0] 1789763967.416866 mapwrite 0x0b3000 thru 0x0bc2a6 (0x92a7 bytes) 000150[0] 1789763967.648477 mapwrite 0x95d000 thru 0x96ccf9 (0xfcfa bytes) 000151[1] 1789763967.889552 trunc from 0x96ccfa to 0x1e1a34 (0x78b2c5 bytes) 000155[0] 1789763968.030347 punch from 0x06accb to 0x06de68 (0x319e bytes) 000156[1] 1789763968.089664 punch from 0x0f30a3 to 0x101be7 (0xeb45 bytes) 000159[1] 1789763968.207761 write 0x815000 thru 0x81a18f (0x5190 bytes) 000160[1] 1789763968.241796 trunc from 0x81a190 to 0x7fac9e (0x1f4f1 bytes) 000162[0] 1789763968.272900 mapread 0x5b3000 thru 0x5b4cd4 (0x1cd5 bytes) 000163[1] 1789763968.334945 punch from 0x4eec20 to 0x4f619d (0x757e bytes) 000164[0] 1789763968.349645 trunc from 0x7fac9f to 0x205266 (0x5f5a38 bytes) 000165[1] 1789763968.462079 punch from 0x15bdb7 to 0x15d2cf (0x1519 bytes) 000166[0] 1789763968.495499 trunc from 0x205267 to 0x3a6921 (0x1a16bb bytes) 000167[1] 1789763968.561599 mapread 0x0e7000 thru 0x0e70dd (0xde bytes) 000168[1] 1789763968.591537 punch from 0x0a572e to 0x0a5ffe (0x8d1 bytes) 000169[1] 1789763968.609213 mapwrite 0x0f8000 thru 0x0fa490 (0x2491 bytes) 000170[0] 1789763968.691395 trunc from 0x3a6922 to 0x6e96cf (0x342dae bytes) 000171[1] 1789763968.727920 punch from 0x1f944f to 0x20666e (0xd220 bytes) 000172[1] 1789763968.752434 read 0x63d000 thru 0x63d2c5 (0x2c6 bytes) 000173[0] 1789763968.783123 mapread 0x6a0000 thru 0x6a6380 (0x6381 bytes) 000174[1] 1789763968.845292 trunc from 0x6e96d0 to 0x1623f2 (0x5872dd bytes) 000175[1] 1789763968.908743 read 0x018000 thru 0x026fa5 (0xefa6 bytes) 000176[1] 1789763968.971356 trunc from 0x1623f3 to 0x2509cd (0xee5db bytes) truncating to largest ever: 0x9b0e2e 000179[1] 1789763969.099714 trunc from 0x2509ce to 0x9b0e2d (0x760460 bytes) 000181[0] 1789763969.170499 read 0x3b6000 thru 0x3c0a4f (0xaa50 bytes) 000182[1] 1789763969.224343 mapwrite 0x31f000 thru 0x32d851 (0xe852 bytes) 000183[1] 1789763969.617133 punch from 0x1475c1 to 0x1501ca (0x8c0a bytes) 000184[1] 1789763969.625494 mapwrite 0x8e3000 thru 0x8e5813 (0x2814 bytes) 000187[1] 1789763969.973208 mapread 0x6e5000 thru 0x6edcdb (0x8cdc bytes) 000188[1] 1789763970.000120 mapwrite 0x5c5000 thru 0x5c7f72 (0x2f73 bytes) 000190[1] 1789763970.230712 punch from 0x268dcf to 0x27507e (0xc2b0 bytes) 000191[1] 1789763970.241100 mapwrite 0x74f000 thru 0x74f035 (0x36 bytes) 000193[1] 1789763970.456651 trunc from 0x9b0e2e to 0x181d7a (0x82f0b3 bytes) 000195[1] 1789763970.578120 write 0x28c000 thru 0x2961f7 (0xa1f8 bytes) 000197[0] 1789763970.580908 read 0x0aa000 thru 0x0aee86 (0x4e87 bytes) 000199[0] 1789763970.745174 read 0x17f000 thru 0x17f07c (0x7d bytes) 000200[0] 1789763970.767216 write 0x145000 thru 0x14acd1 (0x5cd2 bytes) 000201[1] 1789763970.883146 mapread 0x240000 thru 0x24dea8 (0xdea9 bytes) 000203[0] 1789763971.172071 trunc from 0x2961f8 to 0x4b6e30 (0x220c39 bytes) 000205[0] 1789763971.366308 trunc from 0x4b6e31 to 0x2d6596 (0x1e089a bytes) 000210[0] 1789763971.490934 mapwrite 0x18d000 thru 0x1920f4 (0x50f5 bytes) 000212[1] 1789763971.756252 read 0x07b000 thru 0x085723 (0xa724 bytes) 000213[0] 1789763971.914603 write 0x393000 thru 0x3963cb (0x33cc bytes) 000214[0] 1789763971.983014 write 0x6db000 thru 0x6e3bbf (0x8bc0 bytes) 000216[1] 1789763972.039137 write 0x76e000 thru 0x7753a3 (0x73a4 bytes) 000217[0] 1789763972.097803 read 0x2ff000 thru 0x30504d (0x604e bytes) 000218[1] 1789763972.206757 mapread 0x5ac000 thru 0x5b1943 (0x5944 bytes) 000220[0] 1789763972.317625 trunc from 0x7753a4 to 0x72cb55 (0x4884e bytes) 000221[0] 1789763972.435470 mapwrite 0x6f9000 thru 0x6fbdfd (0x2dfe bytes) 000222[1] 1789763972.555873 trunc from 0x72cb56 to 0x0e6134 (0x646a21 bytes) 000224[1] 1789763972.627785 write 0x168000 thru 0x1732ef (0xb2f0 bytes) 000225[0] 1789763972.641175 mapread 0x036000 thru 0x040713 (0xa714 bytes) 000226[0] 1789763972.696550 read 0x0fe000 thru 0x109285 (0xb286 bytes) 000228[1] 1789763972.724118 trunc from 0x1732f0 to 0x48675f (0x313470 bytes) 000229[0] 1789763972.798126 write 0x387000 thru 0x38e996 (0x7997 bytes) 000231[0] 1789763972.981264 punch from 0x35e217 to 0x36cc90 (0xea7a bytes) 000233[0] 1789763973.097861 punch from 0x41e7df to 0x423a43 (0x5265 bytes) 000234[0] 1789763973.148416 mapwrite 0x290000 thru 0x290fe3 (0xfe4 bytes) 000237[1] 1789763973.356945 trunc from 0x486760 to 0x7767f7 (0x2f0098 bytes) 000238[0] 1789763973.466747 read 0x377000 thru 0x37e38e (0x738f bytes) 000240[1] 1789763973.493296 mapread 0x59f000 thru 0x5a6ff0 (0x7ff1 bytes) 000241[1] 1789763973.638411 punch from 0x503b24 to 0x50558f (0x1a6c bytes) 000242[1] 1789763973.670166 trunc from 0x7767f8 to 0x2ab456 (0x4cb3a1 bytes) 000243[0] 1789763973.709452 trunc from 0x2ab457 to 0x7ee578 (0x543122 bytes) 000245[0] 1789763973.851189 trunc from 0x7ee579 to 0x12194c (0x6ccc2c bytes) 000249[1] 1789763974.021972 write 0x902000 thru 0x91072e (0xe72f bytes) 000250[0] 1789763974.053079 mapwrite 0x9b2000 thru 0x9b3786 (0x1787 bytes) 000251[1] 1789763974.316925 mapwrite 0x356000 thru 0x361142 (0xb143 bytes) 000253[0] 1789763974.660825 mapwrite 0x296000 thru 0x2a1efe (0xbeff bytes) 000254[0] 1789763974.996802 write 0x558000 thru 0x5601ed (0x81ee bytes) 000256[0] 1789763975.103160 trunc from 0x9b3787 to 0x055954 (0x95de32 bytes) 000257[1] 1789763975.152627 read 0x01a000 thru 0x020987 (0x6988 bytes) 000258[0] 1789763975.243527 mapread 0x036000 thru 0x037c51 (0x1c52 bytes) 000260[1] 1789763975.291927 trunc from 0x055955 to 0x3f03fe (0x39aaaa bytes) 000264[0] 1789763975.422939 trunc from 0x3f03ff to 0x6f8b58 (0x30875a bytes) 000265[0] 1789763975.469389 mapwrite 0x62b000 thru 0x633887 (0x8888 bytes) 000266[0] 1789763975.611531 punch from 0x19724a to 0x1a20ab (0xae62 bytes) 000267[0] 1789763975.648724 punch from 0x10fc44 to 0x11b0f3 (0xb4b0 bytes) 000268[0] 1789763975.682632 punch from 0x101c6c to 0x106a67 (0x4dfc bytes) 000272[0] 1789763975.727850 trunc from 0x6f8b59 to 0x848fb7 (0x15045f bytes) 000273[0] 1789763975.760457 punch from 0x317420 to 0x31ce66 (0x5a47 bytes) 000274[1] 1789763975.794907 punch from 0x2c3ff3 to 0x2cee83 (0xae91 bytes) 000276[1] 1789763975.841406 read 0x267000 thru 0x271a65 (0xaa66 bytes) 000279[1] 1789763975.960551 trunc from 0x848fb8 to 0x2cadb9 (0x57e1fe bytes) 000280[1] 1789763976.025340 read 0x143000 thru 0x14e154 (0xb155 bytes) 000281[0] 1789763976.164710 trunc from 0x2cadba to 0x75c216 (0x49145d bytes) 000285[0] 1789763976.243667 write 0x64d000 thru 0x655abc (0x8abd bytes) 000287[0] 1789763976.279587 mapread 0x438000 thru 0x44279b (0xa79c bytes) 000288[1] 1789763976.347792 read 0x3bd000 thru 0x3ca661 (0xd662 bytes) 000289[0] 1789763976.391922 trunc from 0x75c217 to 0x3bca26 (0x39f7f0 bytes) 000291[0] 1789763976.593805 mapwrite 0x0f7000 thru 0x1033ee (0xc3ef bytes) 000292[1] 1789763976.750803 read 0x380000 thru 0x387352 (0x7353 bytes) 000293[0] 1789763976.858956 punch from 0x1f1b0f to 0x1ff868 (0xdd5a bytes) 000295[1] 1789763976.939851 mapwrite 0x419000 thru 0x425c05 (0xcc06 bytes) 000296[1] 1789763977.130743 mapread 0x216000 thru 0x21dd26 (0x7d27 bytes) 000297[0] 1789763977.196294 read 0x04f000 thru 0x05816e (0x916f bytes) 000298[1] 1789763977.305584 mapread 0x0a2000 thru 0x0a3ba2 (0x1ba3 bytes) 000299[1] 1789763977.326318 punch from 0x0639cb to 0x0738c2 (0xfef8 bytes) 000302[0] 1789763977.403291 mapwrite 0x66a000 thru 0x66f449 (0x544a bytes) 000303[0] 1789763977.788171 write 0x5c9000 thru 0x5d01d7 (0x71d8 bytes) 000304[0] 1789763977.817902 trunc from 0x66f44a to 0x87fac5 (0x21067c bytes) 000307[1] 1789763977.887748 mapread 0x49c000 thru 0x4a61ee (0xa1ef bytes) 000308[1] 1789763978.081258 read 0x2fa000 thru 0x2fcd9f (0x2da0 bytes) 000309[0] 1789763978.105353 read 0x1c7000 thru 0x1d654d (0xf54e bytes) 000310[0] 1789763978.145269 punch from 0x2d3165 to 0x2e220d (0xf0a9 bytes) 000312[0] 1789763978.214563 punch from 0x29c053 to 0x29dae4 (0x1a92 bytes) 000313[0] 1789763978.221127 mapwrite 0x4c6000 thru 0x4cb96b (0x596c bytes) 000316[0] 1789763978.473327 punch from 0x5bfe76 to 0x5c26ab (0x2836 bytes) 000317[0] 1789763978.488809 mapread 0x35c000 thru 0x361956 (0x5957 bytes) 000319[1] 1789763978.556435 trunc from 0x87fac6 to 0x8144ba (0x6b60b bytes) 000320[0] 1789763978.632425 mapread 0x32e000 thru 0x339e43 (0xbe44 bytes) 000322[1] 1789763978.731853 punch from 0x68441e to 0x687b77 (0x375a bytes) 000323[0] 1789763978.803010 write 0x5a1000 thru 0x5a95c9 (0x85ca bytes) 000325[0] 1789763978.846398 mapread 0x731000 thru 0x73514a (0x414b bytes) 000326[1] 1789763978.882287 punch from 0x0d8a9a to 0x0de24c (0x57b3 bytes) 000327[1] 1789763979.003284 write 0x913000 thru 0x918f9f (0x5fa0 bytes) 000328[1] 1789763979.011972 mapread 0x851000 thru 0x851aa4 (0xaa5 bytes) 000329[1] 1789763979.039249 mapwrite 0x80e000 thru 0x81b08b (0xd08c bytes) 000330[0] 1789763979.257180 write 0x5a1000 thru 0x5ab360 (0xa361 bytes) 000332[0] 1789763979.391859 mapwrite 0x753000 thru 0x75c146 (0x9147 bytes) 000333[0] 1789763979.603350 mapwrite 0x2e2000 thru 0x2e7bb5 (0x5bb6 bytes) 000335[0] 1789763979.792926 mapwrite 0x390000 thru 0x3901b6 (0x1b7 bytes) 000336[1] 1789763979.935337 mapread 0x5d9000 thru 0x5e49fa (0xb9fb bytes) 000337[0] 1789763980.010769 trunc from 0x918fa0 to 0x46abcb (0x4ae3d4 bytes) 000341[1] 1789763980.075311 mapwrite 0x93d000 thru 0x940ec9 (0x3eca bytes) 000346[1] 1789763980.254304 trunc from 0x940eca to 0x03ff89 (0x900f40 bytes) 000347[1] 1789763980.304145 write 0x524000 thru 0x52cb34 (0x8b35 bytes) 000348[0] 1789763980.335956 mapwrite 0x0ed000 thru 0x0f4bed (0x7bee bytes) 000350[0] 1789763980.508974 punch from 0x08e51a to 0x098a8c (0xa573 bytes) 000352[1] 1789763980.569808 mapwrite 0x9f2000 thru 0x9fd277 (0xb278 bytes) 000353[1] 1789763980.766697 trunc from 0x9fd278 to 0x62580a (0x3d7a6d bytes) 000354[0] 1789763980.802551 write 0x922000 thru 0x92f6c6 (0xd6c7 bytes) 000355[1] 1789763980.876102 punch from 0x35f49e to 0x369644 (0xa1a7 bytes) 000357[0] 1789763980.943017 punch from 0x49c816 to 0x4a872e (0xbf19 bytes) 000360[0] 1789763981.041733 mapwrite 0x4f2000 thru 0x4f8a19 (0x6a1a bytes) 000365[1] 1789763981.285304 mapread 0x7fe000 thru 0x80c609 (0xe60a bytes) 000367[1] 1789763981.369463 punch from 0x45a927 to 0x46613a (0xb814 bytes) 000368[1] 1789763981.404907 write 0x097000 thru 0x09e496 (0x7497 bytes) 000370[1] 1789763981.411202 read 0x66e000 thru 0x67ca0d (0xea0e bytes) 000373[1] 1789763981.504081 mapwrite 0x8e4000 thru 0x8e4694 (0x695 bytes) 000378[1] 1789763981.700748 punch from 0x5c41b3 to 0x5d0745 (0xc593 bytes) 000379[1] 1789763981.743663 trunc from 0x92f6c7 to 0x01bb65 (0x913b61 bytes) 000380[0] 1789763981.788816 mapread 0x00a000 thru 0x00c5a8 (0x25a9 bytes) 000381[1] 1789763981.865968 read 0x00a000 thru 0x00ee21 (0x4e22 bytes) 000383[1] 1789763981.889942 mapwrite 0x42e000 thru 0x43b1f5 (0xd1f6 bytes) 000385[1] 1789763982.453261 read 0x2c3000 thru 0x2d1adc (0xeadd bytes) 000386[1] 1789763982.453761 write 0x645000 thru 0x64c5a4 (0x75a5 bytes) 000387[1] 1789763982.455401 mapread 0x1e2000 thru 0x1e4250 (0x2251 bytes) 000388[0] 1789763982.455876 mapread 0x3aa000 thru 0x3aa8f0 (0x8f1 bytes) 000389[1] 1789763982.562971 trunc from 0x64c5a5 to 0x0f2886 (0x559d1e bytes) 000390[0] 1789763982.664886 write 0x7b8000 thru 0x7c7586 (0xf587 bytes) 000391[0] 1789763982.733981 punch from 0x677dc7 to 0x67a0a1 (0x22db bytes) 000393[0] 1789763982.764493 trunc from 0x7c7587 to 0x520764 (0x2a6e22 bytes) 000394[1] 1789763982.818286 trunc from 0x520765 to 0x6785ca (0x157e66 bytes) 000397[1] 1789763983.097164 mapread 0x220000 thru 0x22bb2c (0xbb2d bytes) 000399[0] 1789763983.235606 write 0x71f000 thru 0x71f3fe (0x3ff bytes) 000401[0] 1789763983.326147 mapwrite 0x742000 thru 0x742210 (0x211 bytes) 000402[1] 1789763983.573206 read 0x29d000 thru 0x2a5722 (0x8723 bytes) 000404[1] 1789763983.626532 trunc from 0x742211 to 0x9a622a (0x26401a bytes) 000405[0] 1789763983.675011 mapread 0x1b7000 thru 0x1be95b (0x795c bytes) 000406[1] 1789763983.763245 mapwrite 0x97f000 thru 0x980111 (0x1112 bytes) 000408[1] 1789763984.023723 trunc from 0x9a622b to 0x8ae332 (0xf7ef8 bytes) 000409[1] 1789763984.076744 punch from 0x2b0d80 to 0x2b8c8b (0x7f0c bytes) 000410[0] 1789763984.094817 write 0x7dc000 thru 0x7e3834 (0x7835 bytes) 000412[0] 1789763984.160349 write 0x25b000 thru 0x25d7ea (0x27eb bytes) 000415[1] 1789763984.195018 mapwrite 0x5a2000 thru 0x5aa02c (0x802d bytes) 000418[0] 1789763984.531831 trunc from 0x8ae333 to 0x9a5dc8 (0xf7a96 bytes) 000423[1] 1789763984.757755 punch from 0x2638fb to 0x270766 (0xce6c bytes) 000426[1] 1789763984.839449 trunc from 0x9a5dc9 to 0x48968a (0x51c73e bytes) 000427[1] 1789763984.928855 read 0x032000 thru 0x03a5fa (0x85fb bytes) 000428[0] 1789763984.967642 mapread 0x0bf000 thru 0x0bf222 (0x223 bytes) 000429[0] 1789763985.025807 read 0x105000 thru 0x1069e0 (0x19e1 bytes) 000430[0] 1789763985.038137 trunc from 0x48968b to 0x0aeb32 (0x3dab58 bytes) 000431[0] 1789763985.133156 mapread 0x09d000 thru 0x0a07ce (0x37cf bytes) 000432[1] 1789763985.183506 punch from 0x03023d to 0x040196 (0xff5a bytes) 000433[0] 1789763985.211372 read 0x059000 thru 0x05b682 (0x2683 bytes) 000435[1] 1789763985.277165 read 0x039000 thru 0x040066 (0x7067 bytes) 000436[1] 1789763985.303721 trunc from 0x0aeb33 to 0x8cf3fb (0x8208c9 bytes) 000438[1] 1789763985.422251 punch from 0x8ca119 to 0x8ca4ee (0x3d6 bytes) 000439[1] 1789763985.476664 mapwrite 0x8b9000 thru 0x8bf02b (0x602c bytes) 000440[0] 1789763985.885361 punch from 0x518c67 to 0x51faa0 (0x6e3a bytes) 000442[0] 1789763985.989134 punch from 0x40d454 to 0x40e8d8 (0x1485 bytes) 000443[1] 1789763986.007206 mapwrite 0x5c5000 thru 0x5d46cf (0xf6d0 bytes) 000446[0] 1789763986.569464 punch from 0x50c075 to 0x5103be (0x434a bytes) 000447[0] 1789763986.639866 mapwrite 0x174000 thru 0x17c38f (0x8390 bytes) 000448[1] 1789763986.970453 read 0x8c2000 thru 0x8ce9c9 (0xc9ca bytes) 000450[1] 1789763987.058831 punch from 0x6da6af to 0x6e8df3 (0xe745 bytes) 000451[0] 1789763987.171701 write 0x4d2000 thru 0x4dddea (0xbdeb bytes) 000455[0] 1789763987.230954 read 0x5e8000 thru 0x5eaf74 (0x2f75 bytes) 000457[1] 1789763987.255988 mapread 0x6d8000 thru 0x6dedf2 (0x6df3 bytes) 000458[1] 1789763987.364636 read 0x877000 thru 0x877535 (0x536 bytes) 000459[0] 1789763987.381926 mapread 0x165000 thru 0x165d32 (0xd33 bytes) 000460[1] 1789763987.392948 trunc from 0x8cf3fc to 0x8c235b (0xd0a0 bytes) 000461[1] 1789763987.468528 write 0x22d000 thru 0x234070 (0x7071 bytes) 000462[0] 1789763987.516119 mapwrite 0x70b000 thru 0x715817 (0xa818 bytes) 000463[0] 1789763987.804681 write 0x0ac000 thru 0x0b10f4 (0x50f5 bytes) 000465[1] 1789763987.848721 mapwrite 0x69c000 thru 0x6a13f4 (0x53f5 bytes) 000467[1] 1789763987.978636 read 0x6ec000 thru 0x6f2b6a (0x6b6b bytes) 000468[1] 1789763987.989424 read 0x298000 thru 0x2a5060 (0xd061 bytes) 000469[1] 1789763988.022910 write 0x3eb000 thru 0x3eb950 (0x951 bytes) 000470[1] 1789763988.032781 punch from 0x657dc2 to 0x6652b1 (0xd4f0 bytes) 000471[1] 1789763988.038872 trunc from 0x8c235c to 0x488a42 (0x439919 bytes) 000477[0] 1789763988.157620 mapread 0x073000 thru 0x077327 (0x4328 bytes) 000478[0] 1789763988.201889 mapwrite 0x018000 thru 0x025caa (0xdcab bytes) 000480[1] 1789763988.322492 mapwrite 0x1b0000 thru 0x1be0ac (0xe0ad bytes) 000484[0] 1789763988.489538 read 0x454000 thru 0x4588d8 (0x48d9 bytes) 000485[1] 1789763988.520881 write 0x55c000 thru 0x560cca (0x4ccb bytes) 000487[1] 1789763988.570825 mapread 0x01b000 thru 0x02aedf (0xfee0 bytes) 000488[0] 1789763988.644929 mapread 0x3c4000 thru 0x3c93da (0x53db bytes) 000489[1] 1789763988.714860 read 0x4d2000 thru 0x4e032a (0xe32b bytes) 000490[1] 1789763988.728674 write 0x236000 thru 0x23dd67 (0x7d68 bytes) 000492[0] 1789763988.765519 punch from 0x2e920f to 0x2f2dbe (0x9bb0 bytes) 000493[1] 1789763988.803621 mapread 0x486000 thru 0x488c2b (0x2c2c bytes) 000496[0] 1789763988.861335 mapwrite 0x180000 thru 0x18e1cd (0xe1ce bytes) 000498[1] 1789763989.005269 mapread 0x4c0000 thru 0x4c08f0 (0x8f1 bytes) 000499[1] 1789763989.059999 mapread 0x130000 thru 0x139928 (0x9929 bytes) All operations completed A-OK! PASS 16a (78s) == sanityn test 16b: 500 iterations of dual-mount fsx at small size ========================================================== 16:39:55 (1789763995) keep default fallocate mode: 2 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.615343 s, 13.6 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.426194 s, 19.7 MB/s Chance of close/open is 1 in 50 Seed set to 3999 fd 0: /mnt/lustre/f16b.sanityn fd 1: /mnt/lustre2/f16b.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read 000004[0] 1789763999.992844 mapwrite 0x9264b1 thru 0x92783a (0x138a bytes) 000007[1] 1789764000.190776 read 0x39f9b9 thru 0x3a1601 (0x1c49 bytes) 000008[0] 1789764000.236666 read 0x3ac1ca thru 0x3ac5e9 (0x420 bytes) truncating to largest ever: 0x6ba228 000009[1] 1789764000.250768 trunc from 0x92783b to 0x6ba227 (0x26d613 bytes) 000010[0] 1789764000.309998 mapread 0x4a1962 thru 0x4a2e77 (0x1516 bytes) 000013[0] 1789764000.386623 read 0x3e5db1 thru 0x3e6932 (0xb82 bytes) 000014[0] 1789764000.410567 read 0x2f06b6 thru 0x2f0b8d (0x4d8 bytes) 000015[1] 1789764000.419762 trunc from 0x6ba228 to 0x65ae5f (0x5f3c8 bytes) 000017[1] 1789764000.526654 mapread 0x4e604e thru 0x4e6f78 (0xf2b bytes) 000022[1] 1789764000.588944 punch from 0x3c0a62 to 0x3c11cd (0x76c bytes) 000023[0] 1789764000.594905 write 0x6f69b4 thru 0x6f881e (0x1e6b bytes) 000025[1] 1789764000.632314 mapwrite 0x3205d4 thru 0x3206db (0x108 bytes) 000027[1] 1789764000.749732 write 0x769ee6 thru 0x769f9f (0xba bytes) 000030[1] 1789764000.789862 read 0x0a9bb4 thru 0x0a9ebc (0x309 bytes) 000033[1] 1789764000.846154 mapwrite 0x0684b7 thru 0x06868b (0x1d5 bytes) 000034[0] 1789764000.893142 mapread 0x5a0ead thru 0x5a1a32 (0xb86 bytes) 000037[0] 1789764001.016536 mapwrite 0x565e7d thru 0x566685 (0x809 bytes) 000038[1] 1789764001.122713 mapread 0x47a4e7 thru 0x47b10d (0xc27 bytes) 000040[1] 1789764001.180849 mapwrite 0x2cbb04 thru 0x2ccdb3 (0x12b0 bytes) 000041[0] 1789764001.254628 trunc from 0x769fa0 to 0x2ad990 (0x4bc60f bytes) 000043[0] 1789764001.311957 mapread 0x101b25 thru 0x102eee (0x13ca bytes) truncating to largest ever: 0x80d051 000045[1] 1789764001.325554 trunc from 0x2ad991 to 0x80d050 (0x55f6c0 bytes) 000046[1] 1789764001.386497 punch from 0x489f45 to 0x48b6f3 (0x17af bytes) 000048[1] 1789764001.463681 write 0x08bc08 thru 0x08d3a2 (0x179b bytes) 000049[1] 1789764001.479928 trunc from 0x80d051 to 0x47a556 (0x392afa bytes) 000050[0] 1789764001.531098 trunc from 0x47a557 to 0x03dcb3 (0x43c8a3 bytes) 000051[0] 1789764001.712774 write 0x1af0ff thru 0x1b0eae (0x1db0 bytes) 000053[0] 1789764001.767428 punch from 0x148905 to 0x14a105 (0x1801 bytes) 000054[1] 1789764001.821910 trunc from 0x1b0eaf to 0x6fd463 (0x54c5b5 bytes) 000055[1] 1789764001.874243 punch from 0x16be78 to 0x16d888 (0x1a11 bytes) 000058[0] 1789764001.900074 mapread 0x28c760 thru 0x28cb54 (0x3f5 bytes) 000059[1] 1789764001.958470 mapwrite 0x87c5c1 thru 0x87d1b7 (0xbf7 bytes) 000061[1] 1789764002.130188 write 0x15072e thru 0x150f4c (0x81f bytes) 000065[0] 1789764002.262911 mapread 0x59a79c thru 0x59b6a5 (0xf0a bytes) 000067[1] 1789764002.356665 trunc from 0x87d1b8 to 0x1cbf2c (0x6b128b bytes) 000069[1] 1789764002.462652 mapread 0x0b7147 thru 0x0b7b36 (0x9f0 bytes) 000070[0] 1789764002.467945 punch from 0x12becd to 0x12d029 (0x115d bytes) 000071[1] 1789764002.512752 mapread 0x07b3a5 thru 0x07b4a5 (0x101 bytes) 000072[0] 1789764002.566819 mapread 0x1810e8 thru 0x182c1d (0x1b36 bytes) 000075[0] 1789764002.612168 read 0x142175 thru 0x143597 (0x1423 bytes) 000076[1] 1789764002.677423 read 0x14ed52 thru 0x15064f (0x18fe bytes) 000077[0] 1789764002.735718 write 0x176c88 thru 0x177d47 (0x10c0 bytes) 000078[0] 1789764002.784432 trunc from 0x1cbf2d to 0x0f79cf (0xd455d bytes) 000079[1] 1789764002.834100 mapread 0x0557b7 thru 0x0571e0 (0x1a2a bytes) 000085[1] 1789764003.074337 mapwrite 0x64344e thru 0x64365a (0x20d bytes) 000086[0] 1789764003.209422 read 0x5037b7 thru 0x504442 (0xc8c bytes) 000087[0] 1789764003.236649 read 0x2154c5 thru 0x216106 (0xc42 bytes) 000088[0] 1789764003.257984 mapread 0x2f6dd5 thru 0x2f79ec (0xc18 bytes) 000090[1] 1789764003.269464 mapread 0x1df708 thru 0x1e0a10 (0x1309 bytes) 000093[0] 1789764003.283338 write 0x7a8e28 thru 0x7a9216 (0x3ef bytes) 000094[1] 1789764003.326367 punch from 0x2567c7 to 0x25710d (0x947 bytes) 000095[0] 1789764003.377229 mapread 0x731598 thru 0x732f35 (0x199e bytes) 000096[0] 1789764003.408873 read 0x596bc3 thru 0x5981b3 (0x15f1 bytes) 000097[0] 1789764003.460997 write 0x936e6c thru 0x937826 (0x9bb bytes) 000098[1] 1789764003.535706 read 0x369d47 thru 0x36bbc2 (0x1e7c bytes) 000099[0] 1789764003.601646 mapread 0x1a3c95 thru 0x1a4236 (0x5a2 bytes) 000101[1] 1789764003.669478 punch from 0x529029 to 0x52a471 (0x1449 bytes) 000102[0] 1789764003.711968 mapread 0x236be6 thru 0x2373cb (0x7e6 bytes) 000103[1] 1789764003.751841 trunc from 0x937827 to 0x6987c5 (0x29f061 bytes) 000106[0] 1789764003.844865 write 0x215135 thru 0x215452 (0x31e bytes) 000110[1] 1789764003.941252 mapread 0x3b5fe8 thru 0x3b703f (0x1058 bytes) 000111[1] 1789764003.998772 read 0x2e2954 thru 0x2e3df3 (0x14a0 bytes) 000112[0] 1789764004.017625 trunc from 0x6987c6 to 0x7e0a8b (0x1482c6 bytes) 000113[1] 1789764004.077393 read 0x404ca6 thru 0x406103 (0x145e bytes) 000114[0] 1789764004.102745 mapwrite 0x1df43d thru 0x1e1436 (0x1ffa bytes) 000115[0] 1789764004.175366 mapread 0x376a20 thru 0x3774ee (0xacf bytes) 000116[1] 1789764004.186465 punch from 0x5df3d8 to 0x5df461 (0x8a bytes) 000117[1] 1789764004.219431 mapwrite 0x895ac8 thru 0x896b69 (0x10a2 bytes) 000123[0] 1789764004.419790 punch from 0x1c4c93 to 0x1c5917 (0xc85 bytes) 000124[0] 1789764004.465794 read 0x45d5aa thru 0x45e63d (0x1094 bytes) 000125[1] 1789764004.512767 read 0x1fe7e0 thru 0x1ffeba (0x16db bytes) 000128[0] 1789764004.537842 mapwrite 0x28e30c thru 0x28edcb (0xac0 bytes) 000130[1] 1789764004.622418 write 0x7ce5b3 thru 0x7cedd9 (0x827 bytes) 000131[1] 1789764004.661010 punch from 0x18707c to 0x188c44 (0x1bc9 bytes) 000135[1] 1789764004.734944 mapwrite 0x87fef7 thru 0x8804cb (0x5d5 bytes) 000138[0] 1789764004.828341 trunc from 0x896b6a to 0x3e6547 (0x4b0622 bytes) 000141[0] 1789764004.994848 mapwrite 0x1b8106 thru 0x1b9889 (0x1784 bytes) 000145[1] 1789764005.152774 punch from 0x0c0057 to 0x0c1444 (0x13ee bytes) 000149[1] 1789764005.285626 trunc from 0x3e6548 to 0x791454 (0x3aaf0d bytes) 000154[0] 1789764005.365178 trunc from 0x791455 to 0x69c081 (0xf53d3 bytes) 000155[1] 1789764005.446784 read 0x46b313 thru 0x46c7c0 (0x14ae bytes) 000156[1] 1789764005.488354 write 0x4d9f7a thru 0x4db87f (0x1906 bytes) 000157[1] 1789764005.574431 read 0x565716 thru 0x566bdb (0x14c6 bytes) 000158[0] 1789764005.620905 trunc from 0x69c082 to 0x3d9354 (0x2c2d2d bytes) 000160[1] 1789764005.675358 mapwrite 0x52b430 thru 0x52b6e8 (0x2b9 bytes) 000161[1] 1789764005.813754 trunc from 0x52b6e9 to 0x0953e2 (0x496306 bytes) 000164[1] 1789764005.889672 mapwrite 0x8425db thru 0x842d77 (0x79d bytes) 000165[1] 1789764005.983178 mapread 0x278003 thru 0x2786c8 (0x6c6 bytes) 000167[1] 1789764006.039777 mapwrite 0x20ca3f thru 0x20d2b4 (0x876 bytes) truncating to largest ever: 0x85bf8d 000168[0] 1789764006.172796 trunc from 0x842d78 to 0x85bf8c (0x19215 bytes) 000169[1] 1789764006.255266 punch from 0x664715 to 0x664d33 (0x61f bytes) 000171[0] 1789764006.332530 punch from 0x1d2e36 to 0x1d473b (0x1906 bytes) 000172[0] 1789764006.371355 read 0x5ff812 thru 0x5ff9e7 (0x1d6 bytes) 000175[0] 1789764006.456560 mapwrite 0x0dad02 thru 0x0db791 (0xa90 bytes) 000178[1] 1789764006.618947 trunc from 0x85bf8d to 0x2b8c83 (0x5a3309 bytes) 000180[1] 1789764006.719472 mapread 0x27e501 thru 0x27ef07 (0xa07 bytes) 000181[1] 1789764006.759854 punch from 0x22e264 to 0x22f4f7 (0x1294 bytes) 000182[1] 1789764006.770381 punch from 0x0062c0 to 0x006d05 (0xa46 bytes) 000183[1] 1789764006.775299 mapwrite 0x8aacb6 thru 0x8abc48 (0xf93 bytes) 000184[0] 1789764006.911672 trunc from 0x8abc49 to 0x095380 (0x8168c8 bytes) 000185[0] 1789764006.969391 read 0x04cc23 thru 0x04d9d8 (0xdb6 bytes) 000190[1] 1789764007.120936 write 0x78831d thru 0x78a00c (0x1cf0 bytes) 000192[1] 1789764007.197653 read 0x297926 thru 0x299427 (0x1b02 bytes) 000195[0] 1789764007.251001 write 0x388ea2 thru 0x3898b4 (0xa13 bytes) 000197[1] 1789764007.299513 write 0x22f67b thru 0x22fc6d (0x5f3 bytes) 000199[0] 1789764007.382636 read 0x5166a2 thru 0x517fb7 (0x1916 bytes) 000200[0] 1789764007.423317 write 0x6081ea thru 0x609983 (0x179a bytes) 000201[1] 1789764007.485555 trunc from 0x78a00d to 0x1e58c3 (0x5a4749 bytes) 000202[0] 1789764007.574282 trunc from 0x1e58c4 to 0x7f338a (0x60dac7 bytes) 000203[1] 1789764007.641897 write 0x387da8 thru 0x3899e7 (0x1c40 bytes) 000204[0] 1789764007.685421 trunc from 0x7f338b to 0x7b4d33 (0x3e657 bytes) 000205[0] 1789764007.745143 write 0x583f8e thru 0x585383 (0x13f6 bytes) 000206[1] 1789764007.807671 write 0x716a02 thru 0x7173a1 (0x9a0 bytes) 000207[1] 1789764007.912074 mapread 0x594f07 thru 0x5964d0 (0x15ca bytes) 000210[0] 1789764008.118730 mapwrite 0x764954 thru 0x765f32 (0x15df bytes) 000211[0] 1789764008.402607 punch from 0x150d9f to 0x152547 (0x17a9 bytes) 000213[1] 1789764008.524829 read 0x43f2ec thru 0x43f4e2 (0x1f7 bytes) 000214[0] 1789764008.599570 mapread 0x6c4e0c thru 0x6c5caa (0xe9f bytes) 000217[0] 1789764008.669096 punch from 0x4c8520 to 0x4c9bb0 (0x1691 bytes) 000218[0] 1789764008.692913 read 0x570ae4 thru 0x5723a2 (0x18bf bytes) 000219[1] 1789764008.699816 mapread 0x76da4c thru 0x76eb61 (0x1116 bytes) 000220[0] 1789764008.760964 mapwrite 0x78d00e thru 0x78e806 (0x17f9 bytes) 000222[1] 1789764008.865611 mapread 0x03a70d thru 0x03b761 (0x1055 bytes) 000223[1] 1789764008.943144 punch from 0x67cf67 to 0x67ed07 (0x1da1 bytes) 000224[0] 1789764008.983601 mapwrite 0x58834f thru 0x58a2fb (0x1fad bytes) 000225[0] 1789764009.202585 read 0x503e0b thru 0x50455f (0x755 bytes) 000228[1] 1789764009.262607 trunc from 0x7b4d34 to 0x7b202d (0x2d06 bytes) 000229[1] 1789764009.439073 mapread 0x0940d9 thru 0x094aa6 (0x9ce bytes) truncating to largest ever: 0x872a4f 000230[1] 1789764009.520591 trunc from 0x7b202e to 0x872a4e (0xc0a21 bytes) 000231[1] 1789764009.580330 trunc from 0x872a4f to 0x5aa952 (0x2c80fc bytes) 000233[1] 1789764009.694415 trunc from 0x5aa953 to 0x2e3b7a (0x2c6dd8 bytes) 000234[1] 1789764009.731657 punch from 0x18bd39 to 0x18d09d (0x1365 bytes) 000236[1] 1789764009.736728 trunc from 0x2e3b7b to 0x3d84c9 (0xf494f bytes) 000237[0] 1789764009.771930 mapwrite 0x74880b thru 0x74a471 (0x1c67 bytes) 000238[0] 1789764009.966202 mapread 0x1d6b62 thru 0x1d76cc (0xb6b bytes) 000241[0] 1789764010.047600 mapread 0x25129d thru 0x252b34 (0x1898 bytes) 000243[1] 1789764010.094925 punch from 0x0ade99 to 0x0aea9d (0xc05 bytes) 000244[0] 1789764010.135838 trunc from 0x74a472 to 0x69d11d (0xad354 bytes) 000246[0] 1789764010.199065 trunc from 0x69d11e to 0x341957 (0x35b7c6 bytes) 000247[1] 1789764010.252515 mapread 0x04e92d thru 0x04f271 (0x945 bytes) 000248[0] 1789764010.298234 trunc from 0x341958 to 0x504fdd (0x1c3686 bytes) 000249[0] 1789764010.336392 write 0x606f28 thru 0x608ebb (0x1f94 bytes) 000252[0] 1789764010.364573 read 0x0f12eb thru 0x0f2d9f (0x1ab5 bytes) 000256[0] 1789764010.423095 mapwrite 0x0d7d97 thru 0x0d8db4 (0x101e bytes) 000258[1] 1789764010.547319 write 0x4e32f9 thru 0x4e4765 (0x146d bytes) 000259[0] 1789764010.615755 write 0x8bfc26 thru 0x8c09fc (0xdd7 bytes) 000262[1] 1789764010.716220 punch from 0x106555 to 0x107f38 (0x19e4 bytes) 000263[1] 1789764010.752945 punch from 0x626be4 to 0x6277dd (0xbfa bytes) 000266[1] 1789764010.798621 write 0x89bede thru 0x89d16f (0x1292 bytes) 000267[0] 1789764010.810938 mapread 0x57ee4e thru 0x580577 (0x172a bytes) 000268[1] 1789764010.877256 mapwrite 0x6787e0 thru 0x678d28 (0x549 bytes) 000271[0] 1789764010.995893 trunc from 0x8c09fd to 0x2eaa76 (0x5d5f86 bytes) 000275[0] 1789764011.140516 punch from 0x023636 to 0x0246e6 (0x10b1 bytes) 000276[0] 1789764011.150652 mapwrite 0x896d4e thru 0x897a8e (0xd41 bytes) 000279[0] 1789764011.280669 mapwrite 0x3ed2e5 thru 0x3edbd3 (0x8ef bytes) 000282[1] 1789764011.471607 write 0x4c4a56 thru 0x4c54f4 (0xa9f bytes) 000286[0] 1789764011.692941 read 0x84373b thru 0x845000 (0x18c6 bytes) 000290[0] 1789764011.782237 read 0x5cfd6c thru 0x5d0a4b (0xce0 bytes) 000291[0] 1789764011.804324 write 0x099d58 thru 0x09abc6 (0xe6f bytes) 000292[0] 1789764011.816520 read 0x05b1b8 thru 0x05bf08 (0xd51 bytes) 000294[0] 1789764011.848574 read 0x7fcaf2 thru 0x7fd408 (0x917 bytes) 000297[1] 1789764011.874782 write 0x3aae41 thru 0x3ab610 (0x7d0 bytes) 000298[1] 1789764011.895664 trunc from 0x897a8f to 0x040f37 (0x856b57 bytes) 000303[1] 1789764011.987351 write 0x6f7726 thru 0x6f8340 (0xc1b bytes) 000304[1] 1789764011.992255 punch from 0x1d6631 to 0x1d82d2 (0x1ca2 bytes) 000305[1] 1789764012.036780 mapread 0x63c667 thru 0x63c81a (0x1b4 bytes) 000306[0] 1789764012.104463 read 0x4cb722 thru 0x4ccf68 (0x1847 bytes) 000307[1] 1789764012.161917 write 0x69f87f thru 0x6a0ddc (0x155e bytes) 000308[1] 1789764012.198804 read 0x5f74b4 thru 0x5f77b7 (0x304 bytes) 000309[0] 1789764012.208878 punch from 0x2418a0 to 0x243505 (0x1c66 bytes) 000311[1] 1789764012.306726 write 0x717f37 thru 0x719e3c (0x1f06 bytes) 000312[1] 1789764012.343593 mapwrite 0x5448c7 thru 0x545e0b (0x1545 bytes) 000313[0] 1789764012.486977 trunc from 0x719e3d to 0x627e21 (0xf201b bytes) 000316[0] 1789764012.675464 read 0x28abd6 thru 0x28b667 (0xa92 bytes) 000317[0] 1789764012.696154 mapread 0x0ddbe8 thru 0x0de662 (0xa7b bytes) 000321[1] 1789764012.865906 mapread 0x1cc8f2 thru 0x1ccf1f (0x62e bytes) 000322[1] 1789764012.940846 mapwrite 0x521758 thru 0x5220a9 (0x952 bytes) 000324[0] 1789764013.042319 mapread 0x1918c0 thru 0x192883 (0xfc4 bytes) 000325[0] 1789764013.100534 punch from 0x52c03b to 0x52c1e0 (0x1a6 bytes) 000326[0] 1789764013.148299 punch from 0x322767 to 0x324282 (0x1b1c bytes) 000328[1] 1789764013.231127 mapread 0x038ced thru 0x0395fb (0x90f bytes) 000329[1] 1789764013.293111 trunc from 0x627e22 to 0x089ac6 (0x59e35b bytes) 000330[0] 1789764013.369815 write 0x73da01 thru 0x73f0a5 (0x16a5 bytes) 000332[1] 1789764013.426686 read 0x10b63f thru 0x10bfe3 (0x9a5 bytes) 000333[1] 1789764013.485818 read 0x41e635 thru 0x41ec4e (0x61a bytes) 000334[0] 1789764013.532896 mapread 0x294696 thru 0x295eff (0x186a bytes) 000335[1] 1789764013.579620 read 0x5d81fe thru 0x5d997c (0x177f bytes) 000337[1] 1789764013.589245 read 0x659a4d thru 0x65b6da (0x1c8e bytes) 000339[1] 1789764013.604385 punch from 0x6fd174 to 0x6fe46c (0x12f9 bytes) 000342[1] 1789764013.692484 read 0x6f7b8e thru 0x6f8b92 (0x1005 bytes) 000343[1] 1789764013.737872 mapwrite 0x59a442 thru 0x59aa3d (0x5fc bytes) 000344[1] 1789764013.878696 trunc from 0x73f0a6 to 0x4bed6f (0x280336 bytes) 000345[1] 1789764013.973355 read 0x1b7128 thru 0x1b7734 (0x60d bytes) 000346[1] 1789764013.990493 trunc from 0x4bed70 to 0x4b2bd3 (0xc19c bytes) 000348[0] 1789764014.073001 mapwrite 0x6ffffa thru 0x700c8b (0xc92 bytes) 000349[0] 1789764014.268540 mapwrite 0x241da1 thru 0x241e87 (0xe7 bytes) 000352[1] 1789764014.372181 write 0x398fbb thru 0x39aae2 (0x1b28 bytes) 000353[0] 1789764014.423401 trunc from 0x700c8c to 0x2792be (0x4879cd bytes) 000354[0] 1789764014.490170 mapwrite 0x3d9ceb thru 0x3db8ce (0x1be4 bytes) 000355[1] 1789764014.570414 mapwrite 0x8e750a thru 0x8e78a8 (0x39f bytes) truncating to largest ever: 0x8e91da 000356[0] 1789764014.760097 trunc from 0x8e78a9 to 0x8e91d9 (0x1931 bytes) 000357[1] 1789764014.816384 write 0x9d73a4 thru 0x9d8868 (0x14c5 bytes) 000359[0] 1789764014.868825 write 0x11fa89 thru 0x120c03 (0x117b bytes) 000361[0] 1789764014.961415 mapread 0x6cc54d thru 0x6cd19b (0xc4f bytes) 000362[0] 1789764015.005929 mapwrite 0x32fc68 thru 0x33041f (0x7b8 bytes) 000363[0] 1789764015.123942 trunc from 0x9d8869 to 0x5b364b (0x42521d bytes) 000366[0] 1789764015.177449 mapwrite 0x83da51 thru 0x83f712 (0x1cc2 bytes) 000367[0] 1789764015.370620 read 0x5c2e70 thru 0x5c48e7 (0x1a78 bytes) 000368[0] 1789764015.398754 read 0x45d1a8 thru 0x45dac9 (0x922 bytes) 000369[1] 1789764015.407786 trunc from 0x83f713 to 0x4cbab0 (0x373c62 bytes) 000370[1] 1789764015.514627 mapwrite 0x5a097e thru 0x5a1800 (0xe83 bytes) 000371[0] 1789764015.582205 mapread 0x2b57f7 thru 0x2b62cb (0xad5 bytes) 000372[0] 1789764015.632656 read 0x3f3414 thru 0x3f4ad7 (0x16c4 bytes) 000373[1] 1789764015.746118 read 0x164d74 thru 0x164e9c (0x129 bytes) 000375[0] 1789764015.787751 punch from 0x466e6d to 0x467a4c (0xbe0 bytes) 000376[0] 1789764015.801494 punch from 0x4dcec8 to 0x4dd405 (0x53e bytes) 000377[1] 1789764015.805762 punch from 0x51aec4 to 0x51b6f8 (0x835 bytes) truncating to largest ever: 0x98e674 000383[1] 1789764015.861878 trunc from 0x5a1801 to 0x98e673 (0x3ece73 bytes) 000384[0] 1789764015.903669 mapread 0x7b2679 thru 0x7b2bce (0x556 bytes) truncating to largest ever: 0x9cdf18 000387[1] 1789764016.041052 trunc from 0x98e674 to 0x9cdf17 (0x3f8a4 bytes) 000389[1] 1789764016.096837 trunc from 0x9cdf18 to 0x059e16 (0x974101 bytes) 000390[1] 1789764016.148264 mapread 0x016d0c thru 0x017cde (0xfd3 bytes) 000393[1] 1789764016.190739 mapread 0x01aabd thru 0x01b676 (0xbba bytes) 000396[0] 1789764016.232394 mapread 0x000431 thru 0x001f7c (0x1b4c bytes) 000399[0] 1789764016.313459 write 0x119da4 thru 0x11a6f8 (0x955 bytes) 000400[1] 1789764016.368153 mapread 0x0f20aa thru 0x0f2b75 (0xacc bytes) 000402[0] 1789764016.448312 mapwrite 0x1a106f thru 0x1a2486 (0x1418 bytes) 000404[1] 1789764016.648538 trunc from 0x1a2487 to 0x3b813d (0x215cb7 bytes) 000405[0] 1789764016.786079 trunc from 0x3b813e to 0x60184f (0x249712 bytes) 000409[1] 1789764017.037510 punch from 0x2f8798 to 0x2fa75f (0x1fc8 bytes) 000410[0] 1789764017.155959 mapwrite 0x4d076d thru 0x4d231d (0x1bb1 bytes) 000413[1] 1789764017.431703 trunc from 0x601850 to 0x35750f (0x2aa340 bytes) 000415[0] 1789764017.562283 punch from 0x0874cc to 0x089163 (0x1c98 bytes) 000421[0] 1789764017.816756 read 0x2d79b3 thru 0x2d82f5 (0x943 bytes) 000423[0] 1789764018.056947 punch from 0x244969 to 0x246954 (0x1fec bytes) 000424[1] 1789764018.085152 read 0x33f94f thru 0x340be3 (0x1295 bytes) 000425[1] 1789764018.183112 mapread 0x2d440f thru 0x2d6182 (0x1d74 bytes) 000426[0] 1789764018.232214 mapwrite 0x005e87 thru 0x0064d2 (0x64c bytes) 000427[0] 1789764018.443866 write 0x8d8d73 thru 0x8d9dfb (0x1089 bytes) 000428[0] 1789764018.477402 mapwrite 0x9cfd6a thru 0x9d1320 (0x15b7 bytes) 000429[1] 1789764018.789349 mapread 0x66146a thru 0x6626f5 (0x128c bytes) 000431[1] 1789764018.831934 mapwrite 0x05bba6 thru 0x05c517 (0x972 bytes) 000433[0] 1789764018.917534 write 0x88933b thru 0x8899b9 (0x67f bytes) 000434[1] 1789764018.932417 punch from 0x87d773 to 0x87df45 (0x7d3 bytes) 000435[1] 1789764018.951971 mapwrite 0x0d3cfc thru 0x0d5b46 (0x1e4b bytes) 000436[1] 1789764019.022936 mapwrite 0x448c1e thru 0x449fec (0x13cf bytes) 000439[0] 1789764019.195906 read 0x0a4563 thru 0x0a4890 (0x32e bytes) 000440[1] 1789764019.224394 mapread 0x34fd5e thru 0x351d4f (0x1ff2 bytes) 000443[0] 1789764019.255684 trunc from 0x9d1321 to 0x789606 (0x247d1a bytes) 000444[1] 1789764019.329346 mapread 0x6c1d60 thru 0x6c3ccb (0x1f6c bytes) 000446[1] 1789764019.371977 read 0x3328bc thru 0x334247 (0x198c bytes) 000447[1] 1789764019.393210 read 0x55fb7a thru 0x560f17 (0x139e bytes) 000448[0] 1789764019.398256 read 0x2ab697 thru 0x2ad59d (0x1f07 bytes) 000449[0] 1789764019.404498 mapwrite 0x9aed66 thru 0x9b089d (0x1b38 bytes) 000451[0] 1789764019.580744 mapread 0x4d7f51 thru 0x4d8042 (0xf2 bytes) 000453[0] 1789764019.689480 mapwrite 0x206835 thru 0x20693e (0x10a bytes) 000456[0] 1789764019.854436 mapwrite 0x5a0fd6 thru 0x5a2701 (0x172c bytes) 000457[1] 1789764019.964989 mapread 0x44f244 thru 0x450193 (0xf50 bytes) 000458[0] 1789764020.007899 trunc from 0x9b089e to 0x0b77f0 (0x8f90ad bytes) 000460[0] 1789764020.060724 mapwrite 0x9b37bd thru 0x9b40dd (0x921 bytes) 000461[0] 1789764020.278714 mapwrite 0x4bd8f6 thru 0x4bf663 (0x1d6e bytes) 000462[0] 1789764020.419477 read 0x786204 thru 0x787745 (0x1542 bytes) 000463[1] 1789764020.439204 write 0x6b9467 thru 0x6b95b6 (0x150 bytes) 000464[0] 1789764020.498875 punch from 0x70b622 to 0x70c205 (0xbe4 bytes) 000466[1] 1789764020.576738 punch from 0x7a6984 to 0x7a6fc7 (0x644 bytes) 000470[1] 1789764020.723286 read 0x3d630d thru 0x3d6d00 (0x9f4 bytes) 000472[1] 1789764020.819993 read 0x2420cb thru 0x243b0e (0x1a44 bytes) 000473[0] 1789764020.871584 write 0x499a6b thru 0x49a616 (0xbac bytes) 000474[1] 1789764020.934110 mapread 0x40b7bc thru 0x40c1a2 (0x9e7 bytes) 000477[0] 1789764021.062899 trunc from 0x9b40de to 0x1a12c6 (0x812e17 bytes) 000478[0] 1789764021.230206 punch from 0x15a4bc to 0x15a6c4 (0x209 bytes) 000479[1] 1789764021.296090 mapwrite 0x6c70a5 thru 0x6c8eb5 (0x1e11 bytes) 000481[1] 1789764021.569344 read 0x46f9ed thru 0x4712f8 (0x190c bytes) 000482[0] 1789764021.608552 mapwrite 0x6ee803 thru 0x6ef26b (0xa69 bytes) 000484[1] 1789764022.042236 read 0x12af03 thru 0x12bb45 (0xc43 bytes) 000485[1] 1789764022.164485 mapwrite 0x64a4c7 thru 0x64abea (0x724 bytes) 000486[1] 1789764022.361618 read 0x4cf679 thru 0x4cfa90 (0x418 bytes) 000489[0] 1789764022.395920 write 0x0e784d thru 0x0e83b3 (0xb67 bytes) 000490[1] 1789764022.437152 mapwrite 0x0f2147 thru 0x0f2290 (0x14a bytes) 000491[0] 1789764022.537468 punch from 0x43da90 to 0x43e240 (0x7b1 bytes) 000492[0] 1789764022.592656 read 0x5586a9 thru 0x558d1a (0x672 bytes) 000493[1] 1789764022.637481 mapwrite 0x48d422 thru 0x48ecb7 (0x1896 bytes) 000494[0] 1789764022.768449 trunc from 0x6ef26c to 0x5d866c (0x116bff bytes) 000495[1] 1789764022.811997 read 0x445314 thru 0x445cbd (0x9aa bytes) 000496[0] 1789764022.833190 punch from 0x1ae623 to 0x1af226 (0xc04 bytes) 000497[1] 1789764022.856011 punch from 0x39614d to 0x396ca2 (0xb56 bytes) 000498[0] 1789764022.895062 mapread 0x4ede94 thru 0x4eed1e (0xe8b bytes) 000499[1] 1789764022.957333 mapwrite 0x964049 thru 0x96529d (0x1255 bytes) 000500[1] 1789764023.106651 read 0x6d219c thru 0x6d3b1e (0x1983 bytes) All operations completed A-OK! PASS 16b (36s) == sanityn test 16c: verify data consistency on ldiskfs with cache disabled (b=17397) ========================================================== 16:40:32 (1789764032) SKIP: sanityn test_16c dio on ldiskfs only SKIP 16c (7s) == sanityn test 16d: Verify DIO and buffer IO with two clients ========================================================== 16:40:38 (1789764038) 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.563139 s, 14.9 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.469953 s, 17.8 MB/s ldlm.namespaces.MGC192.168.204.142@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff9a1004e5e800.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff9a100aa27800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff9a1004e5e800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff9a100aa27800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff9a1004e5e800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff9a100aa27800.lru_size=clear 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 17.2664 s, 6.1 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.62386 s, 22.7 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 3.11868 s, 33.6 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.19004 s, 12.8 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 12.7322 s, 8.2 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.30909 s, 19.8 MB/s PASS 16d (69s) == sanityn test 16e: Verify size consistency for O_DIRECT write ========================================================== 16:41:47 (1789764107) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.451518 s, 23.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155258 s, 6.8 MB/s PASS 16e (10s) == sanityn test 16f: rw sequential consistency vs drop_caches ========================================================== 16:41:57 (1789764117) TIMEOUT PASS 16f (31s) == sanityn test 16g: mmap rw sequential consistency vs drop_caches ========================================================== 16:42:28 (1789764148) TIMEOUT PASS 16g (31s) == sanityn test 16h: mmap read after truncate file ======= 16:42:59 (1789764179) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16h (10s) == sanityn test 16i: read after truncate file ============ 16:43:09 (1789764189) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16i (7s) == sanityn test 16j: race dio with buffered i/o ========== 16:43:16 (1789764196) 1+0 records in 1+0 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.05788 s, 12.7 MB/s bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.40997 s, 18.6 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.66804 s, 7.1 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.1325 s, 12.3 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.96405 s, 8.8 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.99108 s, 13.2 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.46399 s, 10.6 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.06079 s, 12.7 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.51488 s, 10.4 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 4.20853 s, 6.2 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 4.36066 s, 6.0 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.30096 s, 7.9 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.50221 s, 7.5 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.70279 s, 7.1 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 4.11449 s, 6.4 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 4.88352 s, 5.4 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 5.10201 s, 5.1 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 5.32233 s, 4.9 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 5.5279 s, 4.7 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK PASS 16j (47s) == sanityn test 16k: Parallel FSX and drop caches should not panic ========================================================== 16:44:03 (1789764243) + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.1 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.2 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.3 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.4 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.5 -l 67108864 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! + fsx 1 done: rc=0 All operations completed A-OK! Fri Sep 18 16:44:56 EDT 2026 + fsx 2 done: rc=0 Fri Sep 18 16:44:56 EDT 2026 + fsx 3 done: rc=0 Fri Sep 18 16:44:56 EDT 2026 + fsx 4 done: rc=0 Fri Sep 18 16:44:56 EDT 2026 + fsx 5 done: rc=0 Fri Sep 18 16:44:56 EDT 2026 PASS 16k (67s) == sanityn test 17: resource creation/LVB creation race ========================================================================= 16:45:11 (1789764311) fail_loc=0x8000030a PASS 17 (13s) == sanityn test 18: mmap sanity check =========================================================================================== 16:45:24 (1789764324) mmap test1: basic mmap operation (PASS, 0.2732s) mmap test2: MAP_PRIVATE not write back (PASS, 0.1718s) mmap test3: concurrent mmap ops on two nodes (PASS, 15.289s) mmap test4: c1 write to f1 from mmapped f2, c2 write to f1 from mmapped f1 (PASS, 4.7596s) mmap test5: read/write file to/from the buffer which mmapped to just this file (PASS, 0.28443s) mmap test6: check mmap write/read content on two nodes (PASS, 0.34215s) mmap test7: file i/o with an unmapped buffer (SKIPPED, 0s) mmap test8: SIGBUS for beyond file size (SKIPPED, 0s) mmap test9: SIGBUS for negative file offset (SKIPPED, 0s) mmap test10: multi-thread mmap access (PASS, 1.2764s) mmap test11: multi-thread shared mmap (PASS, 1.0145s) mmap test10: mtime not change for readonly mmap access (PASS, 1.1166s) PASS 18 (37s) == sanityn test 19: test concurrent uncached read races ========================================================================= 16:46:03 (1789764363) oleg442-server: error: get_param: param_path 'osd-*/lustre-OST*/read_cache_enable': No such file or directory pdsh@oleg442-client: oleg442-server: ssh exited with exit code 2 SKIP: sanityn test_19 not cache-capable obdfilter SKIP 19 (5s) == sanityn test 20: test extra readahead page left in cache ============================================================== 16:46:08 (1789764368) PASS 20 (9s) == sanityn test 21: Try to remove mountpoint on another dir ============================================================== 16:46:18 (1789764378) rmdir: removing directory, '/mnt/lustre/d21.sanityn' rmdir: failed to remove '/mnt/lustre/d21.sanityn': Device or resource busy rmdir: removing directory, '/mnt/lustre2/d21.sanityn' Removed mounted directory from another mountpoint, needs to be fixed PASS 21 (8s) == sanityn test 23: others should see updated atime while another read============================================================== 16:46:27 (1789764387) now is 1789764390 starting reads multiop /mnt/lustre/f23.sanityn vor20_c TMPPIPE=/tmp/multiop_open_wait_pipe.7652 new atime is 1789764451 PASS 23 (72s) == sanityn test 24a: lfs df [-ih] [path] test =================================================================================== 16:47:39 (1789764459) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 6144 3763200 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 6144 3763200 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre2 UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 115.3K 403 114.9K 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 114.5K 399 114.1K 1% /mnt/lustre[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.3K 403 114.9K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 399 114.1K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre2 UUID bytes Used Available Use% Mounted on lustre-MDT0000_UUID 2.1G 2.1M 2.1G 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3.6G 6.0M 3.6G 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3.6G 32.0M 3.6G 1% /mnt/lustre[OST:1] filesystem_summary: 7.2G 38.0M 7.2G 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118067 403 117664 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117231 399 116832 1% /mnt/lustre2[OST:1] filesystem_summary: 234882 386 234496 1% /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 6144 3763200 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.3K 403 114.9K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 399 114.1K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 118067 403 117664 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 117231 399 116832 1% /mnt/lustre[OST:1] filesystem_summary: 234882 386 234496 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118067 403 117664 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117231 399 116832 1% /mnt/lustre2[OST:1] filesystem_summary: 234882 386 234496 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 6144 3763200 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 6144 3763200 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre2 PASS 24a (11s) == sanityn test 24b: lfs df should show both filesystems ========================================================================= 16:47:50 (1789764470) PASS 24b (10s) == sanityn test 25a: change ACL on one mountpoint be seen on another ============================================================= 16:48:00 (1789764480) running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] PASS 25a (9s) == sanityn test 25b: change ACL under remote dir on one mountpoint be seen on another ========================================================== 16:48:09 (1789764489) SKIP: sanityn test_25b needs >= 2 MDTs SKIP 25b (4s) == sanityn test 26a: allow mtime to get older ============ 16:48:13 (1789764493) utime: good mknod times 1789764494 <= 1789764495 <= 1789764495 for /mnt/lustre/f26a utime: good mknod times 1789764494 <= 1789764495 <= 1789764495 for /mnt/lustre2/f26a utime: good utime mtimes 100000, atime 200000 utime: updated times for /mnt/lustre2/f26a PASS 26a (9s) == sanityn test 26b: sync mtime between ost and mds ====== 16:48:22 (1789764502) PASS 26b (9s) == sanityn test 26c: set-in-past on open file is not lost on close ========================================================== 16:48:31 (1789764511) multiop /mnt/lustre/f26c.sanityn vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7652 File: /mnt/lustre/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502398 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-09-18 16:48:33.000000000 -0400 Birth: 2026-09-18 16:48:33.000000000 -0400 File: /mnt/lustre2/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502398 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-09-18 16:48:33.000000000 -0400 Birth: 2026-09-18 16:48:33.000000000 -0400 PASS 26c (10s) SKIP: sanityn test_27 skipping excluded test 27 == sanityn test 30: recreate file race =================== 16:48:43 (1789764523) 0 PASS 30 (11s) == sanityn test 31a: voluntary cancel / blocking ast race======================================================================== 16:48:54 (1789764534) fail_loc=0x314 PASS 31a (8s) == sanityn test 31b: voluntary OST cancel / blocking ast race======================================================================== 16:49:03 (1789764543) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg442-server mds-ost sync done. sleep 5 for ZFS MDS Waiting for MDT destroys to complete fail_loc=0x314 fail_loc=0x316 fail_loc=0 fail_loc=0 Filesystem 1K-blocks Used Available Use% Mounted on 192.168.204.142@tcp:/lustre 7542784 39936 7496704 1% /mnt/lustre2 PASS 31b (23s) == sanityn test 31r: open-rename(replace) race =========== 16:49:26 (1789764566) -rw-r--r-- 1 root root 0 Sep 18 16:49 /mnt/lustre/f31r.sanityn.target fail_loc=0x1419 fail_val=3 PASS 31r (10s) == sanityn test 31s: open should not revalidate invalid dentry ========================================================== 16:49:37 (1789764577) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 1205: 43213 Terminated tail -f $tfile PASS 31s (9s) == sanityn test 31t: getattr should not revalidate invalid dentry ========================================================== 16:49:45 (1789764585) PASS 31t (10s) == sanityn test 32b: lockless i/o ======================== 16:49:56 (1789764596) SKIP: sanityn test_32b max_nolock_bytes is removed >= 2.17.53 SKIP 32b (7s) == sanityn test 32c: contention detection ================ 16:50:02 (1789764602) ldlm.namespaces.filter-lustre-OST0000_UUID.contended_locks=2 ldlm.namespaces.filter-lustre-OST0001_UUID.contended_locks=2 ldlm.namespaces.filter-lustre-OST0000_UUID.contention_events=0 ldlm.namespaces.filter-lustre-OST0001_UUID.contention_events=0 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.79945 s, 29.1 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.80242 s, 29.1 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.5203 s, 20.8 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.9982 s, 17.5 MB/s contention_seconds 4, contention_events 27 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.40311 s, 21.8 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.93855 s, 17.8 MB/s ldlm.namespaces.filter-lustre-OST0000_UUID.contention_events=0 ldlm.namespaces.filter-lustre-OST0001_UUID.contention_events=0 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.00037 s, 26.2 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.06984 s, 25.3 MB/s PASS 32c (40s) SKIP: sanityn test_33a skipping SLOW test 33a == sanityn test 33b: COS: cross create/delete, 2 clients, benchmark under remote dir ========================================================== 16:50:44 (1789764644) SKIP: sanityn test_33b Need two or more clients, have 1 SKIP 33b (4s) == sanityn test 33c: Cancel cross-MDT lock should trigger Sync-on-Lock-Cancel ========================================================== 16:50:48 (1789764648) SKIP: sanityn test_33c needs >= 2 MDTs SKIP 33c (5s) == sanityn test 33d: dependent transactions should trigger COS ========================================================== 16:50:53 (1789764653) SKIP: sanityn test_33d needs >= 2 MDTs SKIP 33d (3s) == sanityn test 33e: independent transactions shouldn't trigger COS ========================================================== 16:50:57 (1789764657) SKIP: sanityn test_33e needs >= 2 MDTs SKIP 33e (4s) == sanityn test 34: no lock timeout under IO ============= 16:51:00 (1789764660) fail_loc=0x512 fail_loc=0x512 lock should not expire writing on client1 reading on client2 success fail_loc=0x511 fail_loc=0x511 lock should expire writing on client1 reading on client2 success oleg442-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9a1004e5e800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9a1004e5e800.ost_server_uuid in IDLE state after 0 sec oleg442-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9a1004e5e800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9a1004e5e800.ost_server_uuid in IDLE state after 0 sec oleg442-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9a1004e5e800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9a1004e5e800.ost_server_uuid in IDLE state after 0 sec oleg442-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9a1004e5e800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9a1004e5e800.ost_server_uuid in IDLE state after 0 sec oleg442-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9a1004e5e800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9a1004e5e800.ost_server_uuid in IDLE state after 0 sec oleg442-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9a1004e5e800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9a1004e5e800.ost_server_uuid in IDLE state after 0 sec PASS 34 (151s) == sanityn test 35: -EINTR cp_ast vs. bl_ast race does not evict client ========================================================== 16:53:32 (1789764812) fail_loc=0x80000317 Race attempt 0 Wait for 50524 50540 for 60 sec... /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 1855: 50524 Killed createmany -o $MOUNT2/$tdir/a 4000 fail_loc=0x0 Filesystem Size Used Avail Use% Mounted on 192.168.204.142@tcp:/lustre 7.2G 39M 7.2G 1% /mnt/lustre 192.168.204.142@tcp:/lustre 7.2G 39M 7.2G 1% /mnt/lustre2 - open/close 441 (time 1789764833.51 total 10.01 last 44.05) - open/close 957 (time 1789764843.52 total 20.03 last 51.51) - open/close 1463 (time 1789764853.54 total 30.05 last 50.50) - open/close 1979 (time 1789764863.55 total 40.06 last 51.54) - open/close 2391 (time 1789764873.58 total 50.08 last 41.11) - open/close 2813 (time 1789764883.60 total 60.11 last 42.09) - open/close 3153 (time 1789764893.62 total 70.12 last 33.95) - open/close 3636 (time 1789764903.63 total 80.14 last 48.24) total: 4000 open/close in 86.26 seconds: 46.37 ops/second PASS 35 (98s) == sanityn test 36: handle ESTALE/open-unlink correctly == 16:55:09 (1789764909) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg442-server mds-ost sync done. Waiting for MDT destroys to complete 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.871738 s, 60.1 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7652 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(0) *** before(7498752) after_dd(7433216) after(7498752) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.820247 s, 63.9 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7652 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(1) *** before(7498752) after_dd(7441408) after(7492608) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.824624 s, 63.6 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7652 sleep 5 for ZFS MDS Waiting for MDT destroys to complete *** cycle(2) *** before(7492608) after_dd(7435264) after(7486464) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.914243 s, 57.3 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7652 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(3) *** before(7486464) after_dd(7440384) after(7491584) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.601385 s, 87.2 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7652 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(4) *** before(7491584) after_dd(7425024) after(7498752) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.09301 s, 48.0 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7652 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(5) *** before(7498752) after_dd(7435264) after(7493632) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.833987 s, 62.9 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7652 sleep 5 for ZFS MDS Waiting for MDT destroys to complete *** cycle(6) *** before(7493632) after_dd(7440384) after(7491584) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.757451 s, 69.2 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7652 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(7) *** before(7491584) after_dd(7440384) after(7491584) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.73414 s, 71.4 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7652 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(8) *** before(7491584) after_dd(7433216) after(7498752) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.805725 s, 65.1 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7652 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(9) *** before(7498752) after_dd(7438336) after(7489536) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.788003 s, 66.5 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7652 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(10) *** before(7489536) after_dd(7438336) after(7489536) PASS 36 (318s) == sanityn test 37: check i_size is not updated for directory on close (bug 18695) ======================================================================== 17:00:27 (1789765227) multiop /mnt/lustre/d37.sanityn vD_c TMPPIPE=/tmp/multiop_open_wait_pipe.7652 - create 1018 (time 1789765241.92 total 10.01 last 101.74) - create 1975 (time 1789765251.93 total 20.01 last 95.69) - create 2888 (time 1789765261.93 total 30.01 last 91.27) - create 3917 (time 1789765271.93 total 40.01 last 102.89) - create 4904 (time 1789765281.93 total 50.01 last 98.68) - create 6006 (time 1789765291.94 total 60.02 last 110.15) - create 7206 (time 1789765301.94 total 70.02 last 119.94) - create 8350 (time 1789765311.95 total 80.03 last 114.29) - create 9616 (time 1789765321.97 total 90.05 last 126.39) total: 10000 create in 92.88 seconds: 107.67 ops/second PASS 37 (105s) == sanityn test 39a: file mtime does not change after rename ========================================================== 17:02:12 (1789765332) repeat after cancel_lru_locks PASS 39a (8s) == sanityn test 39b: file mtime the same on clients with/out lock ========================================================== 17:02:20 (1789765340) repeat after cancel_lru_locks PASS 39b (7s) == sanityn test 39c: check truncate mtime update ================================================================================ 17:02:28 (1789765348) repeat after cancel_lru_locks PASS 39c (6s) == sanityn test 39d: sync write should update mtime ====== 17:02:35 (1789765355) fail_loc=0x411 fail_loc=0 PASS 39d (6s) SKIP: sanityn test_40a skipping ALWAYS excluded test 40a == sanityn test 40b: pdirops: open|create and others ======================================================================== 17:02:42 (1789765362) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40b (16s) == sanityn test 40c: pdirops: link and others ======================================================================== 17:02:58 (1789765378) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40c (16s) == sanityn test 40d: pdirops: unlink and others ======================================================================== 17:03:14 (1789765394) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40d (17s) == sanityn test 40e: pdirops: rename and others ======================================================================== 17:03:32 (1789765412) No conflict No conflict No conflict No conflict No conflict No conflict PASS 40e (19s) == sanityn test 41a: pdirops: create vs mkdir ======================================================================== 17:03:51 (1789765431) mkdir: cannot create directory '/mnt/lustre2/f41a.sanityn': File exists Conflict PASS 41a (16s) == sanityn test 41b: pdirops: create vs create ======================================================================== 17:04:07 (1789765447) open: File exists Conflict PASS 41b (15s) == sanityn test 41c: pdirops: create vs link ======================================================================== 17:04:22 (1789765462) link: cannot create link '/mnt/lustre2/f41c.sanityn' to '/mnt/lustre2/f41c.sanityn-2': File exists Conflict PASS 41c (14s) == sanityn test 41d: pdirops: create vs unlink ======================================================================== 17:04:36 (1789765476) Conflict PASS 41d (15s) == sanityn test 41e: pdirops: create and rename (tgt) ======================================================================== 17:04:52 (1789765492) Conflict PASS 41e (20s) == sanityn test 41f: pdirops: create and rename (src) ======================================================================== 17:05:12 (1789765512) Conflict PASS 41f (16s) == sanityn test 41g: pdirops: create vs getattr ======================================================================== 17:05:28 (1789765528) Conflict PASS 41g (18s) == sanityn test 41h: pdirops: create vs readdir ======================================================================== 17:05:47 (1789765547) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 2427: lctl set_param -n fail_loc=0 2>/dev/null || true: No such file or directory Conflict PASS 41h (17s) == sanityn test 41i: reint_open: create vs create ======== 17:06:03 (1789765563) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 41i (1128s) == sanityn test 42a: pdirops: mkdir vs mkdir ======================================================================== 17:24:52 (1789766692) mkdir: cannot create directory '/mnt/lustre2/f42a.sanityn': File exists Conflict PASS 42a (16s) == sanityn test 42b: pdirops: mkdir vs create ======================================================================== 17:25:08 (1789766708) open: File exists Conflict PASS 42b (17s) == sanityn test 42c: pdirops: mkdir vs link ======================================================================== 17:25:25 (1789766725) link: cannot create link '/mnt/lustre2/f42c.sanityn' to '/mnt/lustre2/f42c.sanityn-2': File exists Conflict PASS 42c (17s) == sanityn test 42d: pdirops: mkdir vs unlink ======================================================================== 17:25:42 (1789766742) Conflict PASS 42d (14s) == sanityn test 42e: pdirops: mkdir and rename (tgt) ======================================================================== 17:25:56 (1789766756) mv: cannot overwrite directory '/mnt/lustre2/f42e.sanityn' with non-directory Conflict PASS 42e (14s) == sanityn test 42f: pdirops: mkdir and rename (src) ======================================================================== 17:26:10 (1789766770) Conflict PASS 42f (14s) == sanityn test 42g: pdirops: mkdir vs getattr ======================================================================== 17:26:24 (1789766784) Conflict PASS 42g (15s) == sanityn test 42h: pdirops: mkdir vs readdir ======================================================================== 17:26:39 (1789766799) Conflict PASS 42h (15s) == sanityn test 43a: rmdir,mkdir doesn't return -EEXIST ======================================================================== 17:26:54 (1789766814) PASS 43a (80s) == sanityn test 43b: pdirops: unlink vs create ======================================================================== 17:28:14 (1789766894) Conflict PASS 43b (14s) == sanityn test 43c: pdirops: unlink vs link ======================================================================== 17:28:28 (1789766908) Conflict PASS 43c (14s) == sanityn test 43d: pdirops: unlink vs unlink ======================================================================== 17:28:42 (1789766922) rm: cannot remove '/mnt/lustre2/f43d.sanityn': No such file or directory Conflict PASS 43d (12s) == sanityn test 43e: pdirops: unlink and rename (tgt) ======================================================================== 17:28:54 (1789766934) Conflict PASS 43e (12s) == sanityn test 43f: pdirops: unlink and rename (src) ======================================================================== 17:29:06 (1789766946) mv: cannot stat '/mnt/lustre2/f43f.sanityn': No such file or directory Conflict PASS 43f (13s) == sanityn test 43g: pdirops: unlink vs getattr ======================================================================== 17:29:19 (1789766959) stat: cannot statx '/mnt/lustre2/f43g.sanityn': No such file or directory Conflict PASS 43g (13s) == sanityn test 43h: pdirops: unlink vs readdir ======================================================================== 17:29:32 (1789766972) Conflict PASS 43h (13s) == sanityn test 43i: pdirops: unlink vs remote mkdir ===== 17:29:45 (1789766985) SKIP: sanityn test_43i needs >= 2 MDTs SKIP 43i (3s) == sanityn test 43j: racy mkdir return EEXIST ======================================================================== 17:29:48 (1789766988) mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists PASS 43j (141s) == sanityn test 43k: unlink vs create ==================== 17:32:10 (1789767130) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 43k (1327s) == sanityn test 44a: pdirops: rename tgt vs mkdir ======================================================================== 17:54:17 (1789768457) mkdir: cannot create directory '/mnt/lustre2/f44a.sanityn': File exists Conflict Fri Sep 18 17:54:29 EDT 2026 PASS 44a (19s) == sanityn test 44b: pdirops: rename tgt vs create ======================================================================== 17:54:37 (1789768477) open: File exists Conflict PASS 44b (15s) == sanityn test 44c: pdirops: rename tgt vs link ======================================================================== 17:54:53 (1789768493) link: cannot create link '/mnt/lustre2/f44c.sanityn' to '/mnt/lustre2/f44c.sanityn-3': File exists Conflict PASS 44c (15s) == sanityn test 44d: pdirops: rename tgt vs unlink ======================================================================== 17:55:08 (1789768508) Conflict PASS 44d (14s) == sanityn test 44e: pdirops: rename tgt and rename (tgt) ======================================================================== 17:55:22 (1789768522) Conflict PASS 44e (14s) == sanityn test 44f: pdirops: rename tgt and rename (src) ======================================================================== 17:55:37 (1789768537) Conflict PASS 44f (14s) == sanityn test 44g: pdirops: rename tgt vs getattr ======================================================================== 17:55:50 (1789768550) Conflict PASS 44g (13s) == sanityn test 44h: pdirops: rename tgt vs readdir ======================================================================== 17:56:03 (1789768563) Conflict PASS 44h (13s) == sanityn test 44i: pdirops: rename tgt vs remote mkdir ========================================================== 17:56:16 (1789768576) SKIP: sanityn test_44i needs >= 2 MDTs SKIP 44i (2s) == sanityn test 45a: rename,mkdir doesn't return -EEXIST ======================================================================== 17:56:18 (1789768578) PASS 45a (119s) == sanityn test 45b: pdirops: rename src vs create ======================================================================== 17:58:17 (1789768697) Conflict PASS 45b (14s) == sanityn test 45c: pdirops: rename src vs link ======================================================================== 17:58:31 (1789768711) Conflict PASS 45c (13s) == sanityn test 45d: pdirops: rename src vs unlink ======================================================================== 17:58:44 (1789768724) rm: cannot remove '/mnt/lustre2/f45d.sanityn': No such file or directory Conflict PASS 45d (14s) == sanityn test 45e: pdirops: rename src and rename (tgt) ======================================================================== 17:58:58 (1789768738) Conflict PASS 45e (13s) == sanityn test 45f: pdirops: rename src and rename (src) ======================================================================== 17:59:11 (1789768751) mv: cannot stat '/mnt/lustre2/f45f.sanityn': No such file or directory Conflict PASS 45f (12s) == sanityn test 45g: pdirops: rename src vs getattr ======================================================================== 17:59:24 (1789768764) stat: cannot statx '/mnt/lustre2/f45g.sanityn': No such file or directory Conflict PASS 45g (12s) == sanityn test 45h: pdirops: unlink vs readdir ======================================================================== 17:59:36 (1789768776) Conflict PASS 45h (11s) == sanityn test 45i: pdirops: rename src vs remote mkdir ========================================================== 17:59:47 (1789768787) SKIP: sanityn test_45i needs >= 2 MDTs SKIP 45i (3s) == sanityn test 45j: read vs rename ====================== 17:59:50 (1789768790) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 45j (836s) == sanityn test 46a: pdirops: link vs mkdir ======================================================================== 18:13:46 (1789769626) mkdir: cannot create directory '/mnt/lustre2/f46a.sanityn': File exists Conflict PASS 46a (10s) == sanityn test 46b: pdirops: link vs create ======================================================================== 18:13:56 (1789769636) open: File exists Conflict PASS 46b (11s) == sanityn test 46c: pdirops: link vs link ======================================================================== 18:14:07 (1789769647) link: cannot create link '/mnt/lustre2/f46c.sanityn' to '/mnt/lustre2/f46c.sanityn': File exists Conflict PASS 46c (10s) == sanityn test 46d: pdirops: link vs unlink ======================================================================== 18:14:17 (1789769657) Conflict PASS 46d (11s) == sanityn test 46e: pdirops: link and rename (tgt) ======================================================================== 18:14:28 (1789769668) Conflict PASS 46e (12s) == sanityn test 46f: pdirops: link and rename (src) ======================================================================== 18:14:40 (1789769680) Conflict PASS 46f (11s) == sanityn test 46g: pdirops: link vs getattr ======================================================================== 18:14:51 (1789769691) Conflict PASS 46g (9s) == sanityn test 46h: pdirops: link vs readdir ======================================================================== 18:15:00 (1789769700) Conflict PASS 46h (11s) == sanityn test 46i: pdirops: link vs remote mkdir ======= 18:15:11 (1789769711) SKIP: sanityn test_46i needs >= 2 MDTs SKIP 46i (2s) == sanityn test 47a: pdirops: remote mkdir vs mkdir ====== 18:15:14 (1789769714) SKIP: sanityn test_47a needs >= 2 MDTs SKIP 47a (2s) == sanityn test 47b: pdirops: remote mkdir vs create ===== 18:15:16 (1789769716) SKIP: sanityn test_47b needs >= 2 MDTs SKIP 47b (3s) == sanityn test 47c: pdirops: remote mkdir vs link ======= 18:15:19 (1789769719) SKIP: sanityn test_47c needs >= 2 MDTs SKIP 47c (3s) == sanityn test 47d: pdirops: remote mkdir vs unlink ===== 18:15:22 (1789769722) SKIP: sanityn test_47d needs >= 2 MDTs SKIP 47d (3s) == sanityn test 47e: pdirops: remote mkdir and rename (tgt) ========================================================== 18:15:25 (1789769725) SKIP: sanityn test_47e needs >= 2 MDTs SKIP 47e (2s) == sanityn test 47f: pdirops: remote mkdir and rename (src) ========================================================== 18:15:28 (1789769728) SKIP: sanityn test_47f needs >= 2 MDTs SKIP 47f (2s) == sanityn test 47g: pdirops: remote mkdir vs getattr ==== 18:15:30 (1789769730) SKIP: sanityn test_47g needs >= 2 MDTs SKIP 47g (2s) == sanityn test 50: osc lvb attrs: enqueue vs. CP AST ======================================================================== 18:15:33 (1789769733) 10+0 records in 10+0 records out 10240 bytes (10 kB, 10 KiB) copied, 0.0883746 s, 116 kB/s fail_loc=0x410 fail_loc=0x0 PASS 50 (9s) == sanityn test 51a: layout lock: refresh layout should work ========================================================== 18:15:42 (1789769742) 0+1 records in 0+1 records out 158 bytes copied, 0.0242591 s, 6.5 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 3690: kill: (218632) - No such process PASS 51a (7s) == sanityn test 51b: layout lock: glimpse should be able to restart if layout changed ========================================================== 18:15:49 (1789769749) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.015127 s, 67.7 kB/s fail_loc=0x1404 fail_val=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0463603 s, 22.6 MB/s 3145728 PASS 51b (17s) == sanityn test 51c: layout lock: IT_LAYOUT blocked and correct layout can be returned ========================================================== 18:16:06 (1789769766) fail_loc=0x172 Setting layout to have 2 stripes ... 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0223165 s, 45.9 kB/s PASS 51c (9s) == sanityn test 51d: layout lock: losing layout lock should clean up memory map region ========================================================== 18:16:16 (1789769776) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0245433 s, 42.7 MB/s Before revoking layout lock: 1024 KB mapped PASS 51d (6s) == sanityn test 51e: lfs getstripe does not break leases, part 2 ========================================================== 18:16:21 (1789769781) /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 2453 0x995 0x240000400 write lease(2) has applied. /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 2453 0x995 0x240000400 read lease(1) has applied. PASS 51e (7s) == sanityn test 54: rename locking ======================= 18:16:28 (1789769788) llite.lustre-ffff9a1004e5e800.xattr_cache=1 llite.lustre-ffff9a100aa27800.xattr_cache=1 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505015 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-18 18:16:29.000000000 -0400 Modify: 2026-09-18 18:16:29.000000000 -0400 Change: 2026-09-18 18:16:29.000000000 -0400 Birth: 2026-09-18 18:16:29.000000000 -0400 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000154 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505018 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-18 18:16:35.000000000 -0400 Modify: 2026-09-18 18:16:36.000000000 -0400 Change: 2026-09-18 18:16:36.000000000 -0400 Birth: 2026-09-18 18:16:35.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505021 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-18 18:16:41.000000000 -0400 Modify: 2026-09-18 18:16:41.000000000 -0400 Change: 2026-09-18 18:16:41.000000000 -0400 Birth: 2026-09-18 18:16:41.000000000 -0400 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505020 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-18 18:16:41.000000000 -0400 Modify: 2026-09-18 18:16:42.000000000 -0400 Change: 2026-09-18 18:16:42.000000000 -0400 Birth: 2026-09-18 18:16:41.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000154 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505024 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-18 18:16:47.000000000 -0400 Modify: 2026-09-18 18:16:48.000000000 -0400 Change: 2026-09-18 18:16:48.000000000 -0400 Birth: 2026-09-18 18:16:47.000000000 -0400 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505023 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-18 18:16:47.000000000 -0400 Modify: 2026-09-18 18:16:48.000000000 -0400 Change: 2026-09-18 18:16:48.000000000 -0400 Birth: 2026-09-18 18:16:47.000000000 -0400 PASS 54 (28s) == sanityn test 55a: rename vs unlink target dir ========= 18:16:56 (1789769816) fail_loc=0x80000156 mv: cannot move '/mnt/lustre/d55a.sanityn/d1/d2' to '/mnt/lustre/d55a.sanityn/d3/d2': No such file or directory PASS 55a (10s) == sanityn test 55b: rename vs unlink source dir ========= 18:17:06 (1789769826) fail_loc=0x80000156 mv: cannot stat '/mnt/lustre/d55b.sanityn/d1/d2': No such file or directory PASS 55b (11s) == sanityn test 55c: rename vs unlink orphan target dir == 18:17:17 (1789769837) fail_loc=0x156 mv: cannot move '/mnt/lustre/d55c.sanityn/d1/d2' to '/mnt/lustre/d55c.sanityn/d3/d2': No such file or directory PASS 55c (15s) == sanityn test 55d: rename file vs link ================= 18:17:32 (1789769852) fail_loc=0x155 mv: '/mnt/lustre/d55d.sanityn/f1' and '/mnt/lustre/d55d.sanityn/d55d.sanityn/f1' are the same file PASS 55d (12s) == sanityn test 55e: rename race AB/BA under the same parent dir ========================================================== 18:17:44 (1789769864) SKIP: sanityn test_55e needs >= 2 MDTs SKIP 55e (2s) == sanityn test 55f: rename: (P1/A -> P2/B) race with (P2/B -> P1/A) ========================================================== 18:17:46 (1789769866) fail_loc=0x155 We survived after P1A_P2B/P2B_P1A race test fail_loc=0x0 PASS 55f (17s) == sanityn test 55g: rename: race with trylock =========== 18:18:03 (1789769883) mdt.lustre-MDT0000.enable_rename_trylock=1 fail_loc=0x155 mv: cannot move '/mnt/lustre2/d55g.sanityn/b1/b2' to a subdirectory of itself, '/mnt/lustre2/d55g.sanityn/a1/a2/a3/a4/b2' We survived race with trylock test fail_loc=0x0 PASS 55g (20s) == sanityn test 56a: test llverdev with single large stripe ========================================================== 18:18:23 (1789769903) run_llverdev /mnt/lustre/f56a.sanityn -p -s 16000000 -c 4k llverdev: /mnt/lustre/f56a.sanityn is 16777216000000 bytes (15625 GB) in size timestamp: 1789769904 chunksize: 4096 size: 16777216000000 write offset: 16384000000kB 0.726656 MB/s read offset: 16384000000kB inf MB/s llverdev: data verified successfully PASS 56a (89s) == sanityn test 56b: test llverdev and partial verify of wide stripe file ========================================================== 18:19:52 (1789769992) run_llverdev /mnt/lustre/f56b.sanityn -p -s 32000000 -c 4k llverdev: /mnt/lustre/f56b.sanityn is 33554432000000 bytes (31250 GB) in size timestamp: 1789769994 chunksize: 4096 size: 33554432000000 write offset: 32768000000kB 2.14165 MB/s read offset: 32768000000kB inf MB/s llverdev: data verified successfully PASS 56b (65s) == sanityn test 60: Verify data_version behaviour ======== 18:20:57 (1789770057) PASS 60 (5s) == sanityn test 70a: cd directory && rm directory ======== 18:21:02 (1789770062) PASS 70a (4s) == sanityn test 70b: remove files after calling rm_entry ========================================================== 18:21:06 (1789770066) PASS 70b (6s) == sanityn test 71a: correct file map just after write operation is finished ========================================================== 18:21:12 (1789770072) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.00472533 s, 13.9 MB/s SKIP: sanityn test_71a ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71a (2s) == sanityn test 71b: check fiemap support for stripecount > 1 ========================================================== 18:21:14 (1789770074) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.00918368 s, 7.1 MB/s SKIP: sanityn test_71b ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71b (2s) == sanityn test 71c: check FIEMAP_EXTENT_LAST flag with different extents number ========================================================== 18:21:16 (1789770076) SKIP: sanityn test_71c support only ldiskfs ost SKIP 71c (3s) == sanityn test 71d: fiemap corruption test with fm_extent_count=0 ========================================================== 18:21:19 (1789770079) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0145578 s, 4.5 MB/s SKIP: sanityn test_71d ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71d (2s) == sanityn test 72: getxattr/setxattr cache should be consistent between nodes ========================================================== 18:21:21 (1789770081) llite.lustre-ffff9a1004e5e800.xattr_cache=1 llite.lustre-ffff9a100aa27800.xattr_cache=1 getfattr: Removing leading '/' from absolute path names user.attr1="value1" getfattr: Removing leading '/' from absolute path names user.attr1="value2" getfattr: Removing leading '/' from absolute path names getfattr: Removing leading '/' from absolute path names b362740ad371eae580e7f85452e8830b - 870e072da1ebfc20ea8c0cd3d6bdf3b6 - PASS 72 (4s) == sanityn test 73: getxattr should not cause xattr lock cancellation ========================================================== 18:21:25 (1789770085) llite.lustre-ffff9a1004e5e800.xattr_cache=1 llite.lustre-ffff9a100aa27800.xattr_cache=1 getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" PASS 73 (5s) == sanityn test 74: flock deadlock: different mounts ======================================================================== 18:21:30 (1789770090) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00966753 s, 106 kB/s 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00999224 s, 102 kB/s 231418: get lock2 231419: get lock1 231418: done 231418: get lock1 231419: done 231419: get lock2 fcntl cmd 7 failed: Resource deadlock avoided 231419: cannot get lock2: Resource deadlock avoided 231419: exit rc=0 231418: done 231418: exit rc=0 PASS 74 (8s) == sanityn test 75: osc: upcall after unuse lock============================================================================= 18:21:38 (1789770098) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0237799 s, 88.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168055 s, 62.4 MB/s fail_loc=0x31d /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 4373: 231999 Killed stat -c %s $DIR1/$tfile fail_loc=0 PASS 75 (9s) == sanityn test 76: Verify MDT open_files listing ======== 18:21:47 (1789770107) opening files: ........ closing files: ........ PASS 76 (36s) == sanityn test 77a: check FIFO NRS policy =============== 18:22:23 (1789770143) ost.OSS.ost.nrs_policies=fifo ost.OSS.ost_create.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_out.nrs_policies=fifo ost.OSS.ost_seq.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.133065 s, 126 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0230462 s, 45.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0436666 s, 24.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0564647 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0779998 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0748277 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0542315 s, 19.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.058245 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0565902 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0500291 s, 21.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0492047 s, 21.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0503747 s, 20.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0551459 s, 19.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0420384 s, 24.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0711694 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0549426 s, 19.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9a1004e5e800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9a1004e5e800.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0496703 s, 21.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0313927 s, 33.4 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183403 s, 57.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0436508 s, 24.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0231883 s, 45.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0233865 s, 44.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0411357 s, 25.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0255526 s, 41.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0261148 s, 40.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0393559 s, 26.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0285749 s, 36.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0296006 s, 35.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0294456 s, 35.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0405379 s, 25.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0294313 s, 35.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00765889 s, 137 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0331572 s, 31.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9a1004e5e800.lock_unused_count=1 removed '/mnt/lustre/d77a.sanityn/nrs_r_oleg442-client.virtnet' removed '/mnt/lustre/d77a.sanityn/nrs_w_oleg442-client.virtnet' removed directory '/mnt/lustre/d77a.sanityn' PASS 77a (5s) == sanityn test 77b: check CRR-N NRS policy ============== 18:22:29 (1789770149) ost.OSS.ost.nrs_policies=crrn ost.OSS.ost_create.nrs_policies=crrn ost.OSS.ost_io.nrs_policies=crrn ost.OSS.ost_out.nrs_policies=crrn ost.OSS.ost_seq.nrs_policies=crrn ost.OSS.ost.nrs_crrn_quantum=1 ost.OSS.ost_create.nrs_crrn_quantum=1 ost.OSS.ost_io.nrs_crrn_quantum=1 ost.OSS.ost_out.nrs_crrn_quantum=1 ost.OSS.ost_seq.nrs_crrn_quantum=1 policy: crr-n, crrn_quantum 1 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.122128 s, 137 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0197537 s, 53.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0342333 s, 30.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0432313 s, 24.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0231994 s, 45.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0188192 s, 55.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0585745 s, 17.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0553675 s, 18.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0708043 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0693584 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0434433 s, 24.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.033017 s, 31.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0881821 s, 11.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0549011 s, 19.1 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0663996 s, 15.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0589807 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0575683 s, 18.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9a1004e5e800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9a1004e5e800.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00406646 s, 258 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0178628 s, 58.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00705143 s, 149 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0444987 s, 23.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00421381 s, 249 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00390607 s, 268 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0405496 s, 25.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0407767 s, 25.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0412546 s, 25.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0443781 s, 23.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.011958 s, 87.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0429034 s, 24.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0462743 s, 22.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0446528 s, 23.5 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0437434 s, 24.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0370448 s, 28.3 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg442-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg442-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost.nrs_crrn_quantum=64 ost.OSS.ost_create.nrs_crrn_quantum=64 ost.OSS.ost_io.nrs_crrn_quantum=64 ost.OSS.ost_out.nrs_crrn_quantum=64 ost.OSS.ost_seq.nrs_crrn_quantum=64 policy: crr-n, crrn_quantum 64 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.131311 s, 128 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0320194 s, 32.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0456735 s, 23.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.025071 s, 41.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0291442 s, 36.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0417424 s, 25.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0244154 s, 42.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0314239 s, 33.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0440458 s, 23.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0319352 s, 32.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0694096 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130539 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115494 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103467 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130707 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144372 s, 7.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9a1004e5e800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9a1004e5e800.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129112 s, 8.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0110262 s, 95.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0211768 s, 49.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0541087 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0575911 s, 18.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0561502 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0361414 s, 29.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0418464 s, 25.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0397101 s, 26.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0564425 s, 18.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0669897 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0338183 s, 31.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9a1004e5e800.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0470754 s, 22.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0459577 s, 22.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00463556 s, 226 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.050475 s, 20.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0577105 s, 18.2 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg442-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg442-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77b (7s) == sanityn test 77c: check ORR NRS policy ================ 18:22:36 (1789770156) ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_orr_quantum=1 ost.OSS.ost_io.nrs_orr_offset_type=physical ost.OSS.ost_io.nrs_orr_supported=reads policy: orr, orr_quantum 1, orr_offset_type physical, orr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.13557 s, 124 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0279969 s, 37.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0202691 s, 51.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0370031 s, 28.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0328953 s, 31.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0668974 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0431689 s, 24.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0769798 s, 13.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0901411 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0831412 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0952252 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.074074 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0884938 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0985276 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0774769 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0979338 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105246 s, 10.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9a1004e5e800.lock_unused_count=5 ldlm.namespaces.lustre-OST0001-osc-ffff9a1004e5e800.lock_unused_count=7 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0395967 s, 26.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0428953 s, 24.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0362787 s, 28.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0314696 s, 33.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0363972 s, 28.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0377766 s, 27.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0322263 s, 32.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0480898 s, 21.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.036896 s, 28.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.027636 s, 37.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0223195 s, 47.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0348158 s, 30.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0319072 s, 32.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0328675 s, 31.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0315345 s, 33.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0320917 s, 32.7 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg442-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg442-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=writes ost.OSS.ost_io.nrs_orr_quantum=64 policy: orr, orr_quantum 64, orr_offset_type physical, orr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.16669 s, 101 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0355915 s, 29.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0492927 s, 21.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0819119 s, 12.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0836121 s, 12.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0990836 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135993 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124798 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126805 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113292 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140244 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141769 s, 7.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136695 s, 7.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144273 s, 7.3 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127595 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15858 s, 6.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9a1004e5e800.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff9a1004e5e800.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151231 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.036526 s, 28.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0351652 s, 29.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0220344 s, 47.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0271512 s, 38.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0298694 s, 35.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0471347 s, 22.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.035237 s, 29.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0467245 s, 22.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0483003 s, 21.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0494551 s, 21.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0420644 s, 24.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0433098 s, 24.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0566638 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0451423 s, 23.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0463447 s, 22.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0500772 s, 20.9 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg442-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg442-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=reads_and_writes ost.OSS.ost_io.nrs_orr_offset_type=logical policy: orr, orr_quantum 64, orr_offset_type logical, orr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.131109 s, 128 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0226312 s, 46.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0341809 s, 30.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158192 s, 66.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0242398 s, 43.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0115993 s, 90.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0326045 s, 32.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129056 s, 81.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0304121 s, 34.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0259183 s, 40.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9a1004e5e800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0634965 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0443336 s, 23.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0976575 s, 10.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0859472 s, 12.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0662933 s, 15.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100836 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124247 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00920246 s, 114 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014644 s, 71.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0311548 s, 33.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0210702 s, 49.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0261827 s, 40.0 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.026848 s, 39.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0275413 s, 38.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0383851 s, 27.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0268728 s, 39.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0300287 s, 34.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0615993 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0434562 s, 24.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0455887 s, 23.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0359165 s, 29.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0631092 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0676539 s, 15.5 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg442-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg442-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77c (8s) == sanityn test 77d: check TRR nrs policy ================ 18:22:44 (1789770164) ost.OSS.ost_io.nrs_policies=trr ost.OSS.ost_io.nrs_trr_quantum=1 ost.OSS.ost_io.nrs_trr_offset_type=physical ost.OSS.ost_io.nrs_trr_supported=reads policy: trr, trr_quantum 1, trr_offset_type physical, trr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.113116 s, 148 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146781 s, 71.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0238322 s, 44.0 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0208839 s, 50.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141423 s, 74.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0317213 s, 33.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0932756 s, 11.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100851 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0681726 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0730614 s, 14.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0929478 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113492 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0832065 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115089 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112697 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122294 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119413 s, 8.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0316852 s, 33.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0332141 s, 31.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0333109 s, 31.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0176828 s, 59.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00758328 s, 138 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0348626 s, 30.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0443388 s, 23.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0037927 s, 276 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00381713 s, 275 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0478828 s, 21.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0592992 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0502526 s, 20.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0507512 s, 20.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0507678 s, 20.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0406587 s, 25.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0123098 s, 85.2 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg442-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg442-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=writes ost.OSS.ost_io.nrs_trr_quantum=64 policy: trr, trr_quantum 64, trr_offset_type physical, trr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.11683 s, 144 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.027171 s, 38.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173585 s, 60.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0494004 s, 21.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.02476 s, 42.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0461837 s, 22.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0663354 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0492457 s, 21.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0870678 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110006 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0839231 s, 12.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0890214 s, 11.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9a1004e5e800.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115853 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0781027 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110104 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115954 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139557 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0283851 s, 36.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00459312 s, 228 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0225005 s, 46.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0229471 s, 45.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0326267 s, 32.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00446289 s, 235 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00523677 s, 200 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0322612 s, 32.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00658815 s, 159 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0506306 s, 20.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0560389 s, 18.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00729828 s, 144 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0354609 s, 29.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.040501 s, 25.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0660619 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0422887 s, 24.8 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg442-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg442-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=reads_and_writes ost.OSS.ost_io.nrs_trr_offset_type=logical policy: trr, trr_quantum 64, trr_offset_type logical, trr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.180169 s, 93.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0263227 s, 39.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0174049 s, 60.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0417978 s, 25.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0722053 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0743453 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0652997 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0581392 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0478197 s, 21.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0848811 s, 12.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0607357 s, 17.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0853585 s, 12.3 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0701967 s, 14.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107683 s, 9.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.084302 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100059 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121364 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0188658 s, 55.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.028116 s, 37.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.04433 s, 23.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0329567 s, 31.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.023454 s, 44.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.04264 s, 24.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0527193 s, 19.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0576655 s, 18.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0443135 s, 23.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0482234 s, 21.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0575856 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0435598 s, 24.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0445693 s, 23.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0489467 s, 21.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0454262 s, 23.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0488967 s, 21.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9a1004e5e800.lock_unused_count=4 removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg442-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg442-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77d (8s) == sanityn test 77e: check TBF NID nrs policy ============ 18:22:52 (1789770172) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_tbf_rule=start localhost nid={0@lo} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start clients nid={192.168.204.42@tcp} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start others nid={*.*.*.*@tcp} rate=50 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.154034 s, 109 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0203098 s, 51.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0382392 s, 27.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0400822 s, 26.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0363857 s, 28.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0583612 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0694709 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0404868 s, 25.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0507277 s, 20.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0728062 s, 14.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0191216 s, 54.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0657436 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0385549 s, 27.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0638467 s, 16.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0821678 s, 12.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0748239 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0772233 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.028477 s, 36.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.010262 s, 102 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00628736 s, 167 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0242965 s, 43.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0231381 s, 45.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0246653 s, 42.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0393826 s, 26.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0324972 s, 32.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0263002 s, 39.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0049181 s, 213 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0370148 s, 28.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00476724 s, 220 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00651445 s, 161 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0616469 s, 17.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173185 s, 60.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0370768 s, 28.3 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg442-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg442-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change localhost rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change clients rate=101 ost.OSS.ost_io.nrs_tbf_rule=change others rate=51 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.0935647 s, 179 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0245312 s, 42.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125071 s, 83.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0322892 s, 32.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0707242 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0837231 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0645048 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.084894 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0885293 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0889716 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0994339 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0795229 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105142 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0914262 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102736 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0969137 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117239 s, 8.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0319435 s, 32.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.026014 s, 40.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0260392 s, 40.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0231551 s, 45.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0442521 s, 23.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.03289 s, 31.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.04498 s, 23.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0398388 s, 26.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.024736 s, 42.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0363751 s, 28.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0507151 s, 20.7 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0521888 s, 20.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0440293 s, 23.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0330518 s, 31.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0363151 s, 28.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0413344 s, 25.4 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg442-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg442-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop localhost ost.OSS.ost_io.nrs_tbf_rule=stop clients ost.OSS.ost_io.nrs_tbf_rule=stop others 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.131827 s, 127 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0257048 s, 40.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0405939 s, 25.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0473006 s, 22.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0655785 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.061233 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0448104 s, 23.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.066915 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0653776 s, 16.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9a1004e5e800.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0545306 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0885951 s, 11.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0762387 s, 13.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0866152 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10734 s, 9.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117613 s, 8.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110416 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114386 s, 9.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0304471 s, 34.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0204394 s, 51.3 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0287925 s, 36.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0221605 s, 47.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0175548 s, 59.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170623 s, 61.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0196815 s, 53.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0284912 s, 36.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0322528 s, 32.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0345692 s, 30.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0350907 s, 29.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0361671 s, 29.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0425721 s, 24.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0363074 s, 28.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0306633 s, 34.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0370725 s, 28.3 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg442-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg442-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.10602 s, 158 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0212686 s, 49.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0241323 s, 43.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0519483 s, 20.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0519149 s, 20.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0393223 s, 26.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0481872 s, 21.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0652124 s, 16.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0643807 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0496573 s, 21.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0423552 s, 24.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0373827 s, 28.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0650373 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.051097 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0614468 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0872469 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101877 s, 10.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9a1004e5e800.lock_unused_count=3 ldlm.namespaces.lustre-OST0001-osc-ffff9a1004e5e800.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0299495 s, 35.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0181755 s, 57.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0369334 s, 28.4 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0321826 s, 32.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0202801 s, 51.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0392117 s, 26.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.034084 s, 30.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0314942 s, 33.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0348737 s, 30.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0373605 s, 28.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0316985 s, 33.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.033989 s, 30.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0386302 s, 27.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0462925 s, 22.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0439706 s, 23.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0412584 s, 25.4 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg442-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg442-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' PASS 77e (12s) == sanityn test 77f: check TBF JobID nrs policy ========== 18:23:04 (1789770184) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start runas jobid={iozone.500 dd.500 tiotest.500} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start iozone_runas jobid={iozone.500} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg442-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.11441 s, 147 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00860512 s, 122 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0285933 s, 36.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0277477 s, 37.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0706902 s, 14.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.062415 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0787029 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.07362 s, 14.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0605049 s, 17.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0893962 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0867254 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102062 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0825894 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109298 s, 9.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102614 s, 10.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0774588 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118036 s, 8.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.030854 s, 34.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0216104 s, 48.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0179021 s, 58.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0315247 s, 33.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0196103 s, 53.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0258578 s, 40.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0217324 s, 48.2 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0213376 s, 49.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0224889 s, 46.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0226398 s, 46.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0411497 s, 25.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0307371 s, 34.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0274824 s, 38.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0272185 s, 38.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.027625 s, 38.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0364023 s, 28.8 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg442-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg442-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change runas rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change iozone_runas rate=101 ost.OSS.ost_io.nrs_tbf_rule=change dd_runas rate=51 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg442-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.156366 s, 107 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152401 s, 68.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0284443 s, 36.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0284808 s, 36.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0627803 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.052773 s, 19.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0462072 s, 22.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0738022 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0744578 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0769766 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0861797 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0804833 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0704233 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0663652 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0824845 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103029 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104579 s, 10.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0253167 s, 41.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0291612 s, 36.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0306469 s, 34.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0502476 s, 20.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0417535 s, 25.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0324101 s, 32.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0391264 s, 26.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.03718 s, 28.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0382291 s, 27.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.038839 s, 27.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0397758 s, 26.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0402349 s, 26.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0443701 s, 23.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0445209 s, 23.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.043699 s, 24.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0568823 s, 18.4 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg442-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg442-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop runas ost.OSS.ost_io.nrs_tbf_rule=stop iozone_runas ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg442-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.1031 s, 163 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0240751 s, 43.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164542 s, 63.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0520426 s, 20.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0438387 s, 23.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0461585 s, 22.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0523796 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0588051 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0542755 s, 19.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0259851 s, 40.4 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9a1004e5e800.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0738039 s, 14.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0776089 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102912 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0988543 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119198 s, 8.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0948472 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116446 s, 9.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9a1004e5e800.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff9a1004e5e800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0610498 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0478404 s, 21.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0402105 s, 26.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0650366 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0478504 s, 21.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0415916 s, 25.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0510874 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0722059 s, 14.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0567314 s, 18.5 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0520721 s, 20.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0484936 s, 21.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0406913 s, 25.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0453651 s, 23.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0553035 s, 19.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0493359 s, 21.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0593508 s, 17.7 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg442-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg442-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_policies=fifo running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg442-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.118186 s, 142 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0218625 s, 48.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0284932 s, 36.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.036242 s, 28.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0534114 s, 19.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0480211 s, 21.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0415119 s, 25.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.043909 s, 23.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0650235 s, 16.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0658024 s, 15.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0622717 s, 16.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.072001 s, 14.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0516925 s, 20.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0580314 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0409907 s, 25.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0672353 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0470189 s, 22.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9a1004e5e800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9a1004e5e800.lock_unused_count=1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0248766 s, 42.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0320362 s, 32.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0277132 s, 37.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0251041 s, 41.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0355817 s, 29.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0220618 s, 47.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0233132 s, 45.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0258622 s, 40.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0279275 s, 37.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0359961 s, 29.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0354378 s, 29.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0285245 s, 36.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0281288 s, 37.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0442402 s, 23.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0322451 s, 32.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0388968 s, 27.0 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg442-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg442-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' PASS 77f (12s) == sanityn test 77g: Change TBF type directly ============ 18:23:16 (1789770196) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 ost.OSS.ost_io.nrs_policies=fifo PASS 77g (6s) == sanityn test 77h: Wrong policy name should report error, not LBUG ========================================================== 18:23:22 (1789770202) oleg442-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=abc: No such file or directory oleg442-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='abc': No such file or directory pdsh@oleg442-client: oleg442-server: ssh exited with exit code 2 oleg442-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc: Invalid argument oleg442-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc': Invalid argument pdsh@oleg442-client: oleg442-server: ssh exited with exit code 22 oleg442-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf reg abc: Invalid argument oleg442-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf reg abc': Invalid argument pdsh@oleg442-client: oleg442-server: ssh exited with exit code 22 oleg442-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc efg: Invalid argument oleg442-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc efg': Invalid argument pdsh@oleg442-client: oleg442-server: ssh exited with exit code 22 PASS 77h (6s) == sanityn test 77i: Change rank of TBF rule ============= 18:23:28 (1789770208) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start before jobid={jobid} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start after jobid={jobid} rate=1000 rank=default ost.OSS.ost_io.nrs_tbf_rule=start target jobid={jobid} rate=1000 rank=after Move before itself ost.OSS.ost_io.nrs_tbf_rule=change target rank=target Move to higher rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=before Move to lower rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=after Move before default ost.OSS.ost_io.nrs_tbf_rule=change target rank=default ost.OSS.ost_io.nrs_policies=fifo PASS 77i (12s) == sanityn test 77j: check TBF-OPCode NRS policy ========= 18:23:40 (1789770220) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf opcode ost.OSS.ost_io.nrs_tbf_rule=start ost_r opcode={ost_read} rate=5 ost.OSS.ost_io.nrs_tbf_rule=start ost_w opcode={ost_write} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.283073 s, 59.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0198996 s, 52.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0262265 s, 40.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167681 s, 62.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.062005 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.059643 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0639859 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0545834 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0619911 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0623359 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0626684 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.240707 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237214 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.287282 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.283344 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.322685 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.325159 s, 3.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0344296 s, 30.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0369443 s, 28.4 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0182016 s, 57.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00574048 s, 183 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0177814 s, 59.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0321003 s, 32.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0417841 s, 25.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0401411 s, 26.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00446909 s, 235 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0230714 s, 45.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0495906 s, 21.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0035993 s, 291 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00446797 s, 235 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0061368 s, 171 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0552339 s, 19.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00642343 s, 163 MB/s removed '/mnt/lustre/d77j.sanityn/nrs_w_oleg442-client.virtnet' removed '/mnt/lustre/d77j.sanityn/nrs_r_oleg442-client.virtnet' removed directory '/mnt/lustre/d77j.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 5, bs: 1048576 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.33572 s, 19.7 MB/s Write runtime is 7 s, speed is 14.285714 vs. 20 Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.5126 s, 5.4 MB/s Read runtime is 20 s, speed is 5.000000 vs. 5 cleanup_tbf /mnt/lustre/d77j.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_r ost.OSS.ost_io.nrs_tbf_rule=stop ost_w ost.OSS.ost_io.nrs_policies=fifo PASS 77j (58s) == sanityn test 77ja: check TBF-UID/GID NRS policy ======= 18:24:38 (1789770278) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg442-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg442-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg442-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg442-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg442-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.23184 s, 13.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0181932 s, 57.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0462123 s, 22.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0396864 s, 26.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0490054 s, 21.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0620717 s, 16.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.039548 s, 26.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0816048 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0850047 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131278 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131678 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132158 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130508 s, 8.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146242 s, 7.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150388 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170009 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13145 s, 8.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0299343 s, 35.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0419177 s, 25.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.03194 s, 32.8 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0263516 s, 39.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0436712 s, 24.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0329839 s, 31.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0343019 s, 30.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0392264 s, 26.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0373652 s, 28.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0396647 s, 26.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0499704 s, 21.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0365374 s, 28.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0338647 s, 31.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0547078 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0412005 s, 25.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0472173 s, 22.2 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg442-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg442-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5, bs: 1048576 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1048576] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.7746 s, 5.3 MB/s Write runtime is 20 s, speed is 5.000000 vs. 5 Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1048576] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9396 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 vs. 5 cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg442-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg442-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg442-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg442-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg442-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.24687 s, 13.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0461712 s, 22.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0318505 s, 32.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0550899 s, 19.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163827 s, 64.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.028965 s, 36.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0777649 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0525626 s, 19.9 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0292453 s, 35.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126975 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129544 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1448 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140471 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157697 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120825 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177105 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154241 s, 6.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0353859 s, 29.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0366499 s, 28.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0364211 s, 28.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.039693 s, 26.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0415444 s, 25.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0346655 s, 30.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0410203 s, 25.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0297022 s, 35.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0369666 s, 28.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0421321 s, 24.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0374394 s, 28.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0376016 s, 27.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0431104 s, 24.3 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0399216 s, 26.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0381214 s, 27.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0430124 s, 24.4 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg442-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg442-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5, bs: 1048576 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1048576] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.7608 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 vs. 5 Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1048576] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9508 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 vs. 5 cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77ja (132s) == sanityn test 77jb: check TBF-UID/GID NRS policy on files that don't belong to us ========================================================== 18:26:50 (1789770410) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg442-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg442-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg442-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg442-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg442-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.2425 s, 13.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0294782 s, 35.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.02792 s, 37.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0481652 s, 21.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0397662 s, 26.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0085053 s, 123 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.02787 s, 37.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0540727 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0637142 s, 16.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128934 s, 8.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128883 s, 8.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159562 s, 6.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145472 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164734 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188589 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180898 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160963 s, 6.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173216 s, 60.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0227098 s, 46.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0207872 s, 50.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0251759 s, 41.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0262267 s, 40.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0236366 s, 44.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0217779 s, 48.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0218265 s, 48.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.018546 s, 56.5 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0250348 s, 41.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0273331 s, 38.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0239223 s, 43.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.029143 s, 36.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0285031 s, 36.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.025999 s, 40.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0294096 s, 35.7 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg442-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg442-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5, bs: 1048576 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1048576] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.756 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 vs. 5 Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1048576] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9636 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 vs. 5 cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg442-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg442-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg442-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg442-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg442-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.23581 s, 13.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0307065 s, 34.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0255093 s, 41.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0361612 s, 29.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0513922 s, 20.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0343136 s, 30.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0533274 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.047091 s, 22.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0407914 s, 25.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135656 s, 7.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135583 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15654 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156786 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173128 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179607 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184977 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174961 s, 6.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0306834 s, 34.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0337101 s, 31.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0333583 s, 31.4 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0280256 s, 37.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0336368 s, 31.2 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0350313 s, 29.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0286581 s, 36.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0310959 s, 33.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0438996 s, 23.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.033075 s, 31.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0336318 s, 31.2 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0307728 s, 34.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0349457 s, 30.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0279836 s, 37.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0360384 s, 29.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0397485 s, 26.4 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg442-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg442-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5, bs: 1048576 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1048576] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.7736 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 vs. 5 Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1048576] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9114 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 vs. 5 cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77jb (139s) == sanityn test 77k: check TBF policy with UID/GID/JobID/OPCode expression ========================================================== 18:29:09 (1789770549) ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg442-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.284121 s, 59.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0182459 s, 57.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0260783 s, 40.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145739 s, 71.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0426736 s, 24.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0565556 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0482074 s, 21.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0375623 s, 27.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0607837 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0809001 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0844687 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0588064 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0642334 s, 16.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0823084 s, 12.7 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0817932 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0586126 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0707208 s, 14.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0248341 s, 42.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0265593 s, 39.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0215207 s, 48.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0188611 s, 55.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0269635 s, 38.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0310349 s, 33.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0232569 s, 45.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0248472 s, 42.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0256709 s, 40.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0255186 s, 41.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.026416 s, 39.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0277276 s, 37.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0278975 s, 37.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.025806 s, 40.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0296013 s, 35.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0378944 s, 27.7 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg442-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg442-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10, bs: 1048576 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1048576] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.01588 s, 20.9 MB/s Write runtime is 6 s, speed is 16.666666 vs. 20 Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1048576] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.77393 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 vs. 10 cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.204.42@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.204.42@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.237994 s, 70.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129647 s, 80.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148703 s, 70.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0440706 s, 23.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0487533 s, 21.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0537971 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0545417 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0454408 s, 23.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0360782 s, 29.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0449308 s, 23.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.03689 s, 28.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0472119 s, 22.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0450033 s, 23.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.248596 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.259017 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.289753 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.300245 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134559 s, 77.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0120151 s, 87.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.018612 s, 56.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162001 s, 64.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0190363 s, 55.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0176655 s, 59.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125368 s, 83.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0108721 s, 96.4 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0208314 s, 50.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0197525 s, 53.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0205488 s, 51.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0251162 s, 41.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00413415 s, 254 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00395346 s, 265 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00667016 s, 157 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00282685 s, 371 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg442-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg442-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10, bs: 1048576 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.98674 s, 21.0 MB/s Write runtime is 6 s, speed is 16.666666 vs. 20 Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.78832 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 vs. 10 cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.204.42@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg442-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.228957 s, 73.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168423 s, 62.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0382329 s, 27.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0334943 s, 31.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0413265 s, 25.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0479106 s, 21.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0424601 s, 24.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0182083 s, 57.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0457533 s, 22.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0340797 s, 30.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0354145 s, 29.6 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0295053 s, 35.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0484128 s, 21.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0498705 s, 21.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0351282 s, 29.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.073585 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0485948 s, 21.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00540183 s, 194 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0199053 s, 52.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0265093 s, 39.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.023917 s, 43.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0229355 s, 45.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.031753 s, 33.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0318664 s, 32.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0218596 s, 48.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0108016 s, 97.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0281109 s, 37.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0347934 s, 30.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0255564 s, 41.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0362006 s, 29.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0314343 s, 33.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0319221 s, 32.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00340863 s, 308 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg442-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg442-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20, bs: 1048576 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1048576] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.96648 s, 21.1 MB/s Write runtime is 6 s, speed is 16.666666 vs. 20 Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1048576] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.89307 s, 21.4 MB/s Read runtime is 6 s, speed is 16.666666 vs. 20 cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg442-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.446138 s, 37.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152871 s, 68.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0278028 s, 37.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0466296 s, 22.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0434462 s, 24.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.029645 s, 35.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0543676 s, 19.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0499573 s, 21.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0390502 s, 26.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0361581 s, 29.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0518174 s, 20.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0738701 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0783456 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.302288 s, 3.5 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.302124 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.360159 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.366464 s, 2.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.029097 s, 36.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0232787 s, 45.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0250621 s, 41.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0262554 s, 39.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0283213 s, 37.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01979 s, 53.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0268918 s, 39.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.035802 s, 29.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0395018 s, 26.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0377079 s, 27.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157724 s, 66.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147763 s, 71.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0385488 s, 27.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0108382 s, 96.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00874881 s, 120 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167077 s, 62.8 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg442-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg442-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10, bs: 1048576 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1048576] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.81718 s, 10.7 MB/s Write runtime is 11 s, speed is 9.090909 vs. 10 Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1048576] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.76305 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 vs. 10 cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10, bs: 1048576 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.9853 s, 21.0 MB/s Write runtime is 6 s, speed is 16.666666 vs. 20 Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.77756 s, 10.7 MB/s Read runtime is 10 s, speed is 10.000000 vs. 10 cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg442-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.831607 s, 20.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0518313 s, 20.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0454666 s, 23.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0437147 s, 24.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0436029 s, 24.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0577827 s, 18.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0484256 s, 21.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0402672 s, 26.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.053035 s, 19.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151398 s, 6.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153455 s, 6.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138204 s, 7.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155778 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14977 s, 7.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169629 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187951 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15332 s, 6.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0320886 s, 32.7 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0263212 s, 39.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0271679 s, 38.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0285652 s, 36.7 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0353785 s, 29.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0299837 s, 35.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0305748 s, 34.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0411649 s, 25.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0294828 s, 35.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0279571 s, 37.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0391684 s, 26.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0371924 s, 28.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0325837 s, 32.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0344553 s, 30.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0384808 s, 27.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0389326 s, 26.9 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg442-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg442-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5, bs: 1048576 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1048576] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.5457 s, 5.4 MB/s Write runtime is 20 s, speed is 5.000000 vs. 5 Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1048576] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.4616 s, 5.4 MB/s Read runtime is 21 s, speed is 4.761904 vs. 5 cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_ug ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg442-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.233408 s, 71.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141203 s, 74.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0345154 s, 30.4 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0314137 s, 33.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.039739 s, 26.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0283782 s, 37.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0542215 s, 19.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0352071 s, 29.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0253536 s, 41.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0409612 s, 25.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0459477 s, 22.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0422857 s, 24.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.04406 s, 23.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0394409 s, 26.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0398214 s, 26.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0548016 s, 19.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0520036 s, 20.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0200354 s, 52.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0124035 s, 84.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131449 s, 79.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153824 s, 68.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168915 s, 62.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0191397 s, 54.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0240417 s, 43.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0188256 s, 55.7 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0178542 s, 58.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0201301 s, 52.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0226086 s, 46.4 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0187409 s, 56.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0229021 s, 45.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.025245 s, 41.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00465167 s, 225 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0266853 s, 39.3 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg442-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg442-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10, bs: 1048576 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1048576] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.96132 s, 21.1 MB/s Write runtime is 6 s, speed is 16.666666 vs. 20 Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1048576] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.76242 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 vs. 10 cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg442-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.435923 s, 38.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0201678 s, 52.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0352739 s, 29.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0394619 s, 26.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0229725 s, 45.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0248524 s, 42.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0411546 s, 25.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0265143 s, 39.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0280554 s, 37.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0616464 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0626893 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0658914 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.074629 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.323126 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.313657 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.369577 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.358248 s, 2.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0357604 s, 29.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0300319 s, 34.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0305815 s, 34.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0342527 s, 30.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0373574 s, 28.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0394999 s, 26.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0377786 s, 27.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0399391 s, 26.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0395913 s, 26.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0409189 s, 25.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0450452 s, 23.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0448194 s, 23.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0421108 s, 24.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0561045 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0515273 s, 20.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0510043 s, 20.6 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg442-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg442-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10, bs: 1048576 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1048576] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.85293 s, 10.6 MB/s Write runtime is 11 s, speed is 9.090909 vs. 10 Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1048576] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.7478 s, 10.8 MB/s Read runtime is 10 s, speed is 10.000000 vs. 10 cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10, bs: 1048576 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1048576] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.78763 s, 10.7 MB/s Write runtime is 11 s, speed is 9.090909 vs. 20 Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1048576] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.74329 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 vs. 10 cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Cleanup TBF ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_policies=fifo Cleanup TBF ost.OSS.ost_io.nrs_policies=fifo PASS 77k (378s) == sanityn test 77kb: Check different granular TBF type combination: jobid+opcode ========================================================== 18:35:27 (1789770927) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/nrs_r_oleg442-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.255849 s, 65.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0196112 s, 53.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0241877 s, 43.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0417577 s, 25.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0498016 s, 21.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0395574 s, 26.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0363404 s, 28.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0575537 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0429392 s, 24.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0537676 s, 19.5 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0435871 s, 24.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0516709 s, 20.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0423773 s, 24.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0251783 s, 41.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0480363 s, 21.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0538963 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0588378 s, 17.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136667 s, 76.7 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0310559 s, 33.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0182915 s, 57.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00571199 s, 184 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0235496 s, 44.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0367384 s, 28.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0373178 s, 28.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0228714 s, 45.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0291104 s, 36.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0444228 s, 23.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0250435 s, 41.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0328182 s, 32.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0344102 s, 30.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0343003 s, 30.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0359372 s, 29.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0354121 s, 29.6 MB/s removed '/mnt/lustre/d77kb.sanityn/nrs_w_oleg442-client.virtnet' removed '/mnt/lustre/d77kb.sanityn/nrs_r_oleg442-client.virtnet' removed directory '/mnt/lustre/d77kb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10, bs: 1048576 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/tbf] [bs=1048576] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.94152 s, 21.2 MB/s Write runtime is 6 s, speed is 16.666666 vs. 20 Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kb.sanityn/tbf] [of=/dev/null] [bs=1048576] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.74321 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 vs. 10 cleanup_tbf /mnt/lustre/d77kb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Cleanup TBF ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77kb (41s) == sanityn test 77kc: Check different granular TBF type combination: nid+opcode ========================================================== 18:36:08 (1789770968) ost.OSS.ost_io.nrs_policies=tbf nid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.204.42@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.204.42@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.261135 s, 64.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0178147 s, 58.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0412964 s, 25.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0397677 s, 26.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0463761 s, 22.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0405953 s, 25.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0490511 s, 21.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0409869 s, 25.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0541743 s, 19.4 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0601016 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0405737 s, 25.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0792315 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0803796 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.228886 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227982 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.290404 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.301487 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0216468 s, 48.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0200396 s, 52.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0203783 s, 51.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0218069 s, 48.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.030992 s, 33.8 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0256737 s, 40.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0232954 s, 45.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0248677 s, 42.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0178981 s, 58.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.023129 s, 45.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0278935 s, 37.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0286144 s, 36.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.029926 s, 35.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0305491 s, 34.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0275479 s, 38.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0308446 s, 34.0 MB/s removed '/mnt/lustre/d77kc.sanityn/nrs_w_oleg442-client.virtnet' removed '/mnt/lustre/d77kc.sanityn/nrs_r_oleg442-client.virtnet' removed directory '/mnt/lustre/d77kc.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10, bs: 1048576 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.95995 s, 21.1 MB/s Write runtime is 6 s, speed is 16.666666 vs. 20 Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.75658 s, 10.7 MB/s Read runtime is 10 s, speed is 10.000000 vs. 10 cleanup_tbf /mnt/lustre/d77kc.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Cleanup TBF ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77kc (47s) == sanityn test 77kd: Check different granular TBF type combination: nid+jobid ========================================================== 18:36:55 (1789771015) ost.OSS.ost_io.nrs_policies=tbf nid+jobid ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.204.42@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/nrs_r_oleg442-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.28019 s, 59.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143298 s, 73.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0111325 s, 94.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161264 s, 65.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0647499 s, 16.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0659711 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0622213 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0801859 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0676788 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0786834 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0693778 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0699167 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0699439 s, 15.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123489 s, 8.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115988 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190555 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.226586 s, 4.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0239649 s, 43.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0288649 s, 36.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0181717 s, 57.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0319402 s, 32.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0223329 s, 47.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0265316 s, 39.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0241872 s, 43.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0247901 s, 42.3 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0192549 s, 54.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0329302 s, 31.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00338936 s, 309 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0324701 s, 32.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0260397 s, 40.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0296154 s, 35.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0269906 s, 38.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0282154 s, 37.2 MB/s removed '/mnt/lustre/d77kd.sanityn/nrs_r_oleg442-client.virtnet' removed '/mnt/lustre/d77kd.sanityn/nrs_w_oleg442-client.virtnet' removed directory '/mnt/lustre/d77kd.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20, bs: 1048576 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/tbf] [bs=1048576] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.07476 s, 20.7 MB/s Write runtime is 6 s, speed is 16.666666 vs. 20 Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kd.sanityn/tbf] [of=/dev/null] [bs=1048576] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.98602 s, 21.0 MB/s Read runtime is 6 s, speed is 16.666666 vs. 20 cleanup_tbf /mnt/lustre/d77kd.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Cleanup TBF ost.OSS.ost_io.nrs_tbf_rule=stop ext ost.OSS.ost_io.nrs_policies=fifo PASS 77kd (41s) == sanityn test 77ke: Check different granular TBF type combination: jobid+opcode ========================================================== 18:37:36 (1789771056) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/nrs_r_oleg442-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.473369 s, 35.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156428 s, 67.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0496245 s, 21.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0368775 s, 28.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.029219 s, 35.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0362449 s, 28.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0889448 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0960465 s, 10.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0855027 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101664 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.092365 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120687 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132417 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186731 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201455 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.252236 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.247227 s, 4.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0210425 s, 49.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013859 s, 75.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0274537 s, 38.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161979 s, 64.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0287203 s, 36.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0203341 s, 51.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0284506 s, 36.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0181202 s, 57.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0289316 s, 36.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0231486 s, 45.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0255246 s, 41.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0262106 s, 40.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0283631 s, 37.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0299678 s, 35.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.031247 s, 33.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0307688 s, 34.1 MB/s removed '/mnt/lustre/d77ke.sanityn/nrs_r_oleg442-client.virtnet' removed '/mnt/lustre/d77ke.sanityn/nrs_w_oleg442-client.virtnet' removed directory '/mnt/lustre/d77ke.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10, bs: 1048576 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/tbf] [bs=1048576] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.87597 s, 10.6 MB/s Write runtime is 11 s, speed is 9.090909 vs. 10 Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ke.sanityn/tbf] [of=/dev/null] [bs=1048576] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.78284 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 vs. 10 cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10, bs: 1048576 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.95313 s, 21.2 MB/s Write runtime is 6 s, speed is 16.666666 vs. 20 Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.75151 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 vs. 10 cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Cleanup TBF ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_policies=fifo PASS 77ke (87s) == sanityn test 77kf: Check different granular TBF type combination: uid+gid ========================================================== 18:39:03 (1789771143) ost.OSS.ost_io.nrs_policies=tbf uid+gid ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/nrs_r_oleg442-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.871843 s, 19.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0189414 s, 55.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0273318 s, 38.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.035216 s, 29.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0334051 s, 31.4 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0388484 s, 27.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0292463 s, 35.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0538225 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0566827 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157806 s, 6.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180035 s, 5.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157751 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189683 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202265 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199899 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187466 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196578 s, 5.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150971 s, 69.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141944 s, 73.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0198215 s, 52.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0185688 s, 56.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0228954 s, 45.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.018482 s, 56.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0114569 s, 91.5 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00453922 s, 231 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0188367 s, 55.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0177299 s, 59.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0188031 s, 55.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.025622 s, 40.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0227562 s, 46.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00582107 s, 180 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00304294 s, 345 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00253744 s, 413 MB/s removed '/mnt/lustre/d77kf.sanityn/nrs_r_oleg442-client.virtnet' removed '/mnt/lustre/d77kf.sanityn/nrs_w_oleg442-client.virtnet' removed directory '/mnt/lustre/d77kf.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5, bs: 1048576 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/tbf] [bs=1048576] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.679 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 vs. 5 Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kf.sanityn/tbf] [of=/dev/null] [bs=1048576] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9599 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 vs. 5 cleanup_tbf /mnt/lustre/d77kf.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Cleanup TBF ost.OSS.ost_io.nrs_tbf_rule=stop ext_ug ost.OSS.ost_io.nrs_policies=fifo PASS 77kf (71s) == sanityn test 77kg: check different granular TBF type combination: uid+opcode ========================================================== 18:40:14 (1789771214) ost.OSS.ost_io.nrs_policies=tbf uid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg442-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.255685 s, 65.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0187871 s, 55.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161575 s, 64.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0364648 s, 28.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0460804 s, 22.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0388688 s, 27.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0439764 s, 23.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0627005 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0669351 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0438187 s, 23.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.047114 s, 22.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0435439 s, 24.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0370624 s, 28.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.076886 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0266516 s, 39.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0653889 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0855101 s, 12.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0239192 s, 43.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0241115 s, 43.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0241826 s, 43.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0211424 s, 49.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0258674 s, 40.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00524606 s, 200 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0210352 s, 49.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00574552 s, 183 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.025145 s, 41.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0278185 s, 37.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0343329 s, 30.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0337282 s, 31.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0307714 s, 34.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00899958 s, 117 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00620821 s, 169 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0367082 s, 28.6 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg442-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg442-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10, bs: 1048576 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1048576] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.96754 s, 21.1 MB/s Write runtime is 6 s, speed is 16.666666 vs. 20 Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1048576] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.74789 s, 10.8 MB/s Read runtime is 10 s, speed is 10.000000 vs. 10 cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg442-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.481912 s, 34.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00950112 s, 110 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0252589 s, 41.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0528018 s, 19.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0779106 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0667614 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0615125 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0760148 s, 13.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0788009 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0803985 s, 13.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0963882 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.088349 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0817402 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.271703 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.272388 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.329776 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.334501 s, 3.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0298805 s, 35.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0329938 s, 31.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0312813 s, 33.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0381607 s, 27.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0428755 s, 24.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0329003 s, 31.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0383856 s, 27.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0385267 s, 27.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0413955 s, 25.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0370649 s, 28.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0308006 s, 34.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.036475 s, 28.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.03864 s, 27.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0459885 s, 22.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.049738 s, 21.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0452601 s, 23.2 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg442-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg442-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10, bs: 1048576 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1048576] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.80588 s, 10.7 MB/s Write runtime is 11 s, speed is 9.090909 vs. 10 Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1048576] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.74372 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 vs. 10 cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10, bs: 1048576 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1048576] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.83231 s, 10.7 MB/s Write runtime is 11 s, speed is 9.090909 vs. 20 Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1048576] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.77581 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 vs. 10 cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Cleanup TBF ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_policies=fifo PASS 77kg (129s) == sanityn test 77kh: Verify that Project ID support for NRS TBF rule ========================================================== 18:42:23 (1789771343) using SAVE_PROJECT_SUPPORTED=0 Stopping clients: oleg442-client.virtnet /mnt/lustre (opts:) Stopping client oleg442-client.virtnet /mnt/lustre opts: Stopping clients: oleg442-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg442-client.virtnet /mnt/lustre2 opts: Stopping /mnt/lustre-mds1 (opts:-f) on oleg442-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg442-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg442-server sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime) proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) devtmpfs on /dev type devtmpfs (rw,nosuid,mode=755) securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime) tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev) devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000) tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755) tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755) cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd) pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime) bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700) cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory) cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio) cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids) cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct) cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma) cgroup on /sys/fs/cgroup/misc type cgroup (rw,nosuid,nodev,noexec,relatime,misc) cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb) cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio) cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices) cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer) cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event) cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset) none on /sys/kernel/tracing type tracefs (rw,relatime) configfs on /sys/kernel/config type configfs (rw,relatime) /dev/nbd0 on / type ext4 (ro,relatime,stripe=32) rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M) debugfs on /sys/kernel/debug type debugfs (rw,relatime) mqueue on /dev/mqueue type mqueue (rw,relatime) none on /mnt type ramfs (rw,relatime) none on /var/lib/stateless/writable type tmpfs (rw,relatime) /dev/vda on /home/green/git/lustre-release type squashfs (ro,relatime) none on /var/cache/man type tmpfs (rw,relatime) none on /var/log type tmpfs (rw,relatime) none on /var/lib/dbus type tmpfs (rw,relatime) none on /tmp type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) none on /var/tmp type tmpfs (rw,relatime) none on /var/lib/NetworkManager type tmpfs (rw,relatime) none on /var/lib/systemd/random-seed type tmpfs (rw,relatime) none on /var/spool type tmpfs (rw,relatime) none on /var/lib/nfs type tmpfs (rw,relatime) none on /var/lib/gssproxy type tmpfs (rw,relatime) none on /var/lib/logrotate type tmpfs (rw,relatime) none on /etc type tmpfs (rw,relatime) none on /var/lib/rsyslog type tmpfs (rw,relatime) none on /root/.ssh type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) 192.168.200.253:/exports/state/oleg442-client.virtnet on /var/lib/stateless/state type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.204.42,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg442-client.virtnet/boot on /boot type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.204.42,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg442-client.virtnet/etc/kdump.conf on /etc/kdump.conf type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.204.42,local_lock=none,addr=192.168.200.253) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=378180k,mode=700) 192.168.200.253:/exports/testreports/70515/testresults/sanityn-zfs-rocky8.10_x86_64-rocky8.10_x86_64 on /tmp/testlogs type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.204.42,local_lock=none,addr=192.168.200.253) /dev/vda on /usr/sbin/mount.lustre type squashfs (ro,relatime) Checking servers environments Checking clients oleg442-client.virtnet environments Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg442-server' oleg442-server: oleg442-server.virtnet: executing load_modules_local oleg442-server: kmemleak disabled oleg442-server: Loading modules from /home/green/git/lustre-release/lustre oleg442-server: detected 4 online CPUs by sysfs oleg442-server: MODOPTS_LIBCFS= oleg442-server: Force libcfs to create 2 CPU partitions oleg442-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Setup mgs, mdt, osts Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg442-server: oleg442-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg442-client: oleg442-server: ssh exited with exit code 1 Started lustre-MDT0000 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg442-server: oleg442-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg442-client: oleg442-server: ssh exited with exit code 1 Started lustre-OST0000 Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg442-server: oleg442-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg442-client: oleg442-server: ssh exited with exit code 1 Started lustre-OST0001 Starting client: oleg442-client.virtnet: -o user_xattr,flock 192.168.204.142@tcp:/lustre /mnt/lustre Starting client oleg442-client.virtnet: -o user_xattr,flock 192.168.204.142@tcp:/lustre /mnt/lustre Started clients oleg442-client.virtnet: 192.168.204.142@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Starting client: oleg442-client.virtnet: -o user_xattr,flock 192.168.204.142@tcp:/lustre /mnt/lustre2 Starting client oleg442-client.virtnet: -o user_xattr,flock 192.168.204.142@tcp:/lustre /mnt/lustre2 Started clients oleg442-client.virtnet: 192.168.204.142@tcp:/lustre on /mnt/lustre2 type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9a1006210000.idle_timeout=debug osc.lustre-OST0000-osc-ffff9a1010a53800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9a1006210000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9a1010a53800.idle_timeout=debug disable quota as required ost.OSS.ost_io.nrs_policies=tbf projid ost.OSS.ost_io.nrs_tbf_rule=start ext_proj_rw projid={100} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.249916 s, 67.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0386439 s, 27.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0207908 s, 50.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0502232 s, 20.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0789948 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0729797 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0637988 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0758696 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0771217 s, 13.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0756686 s, 13.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0834105 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145042 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154163 s, 6.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.260395 s, 4.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.233616 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.296572 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.308958 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016102 s, 65.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155638 s, 67.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.019206 s, 54.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0179255 s, 58.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0250732 s, 41.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0185153 s, 56.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0236966 s, 44.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0203693 s, 51.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166065 s, 63.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0268728 s, 39.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.025579 s, 41.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0252114 s, 41.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0220231 s, 47.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0298811 s, 35.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0261386 s, 40.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0272262 s, 38.5 MB/s removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg442-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg442-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20, bs: 1048576 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.96224 s, 21.1 MB/s Write runtime is 6 s, speed is 16.666666 vs. 20 Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.97248 s, 21.1 MB/s Read runtime is 6 s, speed is 16.666666 vs. 20 cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Cleanup TBF ost.OSS.ost_io.nrs_tbf_rule=stop ext_proj_rw ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf projid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w projid={100}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r projid={100}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.304186 s, 55.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.043087 s, 24.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0514146 s, 20.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0770114 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.081857 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0658415 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0760072 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0733931 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0961761 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0822713 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0842073 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11097 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119916 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.229713 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.225832 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212522 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215251 s, 4.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149389 s, 70.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0177062 s, 59.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0290118 s, 36.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0208167 s, 50.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0216251 s, 48.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0232053 s, 45.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0227059 s, 46.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0258692 s, 40.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0272014 s, 38.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0293057 s, 35.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0455686 s, 23.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0282781 s, 37.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0287732 s, 36.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0318865 s, 32.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0328585 s, 31.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0350874 s, 29.9 MB/s removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg442-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg442-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10, bs: 1048576 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.9606 s, 21.1 MB/s Write runtime is 6 s, speed is 16.666666 vs. 20 Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.74065 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 vs. 10 cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Cleanup TBF ost.OSS.ost_io.nrs_tbf_rule=stop ext_proj_rw ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77kh (138s) == sanityn test 77ki: Add rule with unsupported TBF type should fail for generic TBF ========================================================== 18:44:41 (1789771481) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 oleg442-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_fail projid={100}&opcode={ost_write} rate=20: Invalid argument oleg442-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_fail projid={100}&opcode={ost_write} rate=20': Invalid argument pdsh@oleg442-client: oleg442-server: ssh exited with exit code 22 Cleanup TBF ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77ki (8s) == sanityn test 77kj: Verify nodemap support for NRS TBF rule ========================================================== 18:44:49 (1789771489) On MGS 192.168.204.142, active = nodemap.active=1 On MGS 192.168.204.142, TBF.id = nodemap.TBF.id=1 ost.OSS.ost_io.nrs_policies=tbf nodemap ost.OSS.ost_io.nrs_tbf_rule=start ext_nm_rw nodemap={TBF} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.264948 s, 63.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.034758 s, 30.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0323113 s, 32.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0392108 s, 26.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0515959 s, 20.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.050917 s, 20.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0466285 s, 22.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0828233 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0538613 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0915593 s, 11.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0985521 s, 10.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103112 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108575 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116094 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106701 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130625 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10899 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0281938 s, 37.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0313033 s, 33.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0282033 s, 37.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0237047 s, 44.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0174907 s, 60.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0197737 s, 53.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0360832 s, 29.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012453 s, 84.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0286476 s, 36.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0223553 s, 46.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00469018 s, 224 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125341 s, 83.7 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0267756 s, 39.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00645893 s, 162 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0323781 s, 32.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0188892 s, 55.5 MB/s removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg442-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg442-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20, bs: 1048576 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.96862 s, 21.1 MB/s Write runtime is 6 s, speed is 16.666666 vs. 20 Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.98599 s, 21.0 MB/s Read runtime is 6 s, speed is 16.666666 vs. 20 cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete On MGS 192.168.204.142, TBF.id = On MGS 192.168.204.142, active = nodemap.active=0 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.253104 s, 66.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0642001 s, 16.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.065232 s, 16.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0894814 s, 11.7 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0832731 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0925696 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0924788 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0895793 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0881242 s, 11.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0940027 s, 11.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0942706 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0876218 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11032 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107714 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0966429 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0936104 s, 11.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9a1010a53800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9a1010a53800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0961298 s, 10.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0241771 s, 43.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0192471 s, 54.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169507 s, 61.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0191357 s, 54.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0226195 s, 46.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.018615 s, 56.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0204621 s, 51.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0272423 s, 38.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0364089 s, 28.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0310861 s, 33.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0275025 s, 38.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0219153 s, 47.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0254362 s, 41.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.026287 s, 39.9 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9a1010a53800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0256288 s, 40.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0278626 s, 37.6 MB/s removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg442-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg442-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 1 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20, bs: 1048576 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 2.53751 s, 41.3 MB/s Write runtime is 3 s, speed is 33.333333 vs. 20 write rate (33.333333) should exceed limit (20 * 1) Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 0.793728 s, 132 MB/s Read runtime is 2 s, speed is 50.000000 vs. 20 read rate (50.000000) should exceed limit (20 * 1) cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Cleanup TBF ost.OSS.ost_io.nrs_tbf_rule=stop ext_nm_rw ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf nodemap+opcode oleg442-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_w nodemap={TBF}&opcode={ost_write} rate=20: Invalid argument oleg442-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_w nodemap={TBF}&opcode={ost_write} rate=20': Invalid argument oleg442-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_r nodemap={TBF}&opcode={ost_read} rate=10: Invalid argument oleg442-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_r nodemap={TBF}&opcode={ost_read} rate=10': Invalid argument pdsh@oleg442-client: oleg442-server: ssh exited with exit code 22 On MGS 192.168.204.142, active = nodemap.active=1 On MGS 192.168.204.142, TBF.id = nodemap.TBF.id=2 ost.OSS.ost_io.nrs_tbf_rule=start ext_w nodemap={TBF}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nodemap={TBF}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.276886 s, 60.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0308393 s, 34.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0344453 s, 30.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0814554 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0804214 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0962362 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100714 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109301 s, 9.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0986376 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0998704 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107981 s, 9.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0936273 s, 11.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10254 s, 10.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.255226 s, 4.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.251595 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.291243 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.276963 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0219843 s, 47.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0206236 s, 50.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0190995 s, 54.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.019613 s, 53.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0222164 s, 47.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0246889 s, 42.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0222681 s, 47.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0339949 s, 30.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00513554 s, 204 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00472083 s, 222 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0190797 s, 55.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0355872 s, 29.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00512985 s, 204 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00932792 s, 112 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015034 s, 69.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0214534 s, 48.9 MB/s removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg442-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg442-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10, bs: 1048576 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.95912 s, 21.1 MB/s Write runtime is 6 s, speed is 16.666666 vs. 20 Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.75657 s, 10.7 MB/s Read runtime is 10 s, speed is 10.000000 vs. 10 cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Cleanup TBF ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf nid classify=dynamic ost.OSS.ost_io.nrs_tbf_rule=start nmdyn nodemap={TBF} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.262247 s, 64.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0279931 s, 37.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0312889 s, 33.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0945071 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.083884 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105884 s, 9.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0978472 s, 10.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109742 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0951312 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.094758 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118148 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106544 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10831 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.241372 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.24677 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.304343 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.291545 s, 3.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0360077 s, 29.1 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0348137 s, 30.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0385059 s, 27.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.037212 s, 28.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0341615 s, 30.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0309727 s, 33.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0365967 s, 28.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0400179 s, 26.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0343269 s, 30.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0324147 s, 32.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0382724 s, 27.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0507623 s, 20.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0369237 s, 28.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0362992 s, 28.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0370145 s, 28.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0358651 s, 29.2 MB/s removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg442-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg442-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20, bs: 1048576 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.98353 s, 21.0 MB/s Write runtime is 6 s, speed is 16.666666 vs. 20 Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.97808 s, 21.1 MB/s Read runtime is 6 s, speed is 16.666666 vs. 20 cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Cleanup TBF ost.OSS.ost_io.nrs_tbf_rule=stop nmdyn ost.OSS.ost_io.nrs_policies=fifo On MGS 192.168.204.142, TBF.id = On MGS 192.168.204.142, active = nodemap.active=0 Cleanup TBF ost.OSS.ost_io.nrs_policies=fifo Cleanup TBF ost.OSS.ost_io.nrs_policies=fifo Cleanup TBF ost.OSS.ost_io.nrs_policies=fifo On MGS 192.168.204.142, active = nodemap.active=0 PASS 77kj (168s) == sanityn test 77l: check the output of NRS policies for generic TBF ========================================================== 18:47:37 (1789771657) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_policies=fifo PASS 77l (4s) == sanityn test 77m: check NRS Delay slows write RPC processing ========================================================== 18:47:41 (1789771661) ost.OSS.ost_io.nrs_policies=delay ost.OSS.ost_io.nrs_delay_min=4 ost.OSS.ost_io.nrs_delay_max=4 ost.OSS.ost_io.nrs_delay_pct=100 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg442-client.virtnet] [bs=1M] [count=1] [oflag=direct] [conv=fdatasync] 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 3.5638 s, 294 kB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg442-client.virtnet] [bs=1M] [count=10] [oflag=direct] [conv=fdatasync] 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 40.0184 s, 262 kB/s ost.OSS.ost_io.nrs_policies=fifo PASS 77m (51s) == sanityn test 77n: check wildcard support for TBF JobID NRS policy ========================================================== 18:48:32 (1789771712) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={*.500} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.16352 s, 103 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0491327 s, 21.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0379197 s, 27.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0535447 s, 19.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0520025 s, 20.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0645829 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.054264 s, 19.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0755774 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116412 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0947448 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0709568 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112029 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0926744 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120731 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100604 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115851 s, 9.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9a1010a53800.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff9a1010a53800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120695 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0199368 s, 52.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0246453 s, 42.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0182003 s, 57.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0218131 s, 48.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0341611 s, 30.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0294304 s, 35.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00918674 s, 114 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0281312 s, 37.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00789627 s, 133 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0175247 s, 59.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0244004 s, 43.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0330401 s, 31.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0377215 s, 27.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.005563 s, 188 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138092 s, 75.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0420469 s, 24.9 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg442-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg442-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20, bs: 1048576 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77n.sanityn/tbf] [bs=1048576] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.94478 s, 21.2 MB/s Write runtime is 6 s, speed is 16.666666 vs. 20 Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77n.sanityn/tbf] [of=/dev/null] [bs=1048576] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.97582 s, 21.1 MB/s Read runtime is 6 s, speed is 16.666666 vs. 20 cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.*} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.245936 s, 68.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.034807 s, 30.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0214986 s, 48.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0805084 s, 13.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0846097 s, 12.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0691865 s, 15.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0783241 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0841496 s, 12.5 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0874591 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0751556 s, 14.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0908794 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0865083 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0987868 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.252573 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.260556 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.30951 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.299723 s, 3.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155206 s, 67.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0196787 s, 53.3 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0186454 s, 56.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0240125 s, 43.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016302 s, 64.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0188269 s, 55.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0189992 s, 55.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0205864 s, 50.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160086 s, 65.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0248277 s, 42.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0106301 s, 98.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00374726 s, 280 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00362636 s, 289 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0266387 s, 39.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00349179 s, 300 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00614728 s, 171 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg442-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg442-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20, bs: 1048576 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.93695 s, 21.2 MB/s Write runtime is 6 s, speed is 16.666666 vs. 20 Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.97636 s, 21.1 MB/s Read runtime is 6 s, speed is 16.666666 vs. 20 cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_policies=fifo PASS 77n (69s) == sanityn test 77o: Changing rank should not panic ====== 18:49:41 (1789771781) mds.MDS.mdt.nrs_policies=tbf nid mds.MDS.mdt.nrs_tbf_rule=start name nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=start name1 nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=change name1 rank=name mds.MDS.mdt.nrs_tbf_rule=stop name mds.MDS.mdt.nrs_policies=fifo PASS 77o (4s) == sanityn test 77q: Parallel TBF rule definitions should not panic ========================================================== 18:49:45 (1789771785) mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_policies=fifo PASS 77q (41s) == sanityn test 77p: Check validity of rule names for TBF policies ========================================================== 18:50:26 (1789771826) mds.MDS.mdt.nrs_policies=tbf oleg442-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_overflo uid={500} rate=500: File name too long oleg442-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_overflo uid={500} rate=500': File name too long pdsh@oleg442-client: oleg442-server: ssh exited with exit code 36 oleg442-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start uid={500} rate=500: Invalid argument oleg442-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start uid={500} rate=500': Invalid argument pdsh@oleg442-client: oleg442-server: ssh exited with exit code 22 oleg442-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_empty: Invalid argument oleg442-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_empty': Invalid argument pdsh@oleg442-client: oleg442-server: ssh exited with exit code 22 oleg442-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p@spec uid={500} rate=500: Invalid argument oleg442-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p@spec uid={500} rate=500': Invalid argument pdsh@oleg442-client: oleg442-server: ssh exited with exit code 22 oleg442-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p.spec uid={500} rate=500: Invalid argument oleg442-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p.spec uid={500} rate=500': Invalid argument pdsh@oleg442-client: oleg442-server: ssh exited with exit code 22 oleg442-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p~spec uid={500} rate=500: Invalid argument oleg442-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p~spec uid={500} rate=500': Invalid argument pdsh@oleg442-client: oleg442-server: ssh exited with exit code 22 oleg442-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p#spec uid={500} rate=500: Invalid argument oleg442-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p#spec uid={500} rate=500': Invalid argument pdsh@oleg442-client: oleg442-server: ssh exited with exit code 22 oleg442-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p/spec uid={500} rate=500: Invalid argument oleg442-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p/spec uid={500} rate=500': Invalid argument pdsh@oleg442-client: oleg442-server: ssh exited with exit code 22 oleg442-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p^spec uid={500} rate=500: Invalid argument oleg442-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p^spec uid={500} rate=500': Invalid argument pdsh@oleg442-client: oleg442-server: ssh exited with exit code 22 oleg442-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p%spec uid={500} rate=500: Invalid argument oleg442-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p%spec uid={500} rate=500': Invalid argument pdsh@oleg442-client: oleg442-server: ssh exited with exit code 22 oleg442-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p*spec uid={500} rate=500: Invalid argument oleg442-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p*spec uid={500} rate=500': Invalid argument pdsh@oleg442-client: oleg442-server: ssh exited with exit code 22 oleg442-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p;spec uid={500} rate=500: Invalid argument oleg442-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p;spec uid={500} rate=500': Invalid argument pdsh@oleg442-client: oleg442-server: ssh exited with exit code 22 oleg442-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p,spec uid={500} rate=500: Invalid argument oleg442-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p,spec uid={500} rate=500': Invalid argument pdsh@oleg442-client: oleg442-server: ssh exited with exit code 22 oleg442-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p?spec uid={500} rate=500: Invalid argument oleg442-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p?spec uid={500} rate=500': Invalid argument pdsh@oleg442-client: oleg442-server: ssh exited with exit code 22 oleg442-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77pspec uid={500} rate=500: Invalid argument oleg442-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p>spec uid={500} rate=500': Invalid argument pdsh@oleg442-client: oleg442-server: ssh exited with exit code 22 oleg442-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p:spec uid={500} rate=500: Invalid argument oleg442-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p:spec uid={500} rate=500': Invalid argument pdsh@oleg442-client: oleg442-server: ssh exited with exit code 22 oleg442-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p+spec uid={500} rate=500: Invalid argument oleg442-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p+spec uid={500} rate=500': Invalid argument pdsh@oleg442-client: oleg442-server: ssh exited with exit code 22 oleg442-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p=spec uid={500} rate=500: Invalid argument oleg442-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p=spec uid={500} rate=500': Invalid argument pdsh@oleg442-client: oleg442-server: ssh exited with exit code 22 oleg442-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p)spec uid={500} rate=500: Invalid argument oleg442-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p)spec uid={500} rate=500': Invalid argument pdsh@oleg442-client: oleg442-server: ssh exited with exit code 22 oleg442-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p(spec uid={500} rate=500: Invalid argument oleg442-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p(spec uid={500} rate=500': Invalid argument pdsh@oleg442-client: oleg442-server: ssh exited with exit code 22 oleg442-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p{spec uid={500} rate=500: Invalid argument oleg442-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p{spec uid={500} rate=500': Invalid argument pdsh@oleg442-client: oleg442-server: ssh exited with exit code 22 oleg442-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p}spec uid={500} rate=500: Invalid argument oleg442-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p}spec uid={500} rate=500': Invalid argument pdsh@oleg442-client: oleg442-server: ssh exited with exit code 22 oleg442-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p|spec uid={500} rate=500: Invalid argument oleg442-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p|spec uid={500} rate=500': Invalid argument pdsh@oleg442-client: oleg442-server: ssh exited with exit code 22 oleg442-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p[spec uid={500} rate=500: Invalid argument oleg442-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p[spec uid={500} rate=500': Invalid argument pdsh@oleg442-client: oleg442-server: ssh exited with exit code 22 oleg442-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p]spec uid={500} rate=500: Invalid argument oleg442-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p]spec uid={500} rate=500': Invalid argument pdsh@oleg442-client: oleg442-server: ssh exited with exit code 22 oleg442-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p!spec uid={500} rate=500: Invalid argument oleg442-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p!spec uid={500} rate=500': Invalid argument pdsh@oleg442-client: oleg442-server: ssh exited with exit code 22 oleg442-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p&spec uid={500} rate=500: Invalid argument oleg442-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p&spec uid={500} rate=500': Invalid argument pdsh@oleg442-client: oleg442-server: ssh exited with exit code 22 oleg442-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p$spec uid={500} rate=500: Invalid argument oleg442-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p$spec uid={500} rate=500': Invalid argument pdsh@oleg442-client: oleg442-server: ssh exited with exit code 22 oleg442-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p`spec uid={500} rate=500: Invalid argument oleg442-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p`spec uid={500} rate=500': Invalid argument pdsh@oleg442-client: oleg442-server: ssh exited with exit code 22 oleg442-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p\spec uid={500} rate=500: Invalid argument oleg442-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p\spec uid={500} rate=500': Invalid argument pdsh@oleg442-client: oleg442-server: ssh exited with exit code 22 mds.MDS.mdt.nrs_policies=fifo PASS 77p (13s) == sanityn test 77r: Change type of tbf policy at run time ========================================================== 18:50:39 (1789771839) mds.MDS.mdt.nrs_policies=tbf uid mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf gid mds.MDS.mdt.nrs_tbf_rule=start md_rule gid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start md_rule jobid={*.500} rate=1 mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=fifo check the number of requests in queue: fifo: 0 crrn: 0 tbf: 0 delay: 0 fifo: 0 crrn: 0 tbf: 0 delay: 0 cleanup 77r 279339 kill md_thread xargs PASS 77r (43s) == sanityn test 77s: Check TBF LRU shrinker ============== 18:51:22 (1789771882) jobid_name=%e.%p jobid_var=TEST_77s_JOBVAR mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start touch jobid={touch.*} rate=1000 mds.MDS.mdt.nrs_tbf_rule=start unlink jobid={unlink.*} rate=1000 touch jobid={touch.*} 1000, seq 2, type=jobid, ref 10 touch jobid={touch.*} 1000, seq 2, type=jobid, ref 0 unlink jobid={unlink.*} 1000, seq 3, type=jobid, ref 10 unlink jobid={unlink.*} 1000, seq 3, type=jobid, ref 0 mds.MDS.mdt.nrs_policies=fifo jobid_var=procname_uid jobid_name=%e.%u PASS 77s (16s) == sanityn test 77ta: Testing for dynamic|static classify parameter setting for TBF ========================================================== 18:51:38 (1789771898) ost.OSS.ost_io.nrs_policies=fifo oleg442-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf nid+uid classify=foo: Invalid argument oleg442-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf nid+uid classify=foo': Invalid argument pdsh@oleg442-client: oleg442-server: ssh exited with exit code 22 oleg442-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf nid+uid foo=static: Invalid argument oleg442-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf nid+uid foo=static': Invalid argument pdsh@oleg442-client: oleg442-server: ssh exited with exit code 22 oleg442-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf nid+uid classify=: Invalid argument oleg442-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf nid+uid classify=': Invalid argument pdsh@oleg442-client: oleg442-server: ssh exited with exit code 22 ost.OSS.ost_io.nrs_policies=tbf nid classify=dynamic ost.OSS.ost_io.nrs_policies=tbf nid classify=static ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf reg nid classify=dynamic ost.OSS.ost_io.nrs_policies=tbf hp nid ost.OSS.ost_io.nrs_tbf_rule=start mixed1 nid={0@lo} rate=100 ost.OSS.ost_io.nrs_tbf_rule=stop mixed1 ost.OSS.ost_io.nrs_policies=tbf reg nid classify=static ost.OSS.ost_io.nrs_policies=tbf hp nid classify=dynamic ost.OSS.ost_io.nrs_tbf_rule=start mixed2 nid={0@lo} rate=100 ost.OSS.ost_io.nrs_tbf_rule=stop mixed2 ost.OSS.ost_io.nrs_policies=tbf reg nid+uid ost.OSS.ost_io.nrs_policies=tbf hp nid oleg442-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start mixed3 nid={0@lo}&uid={500} rate=100: Invalid argument oleg442-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start mixed3 nid={0@lo}&uid={500} rate=100': Invalid argument pdsh@oleg442-client: oleg442-server: ssh exited with exit code 22 ost.OSS.ost_io.nrs_policies=tbf nid classify=static classify=dynamic ost.OSS.ost_io.nrs_policies=tbf reg nid classify=dynamic ost.OSS.ost_io.nrs_policies= regular_requests: - name: fifo state: started fallback: yes queued: 0 active: 0 - name: crrn state: stopped fallback: no queued: 0 active: 0 - name: orr state: stopped fallback: no queued: 0 active: 0 - name: trr state: stopped fallback: no queued: 0 active: 0 - name: tbf nid classify=dynamic state: started fallback: no queued: 0 active: 0 - name: delay state: stopped fallback: no queued: 0 active: 0 high_priority_requests: - name: fifo state: started fallback: yes queued: 0 active: 0 - name: crrn state: stopped fallback: no queued: 0 active: 0 - name: orr state: stopped fallback: no queued: 0 active: 0 - name: trr state: stopped fallback: no queued: 0 active: 0 - name: tbf nid classify=static classify=dynamic state: started fallback: no queued: 0 active: 0 - name: delay state: stopped fallback: no queued: 0 active: 0 ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf hp nid classify=static ost.OSS.ost_io.nrs_policies= regular_requests: - name: fifo state: started fallback: yes queued: 0 active: 0 - name: crrn state: stopped fallback: no queued: 0 active: 0 - name: orr state: stopped fallback: no queued: 0 active: 0 - name: trr state: stopped fallback: no queued: 0 active: 0 - name: tbf state: stopped fallback: no queued: 0 active: 0 - name: delay state: stopped fallback: no queued: 0 active: 0 high_priority_requests: - name: fifo state: started fallback: yes queued: 0 active: 0 - name: crrn state: stopped fallback: no queued: 0 active: 0 - name: orr state: stopped fallback: no queued: 0 active: 0 - name: trr state: stopped fallback: no queued: 0 active: 0 - name: tbf nid classify=static state: started fallback: no queued: 0 active: 0 - name: delay state: stopped fallback: no queued: 0 active: 0 ost.OSS.ost_io.nrs_policies=fifo PASS 77ta (15s) == sanityn test 77tB: Request falling to default rule should not panic for dynamic TBF ========================================================== 18:51:53 (1789771913) ost.OSS.ost_io.nrs_policies=tbf nid classify=dynamic not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5, bs: 1048576 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.4944 s, 5.4 MB/s Write runtime is 21 s, speed is 4.761904 vs. 5 Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9571 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 vs. 5 cleanup_tbf /mnt/lustre/d77tB.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=start unmatchrule nid={192.0.2.1@tcp} rate=100 not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5, bs: 1048576 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.476 s, 5.4 MB/s Write runtime is 21 s, speed is 4.761904 vs. 5 Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9544 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 vs. 5 cleanup_tbf /mnt/lustre/d77tB.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop unmatchrule ost.OSS.ost_io.nrs_policies=fifo PASS 77tB (127s) == sanityn test 77th: TBF: a mixed classify head pair keeps each head type ========================================================== 18:54:00 (1789772040) ost.OSS.ost_io.nrs_policies=tbf reg nid+opcode ost.OSS.ost_io.nrs_policies=tbf hp nid classify=dynamic ost.OSS.ost_io.nrs_tbf_rule=start mixedhp nid={0@lo} rate=100 regular_requests: CPT 0: mixedhp nid={0@lo} 100, seq 2, type=nid+opcode, ref 0 default * 10000, seq 1, type=nid+opcode, ref 0 high_priority_requests: CPT 0: mixedhp nid={0@lo} 100, seq 2, type=nid, ref 0 default * 10000, seq 1, type=nid, ref 0 ost.OSS.ost_io.nrs_tbf_rule=stop mixedhp ost.OSS.ost_io.nrs_policies=fifo PASS 77th (4s) == sanityn test 77tb: Testing for dynamic classify TBF Scheduler ========================================================== 18:54:04 (1789772044) ost.OSS.ost_io.nrs_policies=tbf nid classify=dynamic ost.OSS.ost_io.nrs_policies= regular_requests: - name: fifo state: started fallback: yes queued: 0 active: 0 - name: crrn state: stopped fallback: no queued: 0 active: 0 - name: orr state: stopped fallback: no queued: 0 active: 0 - name: trr state: stopped fallback: no queued: 0 active: 0 - name: tbf nid classify=dynamic state: started fallback: no queued: 0 active: 0 - name: delay state: stopped fallback: no queued: 0 active: 0 high_priority_requests: - name: fifo state: started fallback: yes queued: 0 active: 0 - name: crrn state: stopped fallback: no queued: 0 active: 0 - name: orr state: stopped fallback: no queued: 0 active: 0 - name: trr state: stopped fallback: no queued: 0 active: 0 - name: tbf nid classify=dynamic state: started fallback: no queued: 0 active: 0 - name: delay state: stopped fallback: no queued: 0 active: 0 ost.OSS.ost_io.nrs_tbf_classify= reg_classify:dynamic hp_classify:dynamic ost.OSS.ost_io.nrs_tbf_rule=start nidall nid={0@lo 192.168.204.42@tcp} rate=10 ost.OSS.ost_io.nrs_tbf_rule=start nidrd nid={0@lo 192.168.204.42@tcp}&opcode={ost_read} rate=5 ost.OSS.ost_io.nrs_tbf_rule=start nidwr nid={0@lo 192.168.204.42@tcp}&opcode={ost_write} rate=15 ost.OSS.ost_io.nrs_tbf_rule= regular_requests: CPT 0: nidwr nid={0@lo 192.168.204.42@tcp}&opcode={ost_write} 15, seq 4, type=nid+opcode, ref 0 nidrd nid={0@lo 192.168.204.42@tcp}&opcode={ost_read} 5, seq 3, type=nid+opcode, ref 0 nidall nid={0@lo 192.168.204.42@tcp} 10, seq 2, type=nid, ref 0 default * 10000, seq 1, type=nid, ref 0 high_priority_requests: CPT 0: nidwr nid={0@lo 192.168.204.42@tcp}&opcode={ost_write} 15, seq 4, type=nid+opcode, ref 0 nidrd nid={0@lo 192.168.204.42@tcp}&opcode={ost_read} 5, seq 3, type=nid+opcode, ref 0 nidall nid={0@lo 192.168.204.42@tcp} 10, seq 2, type=nid, ref 0 default * 10000, seq 1, type=nid, ref 0 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.334963 s, 50.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0281286 s, 37.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.039121 s, 26.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110316 s, 9.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122975 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123694 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122835 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113516 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0986469 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118082 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123262 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126618 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118632 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.335476 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.341569 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.418776 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.398129 s, 2.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0118782 s, 88.3 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.021191 s, 49.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0179319 s, 58.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0101115 s, 104 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0174099 s, 60.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0100705 s, 104 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0208034 s, 50.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0254707 s, 41.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0191454 s, 54.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0268408 s, 39.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0181873 s, 57.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0293952 s, 35.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0197832 s, 53.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00368031 s, 285 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0229633 s, 45.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0273693 s, 38.3 MB/s removed '/mnt/lustre/d77tb.sanityn/nrs_w_oleg442-client.virtnet' removed '/mnt/lustre/d77tb.sanityn/nrs_r_oleg442-client.virtnet' removed directory '/mnt/lustre/d77tb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 15, read rate: 5, bs: 1048576 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.55014 s, 16.0 MB/s Write runtime is 8 s, speed is 12.500000 vs. 15 Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.4308 s, 5.4 MB/s Read runtime is 20 s, speed is 5.000000 vs. 5 cleanup_tbf /mnt/lustre/d77tb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Cleanup TBF ost.OSS.ost_io.nrs_tbf_rule=stop nidall ost.OSS.ost_io.nrs_tbf_rule=stop nidrd ost.OSS.ost_io.nrs_tbf_rule=stop nidwr ost.OSS.ost_io.nrs_policies=fifo PASS 77tb (51s) == sanityn test 77tc: Testing for the setting of allowed TBF classification type ========================================================== 18:54:55 (1789772095) oleg442-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf nid classify=dynamic allow=nid allow=nid: File exists oleg442-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf nid classify=dynamic allow=nid allow=nid': File exists pdsh@oleg442-client: oleg442-server: ssh exited with exit code 17 oleg442-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf nid allow=nid allow=uid: Invalid argument oleg442-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf nid allow=nid allow=uid': Invalid argument pdsh@oleg442-client: oleg442-server: ssh exited with exit code 22 ost.OSS.ost_io.nrs_policies=tbf nid classify=dynamic allow=nid+opcode allow=nid allow=uid+opcode oleg442-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start another uid={500} rate=10: Operation not permitted oleg442-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start another uid={500} rate=10': Operation not permitted pdsh@oleg442-client: oleg442-server: ssh exited with exit code 1 oleg442-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start another uid={500}&opcode={ost_read}&gid={500} rate=10: Operation not permitted oleg442-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start another uid={500}&opcode={ost_read}&gid={500} rate=10': Operation not permitted pdsh@oleg442-client: oleg442-server: ssh exited with exit code 1 ost.OSS.ost_io.nrs_tbf_rule=start nidall nid={0@lo} rate=10 ost.OSS.ost_io.nrs_tbf_rule=start nidrd nid={0@lo}&opcode={ost_read} rate=5 Cleanup TBF ost.OSS.ost_io.nrs_tbf_rule=stop nidall ost.OSS.ost_io.nrs_tbf_rule=stop nidrd ost.OSS.ost_io.nrs_policies=fifo oleg442-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf nodemap classify=dynamic allow=nid allow=uid allow=gid allow=jobid allow=opcode allow=projid allow=nid+uid allow=nid+gid allow=uid+gid allow=nid+jobid allow=uid+jobid allow=gid+jobid allow=nid+opcode allow=uid+opcode allow=gid+opcode allow=nid+projid: Argument list too long oleg442-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf nodemap classify=dynamic allow=nid allow=uid allow=gid allow=jobid allow=opcode allow=projid allow=nid+uid allow=nid+gid allow=uid+gid allow=nid+jobid allow=uid+jobid allow=gid+jobid allow=nid+opcode allow=uid+opcode allow=gid+opcode allow=nid+projid': Argument list too long pdsh@oleg442-client: oleg442-server: ssh exited with exit code 7 Cleanup TBF ost.OSS.ost_io.nrs_policies=fifo PASS 77tc (14s) == sanityn test 77td: Testing Number of allowed rules larger or smaller than threshold ========================================================== 18:55:09 (1789772109) using SAVE_PROJECT_SUPPORTED=0 Stopping clients: oleg442-client.virtnet /mnt/lustre (opts:) Stopping client oleg442-client.virtnet /mnt/lustre opts: Stopping clients: oleg442-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg442-client.virtnet /mnt/lustre2 opts: Stopping /mnt/lustre-mds1 (opts:-f) on oleg442-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg442-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg442-server sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime) proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) devtmpfs on /dev type devtmpfs (rw,nosuid,mode=755) securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime) tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev) devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000) tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755) tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755) cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd) pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime) bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700) cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory) cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio) cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids) cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct) cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma) cgroup on /sys/fs/cgroup/misc type cgroup (rw,nosuid,nodev,noexec,relatime,misc) cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb) cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio) cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices) cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer) cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event) cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset) none on /sys/kernel/tracing type tracefs (rw,relatime) configfs on /sys/kernel/config type configfs (rw,relatime) /dev/nbd0 on / type ext4 (ro,relatime,stripe=32) rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M) debugfs on /sys/kernel/debug type debugfs (rw,relatime) mqueue on /dev/mqueue type mqueue (rw,relatime) none on /mnt type ramfs (rw,relatime) none on /var/lib/stateless/writable type tmpfs (rw,relatime) /dev/vda on /home/green/git/lustre-release type squashfs (ro,relatime) none on /var/cache/man type tmpfs (rw,relatime) none on /var/log type tmpfs (rw,relatime) none on /var/lib/dbus type tmpfs (rw,relatime) none on /tmp type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) none on /var/tmp type tmpfs (rw,relatime) none on /var/lib/NetworkManager type tmpfs (rw,relatime) none on /var/lib/systemd/random-seed type tmpfs (rw,relatime) none on /var/spool type tmpfs (rw,relatime) none on /var/lib/nfs type tmpfs (rw,relatime) none on /var/lib/gssproxy type tmpfs (rw,relatime) none on /var/lib/logrotate type tmpfs (rw,relatime) none on /etc type tmpfs (rw,relatime) none on /var/lib/rsyslog type tmpfs (rw,relatime) none on /root/.ssh type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) 192.168.200.253:/exports/state/oleg442-client.virtnet on /var/lib/stateless/state type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.204.42,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg442-client.virtnet/boot on /boot type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.204.42,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg442-client.virtnet/etc/kdump.conf on /etc/kdump.conf type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.204.42,local_lock=none,addr=192.168.200.253) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=378180k,mode=700) 192.168.200.253:/exports/testreports/70515/testresults/sanityn-zfs-rocky8.10_x86_64-rocky8.10_x86_64 on /tmp/testlogs type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.204.42,local_lock=none,addr=192.168.200.253) /dev/vda on /usr/sbin/mount.lustre type squashfs (ro,relatime) Checking servers environments Checking clients oleg442-client.virtnet environments Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg442-server' oleg442-server: oleg442-server.virtnet: executing load_modules_local oleg442-server: kmemleak disabled oleg442-server: Loading modules from /home/green/git/lustre-release/lustre oleg442-server: detected 4 online CPUs by sysfs oleg442-server: MODOPTS_LIBCFS= oleg442-server: Force libcfs to create 2 CPU partitions oleg442-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Setup mgs, mdt, osts Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg442-server: oleg442-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg442-client: oleg442-server: ssh exited with exit code 1 Started lustre-MDT0000 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg442-server: oleg442-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg442-client: oleg442-server: ssh exited with exit code 1 Started lustre-OST0000 Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg442-server: oleg442-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg442-client: oleg442-server: ssh exited with exit code 1 Started lustre-OST0001 Starting client: oleg442-client.virtnet: -o user_xattr,flock 192.168.204.142@tcp:/lustre /mnt/lustre Starting client oleg442-client.virtnet: -o user_xattr,flock 192.168.204.142@tcp:/lustre /mnt/lustre Started clients oleg442-client.virtnet: 192.168.204.142@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Starting client: oleg442-client.virtnet: -o user_xattr,flock 192.168.204.142@tcp:/lustre /mnt/lustre2 Starting client oleg442-client.virtnet: -o user_xattr,flock 192.168.204.142@tcp:/lustre /mnt/lustre2 Started clients oleg442-client.virtnet: 192.168.204.142@tcp:/lustre on /mnt/lustre2 type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9a1004199800.idle_timeout=debug osc.lustre-OST0000-osc-ffff9a1004e5e800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9a1004199800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9a1004e5e800.idle_timeout=debug disable quota as required ost.OSS.ost_io.nrs_policies=tbf nid classify=dynamic allow=projid+opcode allow=projid ost.OSS.ost_io.nrs_tbf_allowed_types= regular_allowed_types: CPT 0: 0: 0x00000024 opcode+projid 1: 0x00000020 projid 2: 0x00000002 nid high_priority_allowed_types: CPT 0: 0: 0x00000024 opcode+projid 1: 0x00000020 projid 2: 0x00000002 nid 8 ost.OSS.ost_io.nrs_tbf_rule=start rulerd0 projid={100}&opcode={ost_read} rate=5 ost.OSS.ost_io.nrs_tbf_rule=start rulewr0 projid={100}&opcode={ost_write} rate=10 ost.OSS.ost_io.nrs_tbf_rule=start ruleall0 projid={200} rate=15 ost.OSS.ost_io.nrs_tbf_rule= regular_requests: CPT 0: ruleall0 projid={200} 15, seq 4, type=projid, ref 0 rulewr0 projid={100}&opcode={ost_write} 10, seq 3, type=opcode+projid, ref 0 rulerd0 projid={100}&opcode={ost_read} 5, seq 2, type=opcode+projid, ref 0 default * 10000, seq 1, type=nid+opcode+projid, ref 0 high_priority_requests: CPT 0: ruleall0 projid={200} 15, seq 4, type=projid, ref 0 rulewr0 projid={100}&opcode={ost_write} 10, seq 3, type=opcode+projid, ref 0 rulerd0 projid={100}&opcode={ost_read} 5, seq 2, type=opcode+projid, ref 0 default * 10000, seq 1, type=nid+opcode+projid, ref 0 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.120613 s, 139 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0224628 s, 46.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0518867 s, 20.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0618286 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0741887 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0745641 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0756453 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0864946 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0646212 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0609692 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0801299 s, 13.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.095223 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0919593 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0819004 s, 12.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107251 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0821222 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0946546 s, 11.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9a1004199800.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015168 s, 69.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0175392 s, 59.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0190642 s, 55.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0185243 s, 56.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0180436 s, 58.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0049929 s, 210 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00502975 s, 208 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016211 s, 64.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0222493 s, 47.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0257435 s, 40.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00985208 s, 106 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0258784 s, 40.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.023994 s, 43.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167243 s, 62.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00395185 s, 265 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0269944 s, 38.8 MB/s removed '/mnt/lustre/d77td.sanityn/nrs_w_oleg442-client.virtnet' removed '/mnt/lustre/d77td.sanityn/nrs_r_oleg442-client.virtnet' removed directory '/mnt/lustre/d77td.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 5, bs: 1048576 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.78249 s, 10.7 MB/s Write runtime is 10 s, speed is 10.000000 vs. 10 Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.4299 s, 5.4 MB/s Read runtime is 21 s, speed is 4.761904 vs. 5 cleanup_tbf /mnt/lustre/d77td.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 15, read rate: 15, bs: 1048576 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.54214 s, 16.0 MB/s Write runtime is 7 s, speed is 14.285714 vs. 15 Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.62645 s, 15.8 MB/s Read runtime is 8 s, speed is 12.500000 vs. 15 cleanup_tbf /mnt/lustre/d77td.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=change default rate=15 ost.OSS.ost_io.nrs_tbf_rule= regular_requests: CPT 0: ruleall0 projid={200} 15, seq 4, type=projid, ref 1 rulewr0 projid={100}&opcode={ost_write} 10, seq 3, type=opcode+projid, ref 1 rulerd0 projid={100}&opcode={ost_read} 5, seq 2, type=opcode+projid, ref 1 default * 15, seq 1, type=nid+opcode+projid, ref 2 high_priority_requests: CPT 0: ruleall0 projid={200} 15, seq 4, type=projid, ref 0 rulewr0 projid={100}&opcode={ost_write} 10, seq 3, type=opcode+projid, ref 0 rulerd0 projid={100}&opcode={ost_read} 5, seq 2, type=opcode+projid, ref 0 default * 15, seq 1, type=nid+opcode+projid, ref 0 not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 15, read rate: 15, bs: 1048576 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.54816 s, 16.0 MB/s Write runtime is 8 s, speed is 12.500000 vs. 15 Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.50501 s, 16.1 MB/s Read runtime is 7 s, speed is 14.285714 vs. 15 cleanup_tbf /mnt/lustre/d77td.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf nid classify=dynamic allow=projid+opcode allow=projid ost.OSS.ost_io.nrs_tbf_allowed_types= regular_allowed_types: CPT 0: 0: 0x00000024 opcode+projid 1: 0x00000020 projid 2: 0x00000002 nid high_priority_allowed_types: CPT 0: 0: 0x00000024 opcode+projid 1: 0x00000020 projid 2: 0x00000002 nid 8 ost.OSS.ost_io.nrs_tbf_rule=start rulerd0 projid={100}&opcode={ost_read} rate=5 ost.OSS.ost_io.nrs_tbf_rule=start rulewr0 projid={100}&opcode={ost_write} rate=10 ost.OSS.ost_io.nrs_tbf_rule=start rulerd1 projid={101}&opcode={ost_read} rate=5 ost.OSS.ost_io.nrs_tbf_rule=start rulewr1 projid={101}&opcode={ost_write} rate=10 ost.OSS.ost_io.nrs_tbf_rule=start rulerd2 projid={102}&opcode={ost_read} rate=5 ost.OSS.ost_io.nrs_tbf_rule=start rulewr2 projid={102}&opcode={ost_write} rate=10 ost.OSS.ost_io.nrs_tbf_rule=start rulerd3 projid={103}&opcode={ost_read} rate=5 ost.OSS.ost_io.nrs_tbf_rule=start rulewr3 projid={103}&opcode={ost_write} rate=10 ost.OSS.ost_io.nrs_tbf_rule=start rulerd4 projid={104}&opcode={ost_read} rate=5 ost.OSS.ost_io.nrs_tbf_rule=start rulewr4 projid={104}&opcode={ost_write} rate=10 ost.OSS.ost_io.nrs_tbf_rule=start ruleall0 projid={200} rate=15 ost.OSS.ost_io.nrs_tbf_rule=start ruleall1 projid={201} rate=15 ost.OSS.ost_io.nrs_tbf_rule=start ruleall2 projid={202} rate=15 ost.OSS.ost_io.nrs_tbf_rule=start ruleall3 projid={203} rate=15 ost.OSS.ost_io.nrs_tbf_rule=start ruleall4 projid={204} rate=15 ost.OSS.ost_io.nrs_tbf_rule= regular_requests: CPT 0: ruleall4 projid={204} 15, seq 16, type=projid, ref 0 ruleall3 projid={203} 15, seq 15, type=projid, ref 0 ruleall2 projid={202} 15, seq 14, type=projid, ref 0 ruleall1 projid={201} 15, seq 13, type=projid, ref 0 ruleall0 projid={200} 15, seq 12, type=projid, ref 0 rulewr4 projid={104}&opcode={ost_write} 10, seq 11, type=opcode+projid, ref 0 rulerd4 projid={104}&opcode={ost_read} 5, seq 10, type=opcode+projid, ref 0 rulewr3 projid={103}&opcode={ost_write} 10, seq 9, type=opcode+projid, ref 0 rulerd3 projid={103}&opcode={ost_read} 5, seq 8, type=opcode+projid, ref 0 rulewr2 projid={102}&opcode={ost_write} 10, seq 7, type=opcode+projid, ref 0 rulerd2 projid={102}&opcode={ost_read} 5, seq 6, type=opcode+projid, ref 0 rulewr1 projid={101}&opcode={ost_write} 10, seq 5, type=opcode+projid, ref 0 rulerd1 projid={101}&opcode={ost_read} 5, seq 4, type=opcode+projid, ref 0 rulewr0 projid={100}&opcode={ost_write} 10, seq 3, type=opcode+projid, ref 0 rulerd0 projid={100}&opcode={ost_read} 5, seq 2, type=opcode+projid, ref 0 default * 10000, seq 1, type=nid+opcode+projid, ref 0 high_priority_requests: CPT 0: ruleall4 projid={204} 15, seq 16, type=projid, ref 0 ruleall3 projid={203} 15, seq 15, type=projid, ref 0 ruleall2 projid={202} 15, seq 14, type=projid, ref 0 ruleall1 projid={201} 15, seq 13, type=projid, ref 0 ruleall0 projid={200} 15, seq 12, type=projid, ref 0 rulewr4 projid={104}&opcode={ost_write} 10, seq 11, type=opcode+projid, ref 0 rulerd4 projid={104}&opcode={ost_read} 5, seq 10, type=opcode+projid, ref 0 rulewr3 projid={103}&opcode={ost_write} 10, seq 9, type=opcode+projid, ref 0 rulerd3 projid={103}&opcode={ost_read} 5, seq 8, type=opcode+projid, ref 0 rulewr2 projid={102}&opcode={ost_write} 10, seq 7, type=opcode+projid, ref 0 rulerd2 projid={102}&opcode={ost_read} 5, seq 6, type=opcode+projid, ref 0 rulewr1 projid={101}&opcode={ost_write} 10, seq 5, type=opcode+projid, ref 0 rulerd1 projid={101}&opcode={ost_read} 5, seq 4, type=opcode+projid, ref 0 rulewr0 projid={100}&opcode={ost_write} 10, seq 3, type=opcode+projid, ref 0 rulerd0 projid={100}&opcode={ost_read} 5, seq 2, type=opcode+projid, ref 0 default * 10000, seq 1, type=nid+opcode+projid, ref 0 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.151221 s, 111 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0234821 s, 44.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0310473 s, 33.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0502924 s, 20.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0420504 s, 24.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0533008 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0817753 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0760307 s, 13.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0759727 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0936802 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0819009 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0816847 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0992998 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121073 s, 8.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0914561 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109058 s, 9.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111389 s, 9.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9a1004199800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9a1004199800.lock_unused_count=7 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00909274 s, 115 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156734 s, 66.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0194253 s, 54.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0185698 s, 56.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170911 s, 61.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153766 s, 68.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0186446 s, 56.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141571 s, 74.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0227099 s, 46.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.024008 s, 43.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0237393 s, 44.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.017999 s, 58.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0225155 s, 46.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0227926 s, 46.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00347747 s, 302 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00564608 s, 186 MB/s removed '/mnt/lustre/d77td.sanityn/nrs_w_oleg442-client.virtnet' removed '/mnt/lustre/d77td.sanityn/nrs_r_oleg442-client.virtnet' removed directory '/mnt/lustre/d77td.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 5, bs: 1048576 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.77275 s, 10.7 MB/s Write runtime is 11 s, speed is 9.090909 vs. 10 Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.4384 s, 5.4 MB/s Read runtime is 21 s, speed is 4.761904 vs. 5 cleanup_tbf /mnt/lustre/d77td.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 15, read rate: 15, bs: 1048576 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.56237 s, 16.0 MB/s Write runtime is 8 s, speed is 12.500000 vs. 15 Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.63335 s, 15.8 MB/s Read runtime is 8 s, speed is 12.500000 vs. 15 cleanup_tbf /mnt/lustre/d77td.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=change default rate=15 ost.OSS.ost_io.nrs_tbf_rule= regular_requests: CPT 0: ruleall4 projid={204} 15, seq 16, type=projid, ref 0 ruleall3 projid={203} 15, seq 15, type=projid, ref 0 ruleall2 projid={202} 15, seq 14, type=projid, ref 0 ruleall1 projid={201} 15, seq 13, type=projid, ref 0 ruleall0 projid={200} 15, seq 12, type=projid, ref 1 rulewr4 projid={104}&opcode={ost_write} 10, seq 11, type=opcode+projid, ref 0 rulerd4 projid={104}&opcode={ost_read} 5, seq 10, type=opcode+projid, ref 0 rulewr3 projid={103}&opcode={ost_write} 10, seq 9, type=opcode+projid, ref 0 rulerd3 projid={103}&opcode={ost_read} 5, seq 8, type=opcode+projid, ref 0 rulewr2 projid={102}&opcode={ost_write} 10, seq 7, type=opcode+projid, ref 0 rulerd2 projid={102}&opcode={ost_read} 5, seq 6, type=opcode+projid, ref 0 rulewr1 projid={101}&opcode={ost_write} 10, seq 5, type=opcode+projid, ref 0 rulerd1 projid={101}&opcode={ost_read} 5, seq 4, type=opcode+projid, ref 0 rulewr0 projid={100}&opcode={ost_write} 10, seq 3, type=opcode+projid, ref 1 rulerd0 projid={100}&opcode={ost_read} 5, seq 2, type=opcode+projid, ref 1 default * 15, seq 1, type=nid+opcode+projid, ref 2 high_priority_requests: CPT 0: ruleall4 projid={204} 15, seq 16, type=projid, ref 0 ruleall3 projid={203} 15, seq 15, type=projid, ref 0 ruleall2 projid={202} 15, seq 14, type=projid, ref 0 ruleall1 projid={201} 15, seq 13, type=projid, ref 0 ruleall0 projid={200} 15, seq 12, type=projid, ref 0 rulewr4 projid={104}&opcode={ost_write} 10, seq 11, type=opcode+projid, ref 0 rulerd4 projid={104}&opcode={ost_read} 5, seq 10, type=opcode+projid, ref 0 rulewr3 projid={103}&opcode={ost_write} 10, seq 9, type=opcode+projid, ref 0 rulerd3 projid={103}&opcode={ost_read} 5, seq 8, type=opcode+projid, ref 0 rulewr2 projid={102}&opcode={ost_write} 10, seq 7, type=opcode+projid, ref 0 rulerd2 projid={102}&opcode={ost_read} 5, seq 6, type=opcode+projid, ref 0 rulewr1 projid={101}&opcode={ost_write} 10, seq 5, type=opcode+projid, ref 0 rulerd1 projid={101}&opcode={ost_read} 5, seq 4, type=opcode+projid, ref 0 rulewr0 projid={100}&opcode={ost_write} 10, seq 3, type=opcode+projid, ref 0 rulerd0 projid={100}&opcode={ost_read} 5, seq 2, type=opcode+projid, ref 0 default * 15, seq 1, type=nid+opcode+projid, ref 1 not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 15, read rate: 15, bs: 1048576 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.53212 s, 16.1 MB/s Write runtime is 7 s, speed is 14.285714 vs. 15 Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.49846 s, 16.1 MB/s Read runtime is 8 s, speed is 12.500000 vs. 15 cleanup_tbf /mnt/lustre/d77td.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=fifo PASS 77td (276s) == sanityn test 77tf: TBF: rule churn under load on a dynamic classify policy ========================================================== 18:59:45 (1789772385) jobid_name=%e.%p jobid_var=TEST_77TF_JOBVAR ost.OSS.ost_io.nrs_policies=tbf jobid+opcode classify=dynamic ost.OSS.ost_io.nrs_tbf_rule=start churn1 jobid={dd.*} rate=2000 ost.OSS.ost_io.nrs_tbf_rule=start churn2 jobid={dd.*}&opcode={ost_write} rate=2000 ost.OSS.ost_io.nrs_tbf_rule=stop churn2 ost.OSS.ost_io.nrs_tbf_rule=stop churn1 ost.OSS.ost_io.nrs_tbf_rule=start churn1 jobid={dd.*} rate=2000 ost.OSS.ost_io.nrs_tbf_rule=start churn2 jobid={dd.*}&opcode={ost_write} rate=2000 ost.OSS.ost_io.nrs_tbf_rule=stop churn2 ost.OSS.ost_io.nrs_tbf_rule=stop churn1 ost.OSS.ost_io.nrs_tbf_rule=start churn1 jobid={dd.*} rate=2000 ost.OSS.ost_io.nrs_tbf_rule=start churn2 jobid={dd.*}&opcode={ost_write} rate=2000 ost.OSS.ost_io.nrs_tbf_rule=stop churn2 ost.OSS.ost_io.nrs_tbf_rule=stop churn1 ost.OSS.ost_io.nrs_tbf_rule=start churn1 jobid={dd.*} rate=2000 ost.OSS.ost_io.nrs_tbf_rule=start churn2 jobid={dd.*}&opcode={ost_write} rate=2000 ost.OSS.ost_io.nrs_tbf_rule=stop churn2 ost.OSS.ost_io.nrs_tbf_rule=stop churn1 ost.OSS.ost_io.nrs_tbf_rule=start churn1 jobid={dd.*} rate=2000 ost.OSS.ost_io.nrs_tbf_rule=start churn2 jobid={dd.*}&opcode={ost_write} rate=2000 ost.OSS.ost_io.nrs_tbf_rule=stop churn2 ost.OSS.ost_io.nrs_tbf_rule=stop churn1 ost.OSS.ost_io.nrs_tbf_rule=start churn1 jobid={dd.*} rate=2000 ost.OSS.ost_io.nrs_tbf_rule=start churn2 jobid={dd.*}&opcode={ost_write} rate=2000 ost.OSS.ost_io.nrs_tbf_rule=stop churn2 ost.OSS.ost_io.nrs_tbf_rule=stop churn1 ost.OSS.ost_io.nrs_tbf_rule=start churn1 jobid={dd.*} rate=2000 ost.OSS.ost_io.nrs_tbf_rule=start churn2 jobid={dd.*}&opcode={ost_write} rate=2000 ost.OSS.ost_io.nrs_tbf_rule=stop churn2 ost.OSS.ost_io.nrs_tbf_rule=stop churn1 ost.OSS.ost_io.nrs_tbf_rule=start churn1 jobid={dd.*} rate=2000 ost.OSS.ost_io.nrs_tbf_rule=start churn2 jobid={dd.*}&opcode={ost_write} rate=2000 ost.OSS.ost_io.nrs_tbf_rule=stop churn2 ost.OSS.ost_io.nrs_tbf_rule=stop churn1 ost.OSS.ost_io.nrs_tbf_rule=start churn1 jobid={dd.*} rate=2000 ost.OSS.ost_io.nrs_tbf_rule=start churn2 jobid={dd.*}&opcode={ost_write} rate=2000 ost.OSS.ost_io.nrs_tbf_rule=stop churn2 ost.OSS.ost_io.nrs_tbf_rule=stop churn1 ost.OSS.ost_io.nrs_tbf_rule=start churn1 jobid={dd.*} rate=2000 ost.OSS.ost_io.nrs_tbf_rule=start churn2 jobid={dd.*}&opcode={ost_write} rate=2000 ost.OSS.ost_io.nrs_tbf_rule=stop churn2 ost.OSS.ost_io.nrs_tbf_rule=stop churn1 Cleanup TBF ost.OSS.ost_io.nrs_tbf_rule=stop churn1 ost.OSS.ost_io.nrs_tbf_rule=stop churn2 ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid jobid_name=%e.%u PASS 77tf (19s) == sanityn test 77te: Dynamic type lookup should obey the rule order ========================================================== 19:00:04 (1789772404) using SAVE_PROJECT_SUPPORTED=0 ost.OSS.ost_io.nrs_policies=tbf projid classify=dynamic allow=projid+opcode allow=uid ost.OSS.ost_io.nrs_tbf_allowed_types= regular_allowed_types: CPT 0: 0: 0x00000024 opcode+projid 1: 0x00000020 projid 2: 0x00000008 uid high_priority_allowed_types: CPT 0: 0: 0x00000024 opcode+projid 1: 0x00000020 projid 2: 0x00000008 uid ost.OSS.ost_io.nrs_tbf_rule=start rule1 projid={100}&opcode={ost_read} rate=10 ost.OSS.ost_io.nrs_tbf_rule=start rule2 projid={100}&opcode={ost_write} rate=10 ost.OSS.ost_io.nrs_tbf_rule=start rule3 uid={500} rate=5 regular_requests: CPT 0: rule3 uid={500} 5, seq 4, type=uid, ref 0 rule2 projid={100}&opcode={ost_write} 10, seq 3, type=opcode+projid, ref 0 rule1 projid={100}&opcode={ost_read} 10, seq 2, type=opcode+projid, ref 0 default * 10000, seq 1, type=opcode+uid+projid, ref 0 high_priority_requests: CPT 0: rule3 uid={500} 5, seq 4, type=uid, ref 0 rule2 projid={100}&opcode={ost_write} 10, seq 3, type=opcode+projid, ref 0 rule1 projid={100}&opcode={ost_read} 10, seq 2, type=opcode+projid, ref 0 default * 10000, seq 1, type=opcode+uid+projid, ref 0 not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10, bs: 1048576 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.77452 s, 10.7 MB/s Write runtime is 11 s, speed is 9.090909 vs. 10 Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73012 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 vs. 10 cleanup_tbf /mnt/lustre/d77te.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5, bs: 1048576 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77te.sanityn/tbf] [bs=1048576] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.4708 s, 5.4 MB/s Write runtime is 20 s, speed is 5.000000 vs. 5 Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77te.sanityn/tbf] [of=/dev/null] [bs=1048576] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9512 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 vs. 5 cleanup_tbf /mnt/lustre/d77te.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_policies=fifo PASS 77te (110s) == sanityn test 77tg: TBF: the default rule holds its rate with many rules ========================================================== 19:01:54 (1789772514) ost.OSS.ost_io.nrs_policies=tbf uid classify=dynamic allow=uid ost.OSS.ost_io.nrs_tbf_rule=start r5000 uid={5000} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5001 uid={5001} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5002 uid={5002} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5003 uid={5003} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5004 uid={5004} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5005 uid={5005} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5006 uid={5006} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5007 uid={5007} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5008 uid={5008} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5009 uid={5009} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5010 uid={5010} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5011 uid={5011} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5012 uid={5012} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5013 uid={5013} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5014 uid={5014} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5015 uid={5015} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5016 uid={5016} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5017 uid={5017} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5018 uid={5018} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5019 uid={5019} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5020 uid={5020} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5021 uid={5021} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5022 uid={5022} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5023 uid={5023} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5024 uid={5024} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5025 uid={5025} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5026 uid={5026} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5027 uid={5027} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5028 uid={5028} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5029 uid={5029} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5030 uid={5030} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5031 uid={5031} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5032 uid={5032} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5033 uid={5033} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5034 uid={5034} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5035 uid={5035} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5036 uid={5036} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5037 uid={5037} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5038 uid={5038} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5039 uid={5039} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5040 uid={5040} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5041 uid={5041} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5042 uid={5042} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5043 uid={5043} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5044 uid={5044} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5045 uid={5045} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5046 uid={5046} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5047 uid={5047} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5048 uid={5048} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5049 uid={5049} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5050 uid={5050} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5051 uid={5051} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5052 uid={5052} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5053 uid={5053} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5054 uid={5054} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5055 uid={5055} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5056 uid={5056} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5057 uid={5057} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5058 uid={5058} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5059 uid={5059} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5060 uid={5060} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5061 uid={5061} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5062 uid={5062} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5063 uid={5063} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5064 uid={5064} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5065 uid={5065} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5066 uid={5066} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5067 uid={5067} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5068 uid={5068} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5069 uid={5069} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5070 uid={5070} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5071 uid={5071} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5072 uid={5072} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5073 uid={5073} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5074 uid={5074} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5075 uid={5075} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5076 uid={5076} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5077 uid={5077} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5078 uid={5078} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5079 uid={5079} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5080 uid={5080} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5081 uid={5081} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5082 uid={5082} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5083 uid={5083} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5084 uid={5084} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5085 uid={5085} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5086 uid={5086} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5087 uid={5087} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5088 uid={5088} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5089 uid={5089} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5090 uid={5090} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5091 uid={5091} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5092 uid={5092} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5093 uid={5093} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5094 uid={5094} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5095 uid={5095} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5096 uid={5096} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5097 uid={5097} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5098 uid={5098} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5099 uid={5099} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5100 uid={5100} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5101 uid={5101} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5102 uid={5102} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5103 uid={5103} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5104 uid={5104} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5105 uid={5105} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5106 uid={5106} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5107 uid={5107} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5108 uid={5108} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5109 uid={5109} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5110 uid={5110} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5111 uid={5111} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5112 uid={5112} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5113 uid={5113} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5114 uid={5114} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5115 uid={5115} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5116 uid={5116} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5117 uid={5117} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5118 uid={5118} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5119 uid={5119} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5120 uid={5120} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5121 uid={5121} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5122 uid={5122} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5123 uid={5123} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5124 uid={5124} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5125 uid={5125} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5126 uid={5126} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5127 uid={5127} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5128 uid={5128} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5129 uid={5129} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5130 uid={5130} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5131 uid={5131} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5132 uid={5132} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5133 uid={5133} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5134 uid={5134} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5135 uid={5135} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5136 uid={5136} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5137 uid={5137} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5138 uid={5138} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5139 uid={5139} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5140 uid={5140} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5141 uid={5141} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5142 uid={5142} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5143 uid={5143} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5144 uid={5144} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5145 uid={5145} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5146 uid={5146} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5147 uid={5147} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5148 uid={5148} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5149 uid={5149} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5150 uid={5150} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5151 uid={5151} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5152 uid={5152} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5153 uid={5153} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5154 uid={5154} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5155 uid={5155} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5156 uid={5156} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5157 uid={5157} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5158 uid={5158} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5159 uid={5159} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5160 uid={5160} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5161 uid={5161} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5162 uid={5162} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5163 uid={5163} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5164 uid={5164} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5165 uid={5165} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5166 uid={5166} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5167 uid={5167} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5168 uid={5168} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5169 uid={5169} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5170 uid={5170} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5171 uid={5171} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5172 uid={5172} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5173 uid={5173} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5174 uid={5174} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5175 uid={5175} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5176 uid={5176} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5177 uid={5177} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5178 uid={5178} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5179 uid={5179} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5180 uid={5180} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5181 uid={5181} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5182 uid={5182} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5183 uid={5183} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5184 uid={5184} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5185 uid={5185} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5186 uid={5186} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5187 uid={5187} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5188 uid={5188} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5189 uid={5189} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5190 uid={5190} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5191 uid={5191} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5192 uid={5192} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5193 uid={5193} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5194 uid={5194} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5195 uid={5195} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5196 uid={5196} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5197 uid={5197} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5198 uid={5198} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5199 uid={5199} rate=1000 not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10, bs: 1048576 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.78263 s, 10.7 MB/s Write runtime is 11 s, speed is 9.090909 vs. 10 Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.95384 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 vs. 10 cleanup_tbf /mnt/lustre/d77tg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Cleanup TBF ost.OSS.ost_io.nrs_policies=fifo PASS 77tg (48s) == sanityn test 77ua: Setting TBF rate performance mode: BPS and IOPS ========================================================== 19:02:42 (1789772562) ost.OSS.ost_io.nrs_policies=tbf nid+opcode ost.OSS.ost_io.nrs_policies= regular_requests: - name: fifo state: started fallback: yes queued: 0 active: 0 - name: crrn state: stopped fallback: no queued: 0 active: 0 - name: orr state: stopped fallback: no queued: 0 active: 0 - name: trr state: stopped fallback: no queued: 0 active: 0 - name: tbf nid+opcode state: started fallback: no queued: 0 active: 0 - name: delay state: stopped fallback: no queued: 0 active: 0 high_priority_requests: - name: fifo state: started fallback: yes queued: 0 active: 0 - name: crrn state: stopped fallback: no queued: 0 active: 0 - name: orr state: stopped fallback: no queued: 0 active: 0 - name: trr state: stopped fallback: no queued: 0 active: 0 - name: tbf nid+opcode state: started fallback: no queued: 0 active: 0 - name: delay state: stopped fallback: no queued: 0 active: 0 ost.OSS.ost_io.nrs_tbf_rule=start nidrd nid={0@lo 192.168.204.42@tcp}&opcode={ost_read} bps=5MiB/s ost.OSS.ost_io.nrs_tbf_rule=start nidwr nid={0@lo 192.168.204.42@tcp}&opcode={ost_write} bps=15MiB/s regular_requests: CPT 0: nidwr nid={0@lo 192.168.204.42@tcp}&opcode={ost_write} 15, seq 3, type=nid+opcode, ref 0 bps=15728640 nidrd nid={0@lo 192.168.204.42@tcp}&opcode={ost_read} 5, seq 2, type=nid+opcode, ref 0 bps=5242880 default * 10000, seq 1, type=nid+opcode, ref 0 high_priority_requests: CPT 0: nidwr nid={0@lo 192.168.204.42@tcp}&opcode={ost_write} 15, seq 3, type=nid+opcode, ref 0 bps=15728640 nidrd nid={0@lo 192.168.204.42@tcp}&opcode={ost_read} 5, seq 2, type=nid+opcode, ref 0 bps=5242880 default * 10000, seq 1, type=nid+opcode, ref 0 ost.OSS.ost_io.nrs_tbf_rule=stop nidrd ost.OSS.ost_io.nrs_tbf_rule=stop nidwr ost.OSS.ost_io.nrs_tbf_rule=start nidrd nid={0@lo 192.168.204.42@tcp}&opcode={ost_read} bps=500K ost.OSS.ost_io.nrs_tbf_rule=start nidwr nid={0@lo 192.168.204.42@tcp}&opcode={ost_write} bps=750K regular_requests: CPT 0: nidwr nid={0@lo 192.168.204.42@tcp}&opcode={ost_write} 1, seq 5, type=nid+opcode, ref 0 bps=768000 nidrd nid={0@lo 192.168.204.42@tcp}&opcode={ost_read} 1, seq 4, type=nid+opcode, ref 0 bps=512000 default * 10000, seq 1, type=nid+opcode, ref 0 high_priority_requests: CPT 0: nidwr nid={0@lo 192.168.204.42@tcp}&opcode={ost_write} 1, seq 5, type=nid+opcode, ref 0 bps=768000 nidrd nid={0@lo 192.168.204.42@tcp}&opcode={ost_read} 1, seq 4, type=nid+opcode, ref 0 bps=512000 default * 10000, seq 1, type=nid+opcode, ref 0 ost.OSS.ost_io.nrs_tbf_rule=stop nidrd ost.OSS.ost_io.nrs_tbf_rule=stop nidwr ost.OSS.ost_io.nrs_tbf_rule=start nidrd nid={0@lo 192.168.204.42@tcp}&opcode={ost_read} rate=5MB/s ost.OSS.ost_io.nrs_tbf_rule=start nidwr nid={0@lo 192.168.204.42@tcp}&opcode={ost_write} rate=15MB/s regular_requests: CPT 0: nidwr nid={0@lo 192.168.204.42@tcp}&opcode={ost_write} 14, seq 7, type=nid+opcode, ref 0 bps=15000000 nidrd nid={0@lo 192.168.204.42@tcp}&opcode={ost_read} 4, seq 6, type=nid+opcode, ref 0 bps=5000000 default * 10000, seq 1, type=nid+opcode, ref 0 high_priority_requests: CPT 0: nidwr nid={0@lo 192.168.204.42@tcp}&opcode={ost_write} 14, seq 7, type=nid+opcode, ref 0 bps=15000000 nidrd nid={0@lo 192.168.204.42@tcp}&opcode={ost_read} 4, seq 6, type=nid+opcode, ref 0 bps=5000000 default * 10000, seq 1, type=nid+opcode, ref 0 ost.OSS.ost_io.nrs_tbf_rule=stop nidrd ost.OSS.ost_io.nrs_tbf_rule=stop nidwr ost.OSS.ost_io.nrs_tbf_rule=start nidrd nid={0@lo 192.168.204.42@tcp}&opcode={ost_read} rate=5MB ost.OSS.ost_io.nrs_tbf_rule=start nidwr nid={0@lo 192.168.204.42@tcp}&opcode={ost_write} rate=15MB regular_requests: CPT 0: nidwr nid={0@lo 192.168.204.42@tcp}&opcode={ost_write} 14, seq 9, type=nid+opcode, ref 0 bps=15000000 nidrd nid={0@lo 192.168.204.42@tcp}&opcode={ost_read} 4, seq 8, type=nid+opcode, ref 0 bps=5000000 default * 10000, seq 1, type=nid+opcode, ref 0 high_priority_requests: CPT 0: nidwr nid={0@lo 192.168.204.42@tcp}&opcode={ost_write} 14, seq 9, type=nid+opcode, ref 0 bps=15000000 nidrd nid={0@lo 192.168.204.42@tcp}&opcode={ost_read} 4, seq 8, type=nid+opcode, ref 0 bps=5000000 default * 10000, seq 1, type=nid+opcode, ref 0 ost.OSS.ost_io.nrs_tbf_rule=stop nidrd ost.OSS.ost_io.nrs_tbf_rule=stop nidwr ost.OSS.ost_io.nrs_tbf_rule=start nidrd nid={0@lo 192.168.204.42@tcp}&opcode={ost_read} bps=5 ost.OSS.ost_io.nrs_tbf_rule=start nidwr nid={0@lo 192.168.204.42@tcp}&opcode={ost_write} bps=15 regular_requests: CPT 0: nidwr nid={0@lo 192.168.204.42@tcp}&opcode={ost_write} 15, seq 11, type=nid+opcode, ref 0 bps=15728640 nidrd nid={0@lo 192.168.204.42@tcp}&opcode={ost_read} 5, seq 10, type=nid+opcode, ref 0 bps=5242880 default * 10000, seq 1, type=nid+opcode, ref 0 high_priority_requests: CPT 0: nidwr nid={0@lo 192.168.204.42@tcp}&opcode={ost_write} 15, seq 11, type=nid+opcode, ref 0 bps=15728640 nidrd nid={0@lo 192.168.204.42@tcp}&opcode={ost_read} 5, seq 10, type=nid+opcode, ref 0 bps=5242880 default * 10000, seq 1, type=nid+opcode, ref 0 ost.OSS.ost_io.nrs_tbf_rule=stop nidrd ost.OSS.ost_io.nrs_tbf_rule=stop nidwr ost.OSS.ost_io.nrs_tbf_rule=start nidrd nid={0@lo 192.168.204.42@tcp}&opcode={ost_read} bps=5 iops=1000 ost.OSS.ost_io.nrs_tbf_rule=start nidwr nid={0@lo 192.168.204.42@tcp}&opcode={ost_write} bps=15 iops=1200 regular_requests: CPT 0: nidwr nid={0@lo 192.168.204.42@tcp}&opcode={ost_write} 1200, seq 13, type=nid+opcode, ref 0 bps=15728640 nidrd nid={0@lo 192.168.204.42@tcp}&opcode={ost_read} 1000, seq 12, type=nid+opcode, ref 0 bps=5242880 default * 10000, seq 1, type=nid+opcode, ref 0 high_priority_requests: CPT 0: nidwr nid={0@lo 192.168.204.42@tcp}&opcode={ost_write} 1200, seq 13, type=nid+opcode, ref 0 bps=15728640 nidrd nid={0@lo 192.168.204.42@tcp}&opcode={ost_read} 1000, seq 12, type=nid+opcode, ref 0 bps=5242880 default * 10000, seq 1, type=nid+opcode, ref 0 ost.OSS.ost_io.nrs_tbf_rule=stop nidrd ost.OSS.ost_io.nrs_tbf_rule=stop nidwr ost.OSS.ost_io.nrs_policies=fifo PASS 77ua (10s) == sanityn test 77ub: Testing for TBF rate control in BPS performance mode ========================================================== 19:02:52 (1789772572) ost.OSS.ost_io.nrs_policies=tbf nid+opcode ost.OSS.ost_io.nrs_tbf_rule=start nidrd nid={0@lo 192.168.204.42@tcp}&opcode={ost_read} bps=5 ost.OSS.ost_io.nrs_tbf_rule=start nidwr nid={0@lo 192.168.204.42@tcp}&opcode={ost_write} bps=15 ost.OSS.ost_io.nrs_tbf_rule= regular_requests: CPT 0: nidwr nid={0@lo 192.168.204.42@tcp}&opcode={ost_write} 15, seq 3, type=nid+opcode, ref 0 bps=15728640 nidrd nid={0@lo 192.168.204.42@tcp}&opcode={ost_read} 5, seq 2, type=nid+opcode, ref 0 bps=5242880 default * 10000, seq 1, type=nid+opcode, ref 0 high_priority_requests: CPT 0: nidwr nid={0@lo 192.168.204.42@tcp}&opcode={ost_write} 15, seq 3, type=nid+opcode, ref 0 bps=15728640 nidrd nid={0@lo 192.168.204.42@tcp}&opcode={ost_read} 5, seq 2, type=nid+opcode, ref 0 bps=5242880 default * 10000, seq 1, type=nid+opcode, ref 0 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.496215 s, 33.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0266172 s, 39.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.033858 s, 31.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0443568 s, 23.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0799616 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0821593 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106062 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136441 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135839 s, 7.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136669 s, 7.7 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138085 s, 7.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.219007 s, 4.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21365 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.353124 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.341138 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.429171 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.425438 s, 2.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21908 s, 4.8 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216374 s, 4.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21345 s, 4.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.214091 s, 4.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.218249 s, 4.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220075 s, 4.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22083 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216031 s, 4.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213601 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.219135 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.218555 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.217447 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221786 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.224929 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220427 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.225523 s, 4.6 MB/s removed '/mnt/lustre/d77ub.sanityn/nrs_r_oleg442-client.virtnet' removed '/mnt/lustre/d77ub.sanityn/nrs_w_oleg442-client.virtnet' removed directory '/mnt/lustre/d77ub.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 15, read rate: 5, bs: 4096 Verify the write rate is under TBF control 25600+0 records in 25600+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 261.042 s, 402 kB/s Write runtime is 262 s, speed is .381679 vs. 15 Verify the read rate is under TBF control 25600+0 records in 25600+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 31.215 s, 3.4 MB/s Read runtime is 32 s, speed is 3.125000 vs. 5 cleanup_tbf /mnt/lustre/d77ub.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 15, read rate: 5, bs: 65536 Verify the write rate is under TBF control 1600+0 records in 1600+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.295 s, 5.4 MB/s Write runtime is 20 s, speed is 5.000000 vs. 15 Verify the read rate is under TBF control 1600+0 records in 1600+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.129 s, 5.2 MB/s Read runtime is 21 s, speed is 4.761904 vs. 5 cleanup_tbf /mnt/lustre/d77ub.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 15, read rate: 5, bs: 131072 Verify the write rate is under TBF control 800+0 records in 800+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.88651 s, 10.6 MB/s Write runtime is 11 s, speed is 9.090909 vs. 15 Verify the read rate is under TBF control 800+0 records in 800+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.0691 s, 5.2 MB/s Read runtime is 21 s, speed is 4.761904 vs. 5 cleanup_tbf /mnt/lustre/d77ub.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 15, read rate: 5, bs: 262144 Verify the write rate is under TBF control 400+0 records in 400+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.70094 s, 15.6 MB/s Write runtime is 8 s, speed is 12.500000 vs. 15 Verify the read rate is under TBF control 400+0 records in 400+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.0831 s, 5.2 MB/s Read runtime is 21 s, speed is 4.761904 vs. 5 cleanup_tbf /mnt/lustre/d77ub.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 15, read rate: 5, bs: 1048576 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.68808 s, 15.7 MB/s Write runtime is 8 s, speed is 12.500000 vs. 15 Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9167 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 vs. 5 cleanup_tbf /mnt/lustre/d77ub.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Cleanup TBF ost.OSS.ost_io.nrs_tbf_rule=stop nidrd ost.OSS.ost_io.nrs_tbf_rule=stop nidwr ost.OSS.ost_io.nrs_policies=fifo Cleanup TBF ost.OSS.ost_io.nrs_policies=fifo PASS 77ub (533s) == sanityn test 77uc: TBF: the realtime key does not work with a byte rate ========================================================== 19:11:45 (1789773105) ost.OSS.ost_io.nrs_policies=tbf nid+opcode oleg442-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start rtwr nid={0@lo 192.168.204.42@tcp}&opcode={ost_write} realtime=1 bps=5MiB/s: Operation not supported oleg442-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start rtwr nid={0@lo 192.168.204.42@tcp}&opcode={ost_write} realtime=1 bps=5MiB/s': Operation not supported pdsh@oleg442-client: oleg442-server: ssh exited with exit code 95 ost.OSS.ost_io.nrs_tbf_rule=start rtwr nid={0@lo 192.168.204.42@tcp}&opcode={ost_write} realtime=1 rate=100 ost.OSS.ost_io.nrs_tbf_rule= regular_requests: CPT 0: rtwr nid={0@lo 192.168.204.42@tcp}&opcode={ost_write} 100, seq 2, type=nid+opcode, ref 0 default * 10000, seq 1, type=nid+opcode, ref 0 high_priority_requests: CPT 0: rtwr nid={0@lo 192.168.204.42@tcp}&opcode={ost_write} 100, seq 2, type=nid+opcode, ref 0 default * 10000, seq 1, type=nid+opcode, ref 0 Cleanup TBF ost.OSS.ost_io.nrs_tbf_rule=stop rtwr ost.OSS.ost_io.nrs_policies=fifo PASS 77uc (10s) == sanityn test 77ud: TBF: a rate change on a BPS rule leaves an IOPS rate ========================================================== 19:11:55 (1789773115) ost.OSS.ost_io.nrs_policies=tbf nid+opcode ost.OSS.ost_io.nrs_tbf_rule=start nidrd nid={0@lo 192.168.204.42@tcp}&opcode={ost_read} bps=5MiB/s ost.OSS.ost_io.nrs_tbf_rule=start nidwr nid={0@lo 192.168.204.42@tcp}&opcode={ost_write} bps=5MiB/s ost.OSS.ost_io.nrs_tbf_rule= regular_requests: CPT 0: nidwr nid={0@lo 192.168.204.42@tcp}&opcode={ost_write} 5, seq 3, type=nid+opcode, ref 0 bps=5242880 nidrd nid={0@lo 192.168.204.42@tcp}&opcode={ost_read} 5, seq 2, type=nid+opcode, ref 0 bps=5242880 default * 10000, seq 1, type=nid+opcode, ref 0 high_priority_requests: CPT 0: nidwr nid={0@lo 192.168.204.42@tcp}&opcode={ost_write} 5, seq 3, type=nid+opcode, ref 0 bps=5242880 nidrd nid={0@lo 192.168.204.42@tcp}&opcode={ost_read} 5, seq 2, type=nid+opcode, ref 0 bps=5242880 default * 10000, seq 1, type=nid+opcode, ref 0 ost.OSS.ost_io.nrs_tbf_rule=change nidrd rate=100 ost.OSS.ost_io.nrs_tbf_rule=change nidwr rate=100 ost.OSS.ost_io.nrs_tbf_rule= regular_requests: CPT 0: nidwr nid={0@lo 192.168.204.42@tcp}&opcode={ost_write} 100, seq 3, type=nid+opcode, ref 0 nidrd nid={0@lo 192.168.204.42@tcp}&opcode={ost_read} 100, seq 2, type=nid+opcode, ref 0 default * 10000, seq 1, type=nid+opcode, ref 0 high_priority_requests: CPT 0: nidwr nid={0@lo 192.168.204.42@tcp}&opcode={ost_write} 100, seq 3, type=nid+opcode, ref 0 nidrd nid={0@lo 192.168.204.42@tcp}&opcode={ost_read} 100, seq 2, type=nid+opcode, ref 0 default * 10000, seq 1, type=nid+opcode, ref 0 ost.OSS.ost_io.nrs_tbf_rule=change nidrd rate=20 ost.OSS.ost_io.nrs_tbf_rule=change nidwr rate=20 not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20, bs: 1048576 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.1184 s, 20.5 MB/s Write runtime is 6 s, speed is 16.666666 vs. 20 Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.98861 s, 21.0 MB/s Read runtime is 6 s, speed is 16.666666 vs. 20 cleanup_tbf /mnt/lustre/d77ud.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Cleanup TBF ost.OSS.ost_io.nrs_tbf_rule=stop nidrd ost.OSS.ost_io.nrs_tbf_rule=stop nidwr ost.OSS.ost_io.nrs_policies=fifo PASS 77ud (42s) == sanityn test 77ti: TBF: a late rule reaches a class bucket that exists ========================================================== 19:12:37 (1789773157) ost.OSS.ost_io.nrs_policies=tbf uid classify=dynamic allow=uid ost.OSS.ost_io.nrs_tbf_rule=start r5000 uid={5000} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5001 uid={5001} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5002 uid={5002} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5003 uid={5003} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5004 uid={5004} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5005 uid={5005} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5006 uid={5006} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5007 uid={5007} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5008 uid={5008} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5009 uid={5009} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5010 uid={5010} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5011 uid={5011} rate=1000 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ti.sanityn/nrs_r_oleg442-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.234955 s, 71.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0532167 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0700199 s, 15.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0582096 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0772064 s, 13.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0820019 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0501575 s, 20.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0854497 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0623295 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0929117 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0687405 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0806851 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133788 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148879 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155676 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161053 s, 6.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups:1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138305 s, 7.6 MB/s 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied ldlm.namespaces.lustre-OST0000-osc-ffff9a1004199800.lock_unused_count=5 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00972424 s, 108 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0293264 s, 35.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0188027 s, 55.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.019006 s, 55.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0238066 s, 44.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0248554 s, 42.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0253854 s, 41.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0320907 s, 32.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0201 s, 52.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0202079 s, 51.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0208216 s, 50.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0233556 s, 44.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.023554 s, 44.5 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0244602 s, 42.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0237869 s, 44.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0274821 s, 38.2 MB/s removed '/mnt/lustre/d77ti.sanityn/nrs_w_oleg442-client.virtnet' removed '/mnt/lustre/d77ti.sanityn/nrs_r_oleg442-client.virtnet' removed directory '/mnt/lustre/d77ti.sanityn' ost.OSS.ost_io.nrs_tbf_rule=start late uid={500} rate=2 not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 2, read rate: 2, bs: 1048576 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ti.sanityn/tbf] [bs=1048576] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 48.6956 s, 2.2 MB/s Write runtime is 50 s, speed is 2.000000 vs. 2 Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ti.sanityn/tbf] [of=/dev/null] [bs=1048576] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 49.9435 s, 2.1 MB/s Read runtime is 51 s, speed is 1.960784 vs. 2 cleanup_tbf /mnt/lustre/d77ti.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Cleanup TBF ost.OSS.ost_io.nrs_policies=fifo PASS 77ti (140s) == sanityn test 78: Enable policy and specify tunings right away ========================================================== 19:14:58 (1789773298) oleg442-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_orr_quantum=1: No such device oleg442-server: error: set_param: setting 'ost/OSS/*/nrs_orr_quantum'='1': No such device pdsh@oleg442-client: oleg442-server: ssh exited with exit code 19 ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_policies=fifo PASS 78 (10s) == sanityn test 79: xattr: intent error ================== 19:15:08 (1789773308) File: /mnt/lustre/d79.sanityn Size: 16384 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115272381366439 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-18 19:15:10.000000000 -0400 Modify: 2026-09-18 19:15:10.000000000 -0400 Change: 2026-09-18 19:15:10.000000000 -0400 Birth: 2026-09-18 19:15:10.000000000 -0400 fail_val=0 fail_loc=0x80000160 fail_val=0 fail_loc=0x80000131 # file: mnt/lustre/d79.sanityn trusted.name1="value1" PASS 79 (19s) == sanityn test 80a: migrate directory when some children is being opened ========================================================== 19:15:28 (1789773328) SKIP: sanityn test_80a needs >= 2 MDTs SKIP 80a (3s) == sanityn test 80b: Accessing directory during migration ========================================================== 19:15:32 (1789773332) SKIP: sanityn test_80b needs >= 2 MDTs SKIP 80b (6s) == sanityn test 81a: rename and stat under striped directory ========================================================== 19:15:38 (1789773338) SKIP: sanityn test_81a needs >= 2 MDTs SKIP 81a (7s) == sanityn test 81b: rename under striped directory doesn't deadlock ========================================================== 19:15:46 (1789773346) SKIP: sanityn test_81b We need at least 2 MDTs for this test SKIP 81b (7s) == sanityn test 81c: rename revoke LOOKUP lock for remote object ========================================================== 19:15:53 (1789773353) SKIP: sanityn test_81c needs >= 4 MDTs SKIP 81c (5s) == sanityn test 81d: parallel rename file cross-dir on same MDT ========================================================== 19:15:59 (1789773359) - open/close 654 (time 1789773375.78 total 10.01 last 65.36) - open/close 1114 (time 1789773385.79 total 20.02 last 45.95) - open/close 1618 (time 1789773395.80 total 30.03 last 50.34) - open/close 1991 (time 1789773405.85 total 40.07 last 37.13) - open/close 2355 (time 1789773415.86 total 50.09 last 36.36) total: 2500 open/close in 52.91 seconds: 47.25 ops/second snapshot_time 1789773528.673291391 secs.nsecs start_time 1789773362.746169973 secs.nsecs elapsed_time 165.927121418 secs.nsecs open 2505 samples [usecs] 136 28454 5811047 33419382205 close 2504 samples [usecs] 189 21370 1139563 2382372525 mknod 2501 samples [usecs] 567 24632 4981534 25581194920 unlink 2500 samples [usecs] 751 107740 6917556 104756975150 mkdir 2 samples [usecs] 1443 3781 5224 16378210 rmdir 2 samples [usecs] 7724 179546 187270 32296426292 rename 2500 samples [usecs] 454 73098 21396068 410152594560 getattr 3761 samples [usecs] 10 18916 311332 1074950754 parallel_rename_file 2500 samples [usecs] 454 73098 21396068 410152594560 crossdir_rename 2500 samples [usecs] 454 73098 21396068 410152594560 readpage 23 samples [usecs] 5717 156482 746162 45489610602 PASS 81d (176s) == sanityn test 82: fsetxattr and fgetxattr on orphan files ========================================================== 19:18:55 (1789773535) multiop /mnt/lustre/f82.sanityn vO_ac TMPPIPE=/tmp/multiop_open_wait_pipe.7652 multiop /mnt/lustre2/f82.sanityn vo_Ac TMPPIPE=/tmp/multiop_open_wait_pipe.7652 PASS 82 (8s) == sanityn test 83: access striped directory while it is being created/unlinked ========================================================== 19:19:02 (1789773542) SKIP: sanityn test_83 needs >= 2 MDTs SKIP 83 (4s) == sanityn test 84: 0-nlink race in lu_object_find() ===== 19:19:06 (1789773546) fail_loc=0x8000060b PASS 84 (13s) == sanityn test 85: Lustre API root cache race =========== 19:19:20 (1789773560) Starting: llapi_root_test /mnt/lustre/f85.sanityn /mnt/lustre2/f85.sanityn Starting test test1 at 1789773562 Finishing test test1 at 1789773567 PASS 85 (12s) == sanityn test 90: open/create and unlink striped directory ========================================================== 19:19:32 (1789773572) SKIP: sanityn test_90 needs >= 2 MDTs SKIP 90 (3s) == sanityn test 91: chmod and unlink striped directory === 19:19:35 (1789773575) SKIP: sanityn test_91 needs >= 2 MDTs SKIP 91 (3s) == sanityn test 92: create remote directory under orphan directory ========================================================== 19:19:38 (1789773578) SKIP: sanityn test_92 needs >= 2 MDTs SKIP 92 (3s) == sanityn test 93: alloc_rr should not allocate on same ost ========================================================== 19:19:42 (1789773582) fail_loc=0x00000163 fail_loc=0x0 /mnt/lustre/f93.sanityn-1/file1 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 6285 0x188d 0x280000400 0 6386 0x18f2 0x240000400 /mnt/lustre2/f93.sanityn-2/file2 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 6286 0x188e 0x280000400 0 6387 0x18f3 0x240000400 rc1=0 and rc2=0 PASS 93 (17s) == sanityn test 94: signal vs CP callback race =========== 19:19:58 (1789773598) osc.lustre-OST0000-osc-ffff9a1004199800.idle_timeout=0 osc.lustre-OST0000-osc-ffff9a1004e5e800.idle_timeout=0 osc.lustre-OST0001-osc-ffff9a1004199800.idle_timeout=0 osc.lustre-OST0001-osc-ffff9a1004e5e800.idle_timeout=0 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.317062 s, 25.8 kB/s fail_val=5 fail_loc=0x80000312 fail_val=6 fail_loc=0x80000329 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 3.1061 s, 1.3 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 7277: 324791 Alarm clock dd if=/dev/zero of=$DIR/$tfile conv=notrunc oflag=append bs=4k count=1 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.102054 s, 40.1 kB/s osc.lustre-OST0000-osc-ffff9a1004199800.idle_timeout=debug osc.lustre-OST0000-osc-ffff9a1004e5e800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9a1004199800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9a1004e5e800.idle_timeout=debug PASS 94 (16s) == sanityn test 95a: Check readpage() on a page that was removed from page cache ========================================================== 19:20:14 (1789773614) llite.lustre-ffff9a1004199800.fast_read=0 llite.lustre-ffff9a1004e5e800.fast_read=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0809747 s, 25.9 MB/s fail_loc=0x80001421 fail_val=7 fail_loc=0x80001422 fail_val=10 llite.lustre-ffff9a1004199800.fast_read=1 llite.lustre-ffff9a1004e5e800.fast_read=1 PASS 95a (21s) == sanityn test 95b: Check readpage() on a page that is no longer uptodate ========================================================== 19:20:35 (1789773635) llite.lustre-ffff9a1004199800.fast_read=0 llite.lustre-ffff9a1004e5e800.fast_read=0 1+0 records in 1+0 records out 12288 bytes (12 kB, 12 KiB) copied, 0.0043168 s, 2.8 MB/s fail_loc=0x80001424 fail_val=5 fail_loc=0 llite.lustre-ffff9a1004199800.fast_read=1 llite.lustre-ffff9a1004e5e800.fast_read=1 PASS 95b (14s) == sanityn test 100a: DoM: glimpse RPCs for stat without IO lock (DoM only file) ========================================================== 19:20:49 (1789773649) SKIP: sanityn test_100a Reserved for glimpse-ahead SKIP 100a (3s) == sanityn test 100b: DoM: no glimpse RPC for stat with IO lock (DoM only file) ========================================================== 19:20:52 (1789773652) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00612458 s, 669 kB/s /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK PASS 100b (7s) == sanityn test 100c: DoM: write vs stat without IO lock (combined file) ========================================================== 19:20:59 (1789773659) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.185095 s, 11.3 MB/s /mnt/lustre/d100c.sanityn/dom has type file OK /mnt/lustre/d100c.sanityn/dom has size 2097152 OK PASS 100c (7s) == sanityn test 100d: DoM: write+truncate vs stat without IO lock (combined file) ========================================================== 19:21:06 (1789773666) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.191263 s, 11.0 MB/s /mnt/lustre/d100d.sanityn/dom has type file OK /mnt/lustre/d100d.sanityn/dom has size 4096 OK PASS 100d (8s) == sanityn test 100e: DoM: read on open and file size ==== 19:21:14 (1789773674) 1+0 records in 1+0 records out 12000 bytes (12 kB, 12 KiB) copied, 0.00653497 s, 1.8 MB/s PASS 100e (8s) == sanityn test 101a: Discard DoM data on unlink ========= 19:21:22 (1789773682) /mnt/lustre/f101a.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00993164 s, 412 kB/s PASS 101a (8s) == sanityn test 101b: Discard DoM data on rename ========= 19:21:30 (1789773690) /mnt/lustre/f101b.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0193006 s, 212 kB/s PASS 101b (8s) == sanityn test 101c: Discard DoM data on close-unlink === 19:21:38 (1789773698) /mnt/lustre/f101c.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0142577 s, 287 kB/s PASS 101c (8s) == sanityn test 102: Test open by handle of unlinked file ========================================================== 19:21:46 (1789773706) file: /mnt/lustre/f102.sanityn fh_bytes: 32 fh_type: 151 fh_data: a1 13 00 00 02 00 00 00 88 0a 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 file's parent FID is [0:0x0:0x0] file FID is [0x2000013a1:0xa88:0x0] access via mount point '/mnt/lustre2' - OK access after unlink - OK check_fhandle_syscalls test Passed! PASS 102 (8s) == sanityn test 103: Test size correctness with lockahead ========================================================== 19:21:54 (1789773714) fail_loc=0x415 fail_loc=0x415 fail_val=2 Incorrect size expected (no glimpse fix): Starting test test23 at 1789773717 lockahead_test: lockahead_test.c:1171: test23: assertion 'sb.st_size == sb2.st_size' failed: size on /mnt/lustre/d103.sanityn/f103.sanityn and /mnt/lustre2/d103.sanityn/f103.sanityn differs: 0 vs 1048576 fail_loc=0 fail_loc=0x214 fail_val=2 Starting test test23 at 1789773720 Finishing test test23 at 1789773720 PASS 103 (13s) == sanityn test 104: Verify that MDS stores atime/mtime/ctime during close ========================================================== 19:22:07 (1789773727) SKIP: sanityn test_104 ldiskfs only test SKIP 104 (3s) == sanityn test 105: Glimpse and lock cancel race ======== 19:22:10 (1789773730) fail_loc=0x416 PASS 105 (27s) == sanityn test 106a: Verify the btime via statx() ======= 19:22:37 (1789773757) SKIP: sanityn test_106a Test only for ldiskfs and statx() supported SKIP 106a (3s) == sanityn test 106b: Glimpse RPCs test for statx ======== 19:22:40 (1789773760) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0116562 s, 87.8 kB/s File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115272381368975 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-18 19:22:42.000000000 -0400 Modify: 2026-09-18 19:22:42.000000000 -0400 Change: 2026-09-18 19:22:42.000000000 -0400 Birth: 2026-09-18 19:22:42.000000000 -0400 /mnt/lustre/f106b.sanityn 144115272381368975 -rw-r--r-- 1 0 0 1789773762 1789773762 1789773762 File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115272381368975 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-18 19:22:42.000000000 -0400 Modify: 2026-09-18 19:22:42.000000000 -0400 Change: 2026-09-18 19:22:42.000000000 -0400 Birth: 2026-09-18 19:22:42.000000000 -0400 1789773762 1024 1 PASS 106b (7s) == sanityn test 106c: Verify statx attributes mask ======= 19:22:47 (1789773767) PASS 106c (5s) == sanityn test 107a: Basic grouplock conflict =========== 19:22:53 (1789773773) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.186498 s, 56.2 MB/s /mnt/lustre/f107a.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 6293 0x1895 0x280000400 multiop /mnt/lustre/f107a.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7652 multiop /mnt/lustre2/f107a.sanityn vO_G16022000r10g16022000c TMPPIPE=/tmp/multiop_open_wait_pipe.7652 First grouplock blocks second one PASS 107a (7s) == sanityn test 107b: Grouplock is added to the head of waiting list ========================================================== 19:23:00 (1789773780) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.20286 s, 51.7 MB/s /mnt/lustre/f107b.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 6396 0x18fc 0x240000400 multiop /mnt/lustre/f107b.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7652 Grouplock blocks IO First grouplock blocks second one Second grouplock blocks IO PASS 107b (11s) == sanityn test 108a: lseek: parallel updates ============ 19:23:12 (1789773792) 1+0 records in 1+0 records out 10000 bytes (10 kB, 9.8 KiB) copied, 0.00633806 s, 1.6 MB/s llite.lustre-ffff9a1004199800.hybrid_io=0 llite.lustre-ffff9a1004e5e800.hybrid_io=0 fail_val=4 fail_loc=0x80000414 1+0 records in 1+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 4.5555 s, 1.8 MB/s PASS 108a (11s) == sanityn test 109: Race with several mount instances on 1 node ========================================================== 19:23:22 (1789773802) 192.168.204.142@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg442-client.virtnet /mnt/lustre (opts:) 192.168.204.142@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg442-client.virtnet /mnt/lustre2 (opts:) Starting race between client mounts (duration 30) Iteration 0 Starting client: oleg442-client.virtnet: -o user_xattr,flock 192.168.204.142@tcp:/lustre /mnt/lustre Starting client: oleg442-client.virtnet: -o user_xattr,flock 192.168.204.142@tcp:/lustre /mnt/lustre2 192.168.204.142@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.142@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg442-client.virtnet /mnt/lustre (opts:) Stopping client oleg442-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg442-server' oleg442-server: oleg442-server.virtnet: executing load_modules_local oleg442-server: kmemleak disabled oleg442-server: Loading modules from /home/green/git/lustre-release/lustre oleg442-server: detected 4 online CPUs by sysfs oleg442-server: MODOPTS_LIBCFS= oleg442-server: Force libcfs to create 2 CPU partitions oleg442-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Iteration 1 Starting client: oleg442-client.virtnet: -o user_xattr,flock 192.168.204.142@tcp:/lustre /mnt/lustre Starting client: oleg442-client.virtnet: -o user_xattr,flock 192.168.204.142@tcp:/lustre /mnt/lustre2 192.168.204.142@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.142@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg442-client.virtnet /mnt/lustre (opts:) Stopping client oleg442-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg442-server' oleg442-server: oleg442-server.virtnet: executing load_modules_local oleg442-server: kmemleak disabled oleg442-server: Loading modules from /home/green/git/lustre-release/lustre oleg442-server: detected 4 online CPUs by sysfs oleg442-server: MODOPTS_LIBCFS= oleg442-server: Force libcfs to create 2 CPU partitions oleg442-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Starting client: oleg442-client.virtnet: -o user_xattr,flock 192.168.204.142@tcp:/lustre /mnt/lustre Starting client: oleg442-client.virtnet: -o user_xattr,flock 192.168.204.142@tcp:/lustre /mnt/lustre2 PASS 109 (42s) == sanityn test 110: do not grant another lock on resend ========================================================== 19:24:04 (1789773844) fail_loc=0x525 fail_val=3 fail_loc=0x534 File: /mnt/lustre/f110.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115272414920707 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-18 19:24:05.000000000 -0400 Modify: 2026-09-18 19:24:05.000000000 -0400 Change: 2026-09-18 19:24:05.000000000 -0400 Birth: 2026-09-18 19:24:05.000000000 -0400 fail_loc=0 fail_val=0 Connected clients: oleg442-client.virtnet oleg442-client.virtnet PASS 110 (66s) == sanityn test 111: A racy rename/link an open file should not cause fs corruption ========================================================== 19:25:10 (1789773910) SKIP: sanityn test_111 needs >= 2 MDTs SKIP 111 (2s) == sanityn test 112: update max-inherit in default LMV === 19:25:12 (1789773912) SKIP: sanityn test_112 We need at least 2 MDTs for this test SKIP 112 (1s) == sanityn test 113: check servers of specified fs ======= 19:25:13 (1789773913) lustre-OST0000-osc-ffff9a1010ae1800 active. lustre-OST0001-osc-ffff9a1010ae1800 active. lustre-OST0000-osc-ffff9a1010ae1800 active. lustre-OST0001-osc-ffff9a1010ae1800 active. MGC192.168.204.142@tcp active. PASS 113 (3s) == sanityn test 114: implicit default LMV inherit ======== 19:25:16 (1789773916) SKIP: sanityn test_114 We need at least 2 MDTs for this test SKIP 114 (1s) == sanityn test 115: ldiskfs doesn't check direntry for uniqueness ========================================================== 19:25:17 (1789773917) SKIP: sanityn test_115 ldiskfs only test SKIP 115 (2s) == sanityn test 116: DNE: Set default LMV layout from a remote client ========================================================== 19:25:19 (1789773919) SKIP: sanityn test_116 needs >= 2 MDTs SKIP 116 (1s) == sanityn test 121: trunc append race =================== 19:25:20 (1789773920) 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.011503 s, 1.8 MB/s fail_loc=0x80001436 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0237923 s, 861 kB/s 0000000 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0005000 0000 0000 0000 0000 0000 0000 0000 0000 * 000c800 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0011800 PASS 121 (5s) == sanityn test 122: directory size is consistent across mounts ========================================================== 19:25:25 (1789773925) PASS 122 (4s) == sanityn test 200: service remains healthy while able to process request ========================================================== 19:25:29 (1789773929) at_max=10 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=12 fail_loc=0x1e1 fail_val=20 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0918 s, 0.2 kB/s 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0964 s, 0.2 kB/s 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0865 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0921 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0881 s, 0.2 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0907 s, 0.2 kB/s 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0947 s, 0.2 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0923 s, 0.2 kB/s 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0972 s, 0.2 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1167 s, 0.1 kB/s 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1462 s, 0.1 kB/s 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1291 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1419 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1268 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1301 s, 0.1 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1766 s, 0.1 kB/s4096 bytes (4.1 kB, 4.0 KiB) copied, 40.136 s, 0.1 kB/s 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1686 s, 0.1 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.214 s, 0.1 kB/s 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2367 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2247 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2309 s, 0.1 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2252 s, 0.1 kB/s4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2152 s, 0.1 kB/s at_max=600 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=116 oleg442-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9a100419a800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9a100419a800.ost_server_uuid in IDLE state after 0 sec PASS 200 (68s) cleanup: ====================================================== == sanityn test complete, duration 10570 sec ============= 19:26:37 (1789773997) === sanityn: start cleanup 19:26:38 (1789773998) === Stopping clients: oleg442-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg442-client.virtnet /mnt/lustre2 opts: === sanityn: finish cleanup 19:27:38 (1789774058) ===