-----============= acceptance-small: sanityn ============----- Fri Sep 18 16:31:02 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg447-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanityn.*ex': No such file or directory excepting tests: 27 40a skipping tests SLOW=no: 33a === sanityn: start setup 16:31:27 (1789763487) === Starting client oleg447-client.virtnet: -o user_xattr,flock 192.168.204.147@tcp:/lustre /mnt/lustre2 Mount client oleg447-client.virtnet: mount -t lustre -o user_xattr,flock 192.168.204.147@tcp:/lustre /mnt/lustre2 Started clients oleg447-client.virtnet: 192.168.204.147@tcp:/lustre on /mnt/lustre2 type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) oleg447-client.virtnet: executing check_config_client /mnt/lustre oleg447-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg447-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9e5846a25800.idle_timeout=debug osc.lustre-OST0000-osc-ffff9e58602b1800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9e5846a25800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9e58602b1800.idle_timeout=debug disable quota as required oleg447-server: oleg447-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanityn: finish setup 16:32:08 (1789763528) === 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0383641 s, 27.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7636] == sanityn test 0: do_node_vp() and do_facet_vp() do the right thing ========================================================== 16:32:11 (1789763531) PASS 0 (9s) == sanityn test 1: Check attribute updates on 2 mount points ========================================================== 16:32:20 (1789763540) /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0777 OK /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0666 OK /mnt/lustre/f1.sanityn: absent OK PASS 1 (9s) == sanityn test 2a: check cached attribute updates on 2 mtpt's ================================================================== 16:32:29 (1789763549) -rw-r--r-- 1 root root 0 Sep 18 16:32 /mnt/lustre2/f2a /mnt/lustre/f2a has type file OK /mnt/lustre/f2a has perms 0777 OK PASS 2a (9s) == sanityn test 2b: check cached attribute updates on 2 mtpt's ================================================================== 16:32:38 (1789763558) -rw-r--r-- 1 root root 0 Sep 18 16:32 /mnt/lustre2/f2b /mnt/lustre2/f2b has type file OK /mnt/lustre2/f2b has perms 0777 OK PASS 2b (9s) == sanityn test 2c: check cached attribute updates on 2 mtpt's root ============================================================= 16:32:48 (1789763568) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0777 OK PASS 2c (13s) == sanityn test 2d: check cached attribute updates on 2 mtpt's root ============================================================= 16:33:00 (1789763580) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0755 OK PASS 2d (7s) == sanityn test 2e: check chmod on root is propagated to others ========================================================== 16:33:08 (1789763588) total 1 -rwxrwxrwx 1 root root 0 Sep 18 16:32 f2a -rwxrwxrwx 1 root root 0 Sep 18 16:32 f2b total 1 -rwxrwxrwx 1 root root 0 Sep 18 16:32 f2a -rwxrwxrwx 1 root root 0 Sep 18 16:32 f2b running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre2/f2e.sanityn] [count=1] 1+0 records in 1+0 records out 512 bytes copied, 0.00983929 s, 52.0 kB/s PASS 2e (8s) == sanityn test 2f: check attr/owner updates on DNE with 2 mtpt's ========================================================== 16:33:15 (1789763595) SKIP: sanityn test_2f needs >= 2 MDTs SKIP 2f (5s) == sanityn test 2g: check blocks update on sync write ==== 16:33:20 (1789763600) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.606366 s, 3.5 MB/s /mnt/lustre/f2g.sanityn has 4103 blocks /mnt/lustre2/f2g.sanityn has 4103 blocks PASS 2g (11s) == sanityn test 3: symlink on one mtpt, readlink on another ===================================================================== 16:33:31 (1789763611) PASS 3 (8s) == sanityn test 4: fstat validation on multiple mount points ==================================================================== 16:33:40 (1789763620) PASS 4 (9s) == sanityn test 5: create a file on one mount, truncate it on the other ========================================================== 16:33:49 (1789763629) /mnt/lustre/f5 has type file OK /mnt/lustre/f5 has size 100 OK PASS 5 (7s) == sanityn test 6: remove of open file on other node ============================================================================ 16:33:56 (1789763636) opening writing unlinking /mnt/lustre2/f6.sanityn accessing (1) seeking (1) accessing (2) fstat... reading comparing data truncating seeking (2) writing again seeking (3) reading again comparing data again closing SUCCESS - goto beer PASS 6 (7s) == sanityn test 7: remove of open directory on other node ======================================================================= 16:34:03 (1789763643) creating directory /mnt/lustre/d7 opening directory unlinking /mnt/lustre/d7 Ok, everything goes well. PASS 7 (8s) == sanityn test 8: remove of open special file on other node ==================================================================== 16:34:11 (1789763651) creating special file /mnt/lustre/f8.sanityn opening file unlinking /mnt/lustre/f8.sanityn Ok, everything goes well. PASS 8 (9s) == sanityn test 9a: append of file with sub-page size on multiple mounts ========================================================== 16:34:20 (1789763660) PASS 9a (9s) == sanityn test 9b: append to striped sparse file ======== 16:34:30 (1789763670) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0794544 s, 13.2 MB/s 3+0 records in 3+0 records out 3 bytes copied, 0.0505589 s, 0.1 kB/s Data read (expecting 'foo'): foo PASS 9b (8s) == sanityn test 10a: write of file with sub-page size on multiple mounts ========================================================== 16:34:39 (1789763679) 1+0 records in 1+0 records out 1 byte copied, 0.00929642 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00877973 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0122816 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0108404 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0108869 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0101478 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0100825 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0129517 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00843453 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00934797 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0110523 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00852963 s, 0.1 kB/s PASS 10a (9s) == sanityn test 10b: write of file with sub-page size on multiple mounts ========================================================== 16:34:47 (1789763687) 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.0109473 s, 281 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0540569 s, 75.8 kB/s 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.00032473 s, 9.5 MB/s PASS 10b (8s) == sanityn test 11: execution of file opened for write should return error ============================================================== 16:34:55 (1789763695) multiop /mnt/lustre/d11/f vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7636 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 298: /mnt/lustre2/d11/f: Text file busy PASS 11 (7s) == sanityn test 12: test lock ordering (link, stat, unlink) ========================================================== 16:35:03 (1789763703) warning: '-runtime' deprecated, use '-t runtime' instead - link 287 (time 1789763715.72 total 10.03 last 28.62) - link 586 (time 1789763725.73 total 20.04 last 29.86) - link 843 (time 1789763735.79 total 30.10 last 25.55) - link 1085 (time 1789763745.80 total 40.11 last 24.17) - link 1374 (time 1789763755.85 total 50.16 last 28.77) - link 1656 (time 1789763765.87 total 60.18 last 28.13) - link 1977 (time 1789763775.89 total 70.20 last 32.04) - link 2297 (time 1789763785.93 total 80.24 last 31.88) - link 2670 (time 1789763795.94 total 90.25 last 37.26) total: 3022 link in 100.02 seconds: 30.22 ops/second using seed 2464848499 running for 100 seconds - stat 4390 (time 1789763717 ; total 11 ; last 11) - stat 9796 (time 1789763728 ; total 22 ; last 11) - stat 10000 (time 1789763728 ; total 22 ; last 0) - stat 14061 (time 1789763739 ; total 33 ; last 11) - stat 18835 (time 1789763750 ; total 44 ; last 11) - stat 20000 (time 1789763752 ; total 46 ; last 2) - stat 23336 (time 1789763763 ; total 57 ; last 11) - stat 25909 (time 1789763774 ; total 68 ; last 11) - stat 28278 (time 1789763785 ; total 79 ; last 11) - stat 30000 (time 1789763792 ; total 86 ; last 7) - stat 32610 (time 1789763803 ; total 97 ; last 11) total: 33488 stats in 100 seconds: 334.880005 stats/second - unlinked 0 (time 1789763756 ; total 0 ; last 0) unlink(/mnt/lustre2/lockdir/lockfile3022) error: No such file or directory total: 3021 unlinks in 89 seconds: 33.943821 unlinks/second /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 77: kill: (21448) - No such process /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 78: kill: (21450) - No such process PASS 12 (148s) == sanityn test 13: test directory page revocation ======= 16:37:30 (1789763850) f13 f13 PASS 13 (11s) == sanityn test 14aa: execution of file open for write returns -ETXTBSY ========================================================== 16:37:41 (1789763861) multiop /mnt/lustre/d14aa.sanityn/f14aa.sanityn vOw_c TMPPIPE=/tmp/multiop_open_wait_pipe.7636 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 333: /mnt/lustre2/d14aa.sanityn/f14aa.sanityn: Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 22635 Terminated $MULTIOP_PROG $FILE v$ARGS > $TMPPIPE (wd: ~) PASS 14aa (8s) == sanityn test 14ab: open(RDWR) of executing file returns -ETXTBSY ========================================================== 16:37:49 (1789763869) open(O_RDWR|O_CREAT): Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 23214 Terminated $DIR1/$tdir/sleep 60 (wd: ~) PASS 14ab (7s) == sanityn test 14b: truncate of executing file returns -ETXTBSY ================================================================ 16:37:56 (1789763876) truncate: cannot truncate '/mnt/lustre2/d14b.sanityn/sleep' to length 60: Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 348: 23793 Terminated $DIR1/$tdir/sleep 60 PASS 14b (9s) == sanityn test 14c: open(O_TRUNC) of executing file return -ETXTBSY ============================================================ 16:38:05 (1789763885) cp: cannot create regular file '/mnt/lustre2/d14c.sanityn/sleep': Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 360: 24376 Terminated $DIR1/$tdir/sleep 60 PASS 14c (8s) == sanityn test 14d: chmod of executing file is still possible ================================================================== 16:38:13 (1789763893) chmod /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 371: 24958 Terminated $DIR1/$tdir/sleep 60 PASS 14d (9s) == sanityn test 15: test out-of-space with multiple writers ===================================================================== 16:38:22 (1789763902) PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin: MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky sleep 5 for ZFS MDS Waiting for MDT destroys to complete fallocate on zfs doesn't consume space STRIPECOUNT=2 ORIGFREE=7531520 MAXFREE=800000 SKIP: oos2 test_15 oos2.sh: 7531520KB free > 800000KB, increase MAXFREE (or reduce fs size) sleep 5 for ZFS MDS Waiting for MDT destroys to complete SKIP 15 (55s) == sanityn test 16a: 500 iterations of dual-mount fsx ==== 16:39:17 (1789763957) keep default fallocate mode: 2 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.946595 s, 8.9 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 1.1133 s, 7.5 MB/s Chance of close/open is 1 in 50 Seed set to 3965 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000001[1] 1789763967.261878 write 0x589088 thru 0x59225d (0x91d6 bytes) 000004[1] 1789763967.403692 read 0x276bdb thru 0x27db6b (0x6f91 bytes) 000005[0] 1789763967.445486 write 0x0cbfe1 thru 0x0d89c7 (0xc9e7 bytes) truncating to largest ever: 0x6cf80c 000008[0] 1789763967.642513 trunc from 0x59225e to 0x6cf80b (0x13d5ae bytes) 000010[1] 1789763967.795722 mapwrite 0x552538 thru 0x560676 (0xe13f bytes) 000011[1] 1789763967.931161 mapread 0x3a678f thru 0x3ae4f9 (0x7d6b bytes) 000012[0] 1789763968.027435 write 0x8185ae thru 0x81c381 (0x3dd4 bytes) 000013[0] 1789763968.057495 punch from 0x64d1a5 to 0x65972f (0xc58b bytes) 000014[1] 1789763968.092186 punch from 0x6c4213 to 0x6c69cc (0x27ba bytes) 000015[1] 1789763968.136697 punch from 0x647413 to 0x64d040 (0x5c2e bytes) 000016[1] 1789763968.221314 read 0x75745a thru 0x764958 (0xd4ff bytes) 000017[1] 1789763968.323721 write 0x8ce5c0 thru 0x8d45a3 (0x5fe4 bytes) 000018[1] 1789763968.342021 mapread 0x34a0ac thru 0x350303 (0x6258 bytes) 000019[0] 1789763968.418357 write 0x24e2f2 thru 0x25a262 (0xbf71 bytes) 000020[0] 1789763968.466186 write 0x8ebfbd thru 0x8edeb5 (0x1ef9 bytes) 000022[0] 1789763968.530684 punch from 0x1e58f7 to 0x1f2079 (0xc783 bytes) 000026[0] 1789763968.660371 trunc from 0x8edeb6 to 0x1cc41d (0x721a98 bytes) 000027[1] 1789763968.733490 write 0x2f41cd thru 0x2f4584 (0x3b8 bytes) truncating to largest ever: 0x72cfb3 000028[0] 1789763968.768493 trunc from 0x2f4585 to 0x72cfb2 (0x438a2e bytes) 000029[1] 1789763968.806448 mapread 0x592fd2 thru 0x59f1d2 (0xc201 bytes) 000030[1] 1789763968.898351 punch from 0x07720e to 0x07d857 (0x664a bytes) 000031[0] 1789763968.995382 mapread 0x06a82e thru 0x06bf31 (0x1704 bytes) 000036[1] 1789763969.039698 write 0x41404f thru 0x41bb18 (0x7aca bytes) 000037[1] 1789763969.094225 read 0x5f68f0 thru 0x6052ba (0xe9cb bytes) 000038[0] 1789763969.121193 trunc from 0x72cfb3 to 0x672bd9 (0xba3d9 bytes) 000040[0] 1789763969.252603 mapread 0x13c877 thru 0x149260 (0xc9ea bytes) 000041[0] 1789763969.333129 punch from 0x03cc12 to 0x042514 (0x5903 bytes) 000044[1] 1789763969.432402 mapread 0x6124fa thru 0x62005c (0xdb63 bytes) 000046[1] 1789763969.637599 write 0x0fb38e thru 0x100f78 (0x5beb bytes) 000048[0] 1789763969.739263 trunc from 0x672bda to 0x38daf2 (0x2e50e7 bytes) 000049[0] 1789763969.915047 mapread 0x0d9030 thru 0x0de76c (0x573d bytes) 000051[1] 1789763970.032253 write 0x6a73ad thru 0x6b01e2 (0x8e36 bytes) 000055[0] 1789763970.244073 mapwrite 0x76213b thru 0x76a9e3 (0x88a9 bytes) 000057[0] 1789763970.589388 punch from 0x2d2ab3 to 0x2d5e7b (0x33c9 bytes) 000058[0] 1789763970.596470 trunc from 0x76a9e4 to 0x5e3d2f (0x186cb4 bytes) 000061[1] 1789763970.704076 read 0x27c7c4 thru 0x27dd6d (0x15aa bytes) 000062[0] 1789763970.759116 mapread 0x3af199 thru 0x3b63a3 (0x720b bytes) 000064[1] 1789763970.844407 mapread 0x19fc5e thru 0x1a4b0f (0x4eb2 bytes) 000066[0] 1789763971.073034 read 0x4a242a thru 0x4a9179 (0x6d50 bytes) 000067[1] 1789763971.129366 write 0x9843e6 thru 0x98e61c (0xa237 bytes) 000070[1] 1789763971.391623 mapread 0x7c7591 thru 0x7cfddb (0x884b bytes) 000073[1] 1789763971.709947 mapwrite 0x94dc25 thru 0x95a608 (0xc9e4 bytes) 000075[0] 1789763971.985486 mapwrite 0x674cf0 thru 0x67cee2 (0x81f3 bytes) 000076[0] 1789763972.642609 mapread 0x088e8c thru 0x0917b1 (0x8926 bytes) 000077[0] 1789763972.812472 punch from 0x9227f0 to 0x92cde0 (0xa5f1 bytes) 000079[1] 1789763972.928145 punch from 0x75d3ca to 0x7620d2 (0x4d09 bytes) 000080[0] 1789763972.997002 punch from 0x6a5a17 to 0x6aa7cf (0x4db9 bytes) 000082[1] 1789763973.067898 write 0x74940b thru 0x75918c (0xfd82 bytes) 000083[0] 1789763973.144971 punch from 0x1da2c5 to 0x1e6de4 (0xcb20 bytes) 000085[1] 1789763973.219572 punch from 0x7f1e7d to 0x7f8b01 (0x6c85 bytes) 000086[0] 1789763973.318614 write 0x9ceadf thru 0x9da3b3 (0xb8d5 bytes) 000087[0] 1789763973.372999 write 0x90be5f thru 0x91359a (0x773c bytes) 000088[0] 1789763973.476850 mapwrite 0x56c14b thru 0x57aba8 (0xea5e bytes) 000089[1] 1789763973.717502 punch from 0x2651e0 to 0x26d628 (0x8449 bytes) 000090[0] 1789763973.773333 mapread 0x22c70b thru 0x22f7c6 (0x30bc bytes) 000093[1] 1789763973.933640 mapread 0x2ce6da thru 0x2d1871 (0x3198 bytes) 000094[1] 1789763974.056660 write 0x10a077 thru 0x10ce81 (0x2e0b bytes) 000095[0] 1789763974.153323 read 0x37ef27 thru 0x382b16 (0x3bf0 bytes) 000097[0] 1789763974.230627 mapwrite 0x25bed3 thru 0x26abda (0xed08 bytes) 000098[0] 1789763974.426291 punch from 0x792aea to 0x79d4ed (0xaa04 bytes) 000100[1] 1789763974.493451 write 0x6a59e3 thru 0x6ad8dc (0x7efa bytes) 000103[1] 1789763974.617038 write 0x8a8366 thru 0x8b5a2f (0xd6ca bytes) 000104[1] 1789763974.655174 read 0x0389e5 thru 0x03f3ab (0x69c7 bytes) 000106[1] 1789763974.689620 read 0x7f9049 thru 0x7fcc36 (0x3bee bytes) 000107[1] 1789763974.746690 mapwrite 0x7587a7 thru 0x7595ae (0xe08 bytes) 000108[0] 1789763975.011619 mapwrite 0x3a7288 thru 0x3b15de (0xa357 bytes) 000109[0] 1789763975.425922 read 0x7f1662 thru 0x7ffec6 (0xe865 bytes) 000110[1] 1789763975.706444 mapwrite 0x203ea3 thru 0x20c023 (0x8181 bytes) 000111[0] 1789763975.905130 mapwrite 0x2923c8 thru 0x298207 (0x5e40 bytes) 000113[0] 1789763976.027964 read 0x287f6a thru 0x2903e7 (0x847e bytes) 000115[0] 1789763976.064960 mapwrite 0x25c03f thru 0x2683c0 (0xc382 bytes) 000117[1] 1789763976.150551 read 0x3f839d thru 0x401cb5 (0x9919 bytes) 000118[1] 1789763976.191479 mapread 0x87c8ad thru 0x883c78 (0x73cc bytes) 000121[0] 1789763976.250655 punch from 0x59c8d6 to 0x59f43c (0x2b67 bytes) 000124[1] 1789763976.409514 read 0x7b2617 thru 0x7c0868 (0xe252 bytes) 000125[1] 1789763976.545624 punch from 0x11b376 to 0x1273f4 (0xc07f bytes) 000132[1] 1789763976.650235 mapwrite 0x1979f3 thru 0x19ff25 (0x8533 bytes) 000133[1] 1789763976.796693 mapread 0x2b180b thru 0x2bf247 (0xda3d bytes) 000135[1] 1789763976.887087 punch from 0x2e2665 to 0x2e3364 (0xd00 bytes) 000136[1] 1789763976.927991 mapread 0x9abb06 thru 0x9b0518 (0x4a13 bytes) 000138[1] 1789763977.050252 mapwrite 0x5aff3f thru 0x5b6473 (0x6535 bytes) 000140[0] 1789763977.264541 punch from 0x014490 to 0x0151ff (0xd70 bytes) 000144[0] 1789763977.499688 trunc from 0x9da3b4 to 0x55a0fd (0x4802b6 bytes) 000145[0] 1789763977.570221 mapwrite 0x67ef6f thru 0x688044 (0x90d6 bytes) 000147[1] 1789763977.815772 read 0x5bbc7b thru 0x5bf887 (0x3c0d bytes) 000148[1] 1789763977.849884 write 0x714124 thru 0x71dc25 (0x9b02 bytes) 000149[1] 1789763977.880998 read 0x3a7153 thru 0x3b6fc3 (0xfe71 bytes) 000152[1] 1789763977.936475 write 0x53b2d2 thru 0x53d695 (0x23c4 bytes) 000153[0] 1789763977.954972 write 0x4aeab7 thru 0x4ba358 (0xb8a2 bytes) truncating to largest ever: 0x9437c5 000154[0] 1789763978.022310 trunc from 0x71dc26 to 0x9437c4 (0x225b9f bytes) 000157[1] 1789763978.090163 punch from 0x4948e3 to 0x4a2d8f (0xe4ad bytes) 000158[0] 1789763978.159095 write 0x517884 thru 0x51a4a8 (0x2c25 bytes) 000161[0] 1789763978.240084 mapread 0x174b1e thru 0x181781 (0xcc64 bytes) 000162[1] 1789763978.317690 mapwrite 0x2a83b1 thru 0x2a87cd (0x41d bytes) 000163[1] 1789763978.563252 punch from 0x74e7db to 0x75a55e (0xbd84 bytes) 000164[1] 1789763978.710379 mapread 0x167966 thru 0x169da5 (0x2440 bytes) 000165[0] 1789763978.763275 punch from 0x8f0b2d to 0x8f303a (0x250e bytes) 000167[0] 1789763978.886964 trunc from 0x9437c5 to 0x7de1e0 (0x1655e4 bytes) 000168[0] 1789763978.961821 trunc from 0x7de1e1 to 0x13782a (0x6a69b6 bytes) truncating to largest ever: 0x9c6da2 000171[0] 1789763979.121515 trunc from 0x13782b to 0x9c6da1 (0x88f577 bytes) 000176[0] 1789763979.371976 read 0x8a05e9 thru 0x8ac4f8 (0xbf10 bytes) 000177[0] 1789763979.392399 write 0x3664d7 thru 0x3748aa (0xe3d4 bytes) 000178[1] 1789763979.460306 punch from 0x13b6d3 to 0x13de64 (0x2792 bytes) 000179[0] 1789763979.554241 mapwrite 0x084161 thru 0x084d10 (0xbb0 bytes) 000180[0] 1789763979.759954 punch from 0x34a1a0 to 0x34ef2b (0x4d8c bytes) 000181[1] 1789763979.802723 punch from 0x9c3006 to 0x9c6da1 (0x3d9c bytes) 000183[1] 1789763979.923187 punch from 0x20e5d3 to 0x212758 (0x4186 bytes) 000184[1] 1789763979.983249 mapread 0x389cb5 thru 0x394932 (0xac7e bytes) 000185[0] 1789763980.091618 read 0x049ec4 thru 0x05351a (0x9657 bytes) 000186[1] 1789763980.184442 write 0x8f5fe9 thru 0x902057 (0xc06f bytes) 000188[1] 1789763980.317880 mapread 0x4e5b7f thru 0x4f5204 (0xf686 bytes) 000191[0] 1789763980.519297 write 0x76434e thru 0x774212 (0xfec5 bytes) 000193[1] 1789763980.618083 trunc from 0x9c6da2 to 0x98791c (0x3f485 bytes) 000194[0] 1789763980.716027 mapwrite 0x6f4683 thru 0x70253b (0xdeb9 bytes) 000195[0] 1789763981.008056 write 0x5442d4 thru 0x551422 (0xd14f bytes) 000196[1] 1789763981.039726 mapread 0x040524 thru 0x04904f (0x8b2c bytes) 000200[1] 1789763981.254446 punch from 0x362165 to 0x36c8c5 (0xa761 bytes) 000201[0] 1789763981.281224 trunc from 0x98791d to 0x62e7f2 (0x35912a bytes) 000202[1] 1789763981.363479 trunc from 0x62e7f3 to 0x3590fa (0x2d56f8 bytes) 000205[1] 1789763981.497972 mapread 0x267950 thru 0x26f66c (0x7d1d bytes) 000206[1] 1789763981.549390 read 0x0ffc98 thru 0x106119 (0x6482 bytes) 000208[0] 1789763981.860911 write 0x8a8f2c thru 0x8b6c58 (0xdd2d bytes) 000211[1] 1789763982.113235 mapread 0x2f041d thru 0x2fe5cc (0xe1b0 bytes) 000212[0] 1789763982.402338 punch from 0x4195b1 to 0x4278db (0xe32b bytes) 000213[0] 1789763982.436350 mapread 0x396b3c thru 0x39ebba (0x807f bytes) 000214[1] 1789763982.480599 write 0x040d21 thru 0x042201 (0x14e1 bytes) 000215[1] 1789763982.596041 punch from 0x835093 to 0x838805 (0x3773 bytes) 000217[0] 1789763982.680976 read 0x17f184 thru 0x17f329 (0x1a6 bytes) 000218[0] 1789763982.709199 mapwrite 0x69721e thru 0x6a493b (0xd71e bytes) 000220[0] 1789763983.002128 write 0x62d23c thru 0x63bcd6 (0xea9b bytes) 000221[0] 1789763983.043817 punch from 0x1d9070 to 0x1de12b (0x50bc bytes) 000222[0] 1789763983.081387 mapwrite 0x4f8821 thru 0x4f8b32 (0x312 bytes) 000223[0] 1789763983.200445 read 0x7c3712 thru 0x7cb3ce (0x7cbd bytes) 000224[1] 1789763983.264113 mapread 0x6b4a02 thru 0x6b98ea (0x4ee9 bytes) 000226[1] 1789763983.533275 punch from 0x82dfb5 to 0x8385e6 (0xa632 bytes) 000227[0] 1789763983.584557 mapread 0x71e8a9 thru 0x72182a (0x2f82 bytes) 000228[1] 1789763983.735442 mapread 0x6e7c00 thru 0x6ec3e7 (0x47e8 bytes) 000231[1] 1789763983.849198 mapwrite 0x9e9781 thru 0x9f20d1 (0x8951 bytes) 000232[1] 1789763984.234061 mapread 0x3d9b7f thru 0x3e12c0 (0x7742 bytes) 000234[0] 1789763984.405892 read 0x31fc32 thru 0x325167 (0x5536 bytes) 000235[1] 1789763984.543074 trunc from 0x9f20d2 to 0x5f9eb7 (0x3f821a bytes) 000237[0] 1789763984.659503 mapwrite 0x73901d thru 0x73dd81 (0x4d65 bytes) 000239[0] 1789763984.907344 trunc from 0x73dd82 to 0x7d64b2 (0x98731 bytes) 000242[0] 1789763984.983613 mapwrite 0x0c830d thru 0x0d0a4d (0x8741 bytes) 000244[0] 1789763985.321975 mapread 0x1ad395 thru 0x1b259f (0x520b bytes) 000245[1] 1789763985.373204 read 0x163b50 thru 0x165fe1 (0x2492 bytes) 000246[1] 1789763985.443264 punch from 0x769118 to 0x77621e (0xd107 bytes) 000247[1] 1789763985.517807 punch from 0x4a535c to 0x4a637f (0x1024 bytes) 000249[0] 1789763985.547740 punch from 0x12fc2e to 0x1381a1 (0x8574 bytes) 000250[1] 1789763985.609889 read 0x081f13 thru 0x08fa82 (0xdb70 bytes) 000252[0] 1789763985.673089 mapread 0x01c7ce thru 0x025a3c (0x926f bytes) 000253[0] 1789763985.722404 write 0x3cf975 thru 0x3d3204 (0x3890 bytes) 000255[0] 1789763985.803708 trunc from 0x7d64b3 to 0x2824aa (0x554008 bytes) 000256[0] 1789763985.868767 punch from 0x21cf53 to 0x228424 (0xb4d2 bytes) 000257[0] 1789763985.911526 write 0x2873ef thru 0x2876df (0x2f1 bytes) 000261[1] 1789763985.990421 mapwrite 0x168a38 thru 0x16995d (0xf26 bytes) 000265[0] 1789763986.341326 trunc from 0x2876e0 to 0x560e1b (0x2d973c bytes) 000266[0] 1789763986.452419 mapwrite 0x1204c1 thru 0x1230aa (0x2bea bytes) 000267[1] 1789763986.607366 mapwrite 0x24d35d thru 0x24d4b0 (0x154 bytes) 000268[0] 1789763986.727029 write 0x95f5b1 thru 0x95fbe8 (0x638 bytes) 000270[1] 1789763986.741113 write 0x7b556b thru 0x7c0d22 (0xb7b8 bytes) 000271[0] 1789763986.769189 mapread 0x75e44b thru 0x7618d5 (0x348b bytes) 000272[1] 1789763986.851086 trunc from 0x95fbe9 to 0x087e66 (0x8d7d82 bytes) 000275[1] 1789763987.030015 read 0x02cf25 thru 0x031b84 (0x4c60 bytes) 000276[0] 1789763987.073444 mapwrite 0x0d3f86 thru 0x0d8784 (0x47ff bytes) 000277[0] 1789763987.284435 mapread 0x0222d1 thru 0x02cc11 (0xa941 bytes) 000278[1] 1789763987.337493 read 0x02b469 thru 0x02ebac (0x3744 bytes) 000281[1] 1789763987.488151 trunc from 0x0d8785 to 0x986a9f (0x8ae31b bytes) 000284[1] 1789763987.649906 punch from 0x4dcdad to 0x4e3b2f (0x6d83 bytes) 000285[0] 1789763987.663699 mapwrite 0x8dd917 thru 0x8e5a8e (0x8178 bytes) 000286[1] 1789763987.865331 punch from 0x356099 to 0x35a563 (0x44cb bytes) 000288[1] 1789763987.959482 write 0x1cf514 thru 0x1d399c (0x4489 bytes) 000289[1] 1789763988.050727 trunc from 0x986aa0 to 0x5bc770 (0x3ca32f bytes) 000290[0] 1789763988.091948 mapread 0x12e631 thru 0x12f587 (0xf57 bytes) 000296[1] 1789763988.167211 punch from 0x1acdca to 0x1b8533 (0xb76a bytes) 000298[1] 1789763988.182127 read 0x2f05d9 thru 0x2f9259 (0x8c81 bytes) 000299[1] 1789763988.191422 mapread 0x4e4f82 thru 0x4e6e1e (0x1e9d bytes) 000304[1] 1789763988.259020 mapwrite 0x8b2e0a thru 0x8becf1 (0xbee8 bytes) 000306[0] 1789763988.401279 read 0x77e113 thru 0x78a9e8 (0xc8d6 bytes) 000307[0] 1789763988.445414 mapread 0x5e268d thru 0x5ef384 (0xccf8 bytes) 000309[0] 1789763988.498546 punch from 0x2c9a3e to 0x2d4ad3 (0xb096 bytes) 000310[0] 1789763988.532894 write 0x2d2322 thru 0x2d9e80 (0x7b5f bytes) 000312[0] 1789763988.573171 read 0x17ae1f thru 0x181865 (0x6a47 bytes) 000314[1] 1789763988.596146 mapwrite 0x92b1d6 thru 0x93546f (0xa29a bytes) 000317[1] 1789763988.791953 mapread 0x26bbe3 thru 0x27224a (0x6668 bytes) 000318[1] 1789763988.858550 punch from 0x5536b1 to 0x553eb7 (0x807 bytes) 000319[0] 1789763988.869308 mapwrite 0x4097f2 thru 0x413c41 (0xa450 bytes) 000320[1] 1789763989.052454 mapwrite 0x71d350 thru 0x7294c9 (0xc17a bytes) 000321[1] 1789763989.200097 mapread 0x56d5a6 thru 0x573f1a (0x6975 bytes) 000323[0] 1789763989.281349 read 0x4325da thru 0x43ee57 (0xc87e bytes) 000325[1] 1789763989.350368 mapread 0x3d355b thru 0x3d901e (0x5ac4 bytes) 000328[1] 1789763989.430778 mapread 0x2c10fe thru 0x2c4b60 (0x3a63 bytes) 000329[1] 1789763989.478446 trunc from 0x935470 to 0x739690 (0x1fbddf bytes) 000330[0] 1789763989.544955 read 0x1c90bb thru 0x1ccaa4 (0x39ea bytes) 000331[0] 1789763989.582739 read 0x303ef3 thru 0x30f698 (0xb7a6 bytes) 000334[0] 1789763989.733761 punch from 0x531797 to 0x531dd0 (0x63a bytes) 000335[0] 1789763989.778552 trunc from 0x739691 to 0x029497 (0x7101f9 bytes) 000337[1] 1789763989.853116 mapread 0x00e5eb thru 0x014f1a (0x6930 bytes) 000340[1] 1789763989.980979 trunc from 0x029498 to 0x4c0f23 (0x497a8c bytes) 000341[0] 1789763990.049404 write 0x3525f3 thru 0x354bf4 (0x2602 bytes) 000342[0] 1789763990.092959 punch from 0x354173 to 0x3582e7 (0x4175 bytes) 000343[0] 1789763990.121977 mapread 0x10ea94 thru 0x118216 (0x9783 bytes) 000344[0] 1789763990.206000 punch from 0x00dda6 to 0x01cbdc (0xee37 bytes) 000345[0] 1789763990.227005 mapwrite 0x483264 thru 0x4856b4 (0x2451 bytes) 000346[1] 1789763990.305780 mapwrite 0x8abfe4 thru 0x8b380c (0x7829 bytes) 000347[1] 1789763990.453079 write 0x05833f thru 0x05939f (0x1061 bytes) 000350[1] 1789763990.508118 mapwrite 0x062165 thru 0x068954 (0x67f0 bytes) 000352[1] 1789763990.593198 mapread 0x217719 thru 0x21b3e2 (0x3cca bytes) 000354[0] 1789763990.653325 read 0x7b38af thru 0x7b3d84 (0x4d6 bytes) 000356[0] 1789763990.686375 punch from 0x01fe1e to 0x023651 (0x3834 bytes) 000358[0] 1789763990.707281 mapread 0x78250c thru 0x78e0ea (0xbbdf bytes) 000359[1] 1789763990.781645 mapwrite 0x0a2d5b thru 0x0a5dc2 (0x3068 bytes) 000360[1] 1789763990.946701 write 0x2a4a02 thru 0x2b242f (0xda2e bytes) 000361[0] 1789763990.998329 mapread 0x6c9aa9 thru 0x6d5ec1 (0xc419 bytes) 000362[0] 1789763991.078995 mapread 0x388118 thru 0x392fc8 (0xaeb1 bytes) 000364[1] 1789763991.159241 trunc from 0x8b380d to 0x900d4b (0x4d53f bytes) 000365[0] 1789763991.285901 write 0x1cb6ad thru 0x1d95f2 (0xdf46 bytes) 000368[1] 1789763991.468565 punch from 0x2819e9 to 0x289cb4 (0x82cc bytes) 000369[1] 1789763991.522952 mapread 0x83b27e thru 0x83b4fe (0x281 bytes) 000370[0] 1789763991.541254 trunc from 0x900d4c to 0x667b6d (0x2991de bytes) 000372[1] 1789763991.686010 mapread 0x5a1241 thru 0x5af4e8 (0xe2a8 bytes) 000373[0] 1789763991.815203 mapread 0x606e22 thru 0x615b68 (0xed47 bytes) 000374[0] 1789763991.855555 trunc from 0x667b6e to 0x15e97d (0x5091f0 bytes) 000375[0] 1789763991.915803 read 0x06d1d6 thru 0x07ac1e (0xda49 bytes) 000376[1] 1789763991.954279 read 0x0fa75c thru 0x104ba0 (0xa445 bytes) 000378[0] 1789763991.997689 mapread 0x035261 thru 0x03cb50 (0x78f0 bytes) 000379[0] 1789763992.046615 read 0x02d569 thru 0x0322c8 (0x4d60 bytes) 000380[1] 1789763992.101942 trunc from 0x15e97e to 0x97f8b2 (0x820f35 bytes) 000381[0] 1789763992.205709 mapwrite 0x39e97a thru 0x3a6606 (0x7c8d bytes) 000383[0] 1789763992.479927 punch from 0x00e39a to 0x00f53c (0x11a3 bytes) 000384[0] 1789763992.503325 mapwrite 0x2bba07 thru 0x2bc050 (0x64a bytes) 000386[0] 1789763992.584857 mapwrite 0x9915c6 thru 0x99b661 (0xa09c bytes) 000388[0] 1789763992.845216 write 0x76aa69 thru 0x76ee02 (0x439a bytes) 000390[0] 1789763992.889319 mapwrite 0x5cd53d thru 0x5cfc8f (0x2753 bytes) 000391[1] 1789763992.973798 read 0x94e9aa thru 0x95a8cd (0xbf24 bytes) 000392[0] 1789763993.014285 punch from 0x4301e0 to 0x43bad2 (0xb8f3 bytes) 000393[0] 1789763993.024184 read 0x5dcbce thru 0x5dd818 (0xc4b bytes) 000394[1] 1789763993.043263 punch from 0x45a1e8 to 0x46131b (0x7134 bytes) 000395[0] 1789763993.070702 mapwrite 0x412995 thru 0x42280a (0xfe76 bytes) 000396[0] 1789763993.276035 mapwrite 0x341b19 thru 0x34c8ea (0xadd2 bytes) 000397[1] 1789763993.407238 mapwrite 0x6ae184 thru 0x6b4438 (0x62b5 bytes) 000398[0] 1789763993.639130 mapread 0x4903ff thru 0x4916a6 (0x12a8 bytes) 000400[0] 1789763993.710037 mapread 0x95ca48 thru 0x9608c1 (0x3e7a bytes) 000401[0] 1789763993.757113 trunc from 0x99b662 to 0x4397c0 (0x561ea1 bytes) 000402[0] 1789763993.845422 punch from 0x21188f to 0x212124 (0x896 bytes) 000403[1] 1789763993.871491 mapread 0x0804bf thru 0x0847c6 (0x4308 bytes) 000404[0] 1789763993.934540 punch from 0x39b5a6 to 0x3a3e4f (0x88aa bytes) 000405[0] 1789763993.955537 mapread 0x25b89f thru 0x264eb1 (0x9613 bytes) 000406[0] 1789763994.004368 mapread 0x1946c1 thru 0x19fd96 (0xb6d6 bytes) 000407[0] 1789763994.027770 mapwrite 0x357868 thru 0x3600da (0x8873 bytes) 000408[0] 1789763994.139801 trunc from 0x4397c1 to 0x915e91 (0x4dc6d1 bytes) 000409[1] 1789763994.159344 read 0x69a8ce thru 0x6a763d (0xcd70 bytes) 000414[0] 1789763994.222926 read 0x6fce89 thru 0x70ca79 (0xfbf1 bytes) 000419[0] 1789763994.271106 write 0x3cdd59 thru 0x3d5ce8 (0x7f90 bytes) 000420[1] 1789763994.295620 mapwrite 0x490999 thru 0x4951bf (0x4827 bytes) 000421[1] 1789763994.464484 mapread 0x7c16fa thru 0x7cc019 (0xa920 bytes) 000424[0] 1789763994.642804 mapread 0x86619a thru 0x873daf (0xdc16 bytes) 000425[1] 1789763994.830790 mapwrite 0x0a443a thru 0x0b1460 (0xd027 bytes) 000426[0] 1789763995.044586 punch from 0x7ac48f to 0x7b9254 (0xcdc6 bytes) 000427[0] 1789763995.078270 trunc from 0x915e92 to 0x2329a5 (0x6e34ec bytes) 000428[1] 1789763995.124284 write 0x19ca81 thru 0x1a182a (0x4daa bytes) 000429[0] 1789763995.179161 mapwrite 0x925fe4 thru 0x93332a (0xd347 bytes) 000430[0] 1789763995.355550 mapread 0x33fdf8 thru 0x342d36 (0x2f3f bytes) 000432[0] 1789763995.456059 read 0x88ea93 thru 0x8943fe (0x596c bytes) 000433[1] 1789763995.469590 write 0x70678b thru 0x715f2c (0xf7a2 bytes) 000434[0] 1789763995.504718 punch from 0x3cdf6e to 0x3d72e1 (0x9374 bytes) 000438[1] 1789763995.576629 read 0x47c3ad thru 0x47fade (0x3732 bytes) 000439[0] 1789763995.607090 mapread 0x39a40c thru 0x3a656d (0xc162 bytes) 000440[1] 1789763995.632893 write 0x2cf533 thru 0x2def6b (0xfa39 bytes) 000441[0] 1789763995.697306 mapread 0x69fa0c thru 0x6a18c1 (0x1eb6 bytes) 000442[1] 1789763995.761749 mapread 0x54d861 thru 0x55584c (0x7fec bytes) 000443[0] 1789763995.787303 read 0x063319 thru 0x065775 (0x245d bytes) 000444[0] 1789763995.809256 mapread 0x791d82 thru 0x7957dc (0x3a5b bytes) 000446[0] 1789763995.835177 trunc from 0x93332b to 0x982c2f (0x4f905 bytes) 000447[0] 1789763995.884443 write 0x9b09d7 thru 0x9be507 (0xdb31 bytes) 000449[0] 1789763995.911004 write 0x77579a thru 0x77ed11 (0x9578 bytes) 000450[0] 1789763995.933238 read 0x552763 thru 0x55e91e (0xc1bc bytes) 000452[0] 1789763995.959786 write 0x25dde9 thru 0x26864d (0xa865 bytes) 000454[1] 1789763995.997048 punch from 0x26fb74 to 0x27060c (0xa99 bytes) 000456[0] 1789763996.093696 punch from 0x85de05 to 0x868157 (0xa353 bytes) 000457[0] 1789763996.128954 mapwrite 0x690a03 thru 0x69f4d8 (0xead6 bytes) 000458[1] 1789763996.283788 mapwrite 0x4d028d thru 0x4de919 (0xe68d bytes) 000459[0] 1789763996.407907 mapread 0x50f018 thru 0x510890 (0x1879 bytes) 000461[1] 1789763996.454213 write 0x815099 thru 0x822223 (0xd18b bytes) 000462[1] 1789763996.482667 write 0x157b8a thru 0x1642ba (0xc731 bytes) 000465[1] 1789763996.561075 write 0x950aba thru 0x95644f (0x5996 bytes) 000466[1] 1789763996.574059 mapwrite 0x0ed18a thru 0x0f440d (0x7284 bytes) 000469[1] 1789763996.755754 trunc from 0x9be508 to 0x4dbb60 (0x4e29a7 bytes) 000470[0] 1789763996.827746 mapread 0x1bba1a thru 0x1c9900 (0xdee7 bytes) 000473[0] 1789763996.929195 write 0x0576b1 thru 0x05c3f3 (0x4d43 bytes) 000476[0] 1789763997.036592 write 0x8783ef thru 0x881787 (0x9399 bytes) 000477[0] 1789763997.059262 trunc from 0x881788 to 0x662552 (0x21f235 bytes) 000478[1] 1789763997.089347 mapwrite 0x94c9a8 thru 0x95535a (0x89b3 bytes) 000480[1] 1789763997.215913 read 0x296aa6 thru 0x2a04f0 (0x9a4b bytes) 000481[0] 1789763997.260537 write 0x28c548 thru 0x28dc6d (0x1726 bytes) 000483[1] 1789763997.313436 write 0x30b2d2 thru 0x3136b5 (0x83e4 bytes) 000484[0] 1789763997.340602 read 0x15c48a thru 0x1687b0 (0xc327 bytes) 000485[0] 1789763997.377239 trunc from 0x95535b to 0x49d181 (0x4b81d9 bytes) 000488[1] 1789763997.468058 mapwrite 0x82defd thru 0x83aa45 (0xcb49 bytes) 000489[1] 1789763997.588871 read 0x6ca15f thru 0x6d12c4 (0x7166 bytes) 000490[1] 1789763997.628525 trunc from 0x83aa46 to 0x65414e (0x1e68f7 bytes) 000492[1] 1789763997.659231 mapwrite 0x00e948 thru 0x01a4b1 (0xbb6a bytes) 000494[0] 1789763997.775459 punch from 0x5b0e4f to 0x5b2b95 (0x1d47 bytes) 000495[1] 1789763997.813444 write 0x7b79f5 thru 0x7c6c52 (0xf25e bytes) 000496[0] 1789763997.831052 write 0x47d27b thru 0x47f7d1 (0x2557 bytes) 000498[0] 1789763997.922444 mapread 0x6159a0 thru 0x6219c5 (0xc026 bytes) 000499[1] 1789763998.010115 read 0x00141e thru 0x007fe2 (0x6bc5 bytes) 000500[1] 1789763998.073334 mapread 0x560f27 thru 0x567af6 (0x6bd0 bytes) All operations completed A-OK! Chance of close/open is 1 in 50 Seed set to 3998 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero length punch hole truncating to largest ever: 0x48ca8d 000003[1] 1789763999.200856 trunc from 00000000 to 0x48ca8c (0x48ca8d bytes) 000005[1] 1789763999.229970 read 0x0e9000 thru 0x0f7ca7 (0xeca8 bytes) truncating to largest ever: 0x979b90 000006[1] 1789763999.279130 trunc from 0x48ca8d to 0x979b8f (0x4ed103 bytes) 000007[0] 1789763999.322521 mapwrite 0x46f000 thru 0x470ccb (0x1ccc bytes) 000009[1] 1789763999.558188 read 0x4eb000 thru 0x4ebae9 (0xaea bytes) 000011[0] 1789763999.601195 trunc from 0x979b90 to 0x33a396 (0x63f7f9 bytes) 000014[1] 1789763999.722577 read 0x10c000 thru 0x10d9db (0x19dc bytes) 000015[1] 1789763999.894529 read 0x149000 thru 0x14c1d2 (0x31d3 bytes) 000018[1] 1789763999.896500 trunc from 0x33a397 to 0x587d40 (0x24d9aa bytes) 000020[1] 1789764000.007941 trunc from 0x587d41 to 0x440c29 (0x147117 bytes) 000023[0] 1789764000.108971 mapwrite 0x32a000 thru 0x336363 (0xc364 bytes) 000024[1] 1789764000.322173 trunc from 0x440c2a to 0x945546 (0x50491d bytes) 000025[0] 1789764000.442189 read 0x092000 thru 0x09c951 (0xa952 bytes) 000026[0] 1789764000.468211 trunc from 0x945547 to 0x30ca7f (0x638ac7 bytes) 000029[1] 1789764000.572554 trunc from 0x30ca80 to 0x14462b (0x1c8454 bytes) 000031[0] 1789764000.681628 write 0x8cc000 thru 0x8cf58d (0x358e bytes) 000032[1] 1789764000.738451 read 0x1aa000 thru 0x1b386f (0x9870 bytes) 000033[0] 1789764000.771724 punch from 0x754c9d to 0x75dcf6 (0x905a bytes) 000034[0] 1789764000.847943 trunc from 0x8cf58e to 0x5519ea (0x37dba3 bytes) 000036[0] 1789764000.944295 mapwrite 0x5a1000 thru 0x5adb15 (0xcb16 bytes) 000039[1] 1789764001.102122 mapread 0x49e000 thru 0x4a4517 (0x6518 bytes) 000040[1] 1789764001.139717 punch from 0x12d244 to 0x1336c1 (0x647e bytes) 000041[1] 1789764001.190212 write 0x691000 thru 0x69b442 (0xa443 bytes) 000042[0] 1789764001.216989 punch from 0x35c27e to 0x36220c (0x5f8f bytes) 000043[0] 1789764001.242027 mapread 0x1a1000 thru 0x1a3a5e (0x2a5f bytes) 000044[1] 1789764001.260541 write 0x0e9000 thru 0x0f053e (0x753f bytes) 000045[1] 1789764001.290799 punch from 0x22e7ae to 0x23e62d (0xfe80 bytes) 000046[0] 1789764001.313591 punch from 0x5dc808 to 0x5e94e3 (0xccdc bytes) 000047[1] 1789764001.375073 trunc from 0x69b443 to 0x4fe593 (0x19ceaf bytes) 000054[0] 1789764001.549299 read 0x131000 thru 0x1391c8 (0x81c9 bytes) 000059[1] 1789764001.592564 mapread 0x4c3000 thru 0x4d1432 (0xe433 bytes) 000061[0] 1789764001.612164 punch from 0x07240a to 0x074c0d (0x2804 bytes) 000062[0] 1789764001.657803 punch from 0x00aafb to 0x01a577 (0xfa7d bytes) 000063[0] 1789764001.683417 mapwrite 0x058000 thru 0x05b6dc (0x36dd bytes) 000064[0] 1789764001.814091 mapread 0x18d000 thru 0x191564 (0x4565 bytes) 000065[1] 1789764001.851752 read 0x060000 thru 0x0687e4 (0x87e5 bytes) 000069[0] 1789764001.932891 mapwrite 0x105000 thru 0x10c80c (0x780d bytes) 000073[0] 1789764002.238766 mapwrite 0x76d000 thru 0x7799e4 (0xc9e5 bytes) 000074[0] 1789764002.425454 write 0x922000 thru 0x92eb22 (0xcb23 bytes) 000075[0] 1789764002.427404 punch from 0x525265 to 0x52787c (0x2618 bytes) 000076[1] 1789764002.440765 write 0x434000 thru 0x443f46 (0xff47 bytes) 000078[0] 1789764002.530435 write 0x51e000 thru 0x523ec1 (0x5ec2 bytes) 000079[0] 1789764002.603756 mapread 0x192000 thru 0x19b3ac (0x93ad bytes) 000080[1] 1789764002.662437 mapread 0x7fe000 thru 0x805cb2 (0x7cb3 bytes) 000081[0] 1789764002.750541 mapread 0x0f2000 thru 0x0fd1ac (0xb1ad bytes) 000082[0] 1789764002.767423 punch from 0x4c6960 to 0x4c9a57 (0x30f8 bytes) 000083[1] 1789764002.792028 mapwrite 0x0e4000 thru 0x0eb513 (0x7514 bytes) 000084[0] 1789764002.998475 trunc from 0x92eb23 to 0x3149c3 (0x61a15f bytes) 000087[0] 1789764003.089421 write 0x65b000 thru 0x665780 (0xa781 bytes) 000088[1] 1789764003.093920 mapread 0x16a000 thru 0x16b758 (0x1759 bytes) 000089[1] 1789764003.168108 mapwrite 0x2f5000 thru 0x2f921d (0x421e bytes) 000095[1] 1789764003.347889 punch from 0x107188 to 0x10a5a0 (0x3419 bytes) 000097[0] 1789764003.354128 punch from 0x2bfe91 to 0x2cd4df (0xd64f bytes) 000098[1] 1789764003.374117 punch from 0x600634 to 0x604171 (0x3b3e bytes) 000099[0] 1789764003.389464 trunc from 0x665781 to 0x4902d4 (0x1d54ac bytes) 000101[0] 1789764003.420584 read 0x3be000 thru 0x3c8071 (0xa072 bytes) 000103[0] 1789764003.428485 trunc from 0x4902d5 to 0x591b40 (0x10186c bytes) 000104[1] 1789764003.457153 mapwrite 0x540000 thru 0x54f024 (0xf025 bytes) 000106[1] 1789764003.593331 mapwrite 0x49c000 thru 0x49d962 (0x1963 bytes) 000112[1] 1789764003.722465 read 0x3d7000 thru 0x3d74f8 (0x4f9 bytes) 000113[0] 1789764003.731725 trunc from 0x591b41 to 0x95b6af (0x3c9b6f bytes) 000115[1] 1789764003.809661 write 0x7fe000 thru 0x802b37 (0x4b38 bytes) 000116[1] 1789764003.832637 mapread 0x59f000 thru 0x5a776c (0x876d bytes) 000119[0] 1789764003.942136 mapread 0x22f000 thru 0x235221 (0x6222 bytes) 000121[0] 1789764003.999660 read 0x5c8000 thru 0x5c9fb2 (0x1fb3 bytes) 000123[1] 1789764004.007505 punch from 0x0838ea to 0x083e48 (0x55f bytes) 000124[1] 1789764004.019661 mapread 0x245000 thru 0x2498c9 (0x48ca bytes) 000125[0] 1789764004.060123 read 0x103000 thru 0x112409 (0xf40a bytes) 000127[0] 1789764004.081458 mapwrite 0x452000 thru 0x45a7e4 (0x87e5 bytes) 000128[0] 1789764004.177290 trunc from 0x95b6b0 to 0x440d52 (0x51a95d bytes) 000131[1] 1789764004.263258 punch from 0x2c6360 to 0x2d3454 (0xd0f5 bytes) 000132[0] 1789764004.285515 mapread 0x1fe000 thru 0x204f54 (0x6f55 bytes) 000133[0] 1789764004.375075 trunc from 0x440d53 to 0x84a02c (0x4092da bytes) 000134[1] 1789764004.445246 mapwrite 0x883000 thru 0x883219 (0x21a bytes) 000135[1] 1789764004.523492 read 0x071000 thru 0x073811 (0x2812 bytes) 000136[1] 1789764004.530092 mapread 0x6d1000 thru 0x6d3a2a (0x2a2b bytes) 000137[1] 1789764004.548276 read 0x2ff000 thru 0x30a969 (0xb96a bytes) 000141[1] 1789764004.588979 read 0x87e000 thru 0x883193 (0x5194 bytes) 000142[1] 1789764004.611400 punch from 0x30f051 to 0x314eda (0x5e8a bytes) 000144[1] 1789764004.632494 write 0x637000 thru 0x64451d (0xd51e bytes) 000145[1] 1789764004.655522 trunc from 0x88321a to 0x22ccb3 (0x656566 bytes) 000146[0] 1789764004.713665 read 0x1eb000 thru 0x1ee5e3 (0x35e4 bytes) 000147[0] 1789764004.821891 mapwrite 0x3e1000 thru 0x3e2e3e (0x1e3f bytes) 000149[0] 1789764005.048700 write 0x91c000 thru 0x92666f (0xa670 bytes) 000150[1] 1789764005.056088 trunc from 0x926670 to 0x5bb6f9 (0x36af76 bytes) 000153[1] 1789764005.150818 read 0x381000 thru 0x388391 (0x7392 bytes) 000155[0] 1789764005.160163 mapwrite 0x63f000 thru 0x643c89 (0x4c8a bytes) 000156[1] 1789764005.425646 write 0x66d000 thru 0x673559 (0x655a bytes) 000157[0] 1789764005.499001 write 0x466000 thru 0x46b26a (0x526b bytes) 000158[1] 1789764005.540801 punch from 0x387819 to 0x393ace (0xc2b6 bytes) 000161[0] 1789764005.613561 mapwrite 0x927000 thru 0x936c32 (0xfc33 bytes) 000162[0] 1789764005.832545 mapwrite 0x4bc000 thru 0x4cac23 (0xec24 bytes) 000165[1] 1789764006.040517 write 0x426000 thru 0x433bc7 (0xdbc8 bytes) 000166[1] 1789764006.117405 write 0x085000 thru 0x08f04e (0xa04f bytes) 000167[0] 1789764006.132019 punch from 0x4470ce to 0x4484ec (0x141f bytes) 000168[1] 1789764006.170398 read 0x555000 thru 0x556b3b (0x1b3c bytes) 000171[0] 1789764006.202160 mapwrite 0x048000 thru 0x056091 (0xe092 bytes) 000172[1] 1789764006.485955 write 0x020000 thru 0x02fed3 (0xfed4 bytes) 000173[0] 1789764006.511889 mapread 0x0e7000 thru 0x0ee1e5 (0x71e6 bytes) 000174[0] 1789764006.631929 write 0x20c000 thru 0x20f21a (0x321b bytes) 000175[0] 1789764006.664546 trunc from 0x936c33 to 0x038aa1 (0x8fe191 bytes) 000176[0] 1789764006.727920 read 0x022000 thru 0x0269fa (0x49fb bytes) 000177[1] 1789764006.761493 read 0x028000 thru 0x02f502 (0x7503 bytes) 000178[1] 1789764006.797090 mapwrite 0x6d4000 thru 0x6dfb05 (0xbb06 bytes) 000180[1] 1789764007.097549 punch from 0x16af75 to 0x1757ea (0xa876 bytes) 000185[1] 1789764007.170034 mapwrite 0x601000 thru 0x60f462 (0xe463 bytes) 000189[1] 1789764007.311238 trunc from 0x6dfb06 to 0x4ac36e (0x233797 bytes) 000190[0] 1789764007.353807 mapread 0x100000 thru 0x10cbc1 (0xcbc2 bytes) 000191[0] 1789764007.537964 write 0x282000 thru 0x28d826 (0xb827 bytes) 000192[1] 1789764007.603881 mapread 0x435000 thru 0x43eb23 (0x9b24 bytes) 000194[0] 1789764007.799342 mapwrite 0x5e0000 thru 0x5e50b4 (0x50b5 bytes) 000195[0] 1789764008.001037 trunc from 0x5e50b5 to 0x79f7c2 (0x1ba70e bytes) 000197[0] 1789764008.140819 punch from 0x2e4717 to 0x2f41ee (0xfad8 bytes) 000198[0] 1789764008.182300 mapread 0x763000 thru 0x76bb94 (0x8b95 bytes) 000199[1] 1789764008.297682 mapwrite 0x49e000 thru 0x4a0b57 (0x2b58 bytes) 000200[1] 1789764008.451276 mapread 0x2db000 thru 0x2dc680 (0x1681 bytes) 000201[0] 1789764008.468239 mapwrite 0x65a000 thru 0x669b69 (0xfb6a bytes) 000202[1] 1789764008.720359 read 0x787000 thru 0x793263 (0xc264 bytes) 000203[0] 1789764008.765622 mapwrite 0x1e9000 thru 0x1f85a4 (0xf5a5 bytes) 000205[1] 1789764009.022225 write 0x31b000 thru 0x32536e (0xa36f bytes) 000206[0] 1789764009.058981 mapread 0x755000 thru 0x75fc21 (0xac22 bytes) 000208[0] 1789764009.223986 trunc from 0x79f7c3 to 0x4357b9 (0x36a009 bytes) 000210[0] 1789764009.296290 write 0x341000 thru 0x34d27b (0xc27c bytes) 000213[0] 1789764009.410119 trunc from 0x4357ba to 0x5b4968 (0x17f1af bytes) 000214[1] 1789764009.485444 punch from 0x5357b2 to 0x540692 (0xaee1 bytes) 000219[0] 1789764009.629349 read 0x018000 thru 0x021598 (0x9599 bytes) 000220[1] 1789764009.669313 write 0x26a000 thru 0x274ede (0xaedf bytes) 000224[0] 1789764009.709714 mapwrite 0x49f000 thru 0x4ad3a2 (0xe3a3 bytes) 000225[0] 1789764009.902172 mapread 0x440000 thru 0x449e94 (0x9e95 bytes) 000226[0] 1789764009.936665 read 0x009000 thru 0x018198 (0xf199 bytes) 000227[0] 1789764009.956980 read 0x2b4000 thru 0x2beef5 (0xaef6 bytes) 000228[0] 1789764009.966592 mapread 0x163000 thru 0x16c50a (0x950b bytes) 000232[0] 1789764010.077584 mapread 0x0bf000 thru 0x0cbb5f (0xcb60 bytes) 000234[0] 1789764010.125451 write 0x7f0000 thru 0x7f8cb1 (0x8cb2 bytes) 000237[0] 1789764010.232006 trunc from 0x7f8cb2 to 0x809af5 (0x10e44 bytes) 000239[0] 1789764010.257690 punch from 0x653b81 to 0x655e56 (0x22d6 bytes) 000240[0] 1789764010.268054 read 0x23a000 thru 0x2431a7 (0x91a8 bytes) 000241[0] 1789764010.280768 mapwrite 0x6c8000 thru 0x6d448b (0xc48c bytes) 000243[1] 1789764010.464489 read 0x3cf000 thru 0x3dc56d (0xd56e bytes) 000244[0] 1789764010.495566 mapwrite 0x9d6000 thru 0x9e0c53 (0xac54 bytes) 000245[0] 1789764010.628458 read 0x7cd000 thru 0x7ce9ff (0x1a00 bytes) 000246[0] 1789764010.654740 read 0x1e2000 thru 0x1e69cc (0x49cd bytes) 000247[0] 1789764010.669665 mapread 0x164000 thru 0x16c878 (0x8879 bytes) 000251[0] 1789764010.702331 mapwrite 0x99b000 thru 0x9aae2b (0xfe2c bytes) 000253[1] 1789764010.864532 mapwrite 0x5c4000 thru 0x5d1461 (0xd462 bytes) 000254[0] 1789764011.042637 read 0x8b4000 thru 0x8b9c04 (0x5c05 bytes) 000255[1] 1789764011.065700 trunc from 0x9e0c54 to 0x4434c8 (0x59d78b bytes) 000256[0] 1789764011.101536 trunc from 0x4434c9 to 0x3c147e (0x8204a bytes) 000257[0] 1789764011.139456 mapwrite 0x135000 thru 0x14160b (0xc60c bytes) 000258[0] 1789764011.279302 write 0x870000 thru 0x8713ac (0x13ad bytes) 000260[1] 1789764011.335271 punch from 0x1def4d to 0x1eeb9d (0xfc51 bytes) 000261[0] 1789764011.388008 mapwrite 0x57e000 thru 0x580c0f (0x2c10 bytes) 000262[0] 1789764011.468545 mapwrite 0x03f000 thru 0x042059 (0x305a bytes) 000266[1] 1789764011.599162 punch from 0x4852d2 to 0x48e58d (0x92bc bytes) truncating to largest ever: 0x97d875 000268[0] 1789764011.662512 trunc from 0x8713ad to 0x97d874 (0x10c4c8 bytes) 000269[1] 1789764011.717126 read 0x229000 thru 0x22d4d0 (0x44d1 bytes) 000270[1] 1789764011.752056 read 0x225000 thru 0x22eeaf (0x9eb0 bytes) 000271[1] 1789764011.760513 mapread 0x60f000 thru 0x614b45 (0x5b46 bytes) 000273[0] 1789764011.781153 mapwrite 0x277000 thru 0x283768 (0xc769 bytes) 000274[1] 1789764011.930776 read 0x55b000 thru 0x5669fa (0xb9fb bytes) 000275[1] 1789764011.967369 mapwrite 0x613000 thru 0x618793 (0x5794 bytes) 000276[1] 1789764012.017416 write 0x4fc000 thru 0x50a1da (0xe1db bytes) 000277[1] 1789764012.037965 mapwrite 0x9c9000 thru 0x9d697d (0xd97e bytes) 000278[0] 1789764012.189076 punch from 0x77346d to 0x7771e6 (0x3d7a bytes) 000280[1] 1789764012.247312 punch from 0x798c07 to 0x7a5519 (0xc913 bytes) 000281[1] 1789764012.265216 punch from 0x9ab203 to 0x9b6651 (0xb44f bytes) 000282[1] 1789764012.279621 mapwrite 0x23e000 thru 0x242183 (0x4184 bytes) truncating to largest ever: 0x9c49df 000283[1] 1789764012.442399 trunc from 0x9d697e to 0x9c49de (0x11f9f bytes) 000285[1] 1789764012.497812 punch from 0x865a2e to 0x86fbfd (0xa1d0 bytes) 000286[1] 1789764012.536941 punch from 0x56cffa to 0x56e943 (0x194a bytes) 000287[0] 1789764012.542513 punch from 0x35d79f to 0x361e8c (0x46ee bytes) 000289[0] 1789764012.562753 write 0x054000 thru 0x056783 (0x2784 bytes) 000291[0] 1789764012.615424 mapwrite 0x491000 thru 0x4a00a7 (0xf0a8 bytes) 000295[0] 1789764012.845554 punch from 0x2e1d6c to 0x2f0705 (0xe99a bytes) 000296[0] 1789764012.871526 read 0x243000 thru 0x2474a2 (0x44a3 bytes) 000299[0] 1789764012.914453 trunc from 0x9c49df to 0x8b92a3 (0x10b73b bytes) 000300[1] 1789764012.975816 trunc from 0x8b92a4 to 0x4325d8 (0x486ccb bytes) 000301[0] 1789764013.028581 read 0x02b000 thru 0x02e791 (0x3792 bytes) 000302[1] 1789764013.071668 punch from 0x3d5a11 to 0x3dec48 (0x9238 bytes) 000303[1] 1789764013.100335 write 0x243000 thru 0x24a5db (0x75dc bytes) 000304[0] 1789764013.111879 read 0x2d6000 thru 0x2d6ef9 (0xefa bytes) 000305[1] 1789764013.148879 trunc from 0x4325d9 to 0x2a6b02 (0x18bad6 bytes) 000309[1] 1789764013.251174 write 0x886000 thru 0x88b7c1 (0x57c2 bytes) 000311[1] 1789764013.253715 trunc from 0x88b7c2 to 0x49818f (0x3f3632 bytes) 000312[0] 1789764013.288882 write 0x985000 thru 0x98abc2 (0x5bc3 bytes) 000314[0] 1789764013.303031 write 0x235000 thru 0x24266a (0xd66b bytes) 000317[0] 1789764013.368331 read 0x239000 thru 0x24019f (0x71a0 bytes) 000319[1] 1789764013.483375 trunc from 0x98abc3 to 0x2759f8 (0x7151ca bytes) 000320[1] 1789764013.555869 mapwrite 0x482000 thru 0x491cae (0xfcaf bytes) 000323[1] 1789764013.892942 write 0x4cf000 thru 0x4da8e3 (0xb8e4 bytes) 000324[1] 1789764013.920473 read 0x2db000 thru 0x2de851 (0x3852 bytes) 000325[0] 1789764013.987088 write 0x2d9000 thru 0x2e0ba9 (0x7baa bytes) 000327[0] 1789764014.051279 read 0x2fc000 thru 0x2fe58e (0x258f bytes) 000328[1] 1789764014.059310 trunc from 0x4da8e4 to 0x2a63c7 (0x23451c bytes) 000329[1] 1789764014.115174 write 0x04f000 thru 0x05dee3 (0xeee4 bytes) 000330[0] 1789764014.136106 mapread 0x1aa000 thru 0x1accef (0x2cf0 bytes) 000331[0] 1789764014.189140 write 0x398000 thru 0x39987a (0x187b bytes) 000333[0] 1789764014.206304 write 0x97f000 thru 0x989a16 (0xaa17 bytes) 000334[0] 1789764014.231931 mapwrite 0x12f000 thru 0x131f33 (0x2f34 bytes) 000335[0] 1789764014.362304 punch from 0x95fc44 to 0x960b25 (0xee2 bytes) 000336[0] 1789764014.390498 punch from 0x1b5e71 to 0x1c0933 (0xaac3 bytes) 000338[1] 1789764014.408294 write 0x735000 thru 0x74338d (0xe38e bytes) 000339[1] 1789764014.431324 mapread 0x1c5000 thru 0x1d3233 (0xe234 bytes) 000340[0] 1789764014.477274 mapwrite 0x7b4000 thru 0x7b7e0e (0x3e0f bytes) 000342[0] 1789764014.604603 read 0x3a5000 thru 0x3abb6e (0x6b6f bytes) 000343[0] 1789764014.621399 trunc from 0x989a17 to 0x3a073e (0x5e92d8 bytes) 000344[1] 1789764014.708731 punch from 0x25d998 to 0x25da65 (0xce bytes) 000345[1] 1789764014.756579 trunc from 0x3a073f to 0x4c5ecb (0x12578d bytes) 000349[1] 1789764014.840357 mapread 0x49e000 thru 0x4a9557 (0xb558 bytes) 000350[0] 1789764014.872461 mapread 0x03a000 thru 0x044fd2 (0xafd3 bytes) 000351[1] 1789764014.933452 mapread 0x3d4000 thru 0x3e31cc (0xf1cd bytes) 000352[1] 1789764014.959941 write 0x744000 thru 0x7512db (0xd2dc bytes) 000353[0] 1789764014.997707 mapread 0x4be000 thru 0x4be303 (0x304 bytes) 000355[1] 1789764015.051679 trunc from 0x7512dc to 0x7045ce (0x4cd0d bytes) 000356[1] 1789764015.092493 read 0x3a9000 thru 0x3b8cdf (0xfce0 bytes) 000359[0] 1789764015.151338 mapwrite 0x0b9000 thru 0x0c65e6 (0xd5e7 bytes) 000360[0] 1789764015.491539 mapread 0x08d000 thru 0x09464a (0x764b bytes) 000361[1] 1789764015.504441 mapwrite 0x4be000 thru 0x4be773 (0x774 bytes) 000362[0] 1789764015.608550 write 0x240000 thru 0x24029a (0x29b bytes) 000363[0] 1789764015.651208 trunc from 0x7045cf to 0x951286 (0x24ccb8 bytes) 000364[1] 1789764015.688580 read 0x855000 thru 0x85589d (0x89e bytes) 000366[1] 1789764015.723787 mapwrite 0x420000 thru 0x42092f (0x930 bytes) 000367[0] 1789764015.882685 trunc from 0x951287 to 0x5e2588 (0x36ecfe bytes) 000368[1] 1789764016.031805 punch from 0x3ba643 to 0x3c737b (0xcd39 bytes) 000369[1] 1789764016.181206 mapread 0x100000 thru 0x10fb22 (0xfb23 bytes) 000370[0] 1789764016.350132 trunc from 0x5e2589 to 0x1e96c8 (0x3f8ec0 bytes) 000371[0] 1789764016.433586 mapwrite 0x691000 thru 0x696e21 (0x5e22 bytes) 000372[1] 1789764016.553409 write 0x737000 thru 0x741268 (0xa269 bytes) 000373[1] 1789764016.564788 mapwrite 0x924000 thru 0x924c91 (0xc92 bytes) 000376[1] 1789764016.677119 read 0x6dd000 thru 0x6e5c26 (0x8c27 bytes) 000378[1] 1789764016.787535 write 0x634000 thru 0x6402c1 (0xc2c2 bytes) 000379[0] 1789764016.798514 write 0x731000 thru 0x7335e0 (0x25e1 bytes) 000381[0] 1789764016.863684 write 0x437000 thru 0x438f79 (0x1f7a bytes) 000384[0] 1789764016.907664 mapwrite 0x7ce000 thru 0x7da1d0 (0xc1d1 bytes) 000385[0] 1789764017.133351 write 0x5d4000 thru 0x5e3aef (0xfaf0 bytes) 000390[1] 1789764017.211655 write 0x179000 thru 0x17f950 (0x6951 bytes) 000392[1] 1789764017.300811 mapread 0x7f6000 thru 0x802328 (0xc329 bytes) 000394[1] 1789764017.397976 trunc from 0x924c92 to 0x87ea8a (0xa6207 bytes) 000395[1] 1789764017.436580 trunc from 0x87ea8b to 0x60af44 (0x273b46 bytes) 000397[0] 1789764017.456084 mapwrite 0x83c000 thru 0x84b0a1 (0xf0a2 bytes) 000398[1] 1789764017.622622 read 0x301000 thru 0x30fdbe (0xedbf bytes) 000399[1] 1789764017.696383 trunc from 0x84b0a2 to 0x64c0eb (0x1fefb6 bytes) 000400[0] 1789764017.802636 mapread 0x32c000 thru 0x330ec3 (0x4ec4 bytes) 000401[1] 1789764017.888608 mapwrite 0x98d000 thru 0x996cfb (0x9cfc bytes) 000403[1] 1789764018.079218 mapread 0x2ce000 thru 0x2d5e75 (0x7e76 bytes) 000404[1] 1789764018.121042 mapread 0x8df000 thru 0x8e4ea1 (0x5ea2 bytes) 000405[0] 1789764018.145911 mapwrite 0x8fa000 thru 0x905027 (0xb028 bytes) 000406[1] 1789764018.398715 write 0x848000 thru 0x850719 (0x871a bytes) 000407[1] 1789764018.429457 trunc from 0x996cfc to 0x0cf614 (0x8c76e7 bytes) 000408[1] 1789764018.455499 punch from 0x07d487 to 0x08bffd (0xeb77 bytes) 000411[1] 1789764018.549486 mapwrite 0x080000 thru 0x08ef9e (0xef9f bytes) 000412[0] 1789764018.693633 trunc from 0x0cf615 to 0x71ab8b (0x64b577 bytes) 000415[1] 1789764018.776345 trunc from 0x71ab8c to 0x2431f3 (0x4d7998 bytes) 000416[1] 1789764018.879656 mapwrite 0x68f000 thru 0x695922 (0x6923 bytes) 000417[0] 1789764019.007118 trunc from 0x695923 to 0x06aa3d (0x62aee5 bytes) 000418[0] 1789764019.032745 write 0x07d000 thru 0x0870ce (0xa0cf bytes) 000423[0] 1789764019.060437 read 0x048000 thru 0x04c023 (0x4024 bytes) 000424[0] 1789764019.098187 punch from 0x072797 to 0x07c51a (0x9d84 bytes) 000426[1] 1789764019.142498 write 0x29c000 thru 0x2a1521 (0x5522 bytes) 000427[0] 1789764019.159343 write 0x347000 thru 0x3526fb (0xb6fc bytes) 000428[1] 1789764019.221209 mapread 0x030000 thru 0x035cdf (0x5ce0 bytes) 000429[0] 1789764019.267539 punch from 0x1d6da0 to 0x1e3ba2 (0xce03 bytes) 000430[1] 1789764019.302461 trunc from 0x3526fc to 0x14d8aa (0x204e51 bytes) 000431[0] 1789764019.328568 mapwrite 0x30c000 thru 0x30d0cc (0x10cd bytes) 000434[1] 1789764019.575846 punch from 0x0128e9 to 0x0195d3 (0x6ceb bytes) 000439[1] 1789764019.640055 mapread 0x0d6000 thru 0x0e4c75 (0xec76 bytes) 000440[1] 1789764019.682537 read 0x152000 thru 0x160656 (0xe657 bytes) 000441[1] 1789764019.756726 trunc from 0x30d0cd to 0x67e839 (0x37176d bytes) 000442[1] 1789764019.781909 write 0x403000 thru 0x40adaf (0x7db0 bytes) 000443[1] 1789764019.837580 trunc from 0x67e83a to 0x3662d5 (0x318564 bytes) 000445[1] 1789764019.930408 read 0x1e7000 thru 0x1f5e3d (0xee3e bytes) 000446[0] 1789764019.951565 read 0x201000 thru 0x2078ea (0x68eb bytes) 000448[0] 1789764020.092368 write 0x55f000 thru 0x56ea6d (0xfa6e bytes) 000450[1] 1789764020.154159 mapread 0x364000 thru 0x36f084 (0xb085 bytes) 000452[1] 1789764020.276731 mapread 0x07d000 thru 0x08c062 (0xf063 bytes) 000453[0] 1789764020.277799 punch from 0x25c4d4 to 0x25c54d (0x7a bytes) 000454[1] 1789764020.304219 punch from 0x21b7d0 to 0x21efc6 (0x37f7 bytes) 000455[0] 1789764020.340270 read_OD 0x152000 thru 0x154fff (0x3000 bytes) 000456[1] 1789764020.368542 punch from 0x3e33f1 to 0x3e65c9 (0x31d9 bytes) 000457[1] 1789764020.383445 trunc from 0x56ea6e to 0x0527ab (0x51c2c2 bytes) 000458[1] 1789764020.403263 mapwrite 0x49b000 thru 0x49d4aa (0x24ab bytes) 000461[0] 1789764020.499770 punch from 0x2ae3d2 to 0x2b85dd (0xa20c bytes) 000462[0] 1789764020.532158 trunc from 0x49d4ab to 0x1063df (0x3970cb bytes) 000463[1] 1789764020.561475 mapread 0x086000 thru 0x08ca03 (0x6a04 bytes) 000464[0] 1789764020.653150 trunc from 0x1063e0 to 0x6a3490 (0x59d0b1 bytes) 000465[1] 1789764020.754307 mapread 0x668000 thru 0x66879f (0x7a0 bytes) 000468[0] 1789764020.862465 trunc from 0x6a3491 to 0x89c3c3 (0x1f8f33 bytes) 000469[1] 1789764020.923015 write 0x21e000 thru 0x22245f (0x4460 bytes) 000470[1] 1789764020.956258 mapread 0x602000 thru 0x61091c (0xe91d bytes) 000471[0] 1789764021.052553 mapwrite 0x48f000 thru 0x492498 (0x3499 bytes) 000472[0] 1789764021.169031 punch from 0x0be024 to 0x0cd719 (0xf6f6 bytes) 000474[1] 1789764021.256730 mapwrite 0x6c5000 thru 0x6c6a80 (0x1a81 bytes) 000479[1] 1789764021.498774 mapread 0x583000 thru 0x58b595 (0x8596 bytes) 000480[0] 1789764021.554949 trunc from 0x89c3c4 to 0x6e927d (0x1b3146 bytes) 000481[0] 1789764021.602148 trunc from 0x6e927e to 0x2bd78b (0x42baf2 bytes) 000483[1] 1789764021.639922 punch from 0x112eee to 0x11d607 (0xa71a bytes) 000485[1] 1789764021.695769 punch from 0x0065e2 to 0x0121e6 (0xbc05 bytes) 000486[0] 1789764021.752192 mapwrite 0x57b000 thru 0x58ae52 (0xfe53 bytes) 000489[1] 1789764021.920382 punch from 0x1c58c2 to 0x1d1e8c (0xc5cb bytes) 000492[1] 1789764022.006517 write 0x662000 thru 0x671763 (0xf764 bytes) 000494[0] 1789764022.025758 punch from 0x08386c to 0x090d26 (0xd4bb bytes) 000495[0] 1789764022.102891 mapwrite 0x9a0000 thru 0x9a66e6 (0x66e7 bytes) 000496[1] 1789764022.363690 mapread 0x8cf000 thru 0x8d551d (0x651e bytes) 000497[0] 1789764022.489237 trunc from 0x9a66e7 to 0x7e3d6b (0x1c297b bytes) All operations completed A-OK! PASS 16a (75s) == sanityn test 16b: 500 iterations of dual-mount fsx at small size ========================================================== 16:40:32 (1789764032) keep default fallocate mode: 2 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.712222 s, 11.8 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.294051 s, 28.5 MB/s Chance of close/open is 1 in 50 Seed set to 4037 fd 0: /mnt/lustre/f16b.sanityn fd 1: /mnt/lustre2/f16b.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported truncating to largest ever: 0x61274a 000002[1] 1789764038.805997 trunc from 00000000 to 0x612749 (0x61274a bytes) truncating to largest ever: 0x988552 000003[1] 1789764038.964089 trunc from 0x61274a to 0x988551 (0x375e08 bytes) 000004[0] 1789764039.054779 trunc from 0x988552 to 0x02e59f (0x959fb2 bytes) 000007[1] 1789764039.128294 punch from 0x009543 to 0x009e83 (0x941 bytes) 000008[1] 1789764039.198416 mapread 0x00a7fe thru 0x00a8a1 (0xa4 bytes) 000009[1] 1789764039.291645 read 0x00f49a thru 0x010fff (0x1b66 bytes) 000010[1] 1789764039.354024 write 0x77b5e4 thru 0x77c89a (0x12b7 bytes) 000011[0] 1789764039.447607 trunc from 0x77c89b to 0x8250e8 (0xa884e bytes) 000012[1] 1789764039.564029 write 0x5ce809 thru 0x5cf4da (0xcd2 bytes) 000014[0] 1789764039.680351 mapwrite 0x0c5ce5 thru 0x0c743a (0x1756 bytes) 000015[0] 1789764039.877607 write 0x2b328e thru 0x2b5132 (0x1ea5 bytes) 000016[0] 1789764039.910010 read 0x43dafc thru 0x43e601 (0xb06 bytes) 000017[0] 1789764039.932130 trunc from 0x8250e9 to 0x8e488d (0xbf7a5 bytes) 000019[0] 1789764039.971377 punch from 0x043232 to 0x04364a (0x419 bytes) 000020[0] 1789764039.982091 trunc from 0x8e488e to 0x56f802 (0x37508b bytes) 000021[0] 1789764040.006426 mapwrite 0x02e937 thru 0x02f7be (0xe88 bytes) 000022[1] 1789764040.087463 mapwrite 0x01bb35 thru 0x01c188 (0x654 bytes) 000023[1] 1789764040.174099 punch from 0x0565de to 0x05807f (0x1aa2 bytes) 000025[0] 1789764040.214370 trunc from 0x56f803 to 0x21fc69 (0x34fb99 bytes) 000026[1] 1789764040.291605 write 0x3995fe thru 0x39a748 (0x114b bytes) 000027[1] 1789764040.322451 mapwrite 0x29a9f4 thru 0x29b9dc (0xfe9 bytes) 000028[0] 1789764040.403801 write 0x229bb1 thru 0x22afd6 (0x1426 bytes) 000031[1] 1789764040.474012 mapread 0x21a100 thru 0x21b66a (0x156b bytes) 000033[0] 1789764040.654728 punch from 0x0ac668 to 0x0ad25f (0xbf8 bytes) 000034[1] 1789764040.728807 trunc from 0x39a749 to 0x5a6b46 (0x20c3fe bytes) 000035[0] 1789764040.939156 mapwrite 0x1e2048 thru 0x1e38f3 (0x18ac bytes) truncating to largest ever: 0x9d35be 000036[1] 1789764041.265771 trunc from 0x5a6b47 to 0x9d35bd (0x42ca77 bytes) 000038[1] 1789764041.435990 read 0x6e7798 thru 0x6e7f38 (0x7a1 bytes) 000039[1] 1789764041.493991 write 0x38e2bf thru 0x38e8f4 (0x636 bytes) 000040[1] 1789764041.569947 read 0x713efe thru 0x71572d (0x1830 bytes) 000041[0] 1789764041.581316 trunc from 0x9d35be to 0x954290 (0x7f32d bytes) 000042[1] 1789764041.769456 punch from 0x4c3246 to 0x4c4521 (0x12dc bytes) 000044[0] 1789764041.897115 mapread 0x3511a1 thru 0x352735 (0x1595 bytes) 000048[1] 1789764042.097956 write 0x4710de thru 0x4728f1 (0x1814 bytes) 000049[0] 1789764042.162320 trunc from 0x954291 to 0x658913 (0x2fb97d bytes) 000050[0] 1789764042.295364 mapread 0x3c1377 thru 0x3c2b94 (0x181e bytes) 000052[0] 1789764042.381605 read 0x60a731 thru 0x60b09e (0x96e bytes) 000054[0] 1789764042.478750 trunc from 0x658914 to 0x4fee88 (0x159a8b bytes) 000055[1] 1789764042.545584 mapwrite 0x40c314 thru 0x40e0d6 (0x1dc3 bytes) 000057[0] 1789764042.643175 read 0x24a263 thru 0x24b69f (0x143d bytes) 000058[1] 1789764042.663400 punch from 0x4ee075 to 0x4ee257 (0x1e3 bytes) 000059[0] 1789764042.683949 mapread 0x2c77b9 thru 0x2c942b (0x1c73 bytes) 000061[0] 1789764042.728598 read 0x2e789b thru 0x2e7de7 (0x54d bytes) 000062[1] 1789764042.818341 mapread 0x31de32 thru 0x31e32a (0x4f9 bytes) 000065[1] 1789764042.898910 write 0x74f8e1 thru 0x750fd1 (0x16f1 bytes) 000069[0] 1789764042.986104 mapread 0x414aa8 thru 0x415254 (0x7ad bytes) 000070[1] 1789764043.097115 trunc from 0x750fd2 to 0x9813b0 (0x2303df bytes) 000071[1] 1789764043.200882 trunc from 0x9813b1 to 0x7e2f1a (0x19e496 bytes) 000075[1] 1789764043.279522 read 0x51dedd thru 0x51ec7c (0xda0 bytes) 000076[0] 1789764043.286023 punch from 0x48550a to 0x486378 (0xe6f bytes) 000077[0] 1789764043.347762 punch from 0x78b084 to 0x78c2da (0x1257 bytes) 000079[1] 1789764043.431137 trunc from 0x7e2f1b to 0x9c9380 (0x1e6466 bytes) 000080[1] 1789764043.597865 trunc from 0x9c9381 to 0x3af6b1 (0x619ccf bytes) 000084[1] 1789764043.657119 mapwrite 0x7b0496 thru 0x7b1c30 (0x179b bytes) 000085[1] 1789764043.833679 read 0x1e6105 thru 0x1e7916 (0x1812 bytes) 000086[1] 1789764043.924923 punch from 0x500718 to 0x5020f4 (0x19dd bytes) 000087[1] 1789764043.931311 write 0x6d74d0 thru 0x6d8a07 (0x1538 bytes) 000088[0] 1789764043.987606 mapread 0x51b4ff thru 0x51c532 (0x1034 bytes) 000089[0] 1789764044.064586 trunc from 0x7b1c31 to 0x651ef4 (0x15fd3c bytes) 000094[1] 1789764044.262066 mapread 0x4b3ff3 thru 0x4b498f (0x99d bytes) 000095[0] 1789764044.353312 trunc from 0x651ef5 to 0x15d18e (0x4f4d66 bytes) 000096[1] 1789764044.407811 write 0x7e7461 thru 0x7e8a54 (0x15f4 bytes) 000098[1] 1789764044.571262 mapread 0x041523 thru 0x042a98 (0x1576 bytes) 000100[1] 1789764044.761201 punch from 0x703f3d to 0x704215 (0x2d9 bytes) 000101[0] 1789764044.796867 punch from 0x6ab5bf to 0x6ad12c (0x1b6e bytes) 000102[1] 1789764044.834736 read 0x7c4900 thru 0x7c5b8d (0x128e bytes) 000104[0] 1789764044.864000 write 0x2b5b25 thru 0x2b654b (0xa27 bytes) 000105[1] 1789764044.923539 mapwrite 0x43859a thru 0x439d89 (0x17f0 bytes) 000106[1] 1789764045.093925 mapread 0x1f453c thru 0x1f5b8f (0x1654 bytes) 000108[0] 1789764045.219089 punch from 0x0d32e9 to 0x0d4f6b (0x1c83 bytes) 000109[1] 1789764045.313069 punch from 0x1ae11f to 0x1af188 (0x106a bytes) 000110[1] 1789764045.395036 write 0x25acab thru 0x25c83c (0x1b92 bytes) 000112[1] 1789764045.487228 mapwrite 0x56f851 thru 0x5710b9 (0x1869 bytes) 000113[0] 1789764045.720685 punch from 0x154fb0 to 0x1557d0 (0x821 bytes) 000114[1] 1789764045.812367 mapread 0x0ef62e thru 0x0eff6e (0x941 bytes) 000115[1] 1789764045.931198 mapwrite 0x276307 thru 0x276863 (0x55d bytes) 000117[0] 1789764046.111739 read 0x12e6fb thru 0x12fd50 (0x1656 bytes) 000118[1] 1789764046.165799 read 0x10aa2e thru 0x10c908 (0x1edb bytes) 000119[1] 1789764046.202216 punch from 0x042787 to 0x04447b (0x1cf5 bytes) 000120[0] 1789764046.310350 trunc from 0x7e8a55 to 0x7758d0 (0x73184 bytes) 000121[0] 1789764046.427710 mapread 0x4a12c9 thru 0x4a1c40 (0x978 bytes) 000122[0] 1789764046.436917 mapread 0x3c42cf thru 0x3c49ec (0x71e bytes) 000125[0] 1789764046.517582 trunc from 0x7758d1 to 0x644c3e (0x130c92 bytes) 000126[1] 1789764046.542235 trunc from 0x644c3f to 0x7cd68e (0x188a50 bytes) 000128[1] 1789764046.754775 trunc from 0x7cd68f to 0x5787ed (0x254ea1 bytes) 000135[1] 1789764047.161767 mapwrite 0x1403b5 thru 0x1403d1 (0x1d bytes) 000136[1] 1789764047.285452 mapread 0x4c7906 thru 0x4c8aac (0x11a7 bytes) 000137[1] 1789764047.373357 read 0x2ef44a thru 0x2f0d16 (0x18cd bytes) 000139[1] 1789764047.394253 mapwrite 0x4b6326 thru 0x4b634b (0x26 bytes) 000140[0] 1789764047.666619 read 0x550df0 thru 0x551a1c (0xc2d bytes) 000143[0] 1789764047.734002 punch from 0x3c2c8f to 0x3c3c55 (0xfc7 bytes) 000145[0] 1789764047.896335 write 0x36d1de thru 0x36e02f (0xe52 bytes) 000147[1] 1789764047.987227 mapread 0x20ae09 thru 0x20b607 (0x7ff bytes) 000148[1] 1789764048.085268 mapwrite 0x4b973b thru 0x4ba359 (0xc1f bytes) 000150[0] 1789764048.305338 punch from 0x11e3d6 to 0x11e957 (0x582 bytes) 000151[0] 1789764048.390393 punch from 0x4fc768 to 0x4fe5bc (0x1e55 bytes) 000152[0] 1789764048.475189 read 0x4c323e thru 0x4c4a74 (0x1837 bytes) 000153[1] 1789764048.485119 mapwrite 0x665c8d thru 0x665e31 (0x1a5 bytes) 000154[1] 1789764048.815255 trunc from 0x665e32 to 0x243d1e (0x422113 bytes) 000155[1] 1789764048.853557 punch from 0x1bbb71 to 0x1bcd26 (0x11b6 bytes) 000160[0] 1789764048.951520 trunc from 0x243d1f to 0x38ec4b (0x14af2d bytes) 000162[0] 1789764049.257540 write 0x819a0e thru 0x81b21e (0x1811 bytes) 000163[0] 1789764049.287870 write 0x0d5aab thru 0x0d5b9b (0xf1 bytes) 000164[1] 1789764049.295206 read 0x41d6b4 thru 0x41e5a4 (0xef1 bytes) 000165[0] 1789764049.362751 punch from 0x26ed5f to 0x2702fd (0x159f bytes) 000168[1] 1789764049.436199 mapwrite 0x677d76 thru 0x6793d7 (0x1662 bytes) 000171[0] 1789764050.023279 mapwrite 0x279c30 thru 0x279cd9 (0xaa bytes) 000172[0] 1789764050.117207 punch from 0x01bea4 to 0x01c88b (0x9e8 bytes) 000175[0] 1789764050.224142 punch from 0x45e4a9 to 0x46033e (0x1e96 bytes) 000182[0] 1789764050.359569 mapwrite 0x09af07 thru 0x09c0d4 (0x11ce bytes) 000183[0] 1789764050.519933 mapread 0x3e7fa1 thru 0x3e93e0 (0x1440 bytes) 000184[0] 1789764050.572818 mapwrite 0x0d0ad9 thru 0x0d1fc5 (0x14ed bytes) 000185[0] 1789764050.717637 write 0x022993 thru 0x022c0e (0x27c bytes) 000186[0] 1789764050.762658 punch from 0x096a10 to 0x097110 (0x701 bytes) 000190[0] 1789764050.890997 trunc from 0x81b21f to 0x28b617 (0x58fc07 bytes) 000192[0] 1789764050.961557 punch from 0x1ef36d to 0x1f0575 (0x1209 bytes) 000194[0] 1789764051.048617 read 0x0a977b thru 0x0aa630 (0xeb6 bytes) 000197[1] 1789764051.253706 mapwrite 0x61e2a4 thru 0x61f64e (0x13ab bytes) 000199[1] 1789764051.899291 read 0x003f1c thru 0x00568d (0x1772 bytes) 000200[1] 1789764051.988089 punch from 0x078793 to 0x079bed (0x145b bytes) 000202[1] 1789764052.158070 trunc from 0x61f64f to 0x78a23e (0x16abf0 bytes) 000203[0] 1789764052.320453 read 0x4e060b thru 0x4e0c93 (0x689 bytes) 000205[1] 1789764052.431472 mapread 0x4839e8 thru 0x483fc1 (0x5da bytes) 000207[0] 1789764052.441765 trunc from 0x78a23f to 0x997c67 (0x20da29 bytes) 000210[0] 1789764052.852651 trunc from 0x997c68 to 0x21e4b7 (0x7797b0 bytes) 000211[1] 1789764052.962961 trunc from 0x21e4b8 to 0x29ad4b (0x7c894 bytes) 000212[1] 1789764053.037860 punch from 0x19c718 to 0x19e384 (0x1c6d bytes) 000213[0] 1789764053.051111 trunc from 0x29ad4c to 0x4383f4 (0x19d6a9 bytes) 000215[1] 1789764053.279509 mapwrite 0x63dd3a thru 0x63de31 (0xf8 bytes) 000216[0] 1789764053.673040 read 0x09e14b thru 0x09fa59 (0x190f bytes) 000218[0] 1789764053.796808 mapread 0x3548c2 thru 0x355162 (0x8a1 bytes) 000220[1] 1789764053.938019 write 0x816ac5 thru 0x816b2f (0x6b bytes) 000221[0] 1789764054.039363 read 0x72be8b thru 0x72c73c (0x8b2 bytes) 000223[0] 1789764054.132016 mapread 0x586027 thru 0x587124 (0x10fe bytes) 000224[1] 1789764054.208423 mapwrite 0x7c0ca7 thru 0x7c14a0 (0x7fa bytes) 000225[0] 1789764054.429246 trunc from 0x816b30 to 0x8c143b (0xaa90c bytes) 000226[1] 1789764054.565055 trunc from 0x8c143c to 0x8a117d (0x202be bytes) 000227[1] 1789764054.611486 mapwrite 0x64661e thru 0x6483ba (0x1d9d bytes) 000229[1] 1789764054.696544 mapread 0x13253f thru 0x134282 (0x1d44 bytes) 000230[1] 1789764054.715374 punch from 0x4f4434 to 0x4f5214 (0xde1 bytes) 000233[0] 1789764054.763428 write 0x2b1cac thru 0x2b2669 (0x9be bytes) 000234[1] 1789764054.860950 read 0x860105 thru 0x861592 (0x148e bytes) 000235[0] 1789764054.925427 read 0x62cc7c thru 0x62e261 (0x15e6 bytes) 000240[1] 1789764054.962718 punch from 0x4e01dc to 0x4e0390 (0x1b5 bytes) 000241[0] 1789764054.983927 read 0x744237 thru 0x745148 (0xf12 bytes) 000248[0] 1789764055.035942 mapread 0x759f82 thru 0x75aa68 (0xae7 bytes) 000249[1] 1789764055.070873 punch from 0x2ca278 to 0x2cb5af (0x1338 bytes) 000251[0] 1789764055.111983 mapwrite 0x0884de thru 0x089940 (0x1463 bytes) 000252[0] 1789764055.336779 read 0x79462b thru 0x794be0 (0x5b6 bytes) 000254[0] 1789764055.351969 read 0x6a34f8 thru 0x6a4849 (0x1352 bytes) 000257[0] 1789764055.416080 trunc from 0x8a117e to 0x23b737 (0x665a46 bytes) 000258[1] 1789764055.505564 mapread 0x08e889 thru 0x08f732 (0xeaa bytes) 000260[1] 1789764055.649710 punch from 0x20ec91 to 0x2101be (0x152e bytes) 000263[0] 1789764055.823345 write 0x7f0f97 thru 0x7f25e0 (0x164a bytes) 000264[0] 1789764055.872875 mapwrite 0x496ee3 thru 0x49838d (0x14ab bytes) 000265[0] 1789764056.144135 read 0x1fe0f2 thru 0x1ff8e0 (0x17ef bytes) 000267[0] 1789764056.216669 mapwrite 0x2b9618 thru 0x2b9bac (0x595 bytes) 000268[1] 1789764056.425796 mapwrite 0x4ec60c thru 0x4eccc3 (0x6b8 bytes) 000270[1] 1789764056.711807 mapread 0x0fdad7 thru 0x0fe1f8 (0x722 bytes) 000271[0] 1789764056.778814 mapwrite 0x435101 thru 0x435623 (0x523 bytes) 000272[0] 1789764057.036332 punch from 0x2ab3cf to 0x2accbf (0x18f1 bytes) 000279[0] 1789764057.228505 mapwrite 0x686b12 thru 0x686c2f (0x11e bytes) 000281[1] 1789764057.551765 write 0x0af5d7 thru 0x0b1345 (0x1d6f bytes) 000282[0] 1789764057.655446 mapread 0x25aed2 thru 0x25b239 (0x368 bytes) 000283[1] 1789764057.758178 write 0x4b1a1c thru 0x4b2232 (0x817 bytes) 000284[0] 1789764057.838119 write 0x2b05fa thru 0x2b0e8e (0x895 bytes) 000285[1] 1789764057.904512 mapwrite 0x5aa7bb thru 0x5ab4ee (0xd34 bytes) 000286[1] 1789764058.088584 mapread 0x7915b7 thru 0x792261 (0xcab bytes) 000287[0] 1789764058.163400 trunc from 0x7f25e1 to 0x430459 (0x3c2187 bytes) 000288[1] 1789764058.218230 mapwrite 0x5eb2e6 thru 0x5ec33e (0x1059 bytes) 000289[1] 1789764058.680193 mapread 0x006e44 thru 0x007818 (0x9d5 bytes) 000291[1] 1789764058.774727 mapread 0x10467b thru 0x104b7f (0x505 bytes) 000292[0] 1789764058.871176 trunc from 0x5ec33f to 0x6967a1 (0xaa463 bytes) 000293[1] 1789764058.958115 trunc from 0x6967a2 to 0x542ee6 (0x1538bb bytes) 000294[1] 1789764059.099123 write 0x479d91 thru 0x47b0d8 (0x1348 bytes) 000296[1] 1789764059.192975 trunc from 0x542ee7 to 0x71bdb9 (0x1d8ed3 bytes) 000297[1] 1789764059.229330 mapread 0x2901eb thru 0x29069a (0x4b0 bytes) 000298[1] 1789764059.270409 mapwrite 0x1c9f5d thru 0x1ca535 (0x5d9 bytes) 000300[0] 1789764059.429011 read 0x1c8ee8 thru 0x1ca630 (0x1749 bytes) 000301[1] 1789764059.509777 mapwrite 0x7fac04 thru 0x7fc614 (0x1a11 bytes) 000302[1] 1789764059.658971 punch from 0x11b0ff to 0x11bd38 (0xc3a bytes) 000303[0] 1789764059.690210 write 0x7714ad thru 0x772da0 (0x18f4 bytes) 000304[0] 1789764059.761767 trunc from 0x7fc615 to 0x5136fe (0x2e8f16 bytes) 000305[0] 1789764059.852563 mapwrite 0x4af6af thru 0x4b1293 (0x1be5 bytes) 000306[0] 1789764059.920537 write 0x6bd5a7 thru 0x6bda78 (0x4d2 bytes) 000307[1] 1789764059.925433 read 0x5f3ad1 thru 0x5f598e (0x1ebe bytes) 000308[0] 1789764059.983526 mapread 0x236720 thru 0x236929 (0x20a bytes) 000311[1] 1789764060.055229 read 0x3626ff thru 0x363ead (0x17af bytes) 000313[1] 1789764060.155990 punch from 0x5af7d7 to 0x5b0aea (0x1314 bytes) 000314[0] 1789764060.194974 write 0x1e5b69 thru 0x1e67ff (0xc97 bytes) 000317[1] 1789764060.372714 trunc from 0x6bda79 to 0x714cb0 (0x57238 bytes) 000318[0] 1789764060.456975 write 0x74fc6c thru 0x7513fb (0x1790 bytes) 000319[0] 1789764060.517341 write 0x8e1233 thru 0x8e22b4 (0x1082 bytes) 000320[0] 1789764060.581046 trunc from 0x8e22b5 to 0x7cb99c (0x116918 bytes) 000323[1] 1789764060.677938 mapread 0x7b4b47 thru 0x7b59bb (0xe75 bytes) 000324[1] 1789764060.801635 read 0x257386 thru 0x258fda (0x1c55 bytes) 000325[0] 1789764060.889970 trunc from 0x7cb99d to 0x90a363 (0x13e9c7 bytes) 000327[1] 1789764060.990159 mapwrite 0x497946 thru 0x4986cb (0xd86 bytes) 000329[1] 1789764061.259482 read 0x1a7d1e thru 0x1a96a8 (0x198b bytes) 000330[0] 1789764061.289336 trunc from 0x90a364 to 0x3898e4 (0x580a7f bytes) 000331[1] 1789764061.355263 trunc from 0x3898e5 to 0x194195 (0x1f574f bytes) 000332[1] 1789764061.482479 mapread 0x12114e thru 0x121efc (0xdaf bytes) 000333[1] 1789764061.496941 mapwrite 0x66b129 thru 0x66c695 (0x156d bytes) 000337[1] 1789764061.786426 mapread 0x392d21 thru 0x3935f9 (0x8d9 bytes) 000340[0] 1789764061.908711 read 0x39c7ce thru 0x39daf0 (0x1323 bytes) 000341[0] 1789764061.939259 punch from 0x141c62 to 0x1427be (0xb5d bytes) 000342[0] 1789764062.009219 write 0x69832f thru 0x699c7c (0x194e bytes) 000343[1] 1789764062.089686 mapread 0x05eb18 thru 0x06043d (0x1926 bytes) 000345[1] 1789764062.204987 mapread 0x1df370 thru 0x1e06a2 (0x1333 bytes) 000347[1] 1789764062.316547 write 0x8b83f8 thru 0x8b9302 (0xf0b bytes) 000348[1] 1789764062.350005 write 0x1dbcd8 thru 0x1dd598 (0x18c1 bytes) 000350[1] 1789764062.477583 mapwrite 0x11d992 thru 0x11f243 (0x18b2 bytes) 000353[1] 1789764062.679537 punch from 0x44ecc4 to 0x44f380 (0x6bd bytes) 000354[0] 1789764062.791898 mapwrite 0x7772bb thru 0x777c5f (0x9a5 bytes) 000355[1] 1789764062.939892 mapread 0x4ac25f thru 0x4acfde (0xd80 bytes) 000357[0] 1789764062.985819 mapread 0x84dcf9 thru 0x84ed51 (0x1059 bytes) 000358[1] 1789764063.027435 write 0x2ddd1d thru 0x2dec0a (0xeee bytes) 000359[1] 1789764063.069261 trunc from 0x8b9303 to 0x30e5e0 (0x5aad22 bytes) 000360[1] 1789764063.111815 write 0x6b24ec thru 0x6b3c25 (0x173a bytes) 000362[0] 1789764063.160295 mapwrite 0x07c431 thru 0x07c8e1 (0x4b1 bytes) 000367[0] 1789764063.455124 punch from 0x6ade36 to 0x6afa65 (0x1c30 bytes) 000369[0] 1789764063.531120 mapread 0x1f321a thru 0x1f38d3 (0x6ba bytes) 000372[1] 1789764063.669765 punch from 0x0e336b to 0x0e505c (0x1cf2 bytes) 000373[0] 1789764063.698149 read 0x4cda2b thru 0x4ce15d (0x733 bytes) 000380[0] 1789764063.832410 mapwrite 0x196ceb thru 0x197de0 (0x10f6 bytes) 000382[1] 1789764063.993268 punch from 0x693406 to 0x694870 (0x146b bytes) 000386[1] 1789764064.110285 read 0x4f8886 thru 0x4f9df0 (0x156b bytes) 000387[0] 1789764064.132540 trunc from 0x6b3c26 to 0x47fdbf (0x233e66 bytes) 000389[1] 1789764064.224944 write 0x192d3e thru 0x19492d (0x1bf0 bytes) 000392[0] 1789764064.339235 read 0x105a2c thru 0x107613 (0x1be8 bytes) 000394[0] 1789764064.412052 trunc from 0x47fdc0 to 0x0ece7a (0x392f45 bytes) 000396[0] 1789764064.505540 trunc from 0x0ece7b to 0x9066a8 (0x81982e bytes) 000398[1] 1789764064.612478 write 0x6065e1 thru 0x607af2 (0x1512 bytes) 000400[1] 1789764064.732519 write 0x7f1222 thru 0x7f1cd6 (0xab5 bytes) 000402[1] 1789764064.806688 write 0x949a92 thru 0x94a69c (0xc0b bytes) 000403[1] 1789764064.834256 write 0x482fb8 thru 0x4849a9 (0x19f2 bytes) 000404[1] 1789764064.864904 mapwrite 0x965f6d thru 0x9676c5 (0x1759 bytes) 000405[1] 1789764065.070752 mapwrite 0x65865e thru 0x659b51 (0x14f4 bytes) 000406[0] 1789764065.195589 trunc from 0x9676c6 to 0x8809a2 (0xe6d23 bytes) 000407[1] 1789764065.336228 read 0x7b8cc3 thru 0x7b9cde (0x101c bytes) 000408[1] 1789764065.401010 read 0x037049 thru 0x038c65 (0x1c1d bytes) 000409[0] 1789764065.442125 mapread 0x6347e1 thru 0x6364e8 (0x1d08 bytes) 000410[0] 1789764065.501470 mapwrite 0x63c6ed thru 0x63e69c (0x1fb0 bytes) 000411[0] 1789764065.650302 trunc from 0x8809a3 to 0x48c7da (0x3f41c8 bytes) 000412[1] 1789764065.748719 mapwrite 0x2aedb2 thru 0x2afc9e (0xeed bytes) 000413[0] 1789764065.882349 mapwrite 0x29cd9c thru 0x29d1d3 (0x438 bytes) 000414[1] 1789764065.989141 punch from 0x1fc680 to 0x1fcde5 (0x766 bytes) 000415[0] 1789764066.022589 read 0x2b433f thru 0x2b4ed8 (0xb9a bytes) 000420[1] 1789764066.160459 mapread 0x251e2e thru 0x2525dc (0x7af bytes) 000422[1] 1789764066.221401 mapread 0x2cc5c4 thru 0x2ce2db (0x1d18 bytes) 000423[1] 1789764066.260790 read 0x0cef9f thru 0x0d0641 (0x16a3 bytes) 000426[1] 1789764066.305612 read 0x212dfb thru 0x2147fc (0x1a02 bytes) 000427[1] 1789764066.341572 trunc from 0x48c7db to 0x0e5bc6 (0x3a6c14 bytes) 000429[1] 1789764066.439883 read 0x08526c thru 0x08701e (0x1db3 bytes) 000430[0] 1789764066.485909 read 0x02a0f1 thru 0x02ba91 (0x19a1 bytes) 000431[0] 1789764066.571430 write 0x08f140 thru 0x090dcb (0x1c8c bytes) 000433[0] 1789764066.662220 mapwrite 0x9ab57c thru 0x9ac9a7 (0x142c bytes) 000435[1] 1789764066.881111 mapwrite 0x5773fb thru 0x5775b1 (0x1b7 bytes) 000436[0] 1789764067.001231 mapwrite 0x01d27f thru 0x01d411 (0x193 bytes) 000437[1] 1789764067.099447 trunc from 0x9ac9a8 to 0x3073a4 (0x6a5603 bytes) 000438[0] 1789764067.194630 mapwrite 0x2ba7b2 thru 0x2bb869 (0x10b8 bytes) 000439[1] 1789764067.343216 mapwrite 0x7bea15 thru 0x7bfc42 (0x122e bytes) truncating to largest ever: 0x9f7891 000440[0] 1789764067.647534 trunc from 0x7bfc43 to 0x9f7890 (0x237c4e bytes) 000442[0] 1789764067.738387 read 0x41fc7e thru 0x4206f5 (0xa78 bytes) 000443[0] 1789764067.745507 write 0x8b26c0 thru 0x8b3a46 (0x1387 bytes) 000444[1] 1789764067.754409 mapread 0x980a74 thru 0x9819db (0xf68 bytes) 000447[1] 1789764067.804268 mapread 0x274f08 thru 0x276ca2 (0x1d9b bytes) 000448[0] 1789764067.869676 write 0x9d543f thru 0x9d6a8e (0x1650 bytes) 000449[1] 1789764067.903430 mapwrite 0x2a2a04 thru 0x2a4594 (0x1b91 bytes) 000450[1] 1789764068.216340 punch from 0x6f8046 to 0x6f9b78 (0x1b33 bytes) 000452[1] 1789764068.266980 read 0x1f937c thru 0x1f9846 (0x4cb bytes) 000453[1] 1789764068.293592 read 0x9805bf thru 0x9808f2 (0x334 bytes) 000455[0] 1789764068.326745 mapread 0x9914b4 thru 0x991529 (0x76 bytes) 000457[1] 1789764068.364436 read 0x89ce32 thru 0x89d287 (0x456 bytes) 000458[0] 1789764068.370429 mapwrite 0x680674 thru 0x6807a5 (0x132 bytes) 000459[1] 1789764068.434404 write 0x768660 thru 0x769715 (0x10b6 bytes) 000460[1] 1789764068.456917 mapwrite 0x754302 thru 0x7544eb (0x1ea bytes) 000463[0] 1789764068.608183 mapread 0x14d9b0 thru 0x14f4d8 (0x1b29 bytes) 000465[0] 1789764068.667571 trunc from 0x9f7891 to 0x26be78 (0x78ba18 bytes) 000466[0] 1789764068.784740 read 0x220c84 thru 0x221a89 (0xe06 bytes) 000468[1] 1789764068.943733 write 0x34f3dd thru 0x350012 (0xc36 bytes) 000472[1] 1789764069.100523 write 0x96ec12 thru 0x96f6d9 (0xac8 bytes) 000473[1] 1789764069.152424 read 0x3210f9 thru 0x32198a (0x892 bytes) 000474[0] 1789764069.219420 write 0x516f24 thru 0x517b25 (0xc02 bytes) 000477[1] 1789764069.329260 mapread 0x5a2402 thru 0x5a2f46 (0xb45 bytes) 000478[0] 1789764069.482690 mapwrite 0x12b81c thru 0x12d6a8 (0x1e8d bytes) 000480[1] 1789764069.855507 mapwrite 0x847876 thru 0x8481e2 (0x96d bytes) 000481[1] 1789764070.207349 write 0x0183b3 thru 0x019fbe (0x1c0c bytes) 000482[1] 1789764070.264903 mapread 0x91df54 thru 0x91ef28 (0xfd5 bytes) 000483[1] 1789764070.296556 write 0x4ff141 thru 0x50044c (0x130c bytes) 000484[1] 1789764070.397210 trunc from 0x96f6da to 0x600596 (0x36f143 bytes) 000485[0] 1789764070.483314 mapwrite 0x6294ae thru 0x62adf0 (0x1943 bytes) 000487[0] 1789764070.798429 write 0x96ef3a thru 0x970359 (0x1420 bytes) 000488[0] 1789764070.804723 mapwrite 0x7799b4 thru 0x77b28b (0x18d8 bytes) 000489[1] 1789764070.891385 mapread 0x08dc0f thru 0x08ed19 (0x110b bytes) 000490[1] 1789764070.964759 read 0x1fdda8 thru 0x1ff340 (0x1599 bytes) 000492[0] 1789764071.078539 read 0x86a256 thru 0x86b13e (0xee9 bytes) 000495[0] 1789764071.200562 write 0x4e5507 thru 0x4e6e8e (0x1988 bytes) 000496[1] 1789764071.276815 trunc from 0x97035a to 0x3dfbc8 (0x590791 bytes) 000498[0] 1789764071.447960 punch from 0x21133f to 0x211a7a (0x73c bytes) 000500[0] 1789764071.598209 punch from 0x38efea to 0x390817 (0x182e bytes) All operations completed A-OK! PASS 16b (47s) == sanityn test 16c: verify data consistency on ldiskfs with cache disabled (b=17397) ========================================================== 16:41:19 (1789764079) SKIP: sanityn test_16c dio on ldiskfs only SKIP 16c (6s) == sanityn test 16d: Verify DIO and buffer IO with two clients ========================================================== 16:41:26 (1789764086) 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.618422 s, 13.6 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.376584 s, 22.3 MB/s ldlm.namespaces.MGC192.168.204.147@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff9e5846a25800.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff9e58602b1800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff9e5846a25800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff9e58602b1800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff9e5846a25800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff9e58602b1800.lru_size=clear 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 16.6812 s, 6.3 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.15013 s, 25.3 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 3.39814 s, 30.9 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.85792 s, 13.3 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 12.5248 s, 8.4 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.71876 s, 22.2 MB/s PASS 16d (66s) == sanityn test 16e: Verify size consistency for O_DIRECT write ========================================================== 16:42:33 (1789764153) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.301068 s, 34.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0827736 s, 12.7 MB/s PASS 16e (10s) == sanityn test 16f: rw sequential consistency vs drop_caches ========================================================== 16:42:42 (1789764162) TIMEOUT PASS 16f (30s) == sanityn test 16g: mmap rw sequential consistency vs drop_caches ========================================================== 16:43:12 (1789764192) TIMEOUT PASS 16g (32s) == sanityn test 16h: mmap read after truncate file ======= 16:43:44 (1789764224) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16h (9s) == sanityn test 16i: read after truncate file ============ 16:43:53 (1789764233) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16i (8s) == sanityn test 16j: race dio with buffered i/o ========== 16:44:01 (1789764241) 1+0 records in 1+0 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.70961 s, 9.7 MB/s bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.27717 s, 20.5 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.6729 s, 9.8 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.79237 s, 14.6 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.44123 s, 10.7 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.39252 s, 11.0 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.88055 s, 9.1 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.09814 s, 23.9 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.55103 s, 16.9 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.40451 s, 7.7 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.65569 s, 7.2 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.48904 s, 10.5 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.5744 s, 10.2 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.63545 s, 7.2 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.75053 s, 7.0 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.51663 s, 7.5 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.66132 s, 7.2 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 4.00682 s, 6.5 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.97387 s, 6.6 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK PASS 16j (42s) == sanityn test 16k: Parallel FSX and drop caches should not panic ========================================================== 16:44:43 (1789764283) + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.1 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.2 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.3 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.4 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.5 -l 67108864 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read skipping zero size read truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! + fsx 1 done: rc=0 All operations completed A-OK! Fri Sep 18 16:45:37 EDT 2026 + fsx 2 done: rc=0 Fri Sep 18 16:45:38 EDT 2026 All operations completed A-OK! + fsx 3 done: rc=0 Fri Sep 18 16:45:38 EDT 2026 + fsx 4 done: rc=0 Fri Sep 18 16:45:38 EDT 2026 + fsx 5 done: rc=0 Fri Sep 18 16:45:39 EDT 2026 PASS 16k (70s) == sanityn test 17: resource creation/LVB creation race ========================================================================= 16:45:54 (1789764354) fail_loc=0x8000030a PASS 17 (17s) == sanityn test 18: mmap sanity check =========================================================================================== 16:46:11 (1789764371) mmap test1: basic mmap operation (PASS, 0.18149s) mmap test2: MAP_PRIVATE not write back (PASS, 0.13599s) mmap test3: concurrent mmap ops on two nodes (PASS, 8.0266s) mmap test4: c1 write to f1 from mmapped f2, c2 write to f1 from mmapped f1 (PASS, 4.7575s) mmap test5: read/write file to/from the buffer which mmapped to just this file (PASS, 0.12899s) mmap test6: check mmap write/read content on two nodes (PASS, 0.22527s) mmap test7: file i/o with an unmapped buffer (SKIPPED, 0s) mmap test8: SIGBUS for beyond file size (SKIPPED, 0s) mmap test9: SIGBUS for negative file offset (SKIPPED, 0s) mmap test10: multi-thread mmap access (PASS, 1.2975s) mmap test11: multi-thread shared mmap (PASS, 1.9538s) mmap test10: mtime not change for readonly mmap access (PASS, 1.1065s) PASS 18 (28s) == sanityn test 19: test concurrent uncached read races ========================================================================= 16:46:39 (1789764399) oleg447-server: error: get_param: param_path 'osd-*/lustre-OST*/read_cache_enable': No such file or directory pdsh@oleg447-client: oleg447-server: ssh exited with exit code 2 SKIP: sanityn test_19 not cache-capable obdfilter SKIP 19 (9s) == sanityn test 20: test extra readahead page left in cache ============================================================== 16:46:51 (1789764411) PASS 20 (9s) == sanityn test 21: Try to remove mountpoint on another dir ============================================================== 16:47:00 (1789764420) rmdir: removing directory, '/mnt/lustre/d21.sanityn' rmdir: failed to remove '/mnt/lustre/d21.sanityn': Device or resource busy rmdir: removing directory, '/mnt/lustre2/d21.sanityn' Removed mounted directory from another mountpoint, needs to be fixed PASS 21 (11s) == sanityn test 23: others should see updated atime while another read============================================================== 16:47:11 (1789764431) now is 1789764434 starting reads multiop /mnt/lustre/f23.sanityn vor20_c TMPPIPE=/tmp/multiop_open_wait_pipe.7636 new atime is 1789764495 PASS 23 (71s) == sanityn test 24a: lfs df [-ih] [path] test =================================================================================== 16:48:23 (1789764503) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 30720 3738624 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 30720 3738624 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre2 UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 115.2K 400 114.8K 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 114.5K 402 114.2K 1% /mnt/lustre[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.2K 400 114.8K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 402 114.2K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre2 UUID bytes Used Available Use% Mounted on lustre-MDT0000_UUID 2.1G 2.1M 2.1G 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3.6G 8.0M 3.6G 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3.6G 30.0M 3.6G 1% /mnt/lustre[OST:1] filesystem_summary: 7.2G 38.0M 7.2G 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118000 400 117600 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117298 402 116896 1% /mnt/lustre2[OST:1] filesystem_summary: 234882 386 234496 1% /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 30720 3738624 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.2K 400 114.8K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 402 114.2K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 118000 400 117600 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 117298 402 116896 1% /mnt/lustre[OST:1] filesystem_summary: 234882 386 234496 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118000 400 117600 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117298 402 116896 1% /mnt/lustre2[OST:1] filesystem_summary: 234882 386 234496 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 30720 3738624 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 30720 3738624 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre2 PASS 24a (8s) == sanityn test 24b: lfs df should show both filesystems ========================================================================= 16:48:31 (1789764511) PASS 24b (8s) == sanityn test 25a: change ACL on one mountpoint be seen on another ============================================================= 16:48:39 (1789764519) running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] PASS 25a (10s) == sanityn test 25b: change ACL under remote dir on one mountpoint be seen on another ========================================================== 16:48:50 (1789764530) SKIP: sanityn test_25b needs >= 2 MDTs SKIP 25b (3s) == sanityn test 26a: allow mtime to get older ============ 16:48:53 (1789764533) utime: good mknod times 1789764534 <= 1789764535 <= 1789764535 for /mnt/lustre/f26a utime: good mknod times 1789764534 <= 1789764535 <= 1789764535 for /mnt/lustre2/f26a utime: good utime mtimes 100000, atime 200000 utime: updated times for /mnt/lustre2/f26a PASS 26a (8s) == sanityn test 26b: sync mtime between ost and mds ====== 16:49:02 (1789764542) PASS 26b (10s) == sanityn test 26c: set-in-past on open file is not lost on close ========================================================== 16:49:11 (1789764551) multiop /mnt/lustre/f26c.sanityn vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7636 File: /mnt/lustre/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502398 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-09-18 16:49:13.000000000 -0400 Birth: 2026-09-18 16:49:13.000000000 -0400 File: /mnt/lustre2/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502398 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-09-18 16:49:13.000000000 -0400 Birth: 2026-09-18 16:49:13.000000000 -0400 PASS 26c (8s) SKIP: sanityn test_27 skipping excluded test 27 == sanityn test 30: recreate file race =================== 16:49:21 (1789764561) 0 PASS 30 (8s) == sanityn test 31a: voluntary cancel / blocking ast race======================================================================== 16:49:29 (1789764569) fail_loc=0x314 PASS 31a (8s) == sanityn test 31b: voluntary OST cancel / blocking ast race======================================================================== 16:49:37 (1789764577) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg447-server mds-ost sync done. sleep 5 for ZFS MDS Waiting for MDT destroys to complete fail_loc=0x314 fail_loc=0x316 fail_loc=0 fail_loc=0 Filesystem 1K-blocks Used Available Use% Mounted on 192.168.204.147@tcp:/lustre 7542784 40960 7497728 1% /mnt/lustre2 PASS 31b (29s) == sanityn test 31r: open-rename(replace) race =========== 16:50:06 (1789764606) -rw-r--r-- 1 root root 0 Sep 18 16:50 /mnt/lustre/f31r.sanityn.target fail_loc=0x1419 fail_val=3 PASS 31r (11s) == sanityn test 31s: open should not revalidate invalid dentry ========================================================== 16:50:18 (1789764618) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 1205: 43209 Terminated tail -f $tfile PASS 31s (10s) == sanityn test 31t: getattr should not revalidate invalid dentry ========================================================== 16:50:27 (1789764627) PASS 31t (11s) == sanityn test 32b: lockless i/o ======================== 16:50:38 (1789764638) SKIP: sanityn test_32b max_nolock_bytes is removed >= 2.17.53 SKIP 32b (5s) == sanityn test 32c: contention detection ================ 16:50:43 (1789764643) ldlm.namespaces.filter-lustre-OST0000_UUID.contended_locks=2 ldlm.namespaces.filter-lustre-OST0001_UUID.contended_locks=2 ldlm.namespaces.filter-lustre-OST0000_UUID.contention_events=0 ldlm.namespaces.filter-lustre-OST0001_UUID.contention_events=0 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.93768 s, 27.1 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.12963 s, 46.4 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.90533 s, 27.5 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.25205 s, 23.3 MB/s contention_seconds 4, contention_events 10 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.86796 s, 28.1 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.52057 s, 20.8 MB/s ldlm.namespaces.filter-lustre-OST0000_UUID.contention_events=0 ldlm.namespaces.filter-lustre-OST0001_UUID.contention_events=0 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.78352 s, 29.4 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.81932 s, 28.8 MB/s PASS 32c (33s) SKIP: sanityn test_33a skipping SLOW test 33a == sanityn test 33b: COS: cross create/delete, 2 clients, benchmark under remote dir ========================================================== 16:51:17 (1789764677) SKIP: sanityn test_33b Need two or more clients, have 1 SKIP 33b (4s) == sanityn test 33c: Cancel cross-MDT lock should trigger Sync-on-Lock-Cancel ========================================================== 16:51:21 (1789764681) SKIP: sanityn test_33c needs >= 2 MDTs SKIP 33c (3s) == sanityn test 33d: dependent transactions should trigger COS ========================================================== 16:51:24 (1789764684) SKIP: sanityn test_33d needs >= 2 MDTs SKIP 33d (3s) == sanityn test 33e: independent transactions shouldn't trigger COS ========================================================== 16:51:27 (1789764687) SKIP: sanityn test_33e needs >= 2 MDTs SKIP 33e (4s) == sanityn test 34: no lock timeout under IO ============= 16:51:31 (1789764691) fail_loc=0x512 fail_loc=0x512 lock should not expire writing on client1 reading on client2 success fail_loc=0x511 fail_loc=0x511 lock should expire writing on client1 reading on client2 success oleg447-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9e5846a25800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9e5846a25800.ost_server_uuid in IDLE state after 0 sec oleg447-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9e5846a25800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9e5846a25800.ost_server_uuid in IDLE state after 0 sec oleg447-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9e5846a25800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9e5846a25800.ost_server_uuid in IDLE state after 0 sec oleg447-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9e5846a25800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9e5846a25800.ost_server_uuid in IDLE state after 0 sec oleg447-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9e5846a25800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9e5846a25800.ost_server_uuid in IDLE state after 0 sec oleg447-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9e5846a25800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9e5846a25800.ost_server_uuid in IDLE state after 0 sec PASS 34 (153s) == sanityn test 35: -EINTR cp_ast vs. bl_ast race does not evict client ========================================================== 16:54:04 (1789764844) fail_loc=0x80000317 Race attempt 0 Wait for 50515 50536 for 60 sec... /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 1855: 50515 Killed createmany -o $MOUNT2/$tdir/a 4000 fail_loc=0x0 Filesystem Size Used Avail Use% Mounted on 192.168.204.147@tcp:/lustre 7.2G 40M 7.2G 1% /mnt/lustre 192.168.204.147@tcp:/lustre 7.2G 40M 7.2G 1% /mnt/lustre2 - open/close 565 (time 1789764863.45 total 10.02 last 56.41) - open/close 1028 (time 1789764873.46 total 20.03 last 46.23) - open/close 1514 (time 1789764883.47 total 30.04 last 48.57) - open/close 2015 (time 1789764893.48 total 40.05 last 50.05) - open/close 2455 (time 1789764903.48 total 50.05 last 43.98) - open/close 3081 (time 1789764913.50 total 60.07 last 62.52) - open/close 3637 (time 1789764923.50 total 70.07 last 55.55) total: 4000 open/close in 75.84 seconds: 52.74 ops/second PASS 35 (85s) == sanityn test 36: handle ESTALE/open-unlink correctly == 16:55:29 (1789764929) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg447-server mds-ost sync done. Waiting for MDT destroys to complete 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.20352 s, 43.6 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7636 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(0) *** before(7497728) after_dd(7439360) after(7490560) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.76037 s, 69.0 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7636 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(1) *** before(7490560) after_dd(7435264) after(7486464) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.818337 s, 64.1 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7636 sleep 5 for ZFS MDS Waiting for MDT destroys to complete *** cycle(2) *** before(7486464) after_dd(7440384) after(7491584) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.811483 s, 64.6 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7636 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(3) *** before(7491584) after_dd(7435264) after(7486464) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.526992 s, 99.5 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7636 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(4) *** before(7486464) after_dd(7432192) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.968765 s, 54.1 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7636 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(5) *** before(7497728) after_dd(7440384) after(7491584) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.784088 s, 66.9 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7636 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(6) *** before(7491584) after_dd(7440384) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.799894 s, 65.5 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7636 sleep 5 for ZFS MDS Waiting for MDT destroys to complete *** cycle(7) *** before(7497728) after_dd(7440384) after(7491584) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.758131 s, 69.2 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7636 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(8) *** before(7491584) after_dd(7434240) after(7485440) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.952205 s, 55.1 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7636 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(9) *** before(7485440) after_dd(7439360) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.694534 s, 75.5 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7636 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(10) *** before(7497728) after_dd(7440384) after(7491584) PASS 36 (307s) == sanityn test 37: check i_size is not updated for directory on close (bug 18695) ======================================================================== 17:00:36 (1789765236) multiop /mnt/lustre/d37.sanityn vD_c TMPPIPE=/tmp/multiop_open_wait_pipe.7636 - create 1049 (time 1789765251.53 total 10.01 last 104.84) - create 1958 (time 1789765261.53 total 20.01 last 90.88) - create 3141 (time 1789765271.53 total 30.01 last 118.26) - create 4355 (time 1789765281.54 total 40.02 last 121.32) - create 5511 (time 1789765291.55 total 50.02 last 115.55) - create 6670 (time 1789765301.55 total 60.02 last 115.88) - create 7937 (time 1789765311.55 total 70.03 last 126.62) - create 9167 (time 1789765321.55 total 80.03 last 122.98) total: 10000 create in 86.06 seconds: 116.19 ops/second PASS 37 (99s) == sanityn test 39a: file mtime does not change after rename ========================================================== 17:02:15 (1789765335) repeat after cancel_lru_locks PASS 39a (7s) == sanityn test 39b: file mtime the same on clients with/out lock ========================================================== 17:02:22 (1789765342) repeat after cancel_lru_locks PASS 39b (7s) == sanityn test 39c: check truncate mtime update ================================================================================ 17:02:29 (1789765349) repeat after cancel_lru_locks PASS 39c (7s) == sanityn test 39d: sync write should update mtime ====== 17:02:36 (1789765356) fail_loc=0x411 fail_loc=0 PASS 39d (6s) SKIP: sanityn test_40a skipping ALWAYS excluded test 40a == sanityn test 40b: pdirops: open|create and others ======================================================================== 17:02:43 (1789765363) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40b (16s) == sanityn test 40c: pdirops: link and others ======================================================================== 17:02:59 (1789765379) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40c (16s) == sanityn test 40d: pdirops: unlink and others ======================================================================== 17:03:16 (1789765396) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40d (18s) == sanityn test 40e: pdirops: rename and others ======================================================================== 17:03:33 (1789765413) No conflict No conflict No conflict No conflict No conflict No conflict PASS 40e (19s) == sanityn test 41a: pdirops: create vs mkdir ======================================================================== 17:03:52 (1789765432) mkdir: cannot create directory '/mnt/lustre2/f41a.sanityn': File exists Conflict PASS 41a (16s) == sanityn test 41b: pdirops: create vs create ======================================================================== 17:04:08 (1789765448) open: File exists Conflict PASS 41b (15s) == sanityn test 41c: pdirops: create vs link ======================================================================== 17:04:24 (1789765464) link: cannot create link '/mnt/lustre2/f41c.sanityn' to '/mnt/lustre2/f41c.sanityn-2': File exists Conflict PASS 41c (14s) == sanityn test 41d: pdirops: create vs unlink ======================================================================== 17:04:38 (1789765478) Conflict PASS 41d (15s) == sanityn test 41e: pdirops: create and rename (tgt) ======================================================================== 17:04:53 (1789765493) Conflict PASS 41e (18s) == sanityn test 41f: pdirops: create and rename (src) ======================================================================== 17:05:11 (1789765511) Conflict PASS 41f (15s) == sanityn test 41g: pdirops: create vs getattr ======================================================================== 17:05:27 (1789765527) Conflict PASS 41g (16s) == sanityn test 41h: pdirops: create vs readdir ======================================================================== 17:05:43 (1789765543) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 2427: lctl set_param -n fail_loc=0 2>/dev/null || true: No such file or directory Conflict PASS 41h (16s) == sanityn test 41i: reint_open: create vs create ======== 17:05:59 (1789765559) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 41i (1113s) == sanityn test 42a: pdirops: mkdir vs mkdir ======================================================================== 17:24:33 (1789766673) mkdir: cannot create directory '/mnt/lustre2/f42a.sanityn': File exists Conflict PASS 42a (13s) == sanityn test 42b: pdirops: mkdir vs create ======================================================================== 17:24:46 (1789766686) open: File exists Conflict PASS 42b (15s) == sanityn test 42c: pdirops: mkdir vs link ======================================================================== 17:25:01 (1789766701) link: cannot create link '/mnt/lustre2/f42c.sanityn' to '/mnt/lustre2/f42c.sanityn-2': File exists Conflict PASS 42c (15s) == sanityn test 42d: pdirops: mkdir vs unlink ======================================================================== 17:25:17 (1789766717) Conflict PASS 42d (16s) == sanityn test 42e: pdirops: mkdir and rename (tgt) ======================================================================== 17:25:32 (1789766732) mv: cannot overwrite directory '/mnt/lustre2/f42e.sanityn' with non-directory Conflict PASS 42e (16s) == sanityn test 42f: pdirops: mkdir and rename (src) ======================================================================== 17:25:48 (1789766748) Conflict PASS 42f (14s) == sanityn test 42g: pdirops: mkdir vs getattr ======================================================================== 17:26:02 (1789766762) Conflict PASS 42g (15s) == sanityn test 42h: pdirops: mkdir vs readdir ======================================================================== 17:26:17 (1789766777) Conflict PASS 42h (14s) == sanityn test 43a: rmdir,mkdir doesn't return -EEXIST ======================================================================== 17:26:32 (1789766792) PASS 43a (73s) == sanityn test 43b: pdirops: unlink vs create ======================================================================== 17:27:45 (1789766865) Conflict PASS 43b (17s) == sanityn test 43c: pdirops: unlink vs link ======================================================================== 17:28:02 (1789766882) Conflict PASS 43c (15s) == sanityn test 43d: pdirops: unlink vs unlink ======================================================================== 17:28:17 (1789766897) rm: cannot remove '/mnt/lustre2/f43d.sanityn': No such file or directory Conflict PASS 43d (13s) == sanityn test 43e: pdirops: unlink and rename (tgt) ======================================================================== 17:28:31 (1789766911) Conflict PASS 43e (12s) == sanityn test 43f: pdirops: unlink and rename (src) ======================================================================== 17:28:43 (1789766923) mv: cannot stat '/mnt/lustre2/f43f.sanityn': No such file or directory Conflict PASS 43f (13s) == sanityn test 43g: pdirops: unlink vs getattr ======================================================================== 17:28:56 (1789766936) stat: cannot statx '/mnt/lustre2/f43g.sanityn': No such file or directory Conflict PASS 43g (13s) == sanityn test 43h: pdirops: unlink vs readdir ======================================================================== 17:29:09 (1789766949) Conflict PASS 43h (13s) == sanityn test 43i: pdirops: unlink vs remote mkdir ===== 17:29:22 (1789766962) SKIP: sanityn test_43i needs >= 2 MDTs SKIP 43i (3s) == sanityn test 43j: racy mkdir return EEXIST ======================================================================== 17:29:25 (1789766965) mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists PASS 43j (138s) == sanityn test 43k: unlink vs create ==================== 17:31:43 (1789767103) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 43k (1302s) == sanityn test 44a: pdirops: rename tgt vs mkdir ======================================================================== 17:53:25 (1789768405) mkdir: cannot create directory '/mnt/lustre2/f44a.sanityn': File exists Conflict Fri Sep 18 17:53:39 EDT 2026 PASS 44a (21s) == sanityn test 44b: pdirops: rename tgt vs create ======================================================================== 17:53:46 (1789768426) open: File exists Conflict PASS 44b (25s) == sanityn test 44c: pdirops: rename tgt vs link ======================================================================== 17:54:11 (1789768451) link: cannot create link '/mnt/lustre2/f44c.sanityn' to '/mnt/lustre2/f44c.sanityn-3': File exists Conflict PASS 44c (18s) == sanityn test 44d: pdirops: rename tgt vs unlink ======================================================================== 17:54:29 (1789768469) Conflict PASS 44d (19s) == sanityn test 44e: pdirops: rename tgt and rename (tgt) ======================================================================== 17:54:48 (1789768488) Conflict PASS 44e (15s) == sanityn test 44f: pdirops: rename tgt and rename (src) ======================================================================== 17:55:04 (1789768504) Conflict PASS 44f (14s) == sanityn test 44g: pdirops: rename tgt vs getattr ======================================================================== 17:55:18 (1789768518) Conflict PASS 44g (15s) == sanityn test 44h: pdirops: rename tgt vs readdir ======================================================================== 17:55:32 (1789768532) Conflict PASS 44h (13s) == sanityn test 44i: pdirops: rename tgt vs remote mkdir ========================================================== 17:55:46 (1789768546) SKIP: sanityn test_44i needs >= 2 MDTs SKIP 44i (3s) == sanityn test 45a: rename,mkdir doesn't return -EEXIST ======================================================================== 17:55:49 (1789768549) PASS 45a (114s) == sanityn test 45b: pdirops: rename src vs create ======================================================================== 17:57:44 (1789768664) Conflict PASS 45b (13s) == sanityn test 45c: pdirops: rename src vs link ======================================================================== 17:57:56 (1789768676) Conflict PASS 45c (11s) == sanityn test 45d: pdirops: rename src vs unlink ======================================================================== 17:58:08 (1789768688) rm: cannot remove '/mnt/lustre2/f45d.sanityn': No such file or directory Conflict PASS 45d (12s) == sanityn test 45e: pdirops: rename src and rename (tgt) ======================================================================== 17:58:20 (1789768700) Conflict PASS 45e (12s) == sanityn test 45f: pdirops: rename src and rename (src) ======================================================================== 17:58:33 (1789768713) mv: cannot stat '/mnt/lustre2/f45f.sanityn': No such file or directory Conflict PASS 45f (12s) == sanityn test 45g: pdirops: rename src vs getattr ======================================================================== 17:58:46 (1789768726) stat: cannot statx '/mnt/lustre2/f45g.sanityn': No such file or directory Conflict PASS 45g (14s) == sanityn test 45h: pdirops: unlink vs readdir ======================================================================== 17:59:00 (1789768740) Conflict PASS 45h (12s) == sanityn test 45i: pdirops: rename src vs remote mkdir ========================================================== 17:59:12 (1789768752) SKIP: sanityn test_45i needs >= 2 MDTs SKIP 45i (3s) == sanityn test 45j: read vs rename ====================== 17:59:16 (1789768756) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 45j (835s) == sanityn test 46a: pdirops: link vs mkdir ======================================================================== 18:13:10 (1789769590) mkdir: cannot create directory '/mnt/lustre2/f46a.sanityn': File exists Conflict PASS 46a (11s) == sanityn test 46b: pdirops: link vs create ======================================================================== 18:13:21 (1789769601) open: File exists Conflict PASS 46b (10s) == sanityn test 46c: pdirops: link vs link ======================================================================== 18:13:31 (1789769611) link: cannot create link '/mnt/lustre2/f46c.sanityn' to '/mnt/lustre2/f46c.sanityn': File exists Conflict PASS 46c (9s) == sanityn test 46d: pdirops: link vs unlink ======================================================================== 18:13:41 (1789769621) Conflict PASS 46d (9s) == sanityn test 46e: pdirops: link and rename (tgt) ======================================================================== 18:13:49 (1789769629) Conflict PASS 46e (10s) == sanityn test 46f: pdirops: link and rename (src) ======================================================================== 18:13:59 (1789769639) Conflict PASS 46f (10s) == sanityn test 46g: pdirops: link vs getattr ======================================================================== 18:14:09 (1789769649) Conflict PASS 46g (10s) == sanityn test 46h: pdirops: link vs readdir ======================================================================== 18:14:19 (1789769659) Conflict PASS 46h (11s) == sanityn test 46i: pdirops: link vs remote mkdir ======= 18:14:30 (1789769670) SKIP: sanityn test_46i needs >= 2 MDTs SKIP 46i (3s) == sanityn test 47a: pdirops: remote mkdir vs mkdir ====== 18:14:34 (1789769674) SKIP: sanityn test_47a needs >= 2 MDTs SKIP 47a (2s) == sanityn test 47b: pdirops: remote mkdir vs create ===== 18:14:37 (1789769677) SKIP: sanityn test_47b needs >= 2 MDTs SKIP 47b (2s) == sanityn test 47c: pdirops: remote mkdir vs link ======= 18:14:39 (1789769679) SKIP: sanityn test_47c needs >= 2 MDTs SKIP 47c (3s) == sanityn test 47d: pdirops: remote mkdir vs unlink ===== 18:14:42 (1789769682) SKIP: sanityn test_47d needs >= 2 MDTs SKIP 47d (2s) == sanityn test 47e: pdirops: remote mkdir and rename (tgt) ========================================================== 18:14:44 (1789769684) SKIP: sanityn test_47e needs >= 2 MDTs SKIP 47e (3s) == sanityn test 47f: pdirops: remote mkdir and rename (src) ========================================================== 18:14:47 (1789769687) SKIP: sanityn test_47f needs >= 2 MDTs SKIP 47f (3s) == sanityn test 47g: pdirops: remote mkdir vs getattr ==== 18:14:50 (1789769690) SKIP: sanityn test_47g needs >= 2 MDTs SKIP 47g (2s) == sanityn test 50: osc lvb attrs: enqueue vs. CP AST ======================================================================== 18:14:53 (1789769693) 10+0 records in 10+0 records out 10240 bytes (10 kB, 10 KiB) copied, 0.182704 s, 56.0 kB/s fail_loc=0x410 fail_loc=0x0 PASS 50 (10s) == sanityn test 51a: layout lock: refresh layout should work ========================================================== 18:15:03 (1789769703) 0+1 records in 0+1 records out 158 bytes copied, 0.021665 s, 7.3 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 3690: kill: (218563) - No such process PASS 51a (7s) == sanityn test 51b: layout lock: glimpse should be able to restart if layout changed ========================================================== 18:15:10 (1789769710) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0146397 s, 69.9 kB/s fail_loc=0x1404 fail_val=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0547627 s, 19.1 MB/s 3145728 PASS 51b (18s) == sanityn test 51c: layout lock: IT_LAYOUT blocked and correct layout can be returned ========================================================== 18:15:28 (1789769728) fail_loc=0x172 Setting layout to have 2 stripes ... 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0210858 s, 48.6 kB/s PASS 51c (10s) == sanityn test 51d: layout lock: losing layout lock should clean up memory map region ========================================================== 18:15:38 (1789769738) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168583 s, 62.2 MB/s Before revoking layout lock: 1024 KB mapped PASS 51d (6s) == sanityn test 51e: lfs getstripe does not break leases, part 2 ========================================================== 18:15:44 (1789769744) /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 2453 0x995 0x240000400 write lease(2) has applied. /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 2453 0x995 0x240000400 read lease(1) has applied. PASS 51e (6s) == sanityn test 54: rename locking ======================= 18:15:50 (1789769750) llite.lustre-ffff9e5846a25800.xattr_cache=1 llite.lustre-ffff9e58602b1800.xattr_cache=1 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505015 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-18 18:15:52.000000000 -0400 Modify: 2026-09-18 18:15:52.000000000 -0400 Change: 2026-09-18 18:15:52.000000000 -0400 Birth: 2026-09-18 18:15:52.000000000 -0400 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000154 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505018 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-18 18:15:57.000000000 -0400 Modify: 2026-09-18 18:15:58.000000000 -0400 Change: 2026-09-18 18:15:58.000000000 -0400 Birth: 2026-09-18 18:15:58.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505021 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-18 18:16:03.000000000 -0400 Modify: 2026-09-18 18:16:03.000000000 -0400 Change: 2026-09-18 18:16:03.000000000 -0400 Birth: 2026-09-18 18:16:03.000000000 -0400 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505020 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-18 18:16:03.000000000 -0400 Modify: 2026-09-18 18:16:04.000000000 -0400 Change: 2026-09-18 18:16:04.000000000 -0400 Birth: 2026-09-18 18:16:03.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000154 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505024 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-18 18:16:09.000000000 -0400 Modify: 2026-09-18 18:16:10.000000000 -0400 Change: 2026-09-18 18:16:10.000000000 -0400 Birth: 2026-09-18 18:16:09.000000000 -0400 File: /mnt/lustre/d1 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505023 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-18 18:16:09.000000000 -0400 Modify: 2026-09-18 18:16:10.000000000 -0400 Change: 2026-09-18 18:16:10.000000000 -0400 Birth: 2026-09-18 18:16:09.000000000 -0400 PASS 54 (29s) == sanityn test 55a: rename vs unlink target dir ========= 18:16:19 (1789769779) fail_loc=0x80000156 mv: cannot move '/mnt/lustre/d55a.sanityn/d1/d2' to '/mnt/lustre/d55a.sanityn/d3/d2': No such file or directory PASS 55a (10s) == sanityn test 55b: rename vs unlink source dir ========= 18:16:29 (1789769789) fail_loc=0x80000156 mv: cannot stat '/mnt/lustre/d55b.sanityn/d1/d2': No such file or directory PASS 55b (10s) == sanityn test 55c: rename vs unlink orphan target dir == 18:16:40 (1789769800) fail_loc=0x156 mv: cannot move '/mnt/lustre/d55c.sanityn/d1/d2' to '/mnt/lustre/d55c.sanityn/d3/d2': No such file or directory PASS 55c (15s) == sanityn test 55d: rename file vs link ================= 18:16:55 (1789769815) fail_loc=0x155 mv: '/mnt/lustre/d55d.sanityn/f1' and '/mnt/lustre/d55d.sanityn/d55d.sanityn/f1' are the same file PASS 55d (13s) == sanityn test 55e: rename race AB/BA under the same parent dir ========================================================== 18:17:08 (1789769828) SKIP: sanityn test_55e needs >= 2 MDTs SKIP 55e (2s) == sanityn test 55f: rename: (P1/A -> P2/B) race with (P2/B -> P1/A) ========================================================== 18:17:10 (1789769830) fail_loc=0x155 We survived after P1A_P2B/P2B_P1A race test fail_loc=0x0 PASS 55f (16s) == sanityn test 55g: rename: race with trylock =========== 18:17:27 (1789769847) mdt.lustre-MDT0000.enable_rename_trylock=1 fail_loc=0x155 mv: cannot move '/mnt/lustre2/d55g.sanityn/b1/b2' to a subdirectory of itself, '/mnt/lustre2/d55g.sanityn/a1/a2/a3/a4/b2' We survived race with trylock test fail_loc=0x0 PASS 55g (18s) == sanityn test 56a: test llverdev with single large stripe ========================================================== 18:17:45 (1789769865) run_llverdev /mnt/lustre/f56a.sanityn -p -s 16000000 -c 4k llverdev: /mnt/lustre/f56a.sanityn is 16777216000000 bytes (15625 GB) in size timestamp: 1789769866 chunksize: 4096 size: 16777216000000 write offset: 16384000000kB 0.753569 MB/s read offset: 16384000000kB inf MB/s llverdev: data verified successfully PASS 56a (86s) == sanityn test 56b: test llverdev and partial verify of wide stripe file ========================================================== 18:19:11 (1789769951) run_llverdev /mnt/lustre/f56b.sanityn -p -s 32000000 -c 4k llverdev: /mnt/lustre/f56b.sanityn is 33554432000000 bytes (31250 GB) in size timestamp: 1789769954 chunksize: 4096 size: 33554432000000 write offset: 32768000000kB 2.1799 MB/s read offset: 32768000000kB inf MB/s llverdev: data verified successfully PASS 56b (64s) == sanityn test 60: Verify data_version behaviour ======== 18:20:15 (1789770015) PASS 60 (6s) == sanityn test 70a: cd directory && rm directory ======== 18:20:21 (1789770021) PASS 70a (4s) == sanityn test 70b: remove files after calling rm_entry ========================================================== 18:20:25 (1789770025) PASS 70b (6s) == sanityn test 71a: correct file map just after write operation is finished ========================================================== 18:20:31 (1789770031) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0100246 s, 6.5 MB/s SKIP: sanityn test_71a ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71a (2s) == sanityn test 71b: check fiemap support for stripecount > 1 ========================================================== 18:20:34 (1789770034) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.00928949 s, 7.1 MB/s SKIP: sanityn test_71b ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71b (2s) == sanityn test 71c: check FIEMAP_EXTENT_LAST flag with different extents number ========================================================== 18:20:36 (1789770036) SKIP: sanityn test_71c support only ldiskfs ost SKIP 71c (2s) == sanityn test 71d: fiemap corruption test with fm_extent_count=0 ========================================================== 18:20:38 (1789770038) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0153437 s, 4.3 MB/s SKIP: sanityn test_71d ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71d (2s) == sanityn test 72: getxattr/setxattr cache should be consistent between nodes ========================================================== 18:20:40 (1789770040) llite.lustre-ffff9e5846a25800.xattr_cache=1 llite.lustre-ffff9e58602b1800.xattr_cache=1 getfattr: Removing leading '/' from absolute path names user.attr1="value1" getfattr: Removing leading '/' from absolute path names user.attr1="value2" getfattr: Removing leading '/' from absolute path names getfattr: Removing leading '/' from absolute path names b362740ad371eae580e7f85452e8830b - 870e072da1ebfc20ea8c0cd3d6bdf3b6 - PASS 72 (5s) == sanityn test 73: getxattr should not cause xattr lock cancellation ========================================================== 18:20:45 (1789770045) llite.lustre-ffff9e5846a25800.xattr_cache=1 llite.lustre-ffff9e58602b1800.xattr_cache=1 getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" PASS 73 (5s) == sanityn test 74: flock deadlock: different mounts ======================================================================== 18:20:50 (1789770050) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00908874 s, 113 kB/s 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0138867 s, 73.7 kB/s 231353: get lock1 231352: get lock2 231353: done 231353: get lock2 231352: done 231352: get lock1 fcntl cmd 7 failed: Resource deadlock avoided 231352: cannot get lock1: Resource deadlock avoided fcntl cmd 7 failed: Resource deadlock avoided 231353: cannot get lock2: Resource deadlock avoided 231353: exit rc=0 231352: done 231352: exit rc=0 PASS 74 (9s) == sanityn test 75: osc: upcall after unuse lock============================================================================= 18:20:59 (1789770059) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0224667 s, 93.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141205 s, 74.3 MB/s fail_loc=0x31d /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 4373: 231934 Killed stat -c %s $DIR1/$tfile fail_loc=0 PASS 75 (9s) == sanityn test 76: Verify MDT open_files listing ======== 18:21:08 (1789770068) opening files: ........ closing files: ........ PASS 76 (40s) == sanityn test 77a: check FIFO NRS policy =============== 18:21:48 (1789770108) ost.OSS.ost.nrs_policies=fifo ost.OSS.ost_create.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_out.nrs_policies=fifo ost.OSS.ost_seq.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.119347 s, 141 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01464 s, 71.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0190079 s, 55.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0623635 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0579446 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0402676 s, 26.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0597527 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.050423 s, 20.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.060121 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0220571 s, 47.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0510825 s, 20.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.034474 s, 30.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0630355 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0751344 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0794055 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0667103 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110572 s, 9.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9e5846a25800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9e5846a25800.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0435238 s, 24.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0310338 s, 33.8 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0346078 s, 30.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0328319 s, 31.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0445121 s, 23.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0327204 s, 32.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0347752 s, 30.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0348715 s, 30.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0358576 s, 29.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0402214 s, 26.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0383601 s, 27.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0325276 s, 32.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0432654 s, 24.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0442047 s, 23.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0451169 s, 23.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.041796 s, 25.1 MB/s removed '/mnt/lustre/d77a.sanityn/nrs_w_oleg447-client.virtnet' removed '/mnt/lustre/d77a.sanityn/nrs_r_oleg447-client.virtnet' removed directory '/mnt/lustre/d77a.sanityn' PASS 77a (6s) == sanityn test 77b: check CRR-N NRS policy ============== 18:21:54 (1789770114) ost.OSS.ost.nrs_policies=crrn ost.OSS.ost_create.nrs_policies=crrn ost.OSS.ost_io.nrs_policies=crrn ost.OSS.ost_out.nrs_policies=crrn ost.OSS.ost_seq.nrs_policies=crrn ost.OSS.ost.nrs_crrn_quantum=1 ost.OSS.ost_create.nrs_crrn_quantum=1 ost.OSS.ost_io.nrs_crrn_quantum=1 ost.OSS.ost_out.nrs_crrn_quantum=1 ost.OSS.ost_seq.nrs_crrn_quantum=1 policy: crr-n, crrn_quantum 1 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.13175 s, 127 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0182683 s, 57.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.02238 s, 46.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0299222 s, 35.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0277483 s, 37.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0577091 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137536 s, 76.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0433216 s, 24.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0438641 s, 23.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0369138 s, 28.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0556649 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0350815 s, 29.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0800535 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0571537 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0516835 s, 20.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.099146 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0945231 s, 11.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9e5846a25800.lock_unused_count=7 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0233437 s, 44.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0226346 s, 46.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0257545 s, 40.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0305846 s, 34.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0398178 s, 26.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.022735 s, 46.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.02765 s, 37.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0216257 s, 48.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0289252 s, 36.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0408355 s, 25.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0223771 s, 46.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.031895 s, 32.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0351495 s, 29.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.032235 s, 32.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0281853 s, 37.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0370486 s, 28.3 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg447-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg447-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost.nrs_crrn_quantum=64 ost.OSS.ost_create.nrs_crrn_quantum=64 ost.OSS.ost_io.nrs_crrn_quantum=64 ost.OSS.ost_out.nrs_crrn_quantum=64 ost.OSS.ost_seq.nrs_crrn_quantum=64 policy: crr-n, crrn_quantum 64 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.108378 s, 155 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.017091 s, 61.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0300908 s, 34.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.036142 s, 29.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0611524 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0615396 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0615649 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0373829 s, 28.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0444929 s, 23.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0344314 s, 30.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.057273 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0330065 s, 31.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0550597 s, 19.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.03846 s, 27.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.091526 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0868589 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118036 s, 8.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9e5846a25800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9e5846a25800.lock_unused_count=3 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160749 s, 65.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152218 s, 68.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0314738 s, 33.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0253877 s, 41.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0218227 s, 48.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0247229 s, 42.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.019054 s, 55.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0217783 s, 48.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0327604 s, 32.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0216464 s, 48.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0245867 s, 42.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0377504 s, 27.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0255751 s, 41.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0234559 s, 44.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0359203 s, 29.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0333033 s, 31.5 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg447-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg447-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77b (7s) == sanityn test 77c: check ORR NRS policy ================ 18:22:01 (1789770121) ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_orr_quantum=1 ost.OSS.ost_io.nrs_orr_offset_type=physical ost.OSS.ost_io.nrs_orr_supported=reads policy: orr, orr_quantum 1, orr_offset_type physical, orr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.119989 s, 140 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0335542 s, 31.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0359574 s, 29.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0633394 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0181111 s, 57.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0291095 s, 36.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0302679 s, 34.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0763015 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.024826 s, 42.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0728688 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0631357 s, 16.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0414564 s, 25.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114691 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10087 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101955 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141959 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0543917 s, 19.3 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9e5846a25800.lock_unused_count=2 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0304779 s, 34.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.04497 s, 23.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0245894 s, 42.6 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0291586 s, 36.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0544993 s, 19.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0289171 s, 36.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.031056 s, 33.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.030811 s, 34.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0315668 s, 33.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.033363 s, 31.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0395614 s, 26.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0396516 s, 26.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.045059 s, 23.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0369305 s, 28.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0373851 s, 28.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0504201 s, 20.8 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg447-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg447-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=writes ost.OSS.ost_io.nrs_orr_quantum=64 policy: orr, orr_quantum 64, orr_offset_type physical, orr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.142356 s, 118 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0184832 s, 56.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0199114 s, 52.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0530206 s, 19.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.063204 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0567801 s, 18.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0584512 s, 17.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.046692 s, 22.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0591428 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0503982 s, 20.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.09134 s, 11.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0744264 s, 14.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0528157 s, 19.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9e5846a25800.lock_unused_count=5 ldlm.namespaces.lustre-OST0001-osc-ffff9e5846a25800.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0956641 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0697849 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0963671 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102861 s, 10.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9e5846a25800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9e5846a25800.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0213105 s, 49.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0212962 s, 49.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00805301 s, 130 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00490292 s, 214 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00393255 s, 267 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00400743 s, 262 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00525939 s, 199 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0302932 s, 34.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.020415 s, 51.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.021816 s, 48.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0354661 s, 29.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0220977 s, 47.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0252986 s, 41.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0463313 s, 22.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0122863 s, 85.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0311715 s, 33.6 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg447-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg447-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=reads_and_writes ost.OSS.ost_io.nrs_orr_offset_type=logical policy: orr, orr_quantum 64, orr_offset_type logical, orr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.110753 s, 151 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160419 s, 65.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0254026 s, 41.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0492083 s, 21.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0598169 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0645205 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0507937 s, 20.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0633965 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0798968 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0678639 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0846996 s, 12.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0414512 s, 25.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.07006 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0259464 s, 40.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0758102 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0692063 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0902058 s, 11.6 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9e5846a25800.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0380906 s, 27.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0220323 s, 47.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0207962 s, 50.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0439316 s, 23.9 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0343176 s, 30.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0477715 s, 21.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0354645 s, 29.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0382983 s, 27.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0514388 s, 20.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0267365 s, 39.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0363598 s, 28.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0377845 s, 27.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0306159 s, 34.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0355268 s, 29.5 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0404514 s, 25.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0398382 s, 26.3 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg447-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg447-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77c (8s) == sanityn test 77d: check TRR nrs policy ================ 18:22:09 (1789770129) ost.OSS.ost_io.nrs_policies=trr ost.OSS.ost_io.nrs_trr_quantum=1 ost.OSS.ost_io.nrs_trr_offset_type=physical ost.OSS.ost_io.nrs_trr_supported=reads policy: trr, trr_quantum 1, trr_offset_type physical, trr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.165866 s, 101 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0211993 s, 49.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167961 s, 62.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0696051 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0843961 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0569559 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0937186 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0992597 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0899666 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0757702 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104454 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0981101 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0931966 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.087177 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100892 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0768335 s, 13.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9e5846a25800.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff9e5846a25800.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102194 s, 10.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0490774 s, 21.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0513938 s, 20.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0537975 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0532859 s, 19.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0285407 s, 36.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0311292 s, 33.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0552324 s, 19.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0343292 s, 30.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0563796 s, 18.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0572488 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0365839 s, 28.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0403325 s, 26.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0612415 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0423245 s, 24.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0430807 s, 24.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0483602 s, 21.7 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg447-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg447-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=writes ost.OSS.ost_io.nrs_trr_quantum=64 policy: trr, trr_quantum 64, trr_offset_type physical, trr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.131642 s, 127 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0224668 s, 46.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0318854 s, 32.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0388391 s, 27.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0663162 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0605349 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0939052 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.08801 s, 11.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0733506 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0689254 s, 15.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0538137 s, 19.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104274 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0967503 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0909346 s, 11.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0695477 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10027 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0960005 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0240885 s, 43.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130926 s, 80.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012761 s, 82.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0321543 s, 32.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0238474 s, 44.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.025447 s, 41.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0252993 s, 41.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.026519 s, 39.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0505768 s, 20.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.031062 s, 33.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0322367 s, 32.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0367764 s, 28.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0373544 s, 28.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0398946 s, 26.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.040532 s, 25.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0440995 s, 23.8 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg447-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg447-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=reads_and_writes ost.OSS.ost_io.nrs_trr_offset_type=logical policy: trr, trr_quantum 64, trr_offset_type logical, trr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.195326 s, 85.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0190744 s, 55.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0249698 s, 42.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0242493 s, 43.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0479712 s, 21.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0371454 s, 28.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0511257 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0426514 s, 24.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0312988 s, 33.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0188096 s, 55.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0807651 s, 13.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0756229 s, 13.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.050698 s, 20.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.077614 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0964433 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0615164 s, 17.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9e5846a25800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9e5846a25800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124895 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00746955 s, 140 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00797597 s, 131 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0276952 s, 37.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00417838 s, 251 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0465805 s, 22.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00554299 s, 189 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.020417 s, 51.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0242341 s, 43.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0105153 s, 99.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00985622 s, 106 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00738626 s, 142 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0422746 s, 24.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00427386 s, 245 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0186822 s, 56.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0303975 s, 34.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00891179 s, 118 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg447-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg447-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77d (8s) == sanityn test 77e: check TBF NID nrs policy ============ 18:22:18 (1789770138) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_tbf_rule=start localhost nid={0@lo} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start clients nid={192.168.204.47@tcp} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start others nid={*.*.*.*@tcp} rate=50 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.111467 s, 151 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0257064 s, 40.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0201991 s, 51.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0182208 s, 57.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0376009 s, 27.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0427076 s, 24.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0276635 s, 37.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0256395 s, 40.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0380921 s, 27.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0486038 s, 21.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0378029 s, 27.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0281428 s, 37.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0248838 s, 42.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0881537 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0939535 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.098016 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0892565 s, 11.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0277176 s, 37.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0381853 s, 27.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0397069 s, 26.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0452075 s, 23.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0344689 s, 30.4 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.03417 s, 30.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0369982 s, 28.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0374937 s, 28.0 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0375134 s, 28.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0260124 s, 40.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0373149 s, 28.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0385226 s, 27.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0414197 s, 25.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0348782 s, 30.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0407711 s, 25.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0447121 s, 23.5 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg447-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg447-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change localhost rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change clients rate=101 ost.OSS.ost_io.nrs_tbf_rule=change others rate=51 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.162822 s, 103 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0239826 s, 43.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0295295 s, 35.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0588097 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.071289 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0404762 s, 25.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0536424 s, 19.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0646422 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0687733 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0607127 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.080202 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0717316 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0491309 s, 21.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0809405 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0549563 s, 19.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0696556 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0680176 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0176588 s, 59.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.03184 s, 32.9 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0417234 s, 25.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0275613 s, 38.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.02001 s, 52.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0300193 s, 34.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0475924 s, 22.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0293114 s, 35.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0305778 s, 34.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0070075 s, 150 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0544131 s, 19.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00512991 s, 204 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0175962 s, 59.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00548751 s, 191 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00552936 s, 190 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00918148 s, 114 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg447-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg447-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop localhost ost.OSS.ost_io.nrs_tbf_rule=stop clients ost.OSS.ost_io.nrs_tbf_rule=stop others 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.116048 s, 145 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0293179 s, 35.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0376892 s, 27.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0737349 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0707967 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0745275 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0889616 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0758384 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0554406 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0781681 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0691661 s, 15.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0932469 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0796735 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0902206 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0231211 s, 45.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0711875 s, 14.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9e5846a25800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9e5846a25800.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101429 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0339142 s, 30.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0344853 s, 30.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0434742 s, 24.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.043341 s, 24.2 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00777101 s, 135 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0462436 s, 22.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.032116 s, 32.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0485791 s, 21.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0363874 s, 28.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0338341 s, 31.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00378071 s, 277 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0498527 s, 21.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0103806 s, 101 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9e5846a25800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00480361 s, 218 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0419988 s, 25.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.049516 s, 21.2 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg447-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg447-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.169112 s, 99.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0244702 s, 42.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0286902 s, 36.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.061594 s, 17.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0810054 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0932054 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0827582 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0630552 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116448 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0925659 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0722561 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0484972 s, 21.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110015 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0829043 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0681918 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0665754 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0989534 s, 10.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9e5846a25800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9e5846a25800.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0181363 s, 57.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.039699 s, 26.4 MB/s 1+0 records in 1+0 records out ldlm.namespaces.lustre-OST0000-osc-ffff9e5846a25800.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0216579 s, 48.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0358087 s, 29.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0400064 s, 26.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0333167 s, 31.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0313749 s, 33.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0357025 s, 29.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0341958 s, 30.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0562084 s, 18.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0402224 s, 26.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0411241 s, 25.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0419982 s, 25.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0478889 s, 21.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0451148 s, 23.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0472559 s, 22.2 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg447-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg447-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' PASS 77e (13s) == sanityn test 77f: check TBF JobID nrs policy ========== 18:22:31 (1789770151) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start runas jobid={iozone.500 dd.500 tiotest.500} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start iozone_runas jobid={iozone.500} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg447-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.138547 s, 121 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0273574 s, 38.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0492902 s, 21.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0284986 s, 36.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0459262 s, 22.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0723905 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0907243 s, 11.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0297047 s, 35.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0265569 s, 39.5 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0304221 s, 34.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.028498 s, 36.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14905 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110089 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129074 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103468 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145807 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168589 s, 6.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0195587 s, 53.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0284219 s, 36.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0306212 s, 34.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0214235 s, 48.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0304332 s, 34.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0225596 s, 46.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0341651 s, 30.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0260342 s, 40.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0609381 s, 17.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0368476 s, 28.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0370953 s, 28.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0382851 s, 27.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0310075 s, 33.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0398461 s, 26.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0321562 s, 32.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0334884 s, 31.3 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg447-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg447-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change runas rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change iozone_runas rate=101 ost.OSS.ost_io.nrs_tbf_rule=change dd_runas rate=51 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg447-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.137373 s, 122 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0480643 s, 21.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0496599 s, 21.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0360622 s, 29.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0535375 s, 19.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0437312 s, 24.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0377291 s, 27.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0303398 s, 34.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0559324 s, 18.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0524144 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0835083 s, 12.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0592405 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0368647 s, 28.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.09004 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.084635 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0928546 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.084549 s, 12.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0464183 s, 22.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0363688 s, 28.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0260239 s, 40.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0372068 s, 28.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.037245 s, 28.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0391207 s, 26.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0444919 s, 23.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0346125 s, 30.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0542315 s, 19.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0471699 s, 22.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0476349 s, 22.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0495328 s, 21.2 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0507452 s, 20.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0531184 s, 19.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0659994 s, 15.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0601903 s, 17.4 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg447-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg447-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop runas ost.OSS.ost_io.nrs_tbf_rule=stop iozone_runas ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg447-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.165159 s, 102 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0324723 s, 32.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0475144 s, 22.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0850565 s, 12.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0828378 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100175 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0453818 s, 23.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0716239 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.083959 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103533 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111217 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0749058 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12621 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129776 s, 8.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9e5846a25800.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff9e5846a25800.lock_unused_count=7 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125678 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120248 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133734 s, 7.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0316994 s, 33.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0219828 s, 47.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0301276 s, 34.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0409478 s, 25.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0426431 s, 24.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0417675 s, 25.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0466628 s, 22.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0595806 s, 17.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0493857 s, 21.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0434856 s, 24.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.036516 s, 28.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0519158 s, 20.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.053105 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0593473 s, 17.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.05717 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0457104 s, 22.9 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg447-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg447-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_policies=fifo running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg447-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.138599 s, 121 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0228446 s, 45.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0106872 s, 98.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126206 s, 83.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0511107 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0532883 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0881335 s, 11.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0800295 s, 13.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.095298 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0958135 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0857768 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104832 s, 10.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108589 s, 9.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101568 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125967 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122781 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110603 s, 9.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied ldlm.namespaces.lustre-OST0000-osc-ffff9e5846a25800.lock_unused_count=3 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0278761 s, 37.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.043231 s, 24.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.037564 s, 27.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0458274 s, 22.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0326819 s, 32.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0391438 s, 26.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0392846 s, 26.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0502896 s, 20.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0628552 s, 16.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0677513 s, 15.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0511752 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0446818 s, 23.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0343586 s, 30.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0427771 s, 24.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0433663 s, 24.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0498594 s, 21.0 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg447-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg447-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' PASS 77f (13s) == sanityn test 77g: Change TBF type directly ============ 18:22:44 (1789770164) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 ost.OSS.ost_io.nrs_policies=fifo PASS 77g (6s) == sanityn test 77h: Wrong policy name should report error, not LBUG ========================================================== 18:22:50 (1789770170) oleg447-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=abc: No such file or directory oleg447-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='abc': No such file or directory pdsh@oleg447-client: oleg447-server: ssh exited with exit code 2 oleg447-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc: Invalid argument oleg447-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc': Invalid argument pdsh@oleg447-client: oleg447-server: ssh exited with exit code 22 oleg447-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf reg abc: Invalid argument oleg447-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf reg abc': Invalid argument pdsh@oleg447-client: oleg447-server: ssh exited with exit code 22 oleg447-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc efg: Invalid argument oleg447-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc efg': Invalid argument pdsh@oleg447-client: oleg447-server: ssh exited with exit code 22 PASS 77h (7s) == sanityn test 77i: Change rank of TBF rule ============= 18:22:57 (1789770177) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start before jobid={jobid} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start after jobid={jobid} rate=1000 rank=default ost.OSS.ost_io.nrs_tbf_rule=start target jobid={jobid} rate=1000 rank=after Move before itself ost.OSS.ost_io.nrs_tbf_rule=change target rank=target Move to higher rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=before Move to lower rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=after Move before default ost.OSS.ost_io.nrs_tbf_rule=change target rank=default ost.OSS.ost_io.nrs_policies=fifo PASS 77i (13s) == sanityn test 77j: check TBF-OPCode NRS policy ========= 18:23:10 (1789770190) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf opcode ost.OSS.ost_io.nrs_tbf_rule=start ost_r opcode={ost_read} rate=5 ost.OSS.ost_io.nrs_tbf_rule=start ost_w opcode={ost_write} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.296964 s, 56.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0221282 s, 47.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0227097 s, 46.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0121106 s, 86.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0597539 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0746723 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0522716 s, 20.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0854473 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0617593 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.064504 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0697476 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0657323 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0627284 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.240212 s, 4.4 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.238905 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.257914 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.29836 s, 3.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0104725 s, 100 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0204467 s, 51.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147141 s, 71.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0348471 s, 30.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0197212 s, 53.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0372205 s, 28.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0216663 s, 48.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0366538 s, 28.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0278759 s, 37.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00603296 s, 174 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0300906 s, 34.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0287512 s, 36.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.029247 s, 35.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0336289 s, 31.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0322259 s, 32.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0342194 s, 30.6 MB/s removed '/mnt/lustre/d77j.sanityn/nrs_r_oleg447-client.virtnet' removed '/mnt/lustre/d77j.sanityn/nrs_w_oleg447-client.virtnet' removed directory '/mnt/lustre/d77j.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 5, bs: 1048576 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.73742 s, 18.3 MB/s Write runtime is 6 s, speed is 16.666666 vs. 20 Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.5155 s, 5.4 MB/s Read runtime is 20 s, speed is 5.000000 vs. 5 cleanup_tbf /mnt/lustre/d77j.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_r ost.OSS.ost_io.nrs_tbf_rule=stop ost_w ost.OSS.ost_io.nrs_policies=fifo PASS 77j (55s) == sanityn test 77ja: check TBF-UID/GID NRS policy ======= 18:24:05 (1789770245) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg447-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg447-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg447-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg447-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg447-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.25255 s, 13.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0214816 s, 48.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.052426 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0543281 s, 19.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0434266 s, 24.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.060592 s, 17.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0699719 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0517058 s, 20.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0554942 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116532 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143546 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136513 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146267 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141391 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147332 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154216 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148296 s, 7.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0396885 s, 26.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0472554 s, 22.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0501576 s, 20.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0399048 s, 26.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0483626 s, 21.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0547577 s, 19.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.035286 s, 29.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0628577 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0545234 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0541863 s, 19.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0553303 s, 19.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0540394 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0521948 s, 20.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0575086 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0597067 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0837827 s, 12.5 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg447-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg447-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5, bs: 1048576 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1048576] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8153 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 vs. 5 Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1048576] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8953 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 vs. 5 cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg447-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg447-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg447-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg447-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg447-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.23595 s, 13.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0387303 s, 27.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0412039 s, 25.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0376824 s, 27.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0528714 s, 19.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0344742 s, 30.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0452809 s, 23.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0442363 s, 23.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0548248 s, 19.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139347 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141725 s, 7.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156474 s, 6.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140556 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125375 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128516 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1423 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17737 s, 5.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139057 s, 75.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0233541 s, 44.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0225076 s, 46.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0242162 s, 43.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0188016 s, 55.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0307397 s, 34.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0223551 s, 46.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126835 s, 82.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0318957 s, 32.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0362284 s, 28.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0301647 s, 34.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.041065 s, 25.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0281748 s, 37.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0416405 s, 25.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0320171 s, 32.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00281552 s, 372 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg447-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg447-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5, bs: 1048576 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1048576] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.7522 s, 5.3 MB/s Write runtime is 20 s, speed is 5.000000 vs. 5 Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1048576] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9666 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 vs. 5 cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77ja (137s) == sanityn test 77jb: check TBF-UID/GID NRS policy on files that don't belong to us ========================================================== 18:26:22 (1789770382) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg447-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg447-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg447-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg447-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg447-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.2216 s, 13.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0231881 s, 45.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0337205 s, 31.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.030946 s, 33.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0390311 s, 26.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0186483 s, 56.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0308487 s, 34.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0360185 s, 29.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0384513 s, 27.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159399 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159851 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177125 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165431 s, 6.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194123 s, 5.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191387 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170781 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17494 s, 6.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.031792 s, 33.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0284925 s, 36.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0363785 s, 28.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0293697 s, 35.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.031886 s, 32.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0387053 s, 27.1 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0336448 s, 31.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0405379 s, 25.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0407955 s, 25.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0389239 s, 26.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0380468 s, 27.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0427767 s, 24.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0413218 s, 25.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0364163 s, 28.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0473553 s, 22.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0411309 s, 25.5 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg447-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg447-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5, bs: 1048576 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1048576] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.7687 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 vs. 5 Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1048576] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.928 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 vs. 5 cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg447-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg447-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg447-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg447-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg447-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.2326 s, 13.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0246532 s, 42.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0333237 s, 31.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0286571 s, 36.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158124 s, 66.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0325524 s, 32.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0257934 s, 40.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0124116 s, 84.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0177986 s, 58.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14489 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148018 s, 7.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155909 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140126 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179318 s, 5.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139303 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162686 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205023 s, 5.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.025306 s, 41.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0255527 s, 41.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0250174 s, 41.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0198945 s, 52.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0315696 s, 33.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0319541 s, 32.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0278825 s, 37.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0213405 s, 49.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0360114 s, 29.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0281772 s, 37.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0266819 s, 39.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0330149 s, 31.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0394193 s, 26.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0301816 s, 34.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0403544 s, 26.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0387436 s, 27.1 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg447-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg447-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5, bs: 1048576 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1048576] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.7561 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 vs. 5 Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1048576] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9365 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 vs. 5 cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77jb (139s) == sanityn test 77k: check TBF policy with UID/GID/JobID/OPCode expression ========================================================== 18:28:41 (1789770521) ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg447-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.283933 s, 59.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162329 s, 64.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0322572 s, 32.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0403432 s, 26.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0374397 s, 28.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0442576 s, 23.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0269387 s, 38.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0443433 s, 23.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0285557 s, 36.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0674262 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0365167 s, 28.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0583892 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0458222 s, 22.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0515629 s, 20.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0724381 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0620672 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0369363 s, 28.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00957156 s, 110 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0112071 s, 93.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129748 s, 80.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0244944 s, 42.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0213523 s, 49.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0217924 s, 48.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0226708 s, 46.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0304473 s, 34.4 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0306772 s, 34.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0342483 s, 30.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0233991 s, 44.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0264563 s, 39.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0348348 s, 30.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.042741 s, 24.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0273046 s, 38.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.029027 s, 36.1 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg447-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg447-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10, bs: 1048576 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1048576] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.03369 s, 20.8 MB/s Write runtime is 6 s, speed is 16.666666 vs. 20 Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1048576] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.77251 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 vs. 10 cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.204.47@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.204.47@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.237409 s, 70.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140439 s, 74.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0193235 s, 54.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0211745 s, 49.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0486976 s, 21.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.062519 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0653606 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0426006 s, 24.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0464529 s, 22.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0383511 s, 27.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0385523 s, 27.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0689862 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0469336 s, 22.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0531855 s, 19.7 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0768984 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.219376 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216372 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0354397 s, 29.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0371078 s, 28.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0253901 s, 41.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0468184 s, 22.4 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0445869 s, 23.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0397627 s, 26.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0358947 s, 29.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0520731 s, 20.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0362592 s, 28.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0415614 s, 25.2 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0371879 s, 28.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0506574 s, 20.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0409098 s, 25.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0383794 s, 27.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0379804 s, 27.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0418309 s, 25.1 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg447-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg447-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10, bs: 1048576 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.05618 s, 20.7 MB/s Write runtime is 7 s, speed is 14.285714 vs. 20 Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.77463 s, 10.7 MB/s Read runtime is 10 s, speed is 10.000000 vs. 10 cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.204.47@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg447-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.232273 s, 72.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0257731 s, 40.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0272571 s, 38.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0352578 s, 29.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0563656 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0409938 s, 25.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0440743 s, 23.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0412058 s, 25.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0567618 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0393937 s, 26.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0694496 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0854604 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0858018 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0908006 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10313 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115514 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118059 s, 8.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0251883 s, 41.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0216488 s, 48.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0222469 s, 47.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0391507 s, 26.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0258549 s, 40.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0243328 s, 43.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0266939 s, 39.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0247131 s, 42.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0272066 s, 38.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0406715 s, 25.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0292113 s, 35.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.032772 s, 32.0 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0300205 s, 34.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0296133 s, 35.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0321601 s, 32.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0362354 s, 28.9 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg447-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg447-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20, bs: 1048576 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1048576] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.07603 s, 20.7 MB/s Write runtime is 6 s, speed is 16.666666 vs. 20 Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1048576] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.92579 s, 21.3 MB/s Read runtime is 6 s, speed is 16.666666 vs. 20 cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg447-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.434933 s, 38.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0203241 s, 51.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0297269 s, 35.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0344377 s, 30.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0331995 s, 31.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0389789 s, 26.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.047559 s, 22.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.033642 s, 31.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0435959 s, 24.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0613786 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0579213 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0866194 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0809296 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.291054 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.303894 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.346785 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.368073 s, 2.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0184766 s, 56.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0235879 s, 44.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0210418 s, 49.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.018637 s, 56.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0248389 s, 42.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0235873 s, 44.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0241393 s, 43.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0175909 s, 59.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0174416 s, 60.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0232384 s, 45.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0171295 s, 61.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0240922 s, 43.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0115008 s, 91.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0261357 s, 40.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00388387 s, 270 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0288347 s, 36.4 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg447-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg447-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10, bs: 1048576 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1048576] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.82302 s, 10.7 MB/s Write runtime is 11 s, speed is 9.090909 vs. 10 Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1048576] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.75965 s, 10.7 MB/s Read runtime is 10 s, speed is 10.000000 vs. 10 cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10, bs: 1048576 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.9444 s, 21.2 MB/s Write runtime is 6 s, speed is 16.666666 vs. 20 Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.7655 s, 10.7 MB/s Read runtime is 10 s, speed is 10.000000 vs. 10 cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg447-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.844577 s, 19.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0305405 s, 34.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0333982 s, 31.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0111146 s, 94.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169632 s, 61.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136154 s, 77.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0282051 s, 37.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0357152 s, 29.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143654 s, 73.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137964 s, 7.6 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134903 s, 7.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135561 s, 7.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157817 s, 6.6 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147694 s, 7.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130518 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170309 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162387 s, 6.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0308961 s, 33.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0181995 s, 57.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0259701 s, 40.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0263294 s, 39.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0219991 s, 47.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0239524 s, 43.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0306481 s, 34.2 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0245339 s, 42.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0268776 s, 39.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0278254 s, 37.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0309476 s, 33.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0314194 s, 33.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0466169 s, 22.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0302996 s, 34.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0337455 s, 31.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0377894 s, 27.7 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg447-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg447-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5, bs: 1048576 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1048576] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.5606 s, 5.4 MB/s Write runtime is 21 s, speed is 4.761904 vs. 5 Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1048576] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.4595 s, 5.4 MB/s Read runtime is 20 s, speed is 5.000000 vs. 5 cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_ug ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg447-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.247279 s, 67.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0185292 s, 56.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0252951 s, 41.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0177306 s, 59.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0374035 s, 28.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00927106 s, 113 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0303427 s, 34.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0514631 s, 20.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0275221 s, 38.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0679273 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00843375 s, 124 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10645 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0947745 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117424 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108478 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125484 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0850798 s, 12.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0239589 s, 43.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0275354 s, 38.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0262517 s, 39.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0274899 s, 38.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0340202 s, 30.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0356411 s, 29.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0291602 s, 36.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0315106 s, 33.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0368037 s, 28.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0334025 s, 31.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0431069 s, 24.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0359972 s, 29.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0391178 s, 26.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0356953 s, 29.4 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.034289 s, 30.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0421241 s, 24.9 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg447-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg447-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10, bs: 1048576 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1048576] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.02602 s, 20.9 MB/s Write runtime is 6 s, speed is 16.666666 vs. 20 Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1048576] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.76505 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 vs. 10 cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg447-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.437532 s, 38.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0207711 s, 50.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0171341 s, 61.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0114587 s, 91.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0213196 s, 49.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0213528 s, 49.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.025087 s, 41.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0347373 s, 30.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0274859 s, 38.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.05645 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.056544 s, 18.5 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0825329 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.067533 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.265476 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.262131 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.298051 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.300141 s, 3.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149293 s, 70.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0191346 s, 54.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0275333 s, 38.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0267453 s, 39.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0209983 s, 49.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00679018 s, 154 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0261665 s, 40.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.027672 s, 37.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0295223 s, 35.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0320897 s, 32.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0248228 s, 42.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0114159 s, 91.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00717043 s, 146 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0373081 s, 28.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0106573 s, 98.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00665353 s, 158 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg447-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg447-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10, bs: 1048576 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1048576] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.82384 s, 10.7 MB/s Write runtime is 11 s, speed is 9.090909 vs. 10 Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1048576] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.75453 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 vs. 10 cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10, bs: 1048576 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1048576] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.82198 s, 10.7 MB/s Write runtime is 11 s, speed is 9.090909 vs. 20 Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1048576] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.75941 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 vs. 10 cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Cleanup TBF ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_policies=fifo Cleanup TBF ost.OSS.ost_io.nrs_policies=fifo PASS 77k (363s) == sanityn test 77kb: Check different granular TBF type combination: jobid+opcode ========================================================== 18:34:44 (1789770884) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/nrs_r_oleg447-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.267291 s, 62.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0104539 s, 100 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0317121 s, 33.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0290123 s, 36.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.033665 s, 31.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0324503 s, 32.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0413412 s, 25.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0359261 s, 29.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0344793 s, 30.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0401834 s, 26.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.034008 s, 30.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0339569 s, 30.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0328529 s, 31.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0401725 s, 26.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0289574 s, 36.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0509584 s, 20.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0541118 s, 19.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0195996 s, 53.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0175159 s, 59.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0207495 s, 50.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0210564 s, 49.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0302088 s, 34.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0206806 s, 50.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0226989 s, 46.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0235094 s, 44.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0287957 s, 36.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0196041 s, 53.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0256141 s, 40.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0227778 s, 46.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0225491 s, 46.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0226645 s, 46.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0254997 s, 41.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00528989 s, 198 MB/s removed '/mnt/lustre/d77kb.sanityn/nrs_r_oleg447-client.virtnet' removed '/mnt/lustre/d77kb.sanityn/nrs_w_oleg447-client.virtnet' removed directory '/mnt/lustre/d77kb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10, bs: 1048576 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/tbf] [bs=1048576] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.94316 s, 21.2 MB/s Write runtime is 6 s, speed is 16.666666 vs. 20 Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kb.sanityn/tbf] [of=/dev/null] [bs=1048576] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.74124 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 vs. 10 cleanup_tbf /mnt/lustre/d77kb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Cleanup TBF ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77kb (37s) == sanityn test 77kc: Check different granular TBF type combination: nid+opcode ========================================================== 18:35:21 (1789770921) ost.OSS.ost_io.nrs_policies=tbf nid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.204.47@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.204.47@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.253599 s, 66.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012179 s, 86.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0272275 s, 38.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0298432 s, 35.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0398292 s, 26.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0418042 s, 25.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0240169 s, 43.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0420841 s, 24.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0482579 s, 21.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.042561 s, 24.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0362064 s, 29.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0814291 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0930732 s, 11.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.250344 s, 4.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.253959 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.285847 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.303862 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0217874 s, 48.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0225186 s, 46.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0199495 s, 52.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0238647 s, 43.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0188075 s, 55.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150008 s, 69.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0109423 s, 95.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152075 s, 69.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.02699 s, 38.9 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00839963 s, 125 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0101552 s, 103 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152834 s, 68.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0279565 s, 37.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00514463 s, 204 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00768131 s, 137 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0196262 s, 53.4 MB/s removed '/mnt/lustre/d77kc.sanityn/nrs_r_oleg447-client.virtnet' removed '/mnt/lustre/d77kc.sanityn/nrs_w_oleg447-client.virtnet' removed directory '/mnt/lustre/d77kc.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10, bs: 1048576 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.99611 s, 21.0 MB/s Write runtime is 6 s, speed is 16.666666 vs. 20 Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.74224 s, 10.8 MB/s Read runtime is 10 s, speed is 10.000000 vs. 10 cleanup_tbf /mnt/lustre/d77kc.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Cleanup TBF ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77kc (46s) == sanityn test 77kd: Check different granular TBF type combination: nid+jobid ========================================================== 18:36:07 (1789770967) ost.OSS.ost_io.nrs_policies=tbf nid+jobid ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.204.47@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/nrs_r_oleg447-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.275963 s, 60.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0280082 s, 37.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0238779 s, 43.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0627328 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0653107 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0622777 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0492415 s, 21.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0597983 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0740057 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.070251 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0655748 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0759902 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.066386 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0622497 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0570268 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0757688 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0839843 s, 12.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0218737 s, 47.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0185877 s, 56.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0222292 s, 47.2 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.019594 s, 53.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0225752 s, 46.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0242271 s, 43.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152699 s, 68.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0245906 s, 42.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0256711 s, 40.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0227836 s, 46.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.024564 s, 42.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0274404 s, 38.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0287992 s, 36.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0298954 s, 35.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.026173 s, 40.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0309164 s, 33.9 MB/s removed '/mnt/lustre/d77kd.sanityn/nrs_w_oleg447-client.virtnet' removed '/mnt/lustre/d77kd.sanityn/nrs_r_oleg447-client.virtnet' removed directory '/mnt/lustre/d77kd.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20, bs: 1048576 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/tbf] [bs=1048576] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.96233 s, 21.1 MB/s Write runtime is 6 s, speed is 16.666666 vs. 20 Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kd.sanityn/tbf] [of=/dev/null] [bs=1048576] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.97858 s, 21.1 MB/s Read runtime is 6 s, speed is 16.666666 vs. 20 cleanup_tbf /mnt/lustre/d77kd.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Cleanup TBF ost.OSS.ost_io.nrs_tbf_rule=stop ext ost.OSS.ost_io.nrs_policies=fifo PASS 77kd (40s) == sanityn test 77ke: Check different granular TBF type combination: jobid+opcode ========================================================== 18:36:47 (1789771007) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/nrs_r_oleg447-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.464616 s, 36.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0248234 s, 42.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0242004 s, 43.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0661704 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0673273 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0735302 s, 14.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0888493 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0743914 s, 14.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0936203 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0639335 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0704533 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102392 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106141 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.316237 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.302989 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.349988 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.351164 s, 3.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158178 s, 66.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162794 s, 64.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00985459 s, 106 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0232043 s, 45.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0219956 s, 47.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0121076 s, 86.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142775 s, 73.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0244721 s, 42.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0204814 s, 51.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0174849 s, 60.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0198783 s, 52.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0201845 s, 51.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0208736 s, 50.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0226556 s, 46.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0258573 s, 40.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00317658 s, 330 MB/s removed '/mnt/lustre/d77ke.sanityn/nrs_w_oleg447-client.virtnet' removed '/mnt/lustre/d77ke.sanityn/nrs_r_oleg447-client.virtnet' removed directory '/mnt/lustre/d77ke.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10, bs: 1048576 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/tbf] [bs=1048576] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.83987 s, 10.7 MB/s Write runtime is 11 s, speed is 9.090909 vs. 10 Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ke.sanityn/tbf] [of=/dev/null] [bs=1048576] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.75092 s, 10.8 MB/s Read runtime is 10 s, speed is 10.000000 vs. 10 cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10, bs: 1048576 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.09227 s, 20.6 MB/s Write runtime is 6 s, speed is 16.666666 vs. 20 Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.76827 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 vs. 10 cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Cleanup TBF ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_policies=fifo PASS 77ke (82s) == sanityn test 77kf: Check different granular TBF type combination: uid+gid ========================================================== 18:38:09 (1789771089) ost.OSS.ost_io.nrs_policies=tbf uid+gid ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/nrs_r_oleg447-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.879426 s, 19.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0230127 s, 45.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0417741 s, 25.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0206286 s, 50.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0444237 s, 23.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0515184 s, 20.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0351906 s, 29.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0418878 s, 25.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0476812 s, 22.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157747 s, 6.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174036 s, 6.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171675 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193706 s, 5.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189747 s, 5.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191984 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206334 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191605 s, 5.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0211299 s, 49.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152675 s, 68.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0237897 s, 44.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.018337 s, 57.2 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0174684 s, 60.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0212141 s, 49.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135366 s, 77.5 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0185115 s, 56.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0295238 s, 35.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.026334 s, 39.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0225175 s, 46.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0209071 s, 50.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0186969 s, 56.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0215357 s, 48.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0234651 s, 44.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0255639 s, 41.0 MB/s removed '/mnt/lustre/d77kf.sanityn/nrs_w_oleg447-client.virtnet' removed '/mnt/lustre/d77kf.sanityn/nrs_r_oleg447-client.virtnet' removed directory '/mnt/lustre/d77kf.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5, bs: 1048576 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/tbf] [bs=1048576] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.7344 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 vs. 5 Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kf.sanityn/tbf] [of=/dev/null] [bs=1048576] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9329 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 vs. 5 cleanup_tbf /mnt/lustre/d77kf.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Cleanup TBF ost.OSS.ost_io.nrs_tbf_rule=stop ext_ug ost.OSS.ost_io.nrs_policies=fifo PASS 77kf (71s) == sanityn test 77kg: check different granular TBF type combination: uid+opcode ========================================================== 18:39:20 (1789771160) ost.OSS.ost_io.nrs_policies=tbf uid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg447-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.248254 s, 67.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145349 s, 72.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.033854 s, 31.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0307578 s, 34.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.044502 s, 23.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0326103 s, 32.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0313608 s, 33.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0342091 s, 30.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0465808 s, 22.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0418019 s, 25.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0411731 s, 25.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0396279 s, 26.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0286269 s, 36.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0369741 s, 28.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0321315 s, 32.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00830793 s, 126 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0122808 s, 85.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00836908 s, 125 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0118281 s, 88.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0186453 s, 56.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143153 s, 73.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0120412 s, 87.1 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0122293 s, 85.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0243368 s, 43.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159924 s, 65.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136809 s, 76.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0181561 s, 57.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0203304 s, 51.6 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183656 s, 57.1 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.019589 s, 53.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00294338 s, 356 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00805615 s, 130 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00302698 s, 346 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg447-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg447-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10, bs: 1048576 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1048576] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.94882 s, 21.2 MB/s Write runtime is 6 s, speed is 16.666666 vs. 20 Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1048576] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.74193 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 vs. 10 cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg447-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.433181 s, 38.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0216567 s, 48.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0368953 s, 28.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0211708 s, 49.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0185835 s, 56.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0260441 s, 40.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0238914 s, 43.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0236119 s, 44.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0258234 s, 40.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0647839 s, 16.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0647776 s, 16.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109857 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100006 s, 10.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0939357 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105053 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.360077 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.35012 s, 3.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143818 s, 72.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0216351 s, 48.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0214647 s, 48.9 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0192351 s, 54.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0211782 s, 49.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.020697 s, 50.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0213 s, 49.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0200812 s, 52.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0203544 s, 51.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0252729 s, 41.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00411392 s, 255 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0171609 s, 61.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0284393 s, 36.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00406182 s, 258 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0256514 s, 40.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00801181 s, 131 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg447-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg447-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10, bs: 1048576 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1048576] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.83073 s, 10.7 MB/s Write runtime is 11 s, speed is 9.090909 vs. 10 Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1048576] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.75724 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 vs. 10 cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10, bs: 1048576 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1048576] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.77921 s, 10.7 MB/s Write runtime is 10 s, speed is 10.000000 vs. 20 Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1048576] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.74398 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 vs. 10 cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Cleanup TBF ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_policies=fifo PASS 77kg (125s) == sanityn test 77kh: Verify that Project ID support for NRS TBF rule ========================================================== 18:41:25 (1789771285) using SAVE_PROJECT_SUPPORTED=0 Stopping clients: oleg447-client.virtnet /mnt/lustre (opts:) Stopping client oleg447-client.virtnet /mnt/lustre opts: Stopping clients: oleg447-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg447-client.virtnet /mnt/lustre2 opts: Stopping /mnt/lustre-mds1 (opts:-f) on oleg447-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg447-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg447-server sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime) proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) devtmpfs on /dev type devtmpfs (rw,nosuid,mode=755) securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime) tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev) devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000) tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755) tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755) cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd) pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime) bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700) cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids) cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event) cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio) cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio) cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma) cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer) cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices) cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct) cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset) cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb) cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory) cgroup on /sys/fs/cgroup/misc type cgroup (rw,nosuid,nodev,noexec,relatime,misc) none on /sys/kernel/tracing type tracefs (rw,relatime) configfs on /sys/kernel/config type configfs (rw,relatime) /dev/nbd0 on / type ext4 (ro,relatime,stripe=32) rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) mqueue on /dev/mqueue type mqueue (rw,relatime) hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M) debugfs on /sys/kernel/debug type debugfs (rw,relatime) none on /mnt type ramfs (rw,relatime) none on /var/lib/stateless/writable type tmpfs (rw,relatime) /dev/vda on /home/green/git/lustre-release type squashfs (ro,relatime) none on /var/cache/man type tmpfs (rw,relatime) none on /var/log type tmpfs (rw,relatime) none on /var/lib/dbus type tmpfs (rw,relatime) none on /tmp type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) none on /var/tmp type tmpfs (rw,relatime) none on /var/lib/NetworkManager type tmpfs (rw,relatime) none on /var/lib/systemd/random-seed type tmpfs (rw,relatime) none on /var/spool type tmpfs (rw,relatime) none on /var/lib/nfs type tmpfs (rw,relatime) none on /var/lib/gssproxy type tmpfs (rw,relatime) none on /var/lib/logrotate type tmpfs (rw,relatime) none on /etc type tmpfs (rw,relatime) none on /var/lib/rsyslog type tmpfs (rw,relatime) none on /root/.ssh type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) 192.168.200.253:/exports/state/oleg447-client.virtnet on /var/lib/stateless/state type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.204.47,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg447-client.virtnet/boot on /boot type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.204.47,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg447-client.virtnet/etc/kdump.conf on /etc/kdump.conf type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.204.47,local_lock=none,addr=192.168.200.253) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=378180k,mode=700) 192.168.200.253:/exports/testreports/70513/testresults/sanityn-zfs-rocky8.10_x86_64-rocky8.10_x86_64 on /tmp/testlogs type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.204.47,local_lock=none,addr=192.168.200.253) /dev/vda on /usr/sbin/mount.lustre type squashfs (ro,relatime) Checking servers environments Checking clients oleg447-client.virtnet environments Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg447-server' oleg447-server: oleg447-server.virtnet: executing load_modules_local oleg447-server: kmemleak disabled oleg447-server: Loading modules from /home/green/git/lustre-release/lustre oleg447-server: detected 4 online CPUs by sysfs oleg447-server: MODOPTS_LIBCFS= oleg447-server: Force libcfs to create 2 CPU partitions oleg447-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Setup mgs, mdt, osts Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg447-server: oleg447-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg447-client: oleg447-server: ssh exited with exit code 1 Started lustre-MDT0000 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg447-server: oleg447-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg447-client: oleg447-server: ssh exited with exit code 1 Started lustre-OST0000 Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg447-server: oleg447-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg447-client: oleg447-server: ssh exited with exit code 1 Started lustre-OST0001 Starting client: oleg447-client.virtnet: -o user_xattr,flock 192.168.204.147@tcp:/lustre /mnt/lustre Starting client oleg447-client.virtnet: -o user_xattr,flock 192.168.204.147@tcp:/lustre /mnt/lustre Started clients oleg447-client.virtnet: 192.168.204.147@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Starting client: oleg447-client.virtnet: -o user_xattr,flock 192.168.204.147@tcp:/lustre /mnt/lustre2 Starting client oleg447-client.virtnet: -o user_xattr,flock 192.168.204.147@tcp:/lustre /mnt/lustre2 Started clients oleg447-client.virtnet: 192.168.204.147@tcp:/lustre on /mnt/lustre2 type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9e5847ff5800.idle_timeout=debug osc.lustre-OST0000-osc-ffff9e587aae3800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9e5847ff5800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9e587aae3800.idle_timeout=debug disable quota as required ost.OSS.ost_io.nrs_policies=tbf projid ost.OSS.ost_io.nrs_tbf_rule=start ext_proj_rw projid={100} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.256216 s, 65.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.031073 s, 33.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0267327 s, 39.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0849367 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0856525 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0827499 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0872304 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0815284 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0921755 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0808788 s, 13.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10291 s, 10.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0907252 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111377 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0963086 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0923361 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.253355 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.226473 s, 4.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0335154 s, 31.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0351131 s, 29.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0181657 s, 57.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.028777 s, 36.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0329891 s, 31.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0208565 s, 50.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0357725 s, 29.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0324051 s, 32.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0301103 s, 34.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0297435 s, 35.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0282537 s, 37.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0334773 s, 31.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0395691 s, 26.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0330878 s, 31.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0333992 s, 31.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0380525 s, 27.6 MB/s removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg447-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg447-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20, bs: 1048576 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.98593 s, 21.0 MB/s Write runtime is 6 s, speed is 16.666666 vs. 20 Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.93874 s, 21.2 MB/s Read runtime is 6 s, speed is 16.666666 vs. 20 cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Cleanup TBF ost.OSS.ost_io.nrs_tbf_rule=stop ext_proj_rw ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf projid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w projid={100}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r projid={100}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.30047 s, 55.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0389547 s, 26.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0447707 s, 23.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0554799 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0996599 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0985323 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121479 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0994375 s, 10.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0954411 s, 11.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111093 s, 9.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131336 s, 8.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113184 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103231 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111227 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106585 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213799 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237365 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0248564 s, 42.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0231517 s, 45.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.030538 s, 34.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.030732 s, 34.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0217001 s, 48.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.021589 s, 48.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0264348 s, 39.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0281061 s, 37.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0320371 s, 32.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0244385 s, 42.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.031248 s, 33.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0312123 s, 33.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0393869 s, 26.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00496553 s, 211 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0345061 s, 30.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160658 s, 65.3 MB/s removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg447-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg447-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10, bs: 1048576 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.97292 s, 21.1 MB/s Write runtime is 6 s, speed is 16.666666 vs. 20 Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.755 s, 10.7 MB/s Read runtime is 10 s, speed is 10.000000 vs. 10 cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Cleanup TBF ost.OSS.ost_io.nrs_tbf_rule=stop ext_proj_rw ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77kh (137s) == sanityn test 77ki: Add rule with unsupported TBF type should fail for generic TBF ========================================================== 18:43:42 (1789771422) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 oleg447-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_fail projid={100}&opcode={ost_write} rate=20: Invalid argument oleg447-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_fail projid={100}&opcode={ost_write} rate=20': Invalid argument pdsh@oleg447-client: oleg447-server: ssh exited with exit code 22 Cleanup TBF ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77ki (9s) == sanityn test 77kj: Verify nodemap support for NRS TBF rule ========================================================== 18:43:51 (1789771431) On MGS 192.168.204.147, active = nodemap.active=1 On MGS 192.168.204.147, TBF.id = nodemap.TBF.id=1 ost.OSS.ost_io.nrs_policies=tbf nodemap ost.OSS.ost_io.nrs_tbf_rule=start ext_nm_rw nodemap={TBF} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.281511 s, 59.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0447688 s, 23.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0649486 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118846 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105451 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119979 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119937 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123964 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125245 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155676 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125215 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136416 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137593 s, 7.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.235618 s, 4.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.230212 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.294535 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.277562 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0244745 s, 42.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0277175 s, 37.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0417822 s, 25.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0323415 s, 32.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0354992 s, 29.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0338419 s, 31.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0303559 s, 34.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0454457 s, 23.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0307941 s, 34.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0319787 s, 32.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0305895 s, 34.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0365567 s, 28.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.041896 s, 25.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0379491 s, 27.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0419677 s, 25.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0426968 s, 24.6 MB/s removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg447-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg447-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20, bs: 1048576 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.96408 s, 21.1 MB/s Write runtime is 6 s, speed is 16.666666 vs. 20 Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.9766 s, 21.1 MB/s Read runtime is 6 s, speed is 16.666666 vs. 20 cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete On MGS 192.168.204.147, TBF.id = On MGS 192.168.204.147, active = nodemap.active=0 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.154577 s, 109 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0438685 s, 23.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.044883 s, 23.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120318 s, 8.7 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9e5847ff5800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168463 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150824 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181194 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174372 s, 6.0 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184361 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189718 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174734 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163074 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188031 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202636 s, 5.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210869 s, 5.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188143 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198421 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0222635 s, 47.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0376067 s, 27.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0263658 s, 39.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0300158 s, 34.9 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out ldlm.namespaces.lustre-OST0000-osc-ffff9e5847ff5800.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00557666 s, 188 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0272698 s, 38.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156831 s, 66.9 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159485 s, 65.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158118 s, 66.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0452242 s, 23.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0421433 s, 24.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147961 s, 70.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0420135 s, 25.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0343927 s, 30.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0380063 s, 27.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0357074 s, 29.4 MB/s removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg447-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg447-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 1 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20, bs: 1048576 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 1.97735 s, 53.0 MB/s Write runtime is 3 s, speed is 33.333333 vs. 20 write rate (33.333333) should exceed limit (20 * 1) Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 0.807223 s, 130 MB/s Read runtime is 2 s, speed is 50.000000 vs. 20 read rate (50.000000) should exceed limit (20 * 1) cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Cleanup TBF ost.OSS.ost_io.nrs_tbf_rule=stop ext_nm_rw ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf nodemap+opcode oleg447-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_w nodemap={TBF}&opcode={ost_write} rate=20: Invalid argument oleg447-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_w nodemap={TBF}&opcode={ost_write} rate=20': Invalid argument oleg447-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_r nodemap={TBF}&opcode={ost_read} rate=10: Invalid argument oleg447-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_r nodemap={TBF}&opcode={ost_read} rate=10': Invalid argument pdsh@oleg447-client: oleg447-server: ssh exited with exit code 22 On MGS 192.168.204.147, active = nodemap.active=1 On MGS 192.168.204.147, TBF.id = nodemap.TBF.id=2 ost.OSS.ost_io.nrs_tbf_rule=start ext_w nodemap={TBF}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nodemap={TBF}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.284798 s, 58.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0487059 s, 21.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0296067 s, 35.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0497171 s, 21.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17887 s, 5.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173341 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177777 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170834 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180407 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170911 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159029 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184521 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18988 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207394 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.228175 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.26019 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.260028 s, 4.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0248574 s, 42.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.029431 s, 35.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0222607 s, 47.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0255222 s, 41.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0265627 s, 39.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0365097 s, 28.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0317361 s, 33.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0255201 s, 41.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0294047 s, 35.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0327819 s, 32.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0240539 s, 43.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0275716 s, 38.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.030936 s, 33.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0254329 s, 41.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0248621 s, 42.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00301621 s, 348 MB/s removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg447-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg447-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10, bs: 1048576 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.96669 s, 21.1 MB/s Write runtime is 6 s, speed is 16.666666 vs. 20 Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.76657 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 vs. 10 cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Cleanup TBF ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf nid classify=dynamic ost.OSS.ost_io.nrs_tbf_rule=start nmdyn nodemap={TBF} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.270333 s, 62.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0407202 s, 25.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0411147 s, 25.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0705476 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0920081 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0979719 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105287 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0859674 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108028 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131367 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115146 s, 9.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123891 s, 8.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120644 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227845 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.249088 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.300078 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.301772 s, 3.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157467 s, 66.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0328433 s, 31.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0238362 s, 44.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0438196 s, 23.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0274735 s, 38.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0288029 s, 36.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0383659 s, 27.3 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0463214 s, 22.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0345396 s, 30.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0302254 s, 34.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0334167 s, 31.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0351344 s, 29.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0363124 s, 28.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0357101 s, 29.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0372645 s, 28.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0392159 s, 26.7 MB/s removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg447-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg447-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20, bs: 1048576 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.98469 s, 21.0 MB/s Write runtime is 6 s, speed is 16.666666 vs. 20 Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.98501 s, 21.0 MB/s Read runtime is 6 s, speed is 16.666666 vs. 20 cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Cleanup TBF ost.OSS.ost_io.nrs_tbf_rule=stop nmdyn ost.OSS.ost_io.nrs_policies=fifo On MGS 192.168.204.147, TBF.id = On MGS 192.168.204.147, active = nodemap.active=0 Cleanup TBF ost.OSS.ost_io.nrs_policies=fifo Cleanup TBF ost.OSS.ost_io.nrs_policies=fifo Cleanup TBF ost.OSS.ost_io.nrs_policies=fifo On MGS 192.168.204.147, active = nodemap.active=0 PASS 77kj (167s) == sanityn test 77l: check the output of NRS policies for generic TBF ========================================================== 18:46:38 (1789771598) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_policies=fifo PASS 77l (4s) == sanityn test 77m: check NRS Delay slows write RPC processing ========================================================== 18:46:42 (1789771602) ost.OSS.ost_io.nrs_policies=delay ost.OSS.ost_io.nrs_delay_min=4 ost.OSS.ost_io.nrs_delay_max=4 ost.OSS.ost_io.nrs_delay_pct=100 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg447-client.virtnet] [bs=1M] [count=1] [oflag=direct] [conv=fdatasync] 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 3.39758 s, 309 kB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg447-client.virtnet] [bs=1M] [count=10] [oflag=direct] [conv=fdatasync] 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 40.0067 s, 262 kB/s ost.OSS.ost_io.nrs_policies=fifo PASS 77m (51s) == sanityn test 77n: check wildcard support for TBF JobID NRS policy ========================================================== 18:47:33 (1789771653) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={*.500} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.153694 s, 109 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0257089 s, 40.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0335758 s, 31.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0921278 s, 11.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.086166 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0970051 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0856237 s, 12.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116769 s, 9.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0976724 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111806 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121094 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115023 s, 9.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9e5847ff5800.lock_unused_count=5 ldlm.namespaces.lustre-OST0001-osc-ffff9e5847ff5800.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108579 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0989927 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105922 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100121 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119959 s, 8.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0185482 s, 56.5 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0190035 s, 55.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0204571 s, 51.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0202209 s, 51.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0218926 s, 47.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0249377 s, 42.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0231039 s, 45.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167369 s, 62.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00406518 s, 258 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0257476 s, 40.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0241185 s, 43.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0229507 s, 45.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0213601 s, 49.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00859213 s, 122 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00759196 s, 138 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.004941 s, 212 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg447-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg447-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20, bs: 1048576 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77n.sanityn/tbf] [bs=1048576] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.97086 s, 21.1 MB/s Write runtime is 6 s, speed is 16.666666 vs. 20 Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77n.sanityn/tbf] [of=/dev/null] [bs=1048576] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.97941 s, 21.1 MB/s Read runtime is 6 s, speed is 16.666666 vs. 20 cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.*} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.173779 s, 96.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0384151 s, 27.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0801206 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112956 s, 9.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136628 s, 7.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119053 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141884 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127663 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134869 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150517 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141949 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160149 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1424 s, 7.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165373 s, 6.3 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170025 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199539 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.236495 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183819 s, 57.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0213894 s, 49.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0209816 s, 50.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00430254 s, 244 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.020689 s, 50.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0189217 s, 55.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00844736 s, 124 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.027861 s, 37.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00444175 s, 236 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0287578 s, 36.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0293288 s, 35.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132653 s, 79.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0105327 s, 99.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00547839 s, 191 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.035695 s, 29.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00559784 s, 187 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg447-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg447-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20, bs: 1048576 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.96219 s, 21.1 MB/s Write runtime is 6 s, speed is 16.666666 vs. 20 Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.98504 s, 21.0 MB/s Read runtime is 6 s, speed is 16.666666 vs. 20 cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_policies=fifo PASS 77n (65s) == sanityn test 77o: Changing rank should not panic ====== 18:48:38 (1789771718) mds.MDS.mdt.nrs_policies=tbf nid mds.MDS.mdt.nrs_tbf_rule=start name nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=start name1 nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=change name1 rank=name mds.MDS.mdt.nrs_tbf_rule=stop name mds.MDS.mdt.nrs_policies=fifo PASS 77o (4s) == sanityn test 77q: Parallel TBF rule definitions should not panic ========================================================== 18:48:42 (1789771722) mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_policies=fifo PASS 77q (44s) == sanityn test 77p: Check validity of rule names for TBF policies ========================================================== 18:49:26 (1789771766) mds.MDS.mdt.nrs_policies=tbf oleg447-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_overflo uid={500} rate=500: File name too long oleg447-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_overflo uid={500} rate=500': File name too long pdsh@oleg447-client: oleg447-server: ssh exited with exit code 36 oleg447-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start uid={500} rate=500: Invalid argument oleg447-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start uid={500} rate=500': Invalid argument pdsh@oleg447-client: oleg447-server: ssh exited with exit code 22 oleg447-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_empty: Invalid argument oleg447-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_empty': Invalid argument pdsh@oleg447-client: oleg447-server: ssh exited with exit code 22 oleg447-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p@spec uid={500} rate=500: Invalid argument oleg447-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p@spec uid={500} rate=500': Invalid argument pdsh@oleg447-client: oleg447-server: ssh exited with exit code 22 oleg447-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p.spec uid={500} rate=500: Invalid argument oleg447-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p.spec uid={500} rate=500': Invalid argument pdsh@oleg447-client: oleg447-server: ssh exited with exit code 22 oleg447-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p~spec uid={500} rate=500: Invalid argument oleg447-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p~spec uid={500} rate=500': Invalid argument pdsh@oleg447-client: oleg447-server: ssh exited with exit code 22 oleg447-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p#spec uid={500} rate=500: Invalid argument oleg447-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p#spec uid={500} rate=500': Invalid argument pdsh@oleg447-client: oleg447-server: ssh exited with exit code 22 oleg447-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p/spec uid={500} rate=500: Invalid argument oleg447-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p/spec uid={500} rate=500': Invalid argument pdsh@oleg447-client: oleg447-server: ssh exited with exit code 22 oleg447-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p^spec uid={500} rate=500: Invalid argument oleg447-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p^spec uid={500} rate=500': Invalid argument pdsh@oleg447-client: oleg447-server: ssh exited with exit code 22 oleg447-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p%spec uid={500} rate=500: Invalid argument oleg447-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p%spec uid={500} rate=500': Invalid argument pdsh@oleg447-client: oleg447-server: ssh exited with exit code 22 oleg447-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p*spec uid={500} rate=500: Invalid argument oleg447-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p*spec uid={500} rate=500': Invalid argument pdsh@oleg447-client: oleg447-server: ssh exited with exit code 22 oleg447-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p;spec uid={500} rate=500: Invalid argument oleg447-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p;spec uid={500} rate=500': Invalid argument pdsh@oleg447-client: oleg447-server: ssh exited with exit code 22 oleg447-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p,spec uid={500} rate=500: Invalid argument oleg447-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p,spec uid={500} rate=500': Invalid argument pdsh@oleg447-client: oleg447-server: ssh exited with exit code 22 oleg447-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p?spec uid={500} rate=500: Invalid argument oleg447-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p?spec uid={500} rate=500': Invalid argument pdsh@oleg447-client: oleg447-server: ssh exited with exit code 22 oleg447-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77pspec uid={500} rate=500: Invalid argument oleg447-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p>spec uid={500} rate=500': Invalid argument pdsh@oleg447-client: oleg447-server: ssh exited with exit code 22 oleg447-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p:spec uid={500} rate=500: Invalid argument oleg447-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p:spec uid={500} rate=500': Invalid argument pdsh@oleg447-client: oleg447-server: ssh exited with exit code 22 oleg447-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p+spec uid={500} rate=500: Invalid argument oleg447-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p+spec uid={500} rate=500': Invalid argument pdsh@oleg447-client: oleg447-server: ssh exited with exit code 22 oleg447-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p=spec uid={500} rate=500: Invalid argument oleg447-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p=spec uid={500} rate=500': Invalid argument pdsh@oleg447-client: oleg447-server: ssh exited with exit code 22 oleg447-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p)spec uid={500} rate=500: Invalid argument oleg447-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p)spec uid={500} rate=500': Invalid argument pdsh@oleg447-client: oleg447-server: ssh exited with exit code 22 oleg447-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p(spec uid={500} rate=500: Invalid argument oleg447-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p(spec uid={500} rate=500': Invalid argument pdsh@oleg447-client: oleg447-server: ssh exited with exit code 22 oleg447-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p{spec uid={500} rate=500: Invalid argument oleg447-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p{spec uid={500} rate=500': Invalid argument pdsh@oleg447-client: oleg447-server: ssh exited with exit code 22 oleg447-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p}spec uid={500} rate=500: Invalid argument oleg447-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p}spec uid={500} rate=500': Invalid argument pdsh@oleg447-client: oleg447-server: ssh exited with exit code 22 oleg447-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p|spec uid={500} rate=500: Invalid argument oleg447-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p|spec uid={500} rate=500': Invalid argument pdsh@oleg447-client: oleg447-server: ssh exited with exit code 22 oleg447-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p[spec uid={500} rate=500: Invalid argument oleg447-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p[spec uid={500} rate=500': Invalid argument pdsh@oleg447-client: oleg447-server: ssh exited with exit code 22 oleg447-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p]spec uid={500} rate=500: Invalid argument oleg447-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p]spec uid={500} rate=500': Invalid argument pdsh@oleg447-client: oleg447-server: ssh exited with exit code 22 oleg447-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p!spec uid={500} rate=500: Invalid argument oleg447-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p!spec uid={500} rate=500': Invalid argument pdsh@oleg447-client: oleg447-server: ssh exited with exit code 22 oleg447-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p&spec uid={500} rate=500: Invalid argument oleg447-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p&spec uid={500} rate=500': Invalid argument pdsh@oleg447-client: oleg447-server: ssh exited with exit code 22 oleg447-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p$spec uid={500} rate=500: Invalid argument oleg447-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p$spec uid={500} rate=500': Invalid argument pdsh@oleg447-client: oleg447-server: ssh exited with exit code 22 oleg447-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p`spec uid={500} rate=500: Invalid argument oleg447-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p`spec uid={500} rate=500': Invalid argument pdsh@oleg447-client: oleg447-server: ssh exited with exit code 22 oleg447-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p\spec uid={500} rate=500: Invalid argument oleg447-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p\spec uid={500} rate=500': Invalid argument pdsh@oleg447-client: oleg447-server: ssh exited with exit code 22 mds.MDS.mdt.nrs_policies=fifo PASS 77p (13s) == sanityn test 77r: Change type of tbf policy at run time ========================================================== 18:49:39 (1789771779) mds.MDS.mdt.nrs_policies=tbf uid mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf gid mds.MDS.mdt.nrs_tbf_rule=start md_rule gid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start md_rule jobid={*.500} rate=1 mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=fifo check the number of requests in queue: fifo: 0 crrn: 0 tbf: 0 delay: 0 fifo: 0 crrn: 0 tbf: 0 delay: 0 cleanup 77r 278170 kill md_thread xargs PASS 77r (41s) == sanityn test 77s: Check TBF LRU shrinker ============== 18:50:20 (1789771820) jobid_name=%e.%p jobid_var=TEST_77s_JOBVAR mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start touch jobid={touch.*} rate=1000 mds.MDS.mdt.nrs_tbf_rule=start unlink jobid={unlink.*} rate=1000 touch jobid={touch.*} 1000, seq 2, type=jobid, ref 10 touch jobid={touch.*} 1000, seq 2, type=jobid, ref 0 unlink jobid={unlink.*} 1000, seq 3, type=jobid, ref 10 unlink jobid={unlink.*} 1000, seq 3, type=jobid, ref 0 mds.MDS.mdt.nrs_policies=fifo jobid_var=procname_uid jobid_name=%e.%u PASS 77s (17s) == sanityn test 77ta: Testing for dynamic|static classify parameter setting for TBF ========================================================== 18:50:37 (1789771837) ost.OSS.ost_io.nrs_policies=fifo oleg447-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf nid+uid classify=foo: Invalid argument oleg447-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf nid+uid classify=foo': Invalid argument pdsh@oleg447-client: oleg447-server: ssh exited with exit code 22 oleg447-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf nid+uid foo=static: Invalid argument oleg447-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf nid+uid foo=static': Invalid argument pdsh@oleg447-client: oleg447-server: ssh exited with exit code 22 oleg447-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf nid+uid classify=: Invalid argument oleg447-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf nid+uid classify=': Invalid argument pdsh@oleg447-client: oleg447-server: ssh exited with exit code 22 ost.OSS.ost_io.nrs_policies=tbf nid classify=dynamic ost.OSS.ost_io.nrs_policies=tbf nid classify=static ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf reg nid classify=dynamic ost.OSS.ost_io.nrs_policies=tbf hp nid ost.OSS.ost_io.nrs_tbf_rule=start mixed1 nid={0@lo} rate=100 ost.OSS.ost_io.nrs_tbf_rule=stop mixed1 ost.OSS.ost_io.nrs_policies=tbf reg nid classify=static ost.OSS.ost_io.nrs_policies=tbf hp nid classify=dynamic ost.OSS.ost_io.nrs_tbf_rule=start mixed2 nid={0@lo} rate=100 ost.OSS.ost_io.nrs_tbf_rule=stop mixed2 ost.OSS.ost_io.nrs_policies=tbf reg nid+uid ost.OSS.ost_io.nrs_policies=tbf hp nid oleg447-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start mixed3 nid={0@lo}&uid={500} rate=100: Invalid argument oleg447-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start mixed3 nid={0@lo}&uid={500} rate=100': Invalid argument pdsh@oleg447-client: oleg447-server: ssh exited with exit code 22 ost.OSS.ost_io.nrs_policies=tbf nid classify=static classify=dynamic ost.OSS.ost_io.nrs_policies=tbf reg nid classify=dynamic ost.OSS.ost_io.nrs_policies= regular_requests: - name: fifo state: started fallback: yes queued: 0 active: 0 - name: crrn state: stopped fallback: no queued: 0 active: 0 - name: orr state: stopped fallback: no queued: 0 active: 0 - name: trr state: stopped fallback: no queued: 0 active: 0 - name: tbf nid classify=dynamic state: started fallback: no queued: 0 active: 0 - name: delay state: stopped fallback: no queued: 0 active: 0 high_priority_requests: - name: fifo state: started fallback: yes queued: 0 active: 0 - name: crrn state: stopped fallback: no queued: 0 active: 0 - name: orr state: stopped fallback: no queued: 0 active: 0 - name: trr state: stopped fallback: no queued: 0 active: 0 - name: tbf nid classify=static classify=dynamic state: started fallback: no queued: 0 active: 0 - name: delay state: stopped fallback: no queued: 0 active: 0 ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf hp nid classify=static ost.OSS.ost_io.nrs_policies= regular_requests: - name: fifo state: started fallback: yes queued: 0 active: 0 - name: crrn state: stopped fallback: no queued: 0 active: 0 - name: orr state: stopped fallback: no queued: 0 active: 0 - name: trr state: stopped fallback: no queued: 0 active: 0 - name: tbf state: stopped fallback: no queued: 0 active: 0 - name: delay state: stopped fallback: no queued: 0 active: 0 high_priority_requests: - name: fifo state: started fallback: yes queued: 0 active: 0 - name: crrn state: stopped fallback: no queued: 0 active: 0 - name: orr state: stopped fallback: no queued: 0 active: 0 - name: trr state: stopped fallback: no queued: 0 active: 0 - name: tbf nid classify=static state: started fallback: no queued: 0 active: 0 - name: delay state: stopped fallback: no queued: 0 active: 0 ost.OSS.ost_io.nrs_policies=fifo PASS 77ta (17s) == sanityn test 77tB: Request falling to default rule should not panic for dynamic TBF ========================================================== 18:50:54 (1789771854) ost.OSS.ost_io.nrs_policies=tbf nid classify=dynamic not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5, bs: 1048576 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.4999 s, 5.4 MB/s Write runtime is 20 s, speed is 5.000000 vs. 5 Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9431 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 vs. 5 cleanup_tbf /mnt/lustre/d77tB.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=start unmatchrule nid={192.0.2.1@tcp} rate=100 not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5, bs: 1048576 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.5143 s, 5.4 MB/s Write runtime is 20 s, speed is 5.000000 vs. 5 Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9544 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 vs. 5 cleanup_tbf /mnt/lustre/d77tB.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop unmatchrule ost.OSS.ost_io.nrs_policies=fifo PASS 77tB (128s) == sanityn test 77th: TBF: a mixed classify head pair keeps each head type ========================================================== 18:53:02 (1789771982) ost.OSS.ost_io.nrs_policies=tbf reg nid+opcode ost.OSS.ost_io.nrs_policies=tbf hp nid classify=dynamic ost.OSS.ost_io.nrs_tbf_rule=start mixedhp nid={0@lo} rate=100 regular_requests: CPT 0: mixedhp nid={0@lo} 100, seq 2, type=nid+opcode, ref 0 default * 10000, seq 1, type=nid+opcode, ref 0 high_priority_requests: CPT 0: mixedhp nid={0@lo} 100, seq 2, type=nid, ref 0 default * 10000, seq 1, type=nid, ref 0 ost.OSS.ost_io.nrs_tbf_rule=stop mixedhp ost.OSS.ost_io.nrs_policies=fifo PASS 77th (4s) == sanityn test 77tb: Testing for dynamic classify TBF Scheduler ========================================================== 18:53:06 (1789771986) ost.OSS.ost_io.nrs_policies=tbf nid classify=dynamic ost.OSS.ost_io.nrs_policies= regular_requests: - name: fifo state: started fallback: yes queued: 0 active: 0 - name: crrn state: stopped fallback: no queued: 0 active: 0 - name: orr state: stopped fallback: no queued: 0 active: 0 - name: trr state: stopped fallback: no queued: 0 active: 0 - name: tbf nid classify=dynamic state: started fallback: no queued: 0 active: 0 - name: delay state: stopped fallback: no queued: 0 active: 0 high_priority_requests: - name: fifo state: started fallback: yes queued: 0 active: 0 - name: crrn state: stopped fallback: no queued: 0 active: 0 - name: orr state: stopped fallback: no queued: 0 active: 0 - name: trr state: stopped fallback: no queued: 0 active: 0 - name: tbf nid classify=dynamic state: started fallback: no queued: 0 active: 0 - name: delay state: stopped fallback: no queued: 0 active: 0 ost.OSS.ost_io.nrs_tbf_classify= reg_classify:dynamic hp_classify:dynamic ost.OSS.ost_io.nrs_tbf_rule=start nidall nid={0@lo 192.168.204.47@tcp} rate=10 ost.OSS.ost_io.nrs_tbf_rule=start nidrd nid={0@lo 192.168.204.47@tcp}&opcode={ost_read} rate=5 ost.OSS.ost_io.nrs_tbf_rule=start nidwr nid={0@lo 192.168.204.47@tcp}&opcode={ost_write} rate=15 ost.OSS.ost_io.nrs_tbf_rule= regular_requests: CPT 0: nidwr nid={0@lo 192.168.204.47@tcp}&opcode={ost_write} 15, seq 4, type=nid+opcode, ref 0 nidrd nid={0@lo 192.168.204.47@tcp}&opcode={ost_read} 5, seq 3, type=nid+opcode, ref 0 nidall nid={0@lo 192.168.204.47@tcp} 10, seq 2, type=nid, ref 0 default * 10000, seq 1, type=nid, ref 0 high_priority_requests: CPT 0: nidwr nid={0@lo 192.168.204.47@tcp}&opcode={ost_write} 15, seq 4, type=nid+opcode, ref 0 nidrd nid={0@lo 192.168.204.47@tcp}&opcode={ost_read} 5, seq 3, type=nid+opcode, ref 0 nidall nid={0@lo 192.168.204.47@tcp} 10, seq 2, type=nid, ref 0 default * 10000, seq 1, type=nid, ref 0 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.3216 s, 52.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0337695 s, 31.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0517598 s, 20.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.07269 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0664085 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0708082 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0913839 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0905687 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0747777 s, 14.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148436 s, 7.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144199 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21013 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202964 s, 5.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.349587 s, 3.0 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.332449 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.406531 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.425629 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0182171 s, 57.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0211321 s, 49.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0222099 s, 47.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.019905 s, 52.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0287813 s, 36.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0245378 s, 42.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0221896 s, 47.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0219983 s, 47.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161067 s, 65.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0274403 s, 38.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0311158 s, 33.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00543249 s, 193 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00713391 s, 147 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0311882 s, 33.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00544362 s, 193 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0063485 s, 165 MB/s removed '/mnt/lustre/d77tb.sanityn/nrs_w_oleg447-client.virtnet' removed '/mnt/lustre/d77tb.sanityn/nrs_r_oleg447-client.virtnet' removed directory '/mnt/lustre/d77tb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 15, read rate: 5, bs: 1048576 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.55089 s, 16.0 MB/s Write runtime is 8 s, speed is 12.500000 vs. 15 Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.4393 s, 5.4 MB/s Read runtime is 20 s, speed is 5.000000 vs. 5 cleanup_tbf /mnt/lustre/d77tb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Cleanup TBF ost.OSS.ost_io.nrs_tbf_rule=stop nidall ost.OSS.ost_io.nrs_tbf_rule=stop nidrd ost.OSS.ost_io.nrs_tbf_rule=stop nidwr ost.OSS.ost_io.nrs_policies=fifo PASS 77tb (51s) == sanityn test 77tc: Testing for the setting of allowed TBF classification type ========================================================== 18:53:57 (1789772037) oleg447-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf nid classify=dynamic allow=nid allow=nid: File exists oleg447-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf nid classify=dynamic allow=nid allow=nid': File exists pdsh@oleg447-client: oleg447-server: ssh exited with exit code 17 oleg447-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf nid allow=nid allow=uid: Invalid argument oleg447-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf nid allow=nid allow=uid': Invalid argument pdsh@oleg447-client: oleg447-server: ssh exited with exit code 22 ost.OSS.ost_io.nrs_policies=tbf nid classify=dynamic allow=nid+opcode allow=nid allow=uid+opcode oleg447-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start another uid={500} rate=10: Operation not permitted oleg447-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start another uid={500} rate=10': Operation not permitted pdsh@oleg447-client: oleg447-server: ssh exited with exit code 1 oleg447-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start another uid={500}&opcode={ost_read}&gid={500} rate=10: Operation not permitted oleg447-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start another uid={500}&opcode={ost_read}&gid={500} rate=10': Operation not permitted pdsh@oleg447-client: oleg447-server: ssh exited with exit code 1 ost.OSS.ost_io.nrs_tbf_rule=start nidall nid={0@lo} rate=10 ost.OSS.ost_io.nrs_tbf_rule=start nidrd nid={0@lo}&opcode={ost_read} rate=5 Cleanup TBF ost.OSS.ost_io.nrs_tbf_rule=stop nidall ost.OSS.ost_io.nrs_tbf_rule=stop nidrd ost.OSS.ost_io.nrs_policies=fifo oleg447-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf nodemap classify=dynamic allow=nid allow=uid allow=gid allow=jobid allow=opcode allow=projid allow=nid+uid allow=nid+gid allow=uid+gid allow=nid+jobid allow=uid+jobid allow=gid+jobid allow=nid+opcode allow=uid+opcode allow=gid+opcode allow=nid+projid: Argument list too long oleg447-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf nodemap classify=dynamic allow=nid allow=uid allow=gid allow=jobid allow=opcode allow=projid allow=nid+uid allow=nid+gid allow=uid+gid allow=nid+jobid allow=uid+jobid allow=gid+jobid allow=nid+opcode allow=uid+opcode allow=gid+opcode allow=nid+projid': Argument list too long pdsh@oleg447-client: oleg447-server: ssh exited with exit code 7 Cleanup TBF ost.OSS.ost_io.nrs_policies=fifo PASS 77tc (15s) == sanityn test 77td: Testing Number of allowed rules larger or smaller than threshold ========================================================== 18:54:12 (1789772052) using SAVE_PROJECT_SUPPORTED=0 Stopping clients: oleg447-client.virtnet /mnt/lustre (opts:) Stopping client oleg447-client.virtnet /mnt/lustre opts: Stopping clients: oleg447-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg447-client.virtnet /mnt/lustre2 opts: Stopping /mnt/lustre-mds1 (opts:-f) on oleg447-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg447-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg447-server sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime) proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) devtmpfs on /dev type devtmpfs (rw,nosuid,mode=755) securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime) tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev) devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000) tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755) tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755) cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd) pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime) bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700) cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids) cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event) cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio) cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio) cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma) cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer) cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices) cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct) cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset) cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb) cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory) cgroup on /sys/fs/cgroup/misc type cgroup (rw,nosuid,nodev,noexec,relatime,misc) none on /sys/kernel/tracing type tracefs (rw,relatime) configfs on /sys/kernel/config type configfs (rw,relatime) /dev/nbd0 on / type ext4 (ro,relatime,stripe=32) rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) mqueue on /dev/mqueue type mqueue (rw,relatime) hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M) debugfs on /sys/kernel/debug type debugfs (rw,relatime) none on /mnt type ramfs (rw,relatime) none on /var/lib/stateless/writable type tmpfs (rw,relatime) /dev/vda on /home/green/git/lustre-release type squashfs (ro,relatime) none on /var/cache/man type tmpfs (rw,relatime) none on /var/log type tmpfs (rw,relatime) none on /var/lib/dbus type tmpfs (rw,relatime) none on /tmp type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) none on /var/tmp type tmpfs (rw,relatime) none on /var/lib/NetworkManager type tmpfs (rw,relatime) none on /var/lib/systemd/random-seed type tmpfs (rw,relatime) none on /var/spool type tmpfs (rw,relatime) none on /var/lib/nfs type tmpfs (rw,relatime) none on /var/lib/gssproxy type tmpfs (rw,relatime) none on /var/lib/logrotate type tmpfs (rw,relatime) none on /etc type tmpfs (rw,relatime) none on /var/lib/rsyslog type tmpfs (rw,relatime) none on /root/.ssh type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) 192.168.200.253:/exports/state/oleg447-client.virtnet on /var/lib/stateless/state type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.204.47,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg447-client.virtnet/boot on /boot type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.204.47,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg447-client.virtnet/etc/kdump.conf on /etc/kdump.conf type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.204.47,local_lock=none,addr=192.168.200.253) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=378180k,mode=700) 192.168.200.253:/exports/testreports/70513/testresults/sanityn-zfs-rocky8.10_x86_64-rocky8.10_x86_64 on /tmp/testlogs type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.204.47,local_lock=none,addr=192.168.200.253) /dev/vda on /usr/sbin/mount.lustre type squashfs (ro,relatime) Checking servers environments Checking clients oleg447-client.virtnet environments Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg447-server' oleg447-server: oleg447-server.virtnet: executing load_modules_local oleg447-server: kmemleak disabled oleg447-server: Loading modules from /home/green/git/lustre-release/lustre oleg447-server: detected 4 online CPUs by sysfs oleg447-server: MODOPTS_LIBCFS= oleg447-server: Force libcfs to create 2 CPU partitions oleg447-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Setup mgs, mdt, osts Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg447-server: oleg447-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg447-client: oleg447-server: ssh exited with exit code 1 Started lustre-MDT0000 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg447-server: oleg447-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg447-client: oleg447-server: ssh exited with exit code 1 Started lustre-OST0000 Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg447-server: oleg447-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg447-client: oleg447-server: ssh exited with exit code 1 Started lustre-OST0001 Starting client: oleg447-client.virtnet: -o user_xattr,flock 192.168.204.147@tcp:/lustre /mnt/lustre Starting client oleg447-client.virtnet: -o user_xattr,flock 192.168.204.147@tcp:/lustre /mnt/lustre Started clients oleg447-client.virtnet: 192.168.204.147@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Starting client: oleg447-client.virtnet: -o user_xattr,flock 192.168.204.147@tcp:/lustre /mnt/lustre2 Starting client oleg447-client.virtnet: -o user_xattr,flock 192.168.204.147@tcp:/lustre /mnt/lustre2 Started clients oleg447-client.virtnet: 192.168.204.147@tcp:/lustre on /mnt/lustre2 type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9e5848c63800.idle_timeout=debug osc.lustre-OST0000-osc-ffff9e5848e10000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9e5848c63800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9e5848e10000.idle_timeout=debug disable quota as required ost.OSS.ost_io.nrs_policies=tbf nid classify=dynamic allow=projid+opcode allow=projid ost.OSS.ost_io.nrs_tbf_allowed_types= regular_allowed_types: CPT 0: 0: 0x00000024 opcode+projid 1: 0x00000020 projid 2: 0x00000002 nid high_priority_allowed_types: CPT 0: 0: 0x00000024 opcode+projid 1: 0x00000020 projid 2: 0x00000002 nid 8 ost.OSS.ost_io.nrs_tbf_rule=start rulerd0 projid={100}&opcode={ost_read} rate=5 ost.OSS.ost_io.nrs_tbf_rule=start rulewr0 projid={100}&opcode={ost_write} rate=10 ost.OSS.ost_io.nrs_tbf_rule=start ruleall0 projid={200} rate=15 ost.OSS.ost_io.nrs_tbf_rule= regular_requests: CPT 0: ruleall0 projid={200} 15, seq 4, type=projid, ref 0 rulewr0 projid={100}&opcode={ost_write} 10, seq 3, type=opcode+projid, ref 0 rulerd0 projid={100}&opcode={ost_read} 5, seq 2, type=opcode+projid, ref 0 default * 10000, seq 1, type=nid+opcode+projid, ref 0 high_priority_requests: CPT 0: ruleall0 projid={200} 15, seq 4, type=projid, ref 0 rulewr0 projid={100}&opcode={ost_write} 10, seq 3, type=opcode+projid, ref 0 rulerd0 projid={100}&opcode={ost_read} 5, seq 2, type=opcode+projid, ref 0 default * 10000, seq 1, type=nid+opcode+projid, ref 0 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.143237 s, 117 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0467147 s, 22.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0701733 s, 14.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0664682 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0932831 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0995016 s, 10.5 MB/s 1+0 records in 1+0 records out ldlm.namespaces.lustre-OST0001-osc-ffff9e5848c63800.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0746051 s, 14.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100897 s, 10.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108531 s, 9.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11555 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0906471 s, 11.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120767 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113165 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101288 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114998 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105737 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10022 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141584 s, 74.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01706 s, 61.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out ldlm.namespaces.lustre-OST0000-osc-ffff9e5848c63800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00399466 s, 262 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0194345 s, 54.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012592 s, 83.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139861 s, 75.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0103826 s, 101 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0272899 s, 38.4 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0116936 s, 89.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148336 s, 70.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0105615 s, 99.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0223832 s, 46.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00754651 s, 139 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0118383 s, 88.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00419544 s, 250 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0295029 s, 35.5 MB/s removed '/mnt/lustre/d77td.sanityn/nrs_w_oleg447-client.virtnet' removed '/mnt/lustre/d77td.sanityn/nrs_r_oleg447-client.virtnet' removed directory '/mnt/lustre/d77td.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 5, bs: 1048576 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.80009 s, 10.7 MB/s Write runtime is 11 s, speed is 9.090909 vs. 10 Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.436 s, 5.4 MB/s Read runtime is 21 s, speed is 4.761904 vs. 5 cleanup_tbf /mnt/lustre/d77td.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 15, read rate: 15, bs: 1048576 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.54286 s, 16.0 MB/s Write runtime is 7 s, speed is 14.285714 vs. 15 Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.62108 s, 15.8 MB/s Read runtime is 8 s, speed is 12.500000 vs. 15 cleanup_tbf /mnt/lustre/d77td.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=change default rate=15 ost.OSS.ost_io.nrs_tbf_rule= regular_requests: CPT 0: ruleall0 projid={200} 15, seq 4, type=projid, ref 1 rulewr0 projid={100}&opcode={ost_write} 10, seq 3, type=opcode+projid, ref 1 rulerd0 projid={100}&opcode={ost_read} 5, seq 2, type=opcode+projid, ref 1 default * 15, seq 1, type=nid+opcode+projid, ref 2 high_priority_requests: CPT 0: ruleall0 projid={200} 15, seq 4, type=projid, ref 0 rulewr0 projid={100}&opcode={ost_write} 10, seq 3, type=opcode+projid, ref 0 rulerd0 projid={100}&opcode={ost_read} 5, seq 2, type=opcode+projid, ref 0 default * 15, seq 1, type=nid+opcode+projid, ref 1 not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 15, read rate: 15, bs: 1048576 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.54177 s, 16.0 MB/s Write runtime is 7 s, speed is 14.285714 vs. 15 Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.50016 s, 16.1 MB/s Read runtime is 8 s, speed is 12.500000 vs. 15 cleanup_tbf /mnt/lustre/d77td.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf nid classify=dynamic allow=projid+opcode allow=projid ost.OSS.ost_io.nrs_tbf_allowed_types= regular_allowed_types: CPT 0: 0: 0x00000024 opcode+projid 1: 0x00000020 projid 2: 0x00000002 nid high_priority_allowed_types: CPT 0: 0: 0x00000024 opcode+projid 1: 0x00000020 projid 2: 0x00000002 nid 8 ost.OSS.ost_io.nrs_tbf_rule=start rulerd0 projid={100}&opcode={ost_read} rate=5 ost.OSS.ost_io.nrs_tbf_rule=start rulewr0 projid={100}&opcode={ost_write} rate=10 ost.OSS.ost_io.nrs_tbf_rule=start rulerd1 projid={101}&opcode={ost_read} rate=5 ost.OSS.ost_io.nrs_tbf_rule=start rulewr1 projid={101}&opcode={ost_write} rate=10 ost.OSS.ost_io.nrs_tbf_rule=start rulerd2 projid={102}&opcode={ost_read} rate=5 ost.OSS.ost_io.nrs_tbf_rule=start rulewr2 projid={102}&opcode={ost_write} rate=10 ost.OSS.ost_io.nrs_tbf_rule=start rulerd3 projid={103}&opcode={ost_read} rate=5 ost.OSS.ost_io.nrs_tbf_rule=start rulewr3 projid={103}&opcode={ost_write} rate=10 ost.OSS.ost_io.nrs_tbf_rule=start rulerd4 projid={104}&opcode={ost_read} rate=5 ost.OSS.ost_io.nrs_tbf_rule=start rulewr4 projid={104}&opcode={ost_write} rate=10 ost.OSS.ost_io.nrs_tbf_rule=start ruleall0 projid={200} rate=15 ost.OSS.ost_io.nrs_tbf_rule=start ruleall1 projid={201} rate=15 ost.OSS.ost_io.nrs_tbf_rule=start ruleall2 projid={202} rate=15 ost.OSS.ost_io.nrs_tbf_rule=start ruleall3 projid={203} rate=15 ost.OSS.ost_io.nrs_tbf_rule=start ruleall4 projid={204} rate=15 ost.OSS.ost_io.nrs_tbf_rule= regular_requests: CPT 0: ruleall4 projid={204} 15, seq 16, type=projid, ref 0 ruleall3 projid={203} 15, seq 15, type=projid, ref 0 ruleall2 projid={202} 15, seq 14, type=projid, ref 0 ruleall1 projid={201} 15, seq 13, type=projid, ref 0 ruleall0 projid={200} 15, seq 12, type=projid, ref 0 rulewr4 projid={104}&opcode={ost_write} 10, seq 11, type=opcode+projid, ref 0 rulerd4 projid={104}&opcode={ost_read} 5, seq 10, type=opcode+projid, ref 0 rulewr3 projid={103}&opcode={ost_write} 10, seq 9, type=opcode+projid, ref 0 rulerd3 projid={103}&opcode={ost_read} 5, seq 8, type=opcode+projid, ref 0 rulewr2 projid={102}&opcode={ost_write} 10, seq 7, type=opcode+projid, ref 0 rulerd2 projid={102}&opcode={ost_read} 5, seq 6, type=opcode+projid, ref 0 rulewr1 projid={101}&opcode={ost_write} 10, seq 5, type=opcode+projid, ref 0 rulerd1 projid={101}&opcode={ost_read} 5, seq 4, type=opcode+projid, ref 0 rulewr0 projid={100}&opcode={ost_write} 10, seq 3, type=opcode+projid, ref 0 rulerd0 projid={100}&opcode={ost_read} 5, seq 2, type=opcode+projid, ref 0 default * 10000, seq 1, type=nid+opcode+projid, ref 0 high_priority_requests: CPT 0: ruleall4 projid={204} 15, seq 16, type=projid, ref 0 ruleall3 projid={203} 15, seq 15, type=projid, ref 0 ruleall2 projid={202} 15, seq 14, type=projid, ref 0 ruleall1 projid={201} 15, seq 13, type=projid, ref 0 ruleall0 projid={200} 15, seq 12, type=projid, ref 0 rulewr4 projid={104}&opcode={ost_write} 10, seq 11, type=opcode+projid, ref 0 rulerd4 projid={104}&opcode={ost_read} 5, seq 10, type=opcode+projid, ref 0 rulewr3 projid={103}&opcode={ost_write} 10, seq 9, type=opcode+projid, ref 0 rulerd3 projid={103}&opcode={ost_read} 5, seq 8, type=opcode+projid, ref 0 rulewr2 projid={102}&opcode={ost_write} 10, seq 7, type=opcode+projid, ref 0 rulerd2 projid={102}&opcode={ost_read} 5, seq 6, type=opcode+projid, ref 0 rulewr1 projid={101}&opcode={ost_write} 10, seq 5, type=opcode+projid, ref 0 rulerd1 projid={101}&opcode={ost_read} 5, seq 4, type=opcode+projid, ref 0 rulewr0 projid={100}&opcode={ost_write} 10, seq 3, type=opcode+projid, ref 0 rulerd0 projid={100}&opcode={ost_read} 5, seq 2, type=opcode+projid, ref 0 default * 10000, seq 1, type=nid+opcode+projid, ref 0 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.179998 s, 93.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0408235 s, 25.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0405665 s, 25.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0480724 s, 21.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0792412 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0838281 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0801144 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0887333 s, 11.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9e5848c63800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126427 s, 8.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1281 s, 8.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137504 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117867 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118663 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135967 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125523 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117532 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133718 s, 7.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0124358 s, 84.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0109462 s, 95.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0124187 s, 84.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126004 s, 83.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132432 s, 79.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159274 s, 65.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168879 s, 62.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0230019 s, 45.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0315648 s, 33.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0315322 s, 33.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0230163 s, 45.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.026729 s, 39.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0253888 s, 41.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0255104 s, 41.1 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0345506 s, 30.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0316643 s, 33.1 MB/s removed '/mnt/lustre/d77td.sanityn/nrs_w_oleg447-client.virtnet' removed '/mnt/lustre/d77td.sanityn/nrs_r_oleg447-client.virtnet' removed directory '/mnt/lustre/d77td.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 5, bs: 1048576 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.80694 s, 10.7 MB/s Write runtime is 11 s, speed is 9.090909 vs. 10 Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.447 s, 5.4 MB/s Read runtime is 20 s, speed is 5.000000 vs. 5 cleanup_tbf /mnt/lustre/d77td.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 15, read rate: 15, bs: 1048576 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.53639 s, 16.0 MB/s Write runtime is 8 s, speed is 12.500000 vs. 15 Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.62379 s, 15.8 MB/s Read runtime is 7 s, speed is 14.285714 vs. 15 cleanup_tbf /mnt/lustre/d77td.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=change default rate=15 ost.OSS.ost_io.nrs_tbf_rule= regular_requests: CPT 0: ruleall4 projid={204} 15, seq 16, type=projid, ref 0 ruleall3 projid={203} 15, seq 15, type=projid, ref 0 ruleall2 projid={202} 15, seq 14, type=projid, ref 0 ruleall1 projid={201} 15, seq 13, type=projid, ref 0 ruleall0 projid={200} 15, seq 12, type=projid, ref 1 rulewr4 projid={104}&opcode={ost_write} 10, seq 11, type=opcode+projid, ref 0 rulerd4 projid={104}&opcode={ost_read} 5, seq 10, type=opcode+projid, ref 0 rulewr3 projid={103}&opcode={ost_write} 10, seq 9, type=opcode+projid, ref 0 rulerd3 projid={103}&opcode={ost_read} 5, seq 8, type=opcode+projid, ref 0 rulewr2 projid={102}&opcode={ost_write} 10, seq 7, type=opcode+projid, ref 0 rulerd2 projid={102}&opcode={ost_read} 5, seq 6, type=opcode+projid, ref 0 rulewr1 projid={101}&opcode={ost_write} 10, seq 5, type=opcode+projid, ref 0 rulerd1 projid={101}&opcode={ost_read} 5, seq 4, type=opcode+projid, ref 0 rulewr0 projid={100}&opcode={ost_write} 10, seq 3, type=opcode+projid, ref 1 rulerd0 projid={100}&opcode={ost_read} 5, seq 2, type=opcode+projid, ref 1 default * 15, seq 1, type=nid+opcode+projid, ref 2 high_priority_requests: CPT 0: ruleall4 projid={204} 15, seq 16, type=projid, ref 0 ruleall3 projid={203} 15, seq 15, type=projid, ref 0 ruleall2 projid={202} 15, seq 14, type=projid, ref 0 ruleall1 projid={201} 15, seq 13, type=projid, ref 0 ruleall0 projid={200} 15, seq 12, type=projid, ref 0 rulewr4 projid={104}&opcode={ost_write} 10, seq 11, type=opcode+projid, ref 0 rulerd4 projid={104}&opcode={ost_read} 5, seq 10, type=opcode+projid, ref 0 rulewr3 projid={103}&opcode={ost_write} 10, seq 9, type=opcode+projid, ref 0 rulerd3 projid={103}&opcode={ost_read} 5, seq 8, type=opcode+projid, ref 0 rulewr2 projid={102}&opcode={ost_write} 10, seq 7, type=opcode+projid, ref 0 rulerd2 projid={102}&opcode={ost_read} 5, seq 6, type=opcode+projid, ref 0 rulewr1 projid={101}&opcode={ost_write} 10, seq 5, type=opcode+projid, ref 0 rulerd1 projid={101}&opcode={ost_read} 5, seq 4, type=opcode+projid, ref 0 rulewr0 projid={100}&opcode={ost_write} 10, seq 3, type=opcode+projid, ref 0 rulerd0 projid={100}&opcode={ost_read} 5, seq 2, type=opcode+projid, ref 0 default * 15, seq 1, type=nid+opcode+projid, ref 1 not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 15, read rate: 15, bs: 1048576 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.53336 s, 16.0 MB/s Write runtime is 8 s, speed is 12.500000 vs. 15 Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.50969 s, 16.1 MB/s Read runtime is 7 s, speed is 14.285714 vs. 15 cleanup_tbf /mnt/lustre/d77td.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=fifo PASS 77td (272s) == sanityn test 77tf: TBF: rule churn under load on a dynamic classify policy ========================================================== 18:58:44 (1789772324) jobid_name=%e.%p jobid_var=TEST_77TF_JOBVAR ost.OSS.ost_io.nrs_policies=tbf jobid+opcode classify=dynamic ost.OSS.ost_io.nrs_tbf_rule=start churn1 jobid={dd.*} rate=2000 ost.OSS.ost_io.nrs_tbf_rule=start churn2 jobid={dd.*}&opcode={ost_write} rate=2000 ost.OSS.ost_io.nrs_tbf_rule=stop churn2 ost.OSS.ost_io.nrs_tbf_rule=stop churn1 ost.OSS.ost_io.nrs_tbf_rule=start churn1 jobid={dd.*} rate=2000 ost.OSS.ost_io.nrs_tbf_rule=start churn2 jobid={dd.*}&opcode={ost_write} rate=2000 ost.OSS.ost_io.nrs_tbf_rule=stop churn2 ost.OSS.ost_io.nrs_tbf_rule=stop churn1 ost.OSS.ost_io.nrs_tbf_rule=start churn1 jobid={dd.*} rate=2000 ost.OSS.ost_io.nrs_tbf_rule=start churn2 jobid={dd.*}&opcode={ost_write} rate=2000 ost.OSS.ost_io.nrs_tbf_rule=stop churn2 ost.OSS.ost_io.nrs_tbf_rule=stop churn1 ost.OSS.ost_io.nrs_tbf_rule=start churn1 jobid={dd.*} rate=2000 ost.OSS.ost_io.nrs_tbf_rule=start churn2 jobid={dd.*}&opcode={ost_write} rate=2000 ost.OSS.ost_io.nrs_tbf_rule=stop churn2 ost.OSS.ost_io.nrs_tbf_rule=stop churn1 ost.OSS.ost_io.nrs_tbf_rule=start churn1 jobid={dd.*} rate=2000 ost.OSS.ost_io.nrs_tbf_rule=start churn2 jobid={dd.*}&opcode={ost_write} rate=2000 ost.OSS.ost_io.nrs_tbf_rule=stop churn2 ost.OSS.ost_io.nrs_tbf_rule=stop churn1 ost.OSS.ost_io.nrs_tbf_rule=start churn1 jobid={dd.*} rate=2000 ost.OSS.ost_io.nrs_tbf_rule=start churn2 jobid={dd.*}&opcode={ost_write} rate=2000 ost.OSS.ost_io.nrs_tbf_rule=stop churn2 ost.OSS.ost_io.nrs_tbf_rule=stop churn1 ost.OSS.ost_io.nrs_tbf_rule=start churn1 jobid={dd.*} rate=2000 ost.OSS.ost_io.nrs_tbf_rule=start churn2 jobid={dd.*}&opcode={ost_write} rate=2000 ost.OSS.ost_io.nrs_tbf_rule=stop churn2 ost.OSS.ost_io.nrs_tbf_rule=stop churn1 ost.OSS.ost_io.nrs_tbf_rule=start churn1 jobid={dd.*} rate=2000 ost.OSS.ost_io.nrs_tbf_rule=start churn2 jobid={dd.*}&opcode={ost_write} rate=2000 ost.OSS.ost_io.nrs_tbf_rule=stop churn2 ost.OSS.ost_io.nrs_tbf_rule=stop churn1 ost.OSS.ost_io.nrs_tbf_rule=start churn1 jobid={dd.*} rate=2000 ost.OSS.ost_io.nrs_tbf_rule=start churn2 jobid={dd.*}&opcode={ost_write} rate=2000 ost.OSS.ost_io.nrs_tbf_rule=stop churn2 ost.OSS.ost_io.nrs_tbf_rule=stop churn1 ost.OSS.ost_io.nrs_tbf_rule=start churn1 jobid={dd.*} rate=2000 ost.OSS.ost_io.nrs_tbf_rule=start churn2 jobid={dd.*}&opcode={ost_write} rate=2000 ost.OSS.ost_io.nrs_tbf_rule=stop churn2 ost.OSS.ost_io.nrs_tbf_rule=stop churn1 Cleanup TBF ost.OSS.ost_io.nrs_tbf_rule=stop churn1 ost.OSS.ost_io.nrs_tbf_rule=stop churn2 ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid jobid_name=%e.%u PASS 77tf (22s) == sanityn test 77te: Dynamic type lookup should obey the rule order ========================================================== 18:59:06 (1789772346) using SAVE_PROJECT_SUPPORTED=0 ost.OSS.ost_io.nrs_policies=tbf projid classify=dynamic allow=projid+opcode allow=uid ost.OSS.ost_io.nrs_tbf_allowed_types= regular_allowed_types: CPT 0: 0: 0x00000024 opcode+projid 1: 0x00000020 projid 2: 0x00000008 uid high_priority_allowed_types: CPT 0: 0: 0x00000024 opcode+projid 1: 0x00000020 projid 2: 0x00000008 uid ost.OSS.ost_io.nrs_tbf_rule=start rule1 projid={100}&opcode={ost_read} rate=10 ost.OSS.ost_io.nrs_tbf_rule=start rule2 projid={100}&opcode={ost_write} rate=10 ost.OSS.ost_io.nrs_tbf_rule=start rule3 uid={500} rate=5 regular_requests: CPT 0: rule3 uid={500} 5, seq 4, type=uid, ref 0 rule2 projid={100}&opcode={ost_write} 10, seq 3, type=opcode+projid, ref 0 rule1 projid={100}&opcode={ost_read} 10, seq 2, type=opcode+projid, ref 0 default * 10000, seq 1, type=opcode+uid+projid, ref 0 high_priority_requests: CPT 0: rule3 uid={500} 5, seq 4, type=uid, ref 0 rule2 projid={100}&opcode={ost_write} 10, seq 3, type=opcode+projid, ref 0 rule1 projid={100}&opcode={ost_read} 10, seq 2, type=opcode+projid, ref 0 default * 10000, seq 1, type=opcode+uid+projid, ref 0 not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10, bs: 1048576 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.77429 s, 10.7 MB/s Write runtime is 11 s, speed is 9.090909 vs. 10 Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73485 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 vs. 10 cleanup_tbf /mnt/lustre/d77te.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5, bs: 1048576 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77te.sanityn/tbf] [bs=1048576] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.4652 s, 5.4 MB/s Write runtime is 21 s, speed is 4.761904 vs. 5 Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77te.sanityn/tbf] [of=/dev/null] [bs=1048576] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9543 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 vs. 5 cleanup_tbf /mnt/lustre/d77te.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_policies=fifo PASS 77te (110s) == sanityn test 77tg: TBF: the default rule holds its rate with many rules ========================================================== 19:00:56 (1789772456) ost.OSS.ost_io.nrs_policies=tbf uid classify=dynamic allow=uid ost.OSS.ost_io.nrs_tbf_rule=start r5000 uid={5000} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5001 uid={5001} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5002 uid={5002} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5003 uid={5003} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5004 uid={5004} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5005 uid={5005} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5006 uid={5006} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5007 uid={5007} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5008 uid={5008} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5009 uid={5009} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5010 uid={5010} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5011 uid={5011} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5012 uid={5012} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5013 uid={5013} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5014 uid={5014} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5015 uid={5015} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5016 uid={5016} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5017 uid={5017} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5018 uid={5018} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5019 uid={5019} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5020 uid={5020} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5021 uid={5021} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5022 uid={5022} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5023 uid={5023} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5024 uid={5024} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5025 uid={5025} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5026 uid={5026} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5027 uid={5027} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5028 uid={5028} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5029 uid={5029} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5030 uid={5030} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5031 uid={5031} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5032 uid={5032} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5033 uid={5033} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5034 uid={5034} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5035 uid={5035} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5036 uid={5036} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5037 uid={5037} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5038 uid={5038} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5039 uid={5039} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5040 uid={5040} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5041 uid={5041} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5042 uid={5042} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5043 uid={5043} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5044 uid={5044} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5045 uid={5045} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5046 uid={5046} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5047 uid={5047} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5048 uid={5048} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5049 uid={5049} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5050 uid={5050} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5051 uid={5051} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5052 uid={5052} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5053 uid={5053} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5054 uid={5054} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5055 uid={5055} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5056 uid={5056} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5057 uid={5057} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5058 uid={5058} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5059 uid={5059} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5060 uid={5060} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5061 uid={5061} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5062 uid={5062} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5063 uid={5063} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5064 uid={5064} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5065 uid={5065} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5066 uid={5066} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5067 uid={5067} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5068 uid={5068} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5069 uid={5069} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5070 uid={5070} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5071 uid={5071} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5072 uid={5072} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5073 uid={5073} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5074 uid={5074} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5075 uid={5075} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5076 uid={5076} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5077 uid={5077} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5078 uid={5078} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5079 uid={5079} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5080 uid={5080} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5081 uid={5081} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5082 uid={5082} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5083 uid={5083} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5084 uid={5084} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5085 uid={5085} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5086 uid={5086} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5087 uid={5087} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5088 uid={5088} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5089 uid={5089} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5090 uid={5090} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5091 uid={5091} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5092 uid={5092} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5093 uid={5093} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5094 uid={5094} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5095 uid={5095} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5096 uid={5096} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5097 uid={5097} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5098 uid={5098} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5099 uid={5099} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5100 uid={5100} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5101 uid={5101} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5102 uid={5102} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5103 uid={5103} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5104 uid={5104} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5105 uid={5105} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5106 uid={5106} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5107 uid={5107} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5108 uid={5108} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5109 uid={5109} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5110 uid={5110} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5111 uid={5111} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5112 uid={5112} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5113 uid={5113} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5114 uid={5114} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5115 uid={5115} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5116 uid={5116} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5117 uid={5117} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5118 uid={5118} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5119 uid={5119} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5120 uid={5120} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5121 uid={5121} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5122 uid={5122} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5123 uid={5123} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5124 uid={5124} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5125 uid={5125} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5126 uid={5126} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5127 uid={5127} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5128 uid={5128} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5129 uid={5129} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5130 uid={5130} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5131 uid={5131} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5132 uid={5132} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5133 uid={5133} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5134 uid={5134} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5135 uid={5135} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5136 uid={5136} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5137 uid={5137} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5138 uid={5138} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5139 uid={5139} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5140 uid={5140} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5141 uid={5141} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5142 uid={5142} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5143 uid={5143} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5144 uid={5144} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5145 uid={5145} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5146 uid={5146} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5147 uid={5147} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5148 uid={5148} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5149 uid={5149} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5150 uid={5150} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5151 uid={5151} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5152 uid={5152} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5153 uid={5153} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5154 uid={5154} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5155 uid={5155} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5156 uid={5156} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5157 uid={5157} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5158 uid={5158} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5159 uid={5159} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5160 uid={5160} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5161 uid={5161} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5162 uid={5162} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5163 uid={5163} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5164 uid={5164} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5165 uid={5165} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5166 uid={5166} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5167 uid={5167} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5168 uid={5168} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5169 uid={5169} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5170 uid={5170} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5171 uid={5171} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5172 uid={5172} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5173 uid={5173} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5174 uid={5174} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5175 uid={5175} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5176 uid={5176} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5177 uid={5177} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5178 uid={5178} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5179 uid={5179} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5180 uid={5180} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5181 uid={5181} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5182 uid={5182} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5183 uid={5183} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5184 uid={5184} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5185 uid={5185} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5186 uid={5186} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5187 uid={5187} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5188 uid={5188} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5189 uid={5189} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5190 uid={5190} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5191 uid={5191} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5192 uid={5192} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5193 uid={5193} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5194 uid={5194} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5195 uid={5195} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5196 uid={5196} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5197 uid={5197} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5198 uid={5198} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5199 uid={5199} rate=1000 not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10, bs: 1048576 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.77397 s, 10.7 MB/s Write runtime is 11 s, speed is 9.090909 vs. 10 Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.92316 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 vs. 10 cleanup_tbf /mnt/lustre/d77tg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Cleanup TBF ost.OSS.ost_io.nrs_policies=fifo PASS 77tg (49s) == sanityn test 77ua: Setting TBF rate performance mode: BPS and IOPS ========================================================== 19:01:45 (1789772505) ost.OSS.ost_io.nrs_policies=tbf nid+opcode ost.OSS.ost_io.nrs_policies= regular_requests: - name: fifo state: started fallback: yes queued: 0 active: 0 - name: crrn state: stopped fallback: no queued: 0 active: 0 - name: orr state: stopped fallback: no queued: 0 active: 0 - name: trr state: stopped fallback: no queued: 0 active: 0 - name: tbf nid+opcode state: started fallback: no queued: 0 active: 0 - name: delay state: stopped fallback: no queued: 0 active: 0 high_priority_requests: - name: fifo state: started fallback: yes queued: 0 active: 0 - name: crrn state: stopped fallback: no queued: 0 active: 0 - name: orr state: stopped fallback: no queued: 0 active: 0 - name: trr state: stopped fallback: no queued: 0 active: 0 - name: tbf nid+opcode state: started fallback: no queued: 0 active: 0 - name: delay state: stopped fallback: no queued: 0 active: 0 ost.OSS.ost_io.nrs_tbf_rule=start nidrd nid={0@lo 192.168.204.47@tcp}&opcode={ost_read} bps=5MiB/s ost.OSS.ost_io.nrs_tbf_rule=start nidwr nid={0@lo 192.168.204.47@tcp}&opcode={ost_write} bps=15MiB/s regular_requests: CPT 0: nidwr nid={0@lo 192.168.204.47@tcp}&opcode={ost_write} 15, seq 3, type=nid+opcode, ref 0 bps=15728640 nidrd nid={0@lo 192.168.204.47@tcp}&opcode={ost_read} 5, seq 2, type=nid+opcode, ref 0 bps=5242880 default * 10000, seq 1, type=nid+opcode, ref 0 high_priority_requests: CPT 0: nidwr nid={0@lo 192.168.204.47@tcp}&opcode={ost_write} 15, seq 3, type=nid+opcode, ref 0 bps=15728640 nidrd nid={0@lo 192.168.204.47@tcp}&opcode={ost_read} 5, seq 2, type=nid+opcode, ref 0 bps=5242880 default * 10000, seq 1, type=nid+opcode, ref 0 ost.OSS.ost_io.nrs_tbf_rule=stop nidrd ost.OSS.ost_io.nrs_tbf_rule=stop nidwr ost.OSS.ost_io.nrs_tbf_rule=start nidrd nid={0@lo 192.168.204.47@tcp}&opcode={ost_read} bps=500K ost.OSS.ost_io.nrs_tbf_rule=start nidwr nid={0@lo 192.168.204.47@tcp}&opcode={ost_write} bps=750K regular_requests: CPT 0: nidwr nid={0@lo 192.168.204.47@tcp}&opcode={ost_write} 1, seq 5, type=nid+opcode, ref 0 bps=768000 nidrd nid={0@lo 192.168.204.47@tcp}&opcode={ost_read} 1, seq 4, type=nid+opcode, ref 0 bps=512000 default * 10000, seq 1, type=nid+opcode, ref 0 high_priority_requests: CPT 0: nidwr nid={0@lo 192.168.204.47@tcp}&opcode={ost_write} 1, seq 5, type=nid+opcode, ref 0 bps=768000 nidrd nid={0@lo 192.168.204.47@tcp}&opcode={ost_read} 1, seq 4, type=nid+opcode, ref 0 bps=512000 default * 10000, seq 1, type=nid+opcode, ref 0 ost.OSS.ost_io.nrs_tbf_rule=stop nidrd ost.OSS.ost_io.nrs_tbf_rule=stop nidwr ost.OSS.ost_io.nrs_tbf_rule=start nidrd nid={0@lo 192.168.204.47@tcp}&opcode={ost_read} rate=5MB/s ost.OSS.ost_io.nrs_tbf_rule=start nidwr nid={0@lo 192.168.204.47@tcp}&opcode={ost_write} rate=15MB/s regular_requests: CPT 0: nidwr nid={0@lo 192.168.204.47@tcp}&opcode={ost_write} 14, seq 7, type=nid+opcode, ref 0 bps=15000000 nidrd nid={0@lo 192.168.204.47@tcp}&opcode={ost_read} 4, seq 6, type=nid+opcode, ref 0 bps=5000000 default * 10000, seq 1, type=nid+opcode, ref 0 high_priority_requests: CPT 0: nidwr nid={0@lo 192.168.204.47@tcp}&opcode={ost_write} 14, seq 7, type=nid+opcode, ref 0 bps=15000000 nidrd nid={0@lo 192.168.204.47@tcp}&opcode={ost_read} 4, seq 6, type=nid+opcode, ref 0 bps=5000000 default * 10000, seq 1, type=nid+opcode, ref 0 ost.OSS.ost_io.nrs_tbf_rule=stop nidrd ost.OSS.ost_io.nrs_tbf_rule=stop nidwr ost.OSS.ost_io.nrs_tbf_rule=start nidrd nid={0@lo 192.168.204.47@tcp}&opcode={ost_read} rate=5MB ost.OSS.ost_io.nrs_tbf_rule=start nidwr nid={0@lo 192.168.204.47@tcp}&opcode={ost_write} rate=15MB regular_requests: CPT 0: nidwr nid={0@lo 192.168.204.47@tcp}&opcode={ost_write} 14, seq 9, type=nid+opcode, ref 0 bps=15000000 nidrd nid={0@lo 192.168.204.47@tcp}&opcode={ost_read} 4, seq 8, type=nid+opcode, ref 0 bps=5000000 default * 10000, seq 1, type=nid+opcode, ref 0 high_priority_requests: CPT 0: nidwr nid={0@lo 192.168.204.47@tcp}&opcode={ost_write} 14, seq 9, type=nid+opcode, ref 0 bps=15000000 nidrd nid={0@lo 192.168.204.47@tcp}&opcode={ost_read} 4, seq 8, type=nid+opcode, ref 0 bps=5000000 default * 10000, seq 1, type=nid+opcode, ref 0 ost.OSS.ost_io.nrs_tbf_rule=stop nidrd ost.OSS.ost_io.nrs_tbf_rule=stop nidwr ost.OSS.ost_io.nrs_tbf_rule=start nidrd nid={0@lo 192.168.204.47@tcp}&opcode={ost_read} bps=5 ost.OSS.ost_io.nrs_tbf_rule=start nidwr nid={0@lo 192.168.204.47@tcp}&opcode={ost_write} bps=15 regular_requests: CPT 0: nidwr nid={0@lo 192.168.204.47@tcp}&opcode={ost_write} 15, seq 11, type=nid+opcode, ref 0 bps=15728640 nidrd nid={0@lo 192.168.204.47@tcp}&opcode={ost_read} 5, seq 10, type=nid+opcode, ref 0 bps=5242880 default * 10000, seq 1, type=nid+opcode, ref 0 high_priority_requests: CPT 0: nidwr nid={0@lo 192.168.204.47@tcp}&opcode={ost_write} 15, seq 11, type=nid+opcode, ref 0 bps=15728640 nidrd nid={0@lo 192.168.204.47@tcp}&opcode={ost_read} 5, seq 10, type=nid+opcode, ref 0 bps=5242880 default * 10000, seq 1, type=nid+opcode, ref 0 ost.OSS.ost_io.nrs_tbf_rule=stop nidrd ost.OSS.ost_io.nrs_tbf_rule=stop nidwr ost.OSS.ost_io.nrs_tbf_rule=start nidrd nid={0@lo 192.168.204.47@tcp}&opcode={ost_read} bps=5 iops=1000 ost.OSS.ost_io.nrs_tbf_rule=start nidwr nid={0@lo 192.168.204.47@tcp}&opcode={ost_write} bps=15 iops=1200 regular_requests: CPT 0: nidwr nid={0@lo 192.168.204.47@tcp}&opcode={ost_write} 1200, seq 13, type=nid+opcode, ref 0 bps=15728640 nidrd nid={0@lo 192.168.204.47@tcp}&opcode={ost_read} 1000, seq 12, type=nid+opcode, ref 0 bps=5242880 default * 10000, seq 1, type=nid+opcode, ref 0 high_priority_requests: CPT 0: nidwr nid={0@lo 192.168.204.47@tcp}&opcode={ost_write} 1200, seq 13, type=nid+opcode, ref 0 bps=15728640 nidrd nid={0@lo 192.168.204.47@tcp}&opcode={ost_read} 1000, seq 12, type=nid+opcode, ref 0 bps=5242880 default * 10000, seq 1, type=nid+opcode, ref 0 ost.OSS.ost_io.nrs_tbf_rule=stop nidrd ost.OSS.ost_io.nrs_tbf_rule=stop nidwr ost.OSS.ost_io.nrs_policies=fifo PASS 77ua (10s) == sanityn test 77ub: Testing for TBF rate control in BPS performance mode ========================================================== 19:01:55 (1789772515) ost.OSS.ost_io.nrs_policies=tbf nid+opcode ost.OSS.ost_io.nrs_tbf_rule=start nidrd nid={0@lo 192.168.204.47@tcp}&opcode={ost_read} bps=5 ost.OSS.ost_io.nrs_tbf_rule=start nidwr nid={0@lo 192.168.204.47@tcp}&opcode={ost_write} bps=15 ost.OSS.ost_io.nrs_tbf_rule= regular_requests: CPT 0: nidwr nid={0@lo 192.168.204.47@tcp}&opcode={ost_write} 15, seq 3, type=nid+opcode, ref 0 bps=15728640 nidrd nid={0@lo 192.168.204.47@tcp}&opcode={ost_read} 5, seq 2, type=nid+opcode, ref 0 bps=5242880 default * 10000, seq 1, type=nid+opcode, ref 0 high_priority_requests: CPT 0: nidwr nid={0@lo 192.168.204.47@tcp}&opcode={ost_write} 15, seq 3, type=nid+opcode, ref 0 bps=15728640 nidrd nid={0@lo 192.168.204.47@tcp}&opcode={ost_read} 5, seq 2, type=nid+opcode, ref 0 bps=5242880 default * 10000, seq 1, type=nid+opcode, ref 0 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.49471 s, 33.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.052633 s, 19.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0651233 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0769316 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182834 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195788 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196599 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209527 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197557 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197787 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20828 s, 5.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.346395 s, 3.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.357903 s, 2.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.401482 s, 2.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.390037 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.467298 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.491384 s, 2.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21503 s, 4.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215039 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21343 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216548 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213406 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221259 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220025 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215563 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.214337 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.219122 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21777 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222131 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.218967 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.225458 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223496 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.224316 s, 4.7 MB/s removed '/mnt/lustre/d77ub.sanityn/nrs_w_oleg447-client.virtnet' removed '/mnt/lustre/d77ub.sanityn/nrs_r_oleg447-client.virtnet' removed directory '/mnt/lustre/d77ub.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 15, read rate: 5, bs: 4096 Verify the write rate is under TBF control 25600+0 records in 25600+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 286.584 s, 366 kB/s Write runtime is 287 s, speed is .348432 vs. 15 Verify the read rate is under TBF control 25600+0 records in 25600+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 32.1928 s, 3.3 MB/s Read runtime is 34 s, speed is 2.941176 vs. 5 cleanup_tbf /mnt/lustre/d77ub.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 15, read rate: 5, bs: 65536 Verify the write rate is under TBF control 1600+0 records in 1600+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.3181 s, 5.2 MB/s Write runtime is 22 s, speed is 4.545454 vs. 15 Verify the read rate is under TBF control 1600+0 records in 1600+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.1213 s, 5.2 MB/s Read runtime is 21 s, speed is 4.761904 vs. 5 cleanup_tbf /mnt/lustre/d77ub.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 15, read rate: 5, bs: 131072 Verify the write rate is under TBF control 800+0 records in 800+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.6981 s, 9.0 MB/s Write runtime is 13 s, speed is 7.692307 vs. 15 Verify the read rate is under TBF control 800+0 records in 800+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.0635 s, 5.2 MB/s Read runtime is 21 s, speed is 4.761904 vs. 5 cleanup_tbf /mnt/lustre/d77ub.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 15, read rate: 5, bs: 262144 Verify the write rate is under TBF control 400+0 records in 400+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.75277 s, 15.5 MB/s Write runtime is 7 s, speed is 14.285714 vs. 15 Verify the read rate is under TBF control 400+0 records in 400+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.004 s, 5.2 MB/s Read runtime is 21 s, speed is 4.761904 vs. 5 cleanup_tbf /mnt/lustre/d77ub.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 15, read rate: 5, bs: 1048576 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.87974 s, 15.2 MB/s Write runtime is 8 s, speed is 12.500000 vs. 15 Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8542 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 vs. 5 cleanup_tbf /mnt/lustre/d77ub.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Cleanup TBF ost.OSS.ost_io.nrs_tbf_rule=stop nidrd ost.OSS.ost_io.nrs_tbf_rule=stop nidwr ost.OSS.ost_io.nrs_policies=fifo Cleanup TBF ost.OSS.ost_io.nrs_policies=fifo PASS 77ub (572s) == sanityn test 77uc: TBF: the realtime key does not work with a byte rate ========================================================== 19:11:27 (1789773087) ost.OSS.ost_io.nrs_policies=tbf nid+opcode oleg447-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start rtwr nid={0@lo 192.168.204.47@tcp}&opcode={ost_write} realtime=1 bps=5MiB/s: Operation not supported oleg447-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start rtwr nid={0@lo 192.168.204.47@tcp}&opcode={ost_write} realtime=1 bps=5MiB/s': Operation not supported pdsh@oleg447-client: oleg447-server: ssh exited with exit code 95 ost.OSS.ost_io.nrs_tbf_rule=start rtwr nid={0@lo 192.168.204.47@tcp}&opcode={ost_write} realtime=1 rate=100 ost.OSS.ost_io.nrs_tbf_rule= regular_requests: CPT 0: rtwr nid={0@lo 192.168.204.47@tcp}&opcode={ost_write} 100, seq 2, type=nid+opcode, ref 0 default * 10000, seq 1, type=nid+opcode, ref 0 high_priority_requests: CPT 0: rtwr nid={0@lo 192.168.204.47@tcp}&opcode={ost_write} 100, seq 2, type=nid+opcode, ref 0 default * 10000, seq 1, type=nid+opcode, ref 0 Cleanup TBF ost.OSS.ost_io.nrs_tbf_rule=stop rtwr ost.OSS.ost_io.nrs_policies=fifo PASS 77uc (14s) == sanityn test 77ud: TBF: a rate change on a BPS rule leaves an IOPS rate ========================================================== 19:11:41 (1789773101) ost.OSS.ost_io.nrs_policies=tbf nid+opcode ost.OSS.ost_io.nrs_tbf_rule=start nidrd nid={0@lo 192.168.204.47@tcp}&opcode={ost_read} bps=5MiB/s ost.OSS.ost_io.nrs_tbf_rule=start nidwr nid={0@lo 192.168.204.47@tcp}&opcode={ost_write} bps=5MiB/s ost.OSS.ost_io.nrs_tbf_rule= regular_requests: CPT 0: nidwr nid={0@lo 192.168.204.47@tcp}&opcode={ost_write} 5, seq 3, type=nid+opcode, ref 0 bps=5242880 nidrd nid={0@lo 192.168.204.47@tcp}&opcode={ost_read} 5, seq 2, type=nid+opcode, ref 0 bps=5242880 default * 10000, seq 1, type=nid+opcode, ref 0 high_priority_requests: CPT 0: nidwr nid={0@lo 192.168.204.47@tcp}&opcode={ost_write} 5, seq 3, type=nid+opcode, ref 0 bps=5242880 nidrd nid={0@lo 192.168.204.47@tcp}&opcode={ost_read} 5, seq 2, type=nid+opcode, ref 0 bps=5242880 default * 10000, seq 1, type=nid+opcode, ref 0 ost.OSS.ost_io.nrs_tbf_rule=change nidrd rate=100 ost.OSS.ost_io.nrs_tbf_rule=change nidwr rate=100 ost.OSS.ost_io.nrs_tbf_rule= regular_requests: CPT 0: nidwr nid={0@lo 192.168.204.47@tcp}&opcode={ost_write} 100, seq 3, type=nid+opcode, ref 0 nidrd nid={0@lo 192.168.204.47@tcp}&opcode={ost_read} 100, seq 2, type=nid+opcode, ref 0 default * 10000, seq 1, type=nid+opcode, ref 0 high_priority_requests: CPT 0: nidwr nid={0@lo 192.168.204.47@tcp}&opcode={ost_write} 100, seq 3, type=nid+opcode, ref 0 nidrd nid={0@lo 192.168.204.47@tcp}&opcode={ost_read} 100, seq 2, type=nid+opcode, ref 0 default * 10000, seq 1, type=nid+opcode, ref 0 ost.OSS.ost_io.nrs_tbf_rule=change nidrd rate=20 ost.OSS.ost_io.nrs_tbf_rule=change nidwr rate=20 not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20, bs: 1048576 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.50431 s, 19.1 MB/s Write runtime is 7 s, speed is 14.285714 vs. 20 Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.95141 s, 21.2 MB/s Read runtime is 6 s, speed is 16.666666 vs. 20 cleanup_tbf /mnt/lustre/d77ud.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Cleanup TBF ost.OSS.ost_io.nrs_tbf_rule=stop nidrd ost.OSS.ost_io.nrs_tbf_rule=stop nidwr ost.OSS.ost_io.nrs_policies=fifo PASS 77ud (41s) == sanityn test 77ti: TBF: a late rule reaches a class bucket that exists ========================================================== 19:12:22 (1789773142) ost.OSS.ost_io.nrs_policies=tbf uid classify=dynamic allow=uid ost.OSS.ost_io.nrs_tbf_rule=start r5000 uid={5000} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5001 uid={5001} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5002 uid={5002} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5003 uid={5003} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5004 uid={5004} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5005 uid={5005} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5006 uid={5006} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5007 uid={5007} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5008 uid={5008} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5009 uid={5009} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5010 uid={5010} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5011 uid={5011} rate=1000 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ti.sanityn/nrs_r_oleg447-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.277004 s, 60.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0579734 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.045971 s, 22.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100469 s, 10.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126101 s, 8.3 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134335 s, 7.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141548 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151867 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143055 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126315 s, 8.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127622 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137363 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136465 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158153 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167908 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178086 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185209 s, 5.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9e5848c63800.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff9e5848c63800.lock_unused_count=7 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00647945 s, 162 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.027461 s, 38.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0179159 s, 58.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0346481 s, 30.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0221475 s, 47.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0105273 s, 99.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0196478 s, 53.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0224062 s, 46.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0237637 s, 44.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00453979 s, 231 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.004463 s, 235 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0202303 s, 51.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00515384 s, 203 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00533567 s, 197 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00437021 s, 240 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0090067 s, 116 MB/s removed '/mnt/lustre/d77ti.sanityn/nrs_w_oleg447-client.virtnet' removed '/mnt/lustre/d77ti.sanityn/nrs_r_oleg447-client.virtnet' removed directory '/mnt/lustre/d77ti.sanityn' ost.OSS.ost_io.nrs_tbf_rule=start late uid={500} rate=2 not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 2, read rate: 2, bs: 1048576 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ti.sanityn/tbf] [bs=1048576] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 48.7885 s, 2.1 MB/s Write runtime is 50 s, speed is 2.000000 vs. 2 Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ti.sanityn/tbf] [of=/dev/null] [bs=1048576] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 49.8672 s, 2.1 MB/s Read runtime is 51 s, speed is 1.960784 vs. 2 cleanup_tbf /mnt/lustre/d77ti.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Cleanup TBF ost.OSS.ost_io.nrs_policies=fifo PASS 77ti (134s) == sanityn test 78: Enable policy and specify tunings right away ========================================================== 19:14:36 (1789773276) ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_orr_quantum=1 ost.OSS.ost_io.nrs_policies=fifo PASS 78 (13s) == sanityn test 79: xattr: intent error ================== 19:14:50 (1789773290) File: /mnt/lustre/d79.sanityn Size: 16384 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115272381366439 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-18 19:14:53.000000000 -0400 Modify: 2026-09-18 19:14:53.000000000 -0400 Change: 2026-09-18 19:14:53.000000000 -0400 Birth: 2026-09-18 19:14:53.000000000 -0400 fail_val=0 fail_loc=0x80000160 fail_val=0 fail_loc=0x80000131 # file: mnt/lustre/d79.sanityn trusted.name1="value1" PASS 79 (20s) == sanityn test 80a: migrate directory when some children is being opened ========================================================== 19:15:10 (1789773310) SKIP: sanityn test_80a needs >= 2 MDTs SKIP 80a (3s) == sanityn test 80b: Accessing directory during migration ========================================================== 19:15:14 (1789773314) SKIP: sanityn test_80b needs >= 2 MDTs SKIP 80b (3s) == sanityn test 81a: rename and stat under striped directory ========================================================== 19:15:18 (1789773318) SKIP: sanityn test_81a needs >= 2 MDTs SKIP 81a (4s) == sanityn test 81b: rename under striped directory doesn't deadlock ========================================================== 19:15:22 (1789773322) SKIP: sanityn test_81b We need at least 2 MDTs for this test SKIP 81b (4s) == sanityn test 81c: rename revoke LOOKUP lock for remote object ========================================================== 19:15:27 (1789773327) SKIP: sanityn test_81c needs >= 4 MDTs SKIP 81c (5s) == sanityn test 81d: parallel rename file cross-dir on same MDT ========================================================== 19:15:32 (1789773332) - open/close 425 (time 1789773353.39 total 10.01 last 42.46) - open/close 982 (time 1789773363.40 total 20.01 last 55.67) - open/close 1643 (time 1789773373.40 total 30.02 last 66.07) - open/close 2262 (time 1789773383.40 total 40.02 last 61.87) total: 2500 open/close in 45.84 seconds: 54.54 ops/second snapshot_time 1789773505.427534958 secs.nsecs start_time 1789773338.342441380 secs.nsecs elapsed_time 167.085093578 secs.nsecs open 2505 samples [usecs] 148 40076 5232789 27855480623 close 2504 samples [usecs] 161 10407 1045259 1453716869 mknod 2501 samples [usecs] 588 39953 4597242 23676955320 unlink 2500 samples [usecs] 739 89600 5869879 42869469239 mkdir 2 samples [usecs] 2439 12918 15357 172823445 rmdir 2 samples [usecs] 12365 17457 29822 457640074 rename 2500 samples [usecs] 474 82973 23470902 525198844362 getattr 3761 samples [usecs] 9 8799 273623 415180443 parallel_rename_file 2500 samples [usecs] 474 82973 23470902 525198844362 crossdir_rename 2500 samples [usecs] 474 82973 23470902 525198844362 readpage 20 samples [usecs] 5607 77070 434084 14845995622 PASS 81d (180s) == sanityn test 82: fsetxattr and fgetxattr on orphan files ========================================================== 19:18:32 (1789773512) multiop /mnt/lustre/f82.sanityn vO_ac TMPPIPE=/tmp/multiop_open_wait_pipe.7636 multiop /mnt/lustre2/f82.sanityn vo_Ac TMPPIPE=/tmp/multiop_open_wait_pipe.7636 PASS 82 (11s) == sanityn test 83: access striped directory while it is being created/unlinked ========================================================== 19:18:44 (1789773524) SKIP: sanityn test_83 needs >= 2 MDTs SKIP 83 (5s) == sanityn test 84: 0-nlink race in lu_object_find() ===== 19:18:48 (1789773528) fail_loc=0x8000060b PASS 84 (13s) == sanityn test 85: Lustre API root cache race =========== 19:19:02 (1789773542) Starting: llapi_root_test /mnt/lustre/f85.sanityn /mnt/lustre2/f85.sanityn Starting test test1 at 1789773543 Finishing test test1 at 1789773548 PASS 85 (10s) == sanityn test 90: open/create and unlink striped directory ========================================================== 19:19:13 (1789773553) SKIP: sanityn test_90 needs >= 2 MDTs SKIP 90 (3s) == sanityn test 91: chmod and unlink striped directory === 19:19:16 (1789773556) SKIP: sanityn test_91 needs >= 2 MDTs SKIP 91 (3s) == sanityn test 92: create remote directory under orphan directory ========================================================== 19:19:19 (1789773559) SKIP: sanityn test_92 needs >= 2 MDTs SKIP 92 (4s) == sanityn test 93: alloc_rr should not allocate on same ost ========================================================== 19:19:23 (1789773563) fail_loc=0x00000163 fail_loc=0x0 /mnt/lustre/f93.sanityn-1/file1 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 6381 0x18ed 0x280000400 0 6450 0x1932 0x240000400 /mnt/lustre2/f93.sanityn-2/file2 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 6382 0x18ee 0x280000400 0 6451 0x1933 0x240000400 rc1=0 and rc2=0 PASS 93 (18s) == sanityn test 94: signal vs CP callback race =========== 19:19:41 (1789773581) osc.lustre-OST0000-osc-ffff9e5848c63800.idle_timeout=0 osc.lustre-OST0000-osc-ffff9e5848e10000.idle_timeout=0 osc.lustre-OST0001-osc-ffff9e5848c63800.idle_timeout=0 osc.lustre-OST0001-osc-ffff9e5848e10000.idle_timeout=0 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.223133 s, 36.7 kB/s fail_val=5 fail_loc=0x80000312 fail_val=6 fail_loc=0x80000329 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 3.00354 s, 1.4 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 7277: 323817 Alarm clock dd if=/dev/zero of=$DIR/$tfile conv=notrunc oflag=append bs=4k count=1 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.110094 s, 37.2 kB/s osc.lustre-OST0000-osc-ffff9e5848c63800.idle_timeout=debug osc.lustre-OST0000-osc-ffff9e5848e10000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9e5848c63800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9e5848e10000.idle_timeout=debug PASS 94 (16s) == sanityn test 95a: Check readpage() on a page that was removed from page cache ========================================================== 19:19:57 (1789773597) llite.lustre-ffff9e5848c63800.fast_read=0 llite.lustre-ffff9e5848e10000.fast_read=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0612839 s, 34.2 MB/s fail_loc=0x80001421 fail_val=7 fail_loc=0x80001422 fail_val=10 llite.lustre-ffff9e5848c63800.fast_read=1 llite.lustre-ffff9e5848e10000.fast_read=1 PASS 95a (20s) == sanityn test 95b: Check readpage() on a page that is no longer uptodate ========================================================== 19:20:18 (1789773618) llite.lustre-ffff9e5848c63800.fast_read=0 llite.lustre-ffff9e5848e10000.fast_read=0 1+0 records in 1+0 records out 12288 bytes (12 kB, 12 KiB) copied, 0.00492164 s, 2.5 MB/s fail_loc=0x80001424 fail_val=5 fail_loc=0 llite.lustre-ffff9e5848c63800.fast_read=1 llite.lustre-ffff9e5848e10000.fast_read=1 PASS 95b (14s) == sanityn test 100a: DoM: glimpse RPCs for stat without IO lock (DoM only file) ========================================================== 19:20:31 (1789773631) SKIP: sanityn test_100a Reserved for glimpse-ahead SKIP 100a (4s) == sanityn test 100b: DoM: no glimpse RPC for stat with IO lock (DoM only file) ========================================================== 19:20:35 (1789773635) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0122815 s, 334 kB/s /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK PASS 100b (8s) == sanityn test 100c: DoM: write vs stat without IO lock (combined file) ========================================================== 19:20:43 (1789773643) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.212298 s, 9.9 MB/s /mnt/lustre/d100c.sanityn/dom has type file OK /mnt/lustre/d100c.sanityn/dom has size 2097152 OK PASS 100c (7s) == sanityn test 100d: DoM: write+truncate vs stat without IO lock (combined file) ========================================================== 19:20:50 (1789773650) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.217162 s, 9.7 MB/s /mnt/lustre/d100d.sanityn/dom has type file OK /mnt/lustre/d100d.sanityn/dom has size 4096 OK PASS 100d (7s) == sanityn test 100e: DoM: read on open and file size ==== 19:20:58 (1789773658) 1+0 records in 1+0 records out 12000 bytes (12 kB, 12 KiB) copied, 0.0117865 s, 1.0 MB/s PASS 100e (8s) == sanityn test 101a: Discard DoM data on unlink ========= 19:21:06 (1789773666) /mnt/lustre/f101a.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0202844 s, 202 kB/s PASS 101a (8s) == sanityn test 101b: Discard DoM data on rename ========= 19:21:15 (1789773675) /mnt/lustre/f101b.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0238024 s, 172 kB/s PASS 101b (8s) == sanityn test 101c: Discard DoM data on close-unlink === 19:21:22 (1789773682) /mnt/lustre/f101c.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0285765 s, 143 kB/s PASS 101c (9s) == sanityn test 102: Test open by handle of unlinked file ========================================================== 19:21:32 (1789773692) file: /mnt/lustre/f102.sanityn fh_bytes: 32 fh_type: 151 fh_data: a1 13 00 00 02 00 00 00 88 0a 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 file's parent FID is [0:0x0:0x0] file FID is [0x2000013a1:0xa88:0x0] access via mount point '/mnt/lustre2' - OK access after unlink - OK check_fhandle_syscalls test Passed! PASS 102 (8s) == sanityn test 103: Test size correctness with lockahead ========================================================== 19:21:39 (1789773699) fail_loc=0x415 fail_loc=0x415 fail_val=2 Incorrect size expected (no glimpse fix): Starting test test23 at 1789773702 lockahead_test: lockahead_test.c:1171: test23: assertion 'sb.st_size == sb2.st_size' failed: size on /mnt/lustre/d103.sanityn/f103.sanityn and /mnt/lustre2/d103.sanityn/f103.sanityn differs: 0 vs 1048576 fail_loc=0 fail_loc=0x214 fail_val=2 Starting test test23 at 1789773705 Finishing test test23 at 1789773705 PASS 103 (14s) == sanityn test 104: Verify that MDS stores atime/mtime/ctime during close ========================================================== 19:21:53 (1789773713) SKIP: sanityn test_104 ldiskfs only test SKIP 104 (3s) == sanityn test 105: Glimpse and lock cancel race ======== 19:21:56 (1789773716) fail_loc=0x416 PASS 105 (27s) == sanityn test 106a: Verify the btime via statx() ======= 19:22:23 (1789773743) SKIP: sanityn test_106a Test only for ldiskfs and statx() supported SKIP 106a (3s) == sanityn test 106b: Glimpse RPCs test for statx ======== 19:22:26 (1789773746) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00750215 s, 136 kB/s File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115272381368975 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-18 19:22:27.000000000 -0400 Modify: 2026-09-18 19:22:27.000000000 -0400 Change: 2026-09-18 19:22:27.000000000 -0400 Birth: 2026-09-18 19:22:27.000000000 -0400 /mnt/lustre/f106b.sanityn 144115272381368975 -rw-r--r-- 1 0 0 1789773747 1789773747 1789773747 File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115272381368975 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-18 19:22:27.000000000 -0400 Modify: 2026-09-18 19:22:27.000000000 -0400 Change: 2026-09-18 19:22:27.000000000 -0400 Birth: 2026-09-18 19:22:27.000000000 -0400 1789773747 1024 1 PASS 106b (6s) == sanityn test 106c: Verify statx attributes mask ======= 19:22:33 (1789773753) PASS 106c (7s) == sanityn test 107a: Basic grouplock conflict =========== 19:22:40 (1789773760) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.321919 s, 32.6 MB/s /mnt/lustre/f107a.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 6389 0x18f5 0x280000400 multiop /mnt/lustre/f107a.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7636 multiop /mnt/lustre2/f107a.sanityn vO_G16022000r10g16022000c TMPPIPE=/tmp/multiop_open_wait_pipe.7636 First grouplock blocks second one PASS 107a (8s) == sanityn test 107b: Grouplock is added to the head of waiting list ========================================================== 19:22:48 (1789773768) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.256437 s, 40.9 MB/s /mnt/lustre/f107b.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 6460 0x193c 0x240000400 multiop /mnt/lustre/f107b.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7636 Grouplock blocks IO First grouplock blocks second one Second grouplock blocks IO PASS 107b (12s) == sanityn test 108a: lseek: parallel updates ============ 19:23:00 (1789773780) 1+0 records in 1+0 records out 10000 bytes (10 kB, 9.8 KiB) copied, 0.00631911 s, 1.6 MB/s llite.lustre-ffff9e5848c63800.hybrid_io=0 llite.lustre-ffff9e5848e10000.hybrid_io=0 fail_val=4 fail_loc=0x80000414 1+0 records in 1+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 4.44345 s, 1.9 MB/s PASS 108a (10s) == sanityn test 109: Race with several mount instances on 1 node ========================================================== 19:23:10 (1789773790) 192.168.204.147@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg447-client.virtnet /mnt/lustre (opts:) 192.168.204.147@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg447-client.virtnet /mnt/lustre2 (opts:) Starting race between client mounts (duration 30) Iteration 0 Starting client: oleg447-client.virtnet: -o user_xattr,flock 192.168.204.147@tcp:/lustre /mnt/lustre2 Starting client: oleg447-client.virtnet: -o user_xattr,flock 192.168.204.147@tcp:/lustre /mnt/lustre 192.168.204.147@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.147@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg447-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg447-client.virtnet /mnt/lustre (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg447-server' oleg447-server: oleg447-server.virtnet: executing load_modules_local oleg447-server: kmemleak disabled oleg447-server: Loading modules from /home/green/git/lustre-release/lustre oleg447-server: detected 4 online CPUs by sysfs oleg447-server: MODOPTS_LIBCFS= oleg447-server: Force libcfs to create 2 CPU partitions oleg447-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Iteration 1 Starting client: oleg447-client.virtnet: -o user_xattr,flock 192.168.204.147@tcp:/lustre /mnt/lustre Starting client: oleg447-client.virtnet: -o user_xattr,flock 192.168.204.147@tcp:/lustre /mnt/lustre2 192.168.204.147@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.147@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg447-client.virtnet /mnt/lustre (opts:) Stopping client oleg447-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg447-server' oleg447-server: oleg447-server.virtnet: executing load_modules_local oleg447-server: kmemleak disabled oleg447-server: Loading modules from /home/green/git/lustre-release/lustre oleg447-server: detected 4 online CPUs by sysfs oleg447-server: MODOPTS_LIBCFS= oleg447-server: Force libcfs to create 2 CPU partitions oleg447-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Starting client: oleg447-client.virtnet: -o user_xattr,flock 192.168.204.147@tcp:/lustre /mnt/lustre Starting client: oleg447-client.virtnet: -o user_xattr,flock 192.168.204.147@tcp:/lustre /mnt/lustre2 PASS 109 (42s) == sanityn test 110: do not grant another lock on resend ========================================================== 19:23:52 (1789773832) fail_loc=0x525 fail_val=3 fail_loc=0x534 File: /mnt/lustre/f110.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115272414920707 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-18 19:23:53.000000000 -0400 Modify: 2026-09-18 19:23:53.000000000 -0400 Change: 2026-09-18 19:23:53.000000000 -0400 Birth: 2026-09-18 19:23:53.000000000 -0400 fail_loc=0 fail_val=0 Connected clients: oleg447-client.virtnet oleg447-client.virtnet PASS 110 (170s) == sanityn test 111: A racy rename/link an open file should not cause fs corruption ========================================================== 19:26:42 (1789774002) SKIP: sanityn test_111 needs >= 2 MDTs SKIP 111 (1s) == sanityn test 112: update max-inherit in default LMV === 19:26:44 (1789774004) SKIP: sanityn test_112 We need at least 2 MDTs for this test SKIP 112 (1s) == sanityn test 113: check servers of specified fs ======= 19:26:45 (1789774005) lustre-OST0000-osc-ffff9e587acbb000 active. lustre-OST0001-osc-ffff9e587acbb000 active. lustre-OST0000-osc-ffff9e587acbb000 active. lustre-OST0001-osc-ffff9e587acbb000 active. MGC192.168.204.147@tcp active. PASS 113 (2s) == sanityn test 114: implicit default LMV inherit ======== 19:26:47 (1789774007) SKIP: sanityn test_114 We need at least 2 MDTs for this test SKIP 114 (1s) == sanityn test 115: ldiskfs doesn't check direntry for uniqueness ========================================================== 19:26:48 (1789774008) SKIP: sanityn test_115 ldiskfs only test SKIP 115 (1s) == sanityn test 116: DNE: Set default LMV layout from a remote client ========================================================== 19:26:49 (1789774009) SKIP: sanityn test_116 needs >= 2 MDTs SKIP 116 (2s) == sanityn test 121: trunc append race =================== 19:26:51 (1789774011) 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0067788 s, 3.0 MB/s fail_loc=0x80001436 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0177702 s, 1.2 MB/s 0000000 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0005000 0000 0000 0000 0000 0000 0000 0000 0000 * 000c800 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0011800 PASS 121 (4s) == sanityn test 122: directory size is consistent across mounts ========================================================== 19:26:55 (1789774015) PASS 122 (3s) == sanityn test 200: service remains healthy while able to process request ========================================================== 19:26:58 (1789774018) at_max=10 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=13 fail_loc=0x1e1 fail_val=20 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0935 s, 0.2 kB/s4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0893 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0941 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0896 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0952 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0932 s, 0.2 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0987 s, 0.2 kB/s 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1048 s, 0.2 kB/s 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1019 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1356 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1176 s, 0.1 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1297 s, 0.1 kB/s 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1472 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1808 s, 0.1 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1543 s, 0.1 kB/s 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1569 s, 0.1 kB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1881 s, 0.1 kB/s 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1512 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2433 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.1975 s, 0.1 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2514 s, 0.1 kB/s4096 bytes (4.1 kB, 4.0 KiB) copied, 60.272 s, 0.1 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2581 s, 0.1 kB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2729 s, 0.1 kB/s 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2433 s, 0.1 kB/s4096 bytes (4.1 kB, 4.0 KiB) copied, 60.276 s, 0.1 kB/s at_max=600 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=116 oleg447-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9e5847ff3000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9e5847ff3000.ost_server_uuid in IDLE state after 0 sec PASS 200 (68s) cleanup: ====================================================== == sanityn test complete, duration 10623 sec ============= 19:28:07 (1789774087) === sanityn: start cleanup 19:28:07 (1789774087) === Stopping clients: oleg447-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg447-client.virtnet /mnt/lustre2 opts: === sanityn: finish cleanup 19:29:00 (1789774140) ===