-----============= acceptance-small: sanityn ============----- Tue Sep 1 07:46:03 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg113-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanityn.*ex': No such file or directory excepting tests: 27 40a skipping tests SLOW=no: 33a === sanityn: start setup 07:46:22 (1788263182) === Starting client oleg113-client.virtnet: -o user_xattr,flock 192.168.201.113@tcp:/lustre /mnt/lustre2 Mount client oleg113-client.virtnet: mount -t lustre -o user_xattr,flock 192.168.201.113@tcp:/lustre /mnt/lustre2 Started clients oleg113-client.virtnet: 192.168.201.113@tcp:/lustre on /mnt/lustre2 type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) oleg113-client.virtnet: executing check_config_client /mnt/lustre oleg113-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg113-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff911148971800.idle_timeout=debug osc.lustre-OST0000-osc-ffff91116186b000.idle_timeout=debug osc.lustre-OST0001-osc-ffff911148971800.idle_timeout=debug osc.lustre-OST0001-osc-ffff91116186b000.idle_timeout=debug disable quota as required oleg113-server: oleg113-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanityn: finish setup 07:46:54 (1788263214) === 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154652 s, 67.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7626] == sanityn test 0: do_node_vp() and do_facet_vp() do the right thing ========================================================== 07:46:55 (1788263215) PASS 0 (7s) == sanityn test 1: Check attribute updates on 2 mount points ========================================================== 07:47:03 (1788263223) /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0777 OK /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0666 OK /mnt/lustre/f1.sanityn: absent OK PASS 1 (6s) == sanityn test 2a: check cached attribute updates on 2 mtpt's ================================================================== 07:47:09 (1788263229) -rw-r--r-- 1 root root 0 Sep 1 07:47 /mnt/lustre2/f2a /mnt/lustre/f2a has type file OK /mnt/lustre/f2a has perms 0777 OK PASS 2a (8s) == sanityn test 2b: check cached attribute updates on 2 mtpt's ================================================================== 07:47:17 (1788263237) -rw-r--r-- 1 root root 0 Sep 1 07:47 /mnt/lustre2/f2b /mnt/lustre2/f2b has type file OK /mnt/lustre2/f2b has perms 0777 OK PASS 2b (8s) == sanityn test 2c: check cached attribute updates on 2 mtpt's root ============================================================= 07:47:25 (1788263245) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0777 OK PASS 2c (7s) == sanityn test 2d: check cached attribute updates on 2 mtpt's root ============================================================= 07:47:32 (1788263252) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0755 OK PASS 2d (7s) == sanityn test 2e: check chmod on root is propagated to others ========================================================== 07:47:39 (1788263259) total 1 -rwxrwxrwx 1 root root 0 Sep 1 07:47 f2a -rwxrwxrwx 1 root root 0 Sep 1 07:47 f2b total 1 -rwxrwxrwx 1 root root 0 Sep 1 07:47 f2a -rwxrwxrwx 1 root root 0 Sep 1 07:47 f2b running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre2/f2e.sanityn] [count=1] 1+0 records in 1+0 records out 512 bytes copied, 0.0126973 s, 40.3 kB/s PASS 2e (5s) == sanityn test 2f: check attr/owner updates on DNE with 2 mtpt's ========================================================== 07:47:44 (1788263264) SKIP: sanityn test_2f needs >= 2 MDTs SKIP 2f (4s) == sanityn test 2g: check blocks update on sync write ==== 07:47:48 (1788263268) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.313118 s, 6.7 MB/s /mnt/lustre/f2g.sanityn has 4103 blocks /mnt/lustre2/f2g.sanityn has 4103 blocks PASS 2g (6s) == sanityn test 3: symlink on one mtpt, readlink on another ===================================================================== 07:47:54 (1788263274) PASS 3 (7s) == sanityn test 4: fstat validation on multiple mount points ==================================================================== 07:48:01 (1788263281) PASS 4 (8s) == sanityn test 5: create a file on one mount, truncate it on the other ========================================================== 07:48:09 (1788263289) /mnt/lustre/f5 has type file OK /mnt/lustre/f5 has size 100 OK PASS 5 (6s) == sanityn test 6: remove of open file on other node ============================================================================ 07:48:15 (1788263295) opening writing unlinking /mnt/lustre2/f6.sanityn accessing (1) seeking (1) accessing (2) fstat... reading comparing data truncating seeking (2) writing again seeking (3) reading again comparing data again closing SUCCESS - goto beer PASS 6 (7s) == sanityn test 7: remove of open directory on other node ======================================================================= 07:48:22 (1788263302) creating directory /mnt/lustre/d7 opening directory unlinking /mnt/lustre/d7 Ok, everything goes well. PASS 7 (6s) == sanityn test 8: remove of open special file on other node ==================================================================== 07:48:29 (1788263309) creating special file /mnt/lustre/f8.sanityn opening file unlinking /mnt/lustre/f8.sanityn Ok, everything goes well. PASS 8 (7s) == sanityn test 9a: append of file with sub-page size on multiple mounts ========================================================== 07:48:35 (1788263315) PASS 9a (6s) == sanityn test 9b: append to striped sparse file ======== 07:48:42 (1788263322) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.054491 s, 19.2 MB/s 3+0 records in 3+0 records out 3 bytes copied, 0.0202168 s, 0.1 kB/s Data read (expecting 'foo'): foo PASS 9b (7s) == sanityn test 10a: write of file with sub-page size on multiple mounts ========================================================== 07:48:48 (1788263328) 1+0 records in 1+0 records out 1 byte copied, 0.00534991 s, 0.2 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0126765 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0194206 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0252777 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0156078 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0149265 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0142146 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0135211 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0140697 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0105239 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0210163 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00897632 s, 0.1 kB/s PASS 10a (7s) == sanityn test 10b: write of file with sub-page size on multiple mounts ========================================================== 07:48:55 (1788263335) 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.0119806 s, 256 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0301423 s, 136 kB/s 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.000406437 s, 7.6 MB/s PASS 10b (7s) == sanityn test 11: execution of file opened for write should return error ============================================================== 07:49:02 (1788263342) multiop /mnt/lustre/d11/f vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7626 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 298: /mnt/lustre2/d11/f: Text file busy PASS 11 (7s) == sanityn test 12: test lock ordering (link, stat, unlink) ========================================================== 07:49:09 (1788263349) warning: '-runtime' deprecated, use '-t runtime' instead - link 386 (time 1788263361.75 total 10.06 last 38.38) - link 758 (time 1788263371.76 total 20.07 last 37.14) - link 1129 (time 1788263381.77 total 30.08 last 37.06) - link 1436 (time 1788263391.79 total 40.10 last 30.64) - link 1797 (time 1788263401.81 total 50.12 last 36.03) - link 2211 (time 1788263411.85 total 60.16 last 41.24) - link 2624 (time 1788263421.88 total 70.19 last 41.19) - link 2972 (time 1788263431.91 total 80.22 last 34.69) - link 3246 (time 1788263441.92 total 90.23 last 27.37) total: 3548 link in 100.02 seconds: 35.47 ops/second using seed 524939157 running for 100 seconds - stat 5680 (time 1788263363 ; total 11 ; last 11) - stat 10000 (time 1788263369 ; total 17 ; last 6) - stat 17282 (time 1788263380 ; total 28 ; last 11) - stat 20000 (time 1788263385 ; total 33 ; last 5) - stat 25529 (time 1788263396 ; total 44 ; last 11) - stat 30000 (time 1788263402 ; total 50 ; last 6) - stat 33499 (time 1788263413 ; total 61 ; last 11) - stat 38072 (time 1788263424 ; total 72 ; last 11) - stat 40000 (time 1788263429 ; total 77 ; last 5) - stat 42367 (time 1788263440 ; total 88 ; last 11) - stat 45275 (time 1788263451 ; total 99 ; last 11) total: 45531 stats in 100 seconds: 455.309998 stats/second - unlinked 0 (time 1788263402 ; total 0 ; last 0) unlink(/mnt/lustre2/lockdir/lockfile3548) error: No such file or directory total: 3547 unlinks in 87 seconds: 40.770115 unlinks/second /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 77: kill: (21423) - No such process /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 78: kill: (21425) - No such process PASS 12 (145s) == sanityn test 13: test directory page revocation ======= 07:51:34 (1788263494) f13 f13 PASS 13 (9s) == sanityn test 14aa: execution of file open for write returns -ETXTBSY ========================================================== 07:51:43 (1788263503) multiop /mnt/lustre/d14aa.sanityn/f14aa.sanityn vOw_c TMPPIPE=/tmp/multiop_open_wait_pipe.7626 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 333: /mnt/lustre2/d14aa.sanityn/f14aa.sanityn: Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 22608 Terminated $MULTIOP_PROG $FILE v$ARGS > $TMPPIPE (wd: ~) PASS 14aa (6s) == sanityn test 14ab: open(RDWR) of executing file returns -ETXTBSY ========================================================== 07:51:49 (1788263509) open(O_RDWR|O_CREAT): Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 23187 Terminated $DIR1/$tdir/sleep 60 (wd: ~) PASS 14ab (7s) == sanityn test 14b: truncate of executing file returns -ETXTBSY ================================================================ 07:51:56 (1788263516) truncate: cannot truncate '/mnt/lustre2/d14b.sanityn/sleep' to length 60: Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 348: 23766 Terminated $DIR1/$tdir/sleep 60 PASS 14b (8s) == sanityn test 14c: open(O_TRUNC) of executing file return -ETXTBSY ============================================================ 07:52:04 (1788263524) cp: cannot create regular file '/mnt/lustre2/d14c.sanityn/sleep': Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 360: 24348 Terminated $DIR1/$tdir/sleep 60 PASS 14c (7s) == sanityn test 14d: chmod of executing file is still possible ================================================================== 07:52:12 (1788263532) chmod /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 371: 24931 Terminated $DIR1/$tdir/sleep 60 PASS 14d (10s) == sanityn test 15: test out-of-space with multiple writers ===================================================================== 07:52:22 (1788263542) PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin: MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky sleep 5 for ZFS MDS Waiting for MDT destroys to complete fallocate on zfs doesn't consume space STRIPECOUNT=2 ORIGFREE=7531520 MAXFREE=800000 SKIP: oos2 test_15 oos2.sh: 7531520KB free > 800000KB, increase MAXFREE (or reduce fs size) sleep 5 for ZFS MDS Waiting for MDT destroys to complete SKIP 15 (53s) == sanityn test 16a: 500 iterations of dual-mount fsx ==== 07:53:15 (1788263595) keep default fallocate mode: 2 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.741287 s, 11.3 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.572974 s, 14.6 MB/s Chance of close/open is 1 in 50 Seed set to 3601 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read skipping zero length punch hole skipping zero size read skipping zero size read skipping zero length punch hole skipping zero length punch hole 000010[0] 1788263602.356582 mapwrite 0x584d3f thru 0x58ef9b (0xa25d bytes) 000011[1] 1788263602.458230 mapread 0x561db0 thru 0x56835f (0x65b0 bytes) 000013[0] 1788263602.549619 write 0x2d617c thru 0x2df2b1 (0x9136 bytes) 000015[0] 1788263602.591571 punch from 0x11ff77 to 0x12c54a (0xc5d4 bytes) 000016[1] 1788263602.611142 mapwrite 0x7773d1 thru 0x77f61a (0x824a bytes) 000018[1] 1788263602.760869 mapwrite 0x9f9255 thru 0x9fffff (0x6dab bytes) 000020[0] 1788263602.967447 read 0x20c5cf thru 0x21a962 (0xe394 bytes) truncating to largest ever: 0x7a4db1 000024[1] 1788263603.065556 trunc from 0xa00000 to 0x7a4db0 (0x25b24f bytes) 000025[1] 1788263603.126384 mapread 0x1583b3 thru 0x163df7 (0xba45 bytes) 000026[0] 1788263603.205123 punch from 0x4fb53a to 0x50351d (0x7fe4 bytes) 000028[0] 1788263603.292561 write 0x4f9c09 thru 0x507f26 (0xe31e bytes) truncating to largest ever: 0x9504f5 000029[1] 1788263603.343581 trunc from 0x7a4db1 to 0x9504f4 (0x1ab744 bytes) 000031[1] 1788263603.465817 write 0x4a7637 thru 0x4a9d52 (0x271c bytes) 000032[0] 1788263603.538703 mapread 0x193c94 thru 0x194271 (0x5de bytes) 000033[0] 1788263603.602388 mapread 0x2c8dd0 thru 0x2c9c54 (0xe85 bytes) 000034[1] 1788263603.678243 read 0x386f35 thru 0x394221 (0xd2ed bytes) 000035[1] 1788263603.690149 trunc from 0x9504f5 to 0x428ade (0x527a16 bytes) 000037[0] 1788263603.775806 write 0x31a4d8 thru 0x32a44e (0xff77 bytes) 000038[0] 1788263603.840167 punch from 0x266e22 to 0x26ad31 (0x3f10 bytes) 000039[1] 1788263603.864805 mapread 0x0b1a09 thru 0x0b7816 (0x5e0e bytes) 000040[0] 1788263603.938387 write 0x78722a thru 0x7878bb (0x692 bytes) 000041[1] 1788263603.965997 mapread 0x585409 thru 0x58a7fa (0x53f2 bytes) 000046[0] 1788263604.018073 mapwrite 0x63707d thru 0x63f4c4 (0x8448 bytes) 000048[1] 1788263604.232086 punch from 0x59b63b to 0x5a0deb (0x57b1 bytes) 000050[0] 1788263604.283243 read 0x727b29 thru 0x730b64 (0x903c bytes) 000051[0] 1788263604.332986 write 0x9d1ac4 thru 0x9dc2df (0xa81c bytes) 000052[1] 1788263604.378506 mapread 0x780185 thru 0x78eef2 (0xed6e bytes) 000053[1] 1788263604.588384 trunc from 0x9dc2e0 to 0x023085 (0x9b925a bytes) 000056[1] 1788263604.762537 write 0x1ad9d8 thru 0x1b6629 (0x8c52 bytes) 000058[1] 1788263604.843855 write 0x731974 thru 0x73bbf5 (0xa282 bytes) 000059[0] 1788263604.877111 write 0x3696bf thru 0x377a43 (0xe385 bytes) 000061[1] 1788263604.949853 trunc from 0x73bbf6 to 0x3c7da3 (0x373e52 bytes) 000062[0] 1788263605.028090 punch from 0x248efb to 0x24a21f (0x1325 bytes) 000063[0] 1788263605.066589 punch from 0x2ed0f7 to 0x2f399b (0x68a5 bytes) 000064[1] 1788263605.084553 read 0x3c1fd6 thru 0x3c49b3 (0x29de bytes) 000065[1] 1788263605.212414 read 0x0c3170 thru 0x0d23cf (0xf260 bytes) 000066[0] 1788263605.230560 trunc from 0x3c7da4 to 0x8279f1 (0x45fc4e bytes) 000068[1] 1788263605.334651 mapread 0x39e202 thru 0x3a85d2 (0xa3d1 bytes) 000070[1] 1788263605.482825 trunc from 0x8279f2 to 0x617e8a (0x20fb67 bytes) 000071[0] 1788263605.552093 mapread 0x4edbf9 thru 0x4f761f (0x9a27 bytes) 000075[0] 1788263605.599611 mapwrite 0x63102a thru 0x6324fe (0x14d5 bytes) 000078[0] 1788263605.754727 mapread 0x2351bc thru 0x243cee (0xeb33 bytes) 000079[0] 1788263605.784139 punch from 0x085364 to 0x08b12f (0x5dcc bytes) 000082[1] 1788263605.845348 trunc from 0x6324ff to 0x1e4dad (0x44d751 bytes) 000083[0] 1788263605.891785 write 0x290aac thru 0x293661 (0x2bb6 bytes) 000087[1] 1788263605.980682 mapwrite 0x82c801 thru 0x82f528 (0x2d28 bytes) 000088[0] 1788263606.300356 mapread 0x5dd18e thru 0x5e9242 (0xc0b5 bytes) 000089[1] 1788263606.386040 read 0x5e97fc thru 0x5ee54c (0x4d51 bytes) 000091[0] 1788263606.450064 mapread 0x1ebb65 thru 0x1f60e7 (0xa583 bytes) 000092[1] 1788263606.569606 mapwrite 0x12b9ad thru 0x130581 (0x4bd5 bytes) 000093[0] 1788263606.750599 mapwrite 0x1859fd thru 0x194849 (0xee4d bytes) 000095[0] 1788263606.950103 read 0x64c7b4 thru 0x654850 (0x809d bytes) 000096[0] 1788263606.970194 read 0x36eeb5 thru 0x379644 (0xa790 bytes) 000097[0] 1788263607.006985 mapread 0x403668 thru 0x40f032 (0xb9cb bytes) 000098[1] 1788263607.063369 mapwrite 0x6ee761 thru 0x6fc887 (0xe127 bytes) 000100[1] 1788263607.202124 mapread 0x6f68b7 thru 0x706259 (0xf9a3 bytes) truncating to largest ever: 0x98f2da 000101[0] 1788263607.239151 trunc from 0x82f529 to 0x98f2d9 (0x15fdb1 bytes) 000102[1] 1788263607.301565 mapread 0x5d6607 thru 0x5df293 (0x8c8d bytes) 000106[0] 1788263607.374943 trunc from 0x98f2da to 0x181a11 (0x80d8c8 bytes) 000107[0] 1788263607.407755 punch from 0x15de28 to 0x16bb5a (0xdd33 bytes) 000108[1] 1788263607.437314 mapread 0x16bc83 thru 0x16f788 (0x3b06 bytes) 000110[1] 1788263607.532922 read 0x02b3a9 thru 0x03a2eb (0xef43 bytes) 000111[0] 1788263607.562295 read 0x0ea65c thru 0x0f56ac (0xb051 bytes) 000112[1] 1788263607.600202 write 0x82d659 thru 0x83a8f6 (0xd29e bytes) 000113[0] 1788263607.652423 punch from 0x17172b to 0x1741fc (0x2ad2 bytes) 000114[0] 1788263607.683410 trunc from 0x83a8f7 to 0x729ceb (0x110c0b bytes) 000115[1] 1788263607.728367 write 0x200602 thru 0x2067b0 (0x61af bytes) 000116[1] 1788263607.763862 read 0x12168b thru 0x122012 (0x988 bytes) 000117[0] 1788263607.785563 punch from 0x0be91f to 0x0c99ae (0xb090 bytes) 000119[1] 1788263607.837337 write 0x642350 thru 0x64d4f9 (0xb1aa bytes) 000120[0] 1788263607.873814 mapwrite 0x015dfb thru 0x0248cd (0xead3 bytes) 000121[1] 1788263608.169802 write 0x7238be thru 0x72dfe4 (0xa727 bytes) 000123[1] 1788263608.236859 write 0x742e0f thru 0x751a21 (0xec13 bytes) 000125[1] 1788263608.293788 read 0x4371f1 thru 0x43e4a3 (0x72b3 bytes) 000126[1] 1788263608.340407 read 0x58dfd1 thru 0x59c696 (0xe6c6 bytes) 000128[0] 1788263608.404807 mapread 0x15dbbb thru 0x16b39d (0xd7e3 bytes) 000129[1] 1788263608.520724 mapread 0x59260f thru 0x59fc0e (0xd600 bytes) 000130[1] 1788263608.574310 mapread 0x2c5b5f thru 0x2c93bb (0x385d bytes) 000132[1] 1788263608.662105 read 0x57aff1 thru 0x58aa36 (0xfa46 bytes) 000133[1] 1788263608.682117 trunc from 0x751a22 to 0x9819bb (0x22ff9a bytes) 000134[1] 1788263608.752332 write 0x192669 thru 0x1a0805 (0xe19d bytes) 000135[1] 1788263608.802398 mapwrite 0x13790f thru 0x142e87 (0xb579 bytes) 000136[0] 1788263608.933918 write 0x1d5720 thru 0x1dad75 (0x5656 bytes) 000138[1] 1788263609.029866 mapwrite 0x6fd34c thru 0x707d5b (0xaa10 bytes) 000139[0] 1788263609.166841 trunc from 0x9819bc to 0x42bd15 (0x555ca6 bytes) 000140[0] 1788263609.190285 mapread 0x25e27e thru 0x26b598 (0xd31b bytes) 000141[1] 1788263609.258915 write 0x6cb513 thru 0x6d932c (0xde1a bytes) 000142[0] 1788263609.317114 punch from 0x178f96 to 0x1790f9 (0x164 bytes) 000143[1] 1788263609.356920 read 0x136977 thru 0x1436e2 (0xcd6c bytes) 000145[1] 1788263609.414282 mapread 0x5e51d5 thru 0x5ebb89 (0x69b5 bytes) 000148[0] 1788263609.447994 mapwrite 0x60f6d9 thru 0x611fac (0x28d4 bytes) 000149[0] 1788263609.599166 mapwrite 0x3cd86d thru 0x3dc860 (0xeff4 bytes) 000152[1] 1788263609.857914 punch from 0x624079 to 0x62e493 (0xa41b bytes) 000153[0] 1788263609.905336 trunc from 0x6d932d to 0x1f4135 (0x4e51f7 bytes) 000154[1] 1788263609.983394 read 0x11c519 thru 0x12295d (0x6445 bytes) 000155[1] 1788263610.050341 mapwrite 0x78acad thru 0x79491e (0x9c72 bytes) 000156[0] 1788263610.350367 write 0x1776d4 thru 0x185492 (0xddbf bytes) 000157[0] 1788263610.454033 mapwrite 0x0332dc thru 0x042318 (0xf03d bytes) 000158[0] 1788263610.776906 mapwrite 0x5ad031 thru 0x5b5662 (0x8632 bytes) 000159[1] 1788263611.000759 punch from 0x5bf5f4 to 0x5c6815 (0x7222 bytes) 000160[0] 1788263611.037908 trunc from 0x79491f to 0x9453ff (0x1b0ae1 bytes) 000163[1] 1788263611.173892 read 0x4015fd thru 0x4075a2 (0x5fa6 bytes) 000164[0] 1788263611.224423 mapread 0x6766e0 thru 0x6779f5 (0x1316 bytes) 000166[0] 1788263611.305405 trunc from 0x945400 to 0x0c2d39 (0x8826c6 bytes) 000167[1] 1788263611.404859 trunc from 0x0c2d3a to 0x7c4f05 (0x7021cc bytes) 000169[1] 1788263611.439889 trunc from 0x7c4f06 to 0x84a49f (0x8559a bytes) 000173[0] 1788263611.491356 mapread 0x4b2eb6 thru 0x4b89d2 (0x5b1d bytes) 000174[1] 1788263611.561247 mapread 0x5a8ad5 thru 0x5b3bf1 (0xb11d bytes) 000176[1] 1788263611.612173 write 0x2de224 thru 0x2de5e7 (0x3c4 bytes) 000177[0] 1788263611.619120 punch from 0x040435 to 0x04bcbe (0xb88a bytes) 000178[1] 1788263611.645289 write 0x29c926 thru 0x2a1d24 (0x53ff bytes) 000180[1] 1788263611.692891 mapread 0x54fa6d thru 0x55af42 (0xb4d6 bytes) 000181[1] 1788263611.711937 punch from 0x12b0a9 to 0x132500 (0x7458 bytes) 000183[1] 1788263611.745152 mapread 0x320f66 thru 0x329016 (0x80b1 bytes) 000184[1] 1788263611.785375 write 0x11c495 thru 0x127a55 (0xb5c1 bytes) 000186[1] 1788263611.821755 punch from 0x65c2a2 to 0x660564 (0x42c3 bytes) 000188[0] 1788263611.863120 punch from 0x0fdf9e to 0x0fff48 (0x1fab bytes) 000189[1] 1788263611.892395 write 0x68048f thru 0x6812f4 (0xe66 bytes) 000190[1] 1788263611.916801 trunc from 0x84a4a0 to 0x592b2f (0x2b7970 bytes) 000191[1] 1788263611.973440 punch from 0x4cbbe6 to 0x4d0361 (0x477c bytes) 000193[1] 1788263612.011778 mapwrite 0x74c4ec thru 0x752222 (0x5d37 bytes) 000196[0] 1788263612.171921 trunc from 0x752223 to 0x75ac62 (0x8a40 bytes) 000197[1] 1788263612.239851 mapwrite 0x27eb2b thru 0x286311 (0x77e7 bytes) 000199[0] 1788263612.356749 write 0x1920a8 thru 0x198df4 (0x6d4d bytes) 000200[1] 1788263612.405432 write 0x72777c thru 0x72c8d2 (0x5157 bytes) 000201[1] 1788263612.438394 mapwrite 0x6ff491 thru 0x70dabe (0xe62e bytes) 000202[0] 1788263612.517878 mapwrite 0x0271c4 thru 0x02cc6b (0x5aa8 bytes) 000205[1] 1788263612.669337 read 0x3f6f68 thru 0x3fccb7 (0x5d50 bytes) 000206[1] 1788263612.737557 read 0x493c90 thru 0x4a127d (0xd5ee bytes) 000207[0] 1788263612.763689 read 0x12f80d thru 0x13219e (0x2992 bytes) 000209[0] 1788263612.789940 mapwrite 0x80b61e thru 0x80d228 (0x1c0b bytes) 000210[1] 1788263612.911725 write 0x8526a7 thru 0x85e1da (0xbb34 bytes) 000211[0] 1788263612.943452 write 0x7ef8dd thru 0x7ff56f (0xfc93 bytes) 000212[0] 1788263612.973921 punch from 0x59c8b6 to 0x5a1d91 (0x54dc bytes) 000213[1] 1788263613.004621 trunc from 0x85e1db to 0x87f939 (0x2175f bytes) 000217[1] 1788263613.119292 punch from 0x25b4b6 to 0x260066 (0x4bb1 bytes) 000218[0] 1788263613.124360 mapwrite 0x7f213f thru 0x7fa2f8 (0x81ba bytes) 000219[1] 1788263613.233282 punch from 0x3c9102 to 0x3d3ce6 (0xabe5 bytes) 000220[1] 1788263613.250764 read 0x5bcde0 thru 0x5be479 (0x169a bytes) 000221[0] 1788263613.288259 mapread 0x5bc7f8 thru 0x5c736f (0xab78 bytes) 000222[1] 1788263613.309678 punch from 0x137764 to 0x13ee8c (0x7729 bytes) 000223[0] 1788263613.324681 trunc from 0x87f93a to 0x3dc68a (0x4a32af bytes) 000224[1] 1788263613.403255 write 0x1f52bd thru 0x1f6aa3 (0x17e7 bytes) 000226[0] 1788263613.454338 read 0x176df1 thru 0x181dff (0xb00f bytes) 000230[1] 1788263613.599275 punch from 0x14669a to 0x14b5da (0x4f41 bytes) 000232[1] 1788263613.647714 punch from 0x242a84 to 0x24f5c7 (0xcb44 bytes) 000233[1] 1788263613.662332 mapread 0x1daee4 thru 0x1e1743 (0x6860 bytes) 000236[1] 1788263613.720870 trunc from 0x3dc68b to 0x48ec7a (0xb25f0 bytes) 000237[1] 1788263613.769508 punch from 0x1004f4 to 0x1081b9 (0x7cc6 bytes) 000239[0] 1788263613.798335 mapwrite 0x6cdccc thru 0x6da612 (0xc947 bytes) 000243[0] 1788263614.006518 read 0x60c7ff thru 0x615d82 (0x9584 bytes) 000244[0] 1788263614.007225 read 0x149da0 thru 0x14bf5f (0x21c0 bytes) 000245[1] 1788263614.012818 trunc from 0x6da613 to 0x7273cd (0x4cdbb bytes) 000246[0] 1788263614.090242 write 0x273dab thru 0x27eb48 (0xad9e bytes) 000248[0] 1788263614.150184 write 0x6e0949 thru 0x6e2698 (0x1d50 bytes) 000249[1] 1788263614.212953 write 0x5d9e5b thru 0x5dc86b (0x2a11 bytes) 000250[0] 1788263614.272663 read 0x4e21e9 thru 0x4e560e (0x3426 bytes) 000253[1] 1788263614.334424 mapread 0x53c2a0 thru 0x54b5b1 (0xf312 bytes) 000257[1] 1788263614.409639 mapread 0x17ee42 thru 0x18345f (0x461e bytes) 000258[1] 1788263614.457647 mapread 0x719124 thru 0x725bf1 (0xcace bytes) 000263[1] 1788263614.480209 mapread 0x57dff5 thru 0x5819ac (0x39b8 bytes) 000264[0] 1788263614.501103 mapwrite 0x670990 thru 0x672f38 (0x25a9 bytes) 000266[0] 1788263614.615503 mapwrite 0x920268 thru 0x925d81 (0x5b1a bytes) 000269[0] 1788263614.818166 trunc from 0x925d82 to 0x10faf8 (0x816289 bytes) 000273[0] 1788263614.872660 mapwrite 0x97974c thru 0x982d49 (0x95fe bytes) 000274[1] 1788263614.979906 mapwrite 0x10e1a7 thru 0x113fac (0x5e06 bytes) 000275[0] 1788263615.088989 mapwrite 0x8bd2e3 thru 0x8ccc74 (0xf992 bytes) 000279[1] 1788263615.296255 mapwrite 0x16b1bc thru 0x1766ef (0xb534 bytes) 000280[0] 1788263615.394680 punch from 0x2851bd to 0x293ca6 (0xeaea bytes) 000282[0] 1788263615.411331 read 0x7e0d3d thru 0x7e4bcd (0x3e91 bytes) 000283[0] 1788263615.448001 read 0x272ee9 thru 0x277bba (0x4cd2 bytes) 000284[1] 1788263615.483885 read 0x4e935a thru 0x4eae6a (0x1b11 bytes) 000285[1] 1788263615.517699 punch from 0x609a67 to 0x6145ea (0xab84 bytes) 000286[1] 1788263615.547626 read 0x4588a8 thru 0x46602f (0xd788 bytes) 000287[0] 1788263615.568298 mapread 0x46a2c1 thru 0x4763e2 (0xc122 bytes) 000288[1] 1788263615.646411 mapread 0x5d4e37 thru 0x5d69c1 (0x1b8b bytes) 000291[0] 1788263615.722342 punch from 0x22b18f to 0x2329e5 (0x7857 bytes) 000292[1] 1788263615.725345 trunc from 0x982d4a to 0x196b0b (0x7ec23e bytes) 000295[1] 1788263615.829298 trunc from 0x196b0c to 0x1e1e10 (0x4b305 bytes) 000297[0] 1788263615.870800 read 0x01500c thru 0x01c0cd (0x70c2 bytes) 000298[1] 1788263615.935505 mapwrite 0x16a3c7 thru 0x177e1a (0xda54 bytes) 000299[0] 1788263616.048054 mapwrite 0x7043fb thru 0x711a2d (0xd633 bytes) 000302[0] 1788263616.325747 read 0x5f69a5 thru 0x603803 (0xce5f bytes) 000305[1] 1788263616.420939 mapwrite 0x0d48a8 thru 0x0e065c (0xbdb5 bytes) 000306[0] 1788263616.745016 mapread 0x0803c3 thru 0x08a6ac (0xa2ea bytes) 000308[0] 1788263616.810819 trunc from 0x711a2e to 0x4c3a5c (0x24dfd1 bytes) 000309[0] 1788263616.903899 trunc from 0x4c3a5d to 0x796516 (0x2d2aba bytes) 000311[0] 1788263616.935989 mapread 0x223e82 thru 0x22fd1d (0xbe9c bytes) 000312[0] 1788263616.997434 punch from 0x291ab4 to 0x2984bd (0x6a0a bytes) 000313[1] 1788263617.030788 mapwrite 0x2cf579 thru 0x2dbba2 (0xc62a bytes) 000314[1] 1788263617.179815 trunc from 0x796517 to 0x266ede (0x52f638 bytes) 000315[0] 1788263617.272918 mapread 0x1e06e5 thru 0x1e8a1f (0x833b bytes) 000316[1] 1788263617.324651 read 0x24ac2c thru 0x24c996 (0x1d6b bytes) 000319[1] 1788263617.349867 read 0x160c1b thru 0x16c22f (0xb615 bytes) 000323[0] 1788263617.410934 read 0x1e4011 thru 0x1f0ea7 (0xce97 bytes) 000330[0] 1788263617.479908 mapwrite 0x4c03d9 thru 0x4c9001 (0x8c29 bytes) 000331[0] 1788263617.704898 punch from 0x3f8482 to 0x401fb9 (0x9b38 bytes) 000334[1] 1788263617.778761 trunc from 0x4c9002 to 0x49a8c7 (0x2e73a bytes) 000336[0] 1788263617.832017 trunc from 0x49a8c8 to 0x6eac75 (0x2503ae bytes) 000340[0] 1788263617.970953 mapread 0x41adab thru 0x41d1eb (0x2441 bytes) 000343[0] 1788263618.082220 write 0x111d55 thru 0x11ea88 (0xcd34 bytes) 000347[0] 1788263618.120250 read 0x153965 thru 0x16335c (0xf9f8 bytes) 000348[0] 1788263618.153518 mapwrite 0x1b9454 thru 0x1b9534 (0xe1 bytes) 000349[1] 1788263618.254883 write 0x90fbe7 thru 0x91a886 (0xaca0 bytes) 000350[1] 1788263618.331368 trunc from 0x91a887 to 0x0d96c0 (0x8411c6 bytes) 000351[1] 1788263618.483128 punch from 0x06a5ea to 0x06f789 (0x51a0 bytes) 000352[0] 1788263618.530417 punch from 0x01929b to 0x01e215 (0x4f7b bytes) 000353[0] 1788263618.570192 mapwrite 0x735ffc thru 0x7369e5 (0x9ea bytes) 000354[0] 1788263618.661180 mapwrite 0x333de7 thru 0x343d65 (0xff7f bytes) 000355[0] 1788263618.743309 trunc from 0x7369e6 to 0x0ed88c (0x649159 bytes) 000356[0] 1788263618.762110 write 0x8ea247 thru 0x8eba8e (0x1848 bytes) 000357[0] 1788263618.803776 trunc from 0x8eba8f to 0x8f3598 (0x7b0a bytes) 000360[1] 1788263618.889520 punch from 0x5f9091 to 0x5fa116 (0x1086 bytes) 000361[1] 1788263618.925550 mapread 0x4335f5 thru 0x43b66c (0x8078 bytes) 000368[1] 1788263619.089208 write 0x9bf946 thru 0x9cea46 (0xf101 bytes) 000369[1] 1788263619.106111 mapread 0x40b215 thru 0x40fcbd (0x4aa9 bytes) 000370[0] 1788263619.173413 punch from 0x65caa9 to 0x665224 (0x877c bytes) 000371[0] 1788263619.212635 read 0x787a66 thru 0x78b70f (0x3caa bytes) 000373[1] 1788263619.250444 punch from 0x812d10 to 0x818a75 (0x5d66 bytes) 000375[1] 1788263619.284806 mapread 0x02b546 thru 0x037e53 (0xc90e bytes) 000377[1] 1788263619.373231 write 0x483ac6 thru 0x48db7e (0xa0b9 bytes) 000379[0] 1788263619.394071 mapwrite 0x0db774 thru 0x0dbb07 (0x394 bytes) 000380[0] 1788263619.495843 read 0x9159f7 thru 0x91fd0c (0xa316 bytes) 000381[1] 1788263619.537691 write 0x9a7414 thru 0x9b497a (0xd567 bytes) 000383[0] 1788263619.625162 trunc from 0x9cea47 to 0x5f7815 (0x3d7231 bytes) 000385[1] 1788263619.752789 trunc from 0x5f7816 to 0x4c4181 (0x133694 bytes) 000387[0] 1788263619.795948 punch from 0x482d6d to 0x4833ef (0x683 bytes) 000388[1] 1788263619.820677 write 0x910729 thru 0x91850e (0x7de6 bytes) 000390[0] 1788263619.849152 write 0x92aa1a thru 0x935f20 (0xb507 bytes) 000391[0] 1788263619.887904 read 0x57686d thru 0x57e02c (0x77c0 bytes) 000392[1] 1788263619.939275 mapread 0x4253ff thru 0x434857 (0xf459 bytes) 000395[1] 1788263620.057709 mapread 0x49cae9 thru 0x4a7765 (0xac7d bytes) 000397[0] 1788263620.136252 trunc from 0x935f21 to 0x7a029d (0x195c83 bytes) 000398[1] 1788263620.201158 write 0x978b37 thru 0x97f37c (0x6846 bytes) 000399[0] 1788263620.237855 read 0x5dbd0f thru 0x5df06c (0x335e bytes) 000401[1] 1788263620.309582 trunc from 0x97f37d to 0x83f5c5 (0x13fdb7 bytes) 000402[1] 1788263620.352363 read 0x0270d7 thru 0x033a5d (0xc987 bytes) 000404[0] 1788263620.421818 write 0x02c0b2 thru 0x035795 (0x96e4 bytes) 000405[0] 1788263620.460220 write 0x7f9def thru 0x7fdfe3 (0x41f5 bytes) 000406[1] 1788263620.509527 read 0x43120f thru 0x434196 (0x2f88 bytes) 000411[0] 1788263620.664860 mapwrite 0x7efee9 thru 0x7f1bd9 (0x1cf1 bytes) 000412[1] 1788263620.792280 mapread 0x12c093 thru 0x1364d8 (0xa446 bytes) 000415[1] 1788263620.889936 trunc from 0x83f5c6 to 0x860c45 (0x21680 bytes) 000416[1] 1788263620.941936 trunc from 0x860c46 to 0x25bba7 (0x60509e bytes) 000417[1] 1788263620.961729 write 0x24adb2 thru 0x25257f (0x77ce bytes) 000418[0] 1788263620.997901 read 0x15f498 thru 0x160301 (0xe6a bytes) 000419[1] 1788263621.086531 write 0x14b410 thru 0x156af6 (0xb6e7 bytes) 000420[0] 1788263621.141187 mapwrite 0x72368b thru 0x725ec7 (0x283d bytes) 000422[1] 1788263621.353439 read 0x3bc940 thru 0x3bf21a (0x28db bytes) 000426[0] 1788263621.417045 punch from 0x05cf5f to 0x069818 (0xc8ba bytes) 000427[1] 1788263621.437033 mapwrite 0x7593b5 thru 0x75fa88 (0x66d4 bytes) 000428[1] 1788263621.639768 punch from 0x198132 to 0x19f397 (0x7266 bytes) 000430[0] 1788263621.668089 mapread 0x4d4916 thru 0x4d80fd (0x37e8 bytes) 000432[1] 1788263621.753339 mapwrite 0x009a16 thru 0x00b924 (0x1f0f bytes) 000433[1] 1788263621.854773 trunc from 0x75fa89 to 0x414fa1 (0x34aae7 bytes) 000435[1] 1788263621.952825 mapread 0x1b6912 thru 0x1c5bbd (0xf2ac bytes) 000436[1] 1788263622.005958 mapwrite 0x16af00 thru 0x175f20 (0xb021 bytes) 000438[0] 1788263622.162883 punch from 0x098317 to 0x09848e (0x178 bytes) 000440[0] 1788263622.252143 read 0x049808 thru 0x04de2a (0x4623 bytes) 000443[1] 1788263622.347817 mapread 0x408a9f thru 0x40b70c (0x2c6e bytes) 000444[0] 1788263622.439425 mapwrite 0x0dfb37 thru 0x0e7004 (0x74ce bytes) 000445[1] 1788263622.578953 trunc from 0x414fa2 to 0x819e53 (0x404eb2 bytes) 000446[0] 1788263622.653834 punch from 0x4de994 to 0x4ec518 (0xdb85 bytes) 000447[1] 1788263622.691424 punch from 0x067240 to 0x0763be (0xf17f bytes) 000448[0] 1788263622.725273 read 0x48da80 thru 0x4991d8 (0xb759 bytes) 000449[1] 1788263622.757346 read 0x74de7d thru 0x75d631 (0xf7b5 bytes) 000451[1] 1788263622.803435 mapread 0x747106 thru 0x74cf8b (0x5e86 bytes) 000452[1] 1788263622.837653 mapread 0x2a3698 thru 0x2a88bc (0x5225 bytes) 000453[1] 1788263622.860256 write 0x6930b5 thru 0x69f0c4 (0xc010 bytes) 000458[0] 1788263623.047799 mapread 0x67b494 thru 0x685deb (0xa958 bytes) 000460[0] 1788263623.169235 read 0x00decc thru 0x01947a (0xb5af bytes) 000462[1] 1788263623.217652 mapwrite 0x3ea2fe thru 0x3ed795 (0x3498 bytes) 000463[1] 1788263623.384579 punch from 0x56f83a to 0x572167 (0x292e bytes) 000464[1] 1788263623.414598 trunc from 0x819e54 to 0x0c0d86 (0x7590cd bytes) 000466[0] 1788263623.490277 write 0x41ae8f thru 0x41f22e (0x43a0 bytes) 000468[1] 1788263623.553869 write 0x2cfc12 thru 0x2d0961 (0xd50 bytes) 000469[1] 1788263623.581253 mapread 0x2cc5ec thru 0x2dac2b (0xe640 bytes) 000470[0] 1788263623.745352 write 0x1298b9 thru 0x12f1a2 (0x58ea bytes) 000471[0] 1788263623.785869 write 0x44ab98 thru 0x44db64 (0x2fcd bytes) 000473[0] 1788263623.788613 mapread 0x063cce thru 0x06c19d (0x84d0 bytes) 000477[0] 1788263623.836196 trunc from 0x44db65 to 0x77ad79 (0x32d215 bytes) 000478[0] 1788263623.862556 mapread 0x13229f thru 0x13469d (0x23ff bytes) 000480[1] 1788263623.873200 trunc from 0x77ad7a to 0x412d30 (0x368049 bytes) 000481[1] 1788263623.973391 read 0x40307b thru 0x411d00 (0xec86 bytes) 000482[0] 1788263624.017689 write 0x8ec3a1 thru 0x8f0bee (0x484e bytes) 000487[0] 1788263624.151682 mapread 0x8d908b thru 0x8d961c (0x592 bytes) 000489[0] 1788263624.187713 mapwrite 0x56c31c thru 0x572702 (0x63e7 bytes) 000490[1] 1788263624.276276 punch from 0x55c46e to 0x569b7b (0xd70e bytes) 000492[1] 1788263624.318582 mapwrite 0x298156 thru 0x29eb91 (0x6a3c bytes) 000493[1] 1788263624.422628 mapread 0x479c9e thru 0x486d36 (0xd099 bytes) 000494[0] 1788263624.495252 mapread 0x21ad93 thru 0x21ca8e (0x1cfc bytes) 000500[0] 1788263624.624586 trunc from 0x8f0bef to 0x5b440a (0x33c7e4 bytes) All operations completed A-OK! Chance of close/open is 1 in 50 Seed set to 3624 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero length punch hole skipping zero size read skipping zero size read skipping zero size read 000008[1] 1788263625.598445 mapwrite 0x2eb000 thru 0x2f117e (0x617f bytes) 000009[1] 1788263625.721929 mapwrite 0x13c000 thru 0x145b69 (0x9b6a bytes) 000010[1] 1788263625.775508 mapread 0x26f000 thru 0x277623 (0x8624 bytes) 000011[0] 1788263625.793699 write 0x613000 thru 0x622e15 (0xfe16 bytes) 000014[0] 1788263625.824165 mapread 0x23c000 thru 0x24b2d1 (0xf2d2 bytes) 000015[0] 1788263625.868101 mapread 0x48f000 thru 0x48f511 (0x512 bytes) truncating to largest ever: 0x45c4e9 000016[0] 1788263625.874884 trunc from 0x622e16 to 0x45c4e8 (0x1c692d bytes) 000017[1] 1788263625.914655 mapwrite 0x068000 thru 0x07617a (0xe17b bytes) 000018[1] 1788263626.015267 mapwrite 0x51a000 thru 0x51d3af (0x33b0 bytes) 000021[1] 1788263626.096835 punch from 0x28286f to 0x28b17a (0x890c bytes) 000023[0] 1788263626.108257 mapread 0x097000 thru 0x098388 (0x1389 bytes) 000024[0] 1788263626.182713 read 0x421000 thru 0x42f53d (0xe53e bytes) 000025[1] 1788263626.223447 mapwrite 0x770000 thru 0x7742e7 (0x42e8 bytes) 000029[0] 1788263626.488691 mapread 0x5a7000 thru 0x5b3389 (0xc38a bytes) 000030[0] 1788263626.559316 write 0x448000 thru 0x4569e4 (0xe9e5 bytes) 000031[0] 1788263626.600291 mapwrite 0x13d000 thru 0x14040d (0x340e bytes) 000032[1] 1788263626.722255 read 0x3ee000 thru 0x3fcc07 (0xec08 bytes) truncating to largest ever: 0x7cda20 000033[1] 1788263626.787802 trunc from 0x7742e8 to 0x7cda1f (0x59738 bytes) 000035[1] 1788263626.837634 write 0x5a3000 thru 0x5ad9da (0xa9db bytes) 000037[1] 1788263626.868707 punch from 0x1bc349 to 0x1c78c6 (0xb57e bytes) 000038[0] 1788263626.875275 punch from 0x3a0223 to 0x3a8b14 (0x88f2 bytes) 000039[0] 1788263626.926080 mapread 0x498000 thru 0x49a5c6 (0x25c7 bytes) 000040[0] 1788263626.970863 read 0x74a000 thru 0x755373 (0xb374 bytes) 000045[0] 1788263627.010798 write 0x9e4000 thru 0x9e8ee1 (0x4ee2 bytes) 000048[0] 1788263627.034255 mapwrite 0x6e3000 thru 0x6e3d07 (0xd08 bytes) 000050[0] 1788263627.144783 mapread 0x38e000 thru 0x39051c (0x251d bytes) 000051[0] 1788263627.168070 read 0x492000 thru 0x49a691 (0x8692 bytes) 000052[0] 1788263627.207534 write 0x583000 thru 0x583413 (0x414 bytes) 000053[0] 1788263627.241449 write 0x65b000 thru 0x6692e7 (0xe2e8 bytes) 000056[0] 1788263627.294258 mapwrite 0x948000 thru 0x952592 (0xa593 bytes) 000058[0] 1788263627.484602 mapread 0x31a000 thru 0x3293dd (0xf3de bytes) 000059[0] 1788263627.608688 punch from 0x0d9be0 to 0x0dc4bc (0x28dd bytes) 000061[0] 1788263627.675562 read 0x507000 thru 0x50b487 (0x4488 bytes) 000063[0] 1788263627.722067 mapwrite 0x48a000 thru 0x495d59 (0xbd5a bytes) 000064[1] 1788263627.905561 mapwrite 0x849000 thru 0x850080 (0x7081 bytes) 000067[1] 1788263628.194137 write 0x463000 thru 0x468fd9 (0x5fda bytes) 000068[0] 1788263628.205650 mapread 0x600000 thru 0x603494 (0x3495 bytes) 000069[1] 1788263628.249833 mapwrite 0x71b000 thru 0x71d16f (0x2170 bytes) 000071[0] 1788263628.356720 mapwrite 0x55c000 thru 0x5681ca (0xc1cb bytes) 000072[1] 1788263628.481019 mapread 0x90c000 thru 0x910385 (0x4386 bytes) 000075[0] 1788263628.528929 write 0x378000 thru 0x37c9f9 (0x49fa bytes) 000077[0] 1788263628.597791 punch from 0x2718da to 0x277a36 (0x615d bytes) 000081[1] 1788263628.671495 mapread 0x3f8000 thru 0x402397 (0xa398 bytes) 000083[1] 1788263628.719721 mapread 0x26d000 thru 0x271503 (0x4504 bytes) 000084[0] 1788263628.738958 write 0x393000 thru 0x39faf7 (0xcaf8 bytes) 000086[1] 1788263628.781560 punch from 0x23c7ca to 0x23dd7c (0x15b3 bytes) 000087[0] 1788263628.807947 mapread 0x711000 thru 0x713bce (0x2bcf bytes) 000088[1] 1788263628.865742 mapwrite 0x1b5000 thru 0x1bcc14 (0x7c15 bytes) 000089[1] 1788263628.982993 trunc from 0x9e8ee2 to 0x2c2321 (0x726bc0 bytes) 000090[0] 1788263629.057034 mapread 0x05d000 thru 0x06b373 (0xe374 bytes) 000093[0] 1788263629.116202 punch from 0x2675b6 to 0x267dd0 (0x81b bytes) 000094[0] 1788263629.155129 mapread 0x1dc000 thru 0x1e86c6 (0xc6c7 bytes) 000095[0] 1788263629.208766 mapwrite 0x8de000 thru 0x8e4f18 (0x6f19 bytes) 000098[1] 1788263629.390452 write 0x220000 thru 0x22c7f5 (0xc7f6 bytes) 000099[1] 1788263629.406099 trunc from 0x8e4f19 to 0x4c7bce (0x41d34a bytes) 000100[0] 1788263629.437677 mapread 0x422000 thru 0x423eb3 (0x1eb4 bytes) 000101[0] 1788263629.475967 mapwrite 0x0b0000 thru 0x0b24a0 (0x24a1 bytes) 000102[1] 1788263629.553088 mapwrite 0x16a000 thru 0x16eee4 (0x4ee5 bytes) 000103[1] 1788263629.678093 read 0x28f000 thru 0x291109 (0x210a bytes) 000106[1] 1788263629.713578 punch from 0x1b5fc3 to 0x1c1337 (0xb375 bytes) 000107[0] 1788263629.728156 write 0x6e2000 thru 0x6e3856 (0x1857 bytes) 000109[1] 1788263629.744086 trunc from 0x6e3857 to 0x08cc18 (0x656c3e bytes) 000111[1] 1788263629.790945 write 0x9f4000 thru 0x9f5a19 (0x1a1a bytes) 000113[1] 1788263629.813941 mapread 0x4da000 thru 0x4e63aa (0xc3ab bytes) 000114[0] 1788263629.844217 punch from 0x7f4d98 to 0x7fe631 (0x989a bytes) 000116[1] 1788263629.879721 mapread 0x946000 thru 0x947fbc (0x1fbd bytes) 000117[1] 1788263629.909031 mapread 0x273000 thru 0x279a2b (0x6a2c bytes) 000118[0] 1788263629.939801 read 0x141000 thru 0x1433e5 (0x23e6 bytes) 000119[1] 1788263629.945020 mapwrite 0x6ef000 thru 0x6f7e43 (0x8e44 bytes) 000121[1] 1788263630.179080 mapread 0x3b8000 thru 0x3c0f00 (0x8f01 bytes) 000123[1] 1788263630.232362 mapwrite 0x158000 thru 0x158665 (0x666 bytes) 000126[1] 1788263630.326639 trunc from 0x9f5a1a to 0x64d6df (0x3a833a bytes) 000128[0] 1788263630.370312 trunc from 0x64d6e0 to 0x6c7362 (0x79c83 bytes) 000130[0] 1788263630.417576 read 0x497000 thru 0x4a169b (0xa69c bytes) 000131[1] 1788263630.424008 mapwrite 0x307000 thru 0x316fbd (0xffbe bytes) 000132[1] 1788263630.521906 punch from 0x1b29c9 to 0x1bc82b (0x9e63 bytes) 000133[0] 1788263630.544893 mapread 0x47f000 thru 0x486553 (0x7554 bytes) 000134[0] 1788263630.601001 mapwrite 0x7ea000 thru 0x7f9df4 (0xfdf5 bytes) 000143[0] 1788263630.917785 write 0x7f1000 thru 0x7f18b5 (0x8b6 bytes) 000144[0] 1788263630.920568 mapread 0x6f3000 thru 0x6fe390 (0xb391 bytes) 000145[0] 1788263630.962825 read 0x00f000 thru 0x011fdf (0x2fe0 bytes) 000149[1] 1788263631.061191 write 0x515000 thru 0x51c7c3 (0x77c4 bytes) 000150[1] 1788263631.095925 punch from 0x0fdd6b to 0x1012bd (0x3553 bytes) 000151[1] 1788263631.100208 read 0x574000 thru 0x579e3f (0x5e40 bytes) 000153[1] 1788263631.121817 read 0x2c5000 thru 0x2d4fda (0xffdb bytes) 000154[0] 1788263631.142062 trunc from 0x7f9df5 to 0x3bfb5b (0x43a299 bytes) truncating to largest ever: 0x93fde3 000155[1] 1788263631.180994 trunc from 0x3bfb5c to 0x93fde2 (0x580287 bytes) 000158[1] 1788263631.243029 write 0x7b1000 thru 0x7b3e6c (0x2e6d bytes) 000159[0] 1788263631.262954 read 0x6f8000 thru 0x7041b6 (0xc1b7 bytes) 000161[1] 1788263631.300753 trunc from 0x93fde3 to 0x005869 (0x93a579 bytes) 000162[0] 1788263631.333130 mapwrite 0x63d000 thru 0x64cc9a (0xfc9b bytes) 000163[0] 1788263631.424866 punch from 0x1a0182 to 0x1a828f (0x810e bytes) 000164[1] 1788263631.429755 write 0x2ae000 thru 0x2b9935 (0xb936 bytes) 000167[1] 1788263631.476155 punch from 0x2f5f36 to 0x3057ec (0xf8b7 bytes) 000168[0] 1788263631.502559 read 0x220000 thru 0x22d451 (0xd452 bytes) 000169[1] 1788263631.525502 mapread 0x0c8000 thru 0x0cf766 (0x7767 bytes) 000170[1] 1788263631.541641 mapread 0x26c000 thru 0x277b66 (0xbb67 bytes) 000171[0] 1788263631.567687 read 0x4ae000 thru 0x4b45af (0x65b0 bytes) 000172[0] 1788263631.574267 mapwrite 0x341000 thru 0x34b133 (0xa134 bytes) 000174[1] 1788263631.714642 mapread 0x0ae000 thru 0x0b7086 (0x9087 bytes) 000175[0] 1788263631.756587 write 0x872000 thru 0x87f562 (0xd563 bytes) 000176[0] 1788263631.775672 write 0x56b000 thru 0x57878f (0xd790 bytes) 000180[0] 1788263631.792966 punch from 0x4474a6 to 0x451e61 (0xa9bc bytes) 000181[1] 1788263631.808419 write 0x35b000 thru 0x366faf (0xbfb0 bytes) 000185[1] 1788263631.891137 trunc from 0x87f563 to 0x7daa2d (0xa4b35 bytes) 000186[1] 1788263631.916233 punch from 0x5697c1 to 0x56b8b4 (0x20f4 bytes) 000187[1] 1788263631.937971 mapread 0x3a3000 thru 0x3ae997 (0xb998 bytes) 000189[0] 1788263631.988149 read 0x46d000 thru 0x47974d (0xc74e bytes) 000194[1] 1788263632.027137 read 0x774000 thru 0x77aa8c (0x6a8d bytes) 000195[1] 1788263632.035534 write 0x446000 thru 0x4471f8 (0x11f9 bytes) 000197[0] 1788263632.070417 read 0x7a3000 thru 0x7b2223 (0xf224 bytes) 000198[0] 1788263632.104028 write 0x3ac000 thru 0x3b208c (0x608d bytes) 000200[0] 1788263632.141957 read 0x4ee000 thru 0x4f0e70 (0x2e71 bytes) 000203[1] 1788263632.160999 read 0x2cf000 thru 0x2d0071 (0x1072 bytes) 000205[0] 1788263632.187945 mapwrite 0x898000 thru 0x89e956 (0x6957 bytes) 000207[0] 1788263632.310601 punch from 0x7b69ee to 0x7c5846 (0xee59 bytes) 000208[0] 1788263632.314875 punch from 0x565520 to 0x567487 (0x1f68 bytes) 000209[0] 1788263632.320370 punch from 0x8140f8 to 0x8212ef (0xd1f8 bytes) 000211[0] 1788263632.334746 mapwrite 0x6fc000 thru 0x707444 (0xb445 bytes) 000214[1] 1788263632.456605 mapread 0x1a7000 thru 0x1b378b (0xc78c bytes) 000216[0] 1788263632.530029 punch from 0x4599ce to 0x45bc30 (0x2263 bytes) 000217[1] 1788263632.580537 punch from 0x4aadb5 to 0x4b1969 (0x6bb5 bytes) 000218[0] 1788263632.631076 punch from 0x2070f1 to 0x2125fa (0xb50a bytes) 000219[1] 1788263632.661720 trunc from 0x89e957 to 0x374806 (0x52a150 bytes) 000221[1] 1788263632.715130 read 0x32e000 thru 0x33bec7 (0xdec8 bytes) 000222[0] 1788263632.771435 punch from 0x1b2d9d to 0x1bed67 (0xbfcb bytes) 000225[1] 1788263632.844662 write 0x0b7000 thru 0x0c046d (0x946e bytes) 000226[1] 1788263632.908581 punch from 0x054c77 to 0x055517 (0x8a1 bytes) 000228[1] 1788263632.938171 write 0x939000 thru 0x93ebb6 (0x5bb7 bytes) truncating to largest ever: 0x960d78 000229[1] 1788263632.942814 trunc from 0x93ebb7 to 0x960d77 (0x221c1 bytes) 000230[1] 1788263632.978885 trunc from 0x960d78 to 0x060a3b (0x90033c bytes) 000231[0] 1788263633.004058 punch from 0x044097 to 0x048b99 (0x4b03 bytes) 000232[0] 1788263633.026313 punch from 0x0234f7 to 0x02c9da (0x94e4 bytes) 000235[1] 1788263633.049067 trunc from 0x060a3c to 0x4d6c2c (0x4761f1 bytes) 000236[0] 1788263633.087613 punch from 0x40daba to 0x41885e (0xada5 bytes) 000237[0] 1788263633.119377 write 0x658000 thru 0x65a24b (0x224c bytes) 000238[0] 1788263633.138798 punch from 0x5b2467 to 0x5bcab6 (0xa650 bytes) 000240[0] 1788263633.180678 mapwrite 0x90b000 thru 0x918e04 (0xde05 bytes) 000241[0] 1788263633.392520 punch from 0x84b701 to 0x84fe56 (0x4756 bytes) 000243[1] 1788263633.403109 read 0x306000 thru 0x307b37 (0x1b38 bytes) 000245[1] 1788263633.426071 punch from 0x006c24 to 0x00d184 (0x6561 bytes) 000246[1] 1788263633.471161 punch from 0x3b5e22 to 0x3bc4bd (0x669c bytes) 000247[1] 1788263633.504847 punch from 0x27324e to 0x2826ac (0xf45f bytes) 000249[1] 1788263633.533006 read 0x8be000 thru 0x8c45cf (0x65d0 bytes) 000252[1] 1788263633.554032 write 0x254000 thru 0x25d020 (0x9021 bytes) 000254[1] 1788263633.619513 mapwrite 0x64d000 thru 0x651a90 (0x4a91 bytes) 000255[0] 1788263633.800819 punch from 0x52d120 to 0x535524 (0x8405 bytes) 000256[0] 1788263633.847577 read 0x2c0000 thru 0x2ce208 (0xe209 bytes) 000258[1] 1788263633.893820 mapread 0x725000 thru 0x7349cf (0xf9d0 bytes) 000259[1] 1788263634.026325 trunc from 0x918e05 to 0x3eb612 (0x52d7f2 bytes) 000260[1] 1788263634.086594 read 0x2b3000 thru 0x2b9593 (0x6594 bytes) 000262[1] 1788263634.183446 trunc from 0x3eb613 to 0x4017da (0x161c8 bytes) 000264[1] 1788263634.238112 mapwrite 0x690000 thru 0x69a1c7 (0xa1c8 bytes) 000266[0] 1788263634.385990 mapwrite 0x7cc000 thru 0x7cd46f (0x1470 bytes) 000269[0] 1788263634.525867 trunc from 0x7cd470 to 0x231c48 (0x59b827 bytes) 000271[0] 1788263634.546645 read 0x05c000 thru 0x06b7db (0xf7dc bytes) 000274[1] 1788263634.622594 mapwrite 0x8b5000 thru 0x8c4dfd (0xfdfe bytes) 000276[0] 1788263634.774664 mapwrite 0x6b1000 thru 0x6bd91c (0xc91d bytes) 000280[1] 1788263635.013190 mapwrite 0x5a9000 thru 0x5aee15 (0x5e16 bytes) 000281[1] 1788263635.118138 punch from 0x64dc93 to 0x64dcaf (0x1d bytes) 000284[1] 1788263635.158894 mapread 0x369000 thru 0x375a13 (0xca14 bytes) 000285[0] 1788263635.232058 read 0x6b4000 thru 0x6b971b (0x571c bytes) 000287[1] 1788263635.255190 trunc from 0x8c4dfe to 0x9007c3 (0x3b9c6 bytes) 000288[1] 1788263635.286116 mapwrite 0x84e000 thru 0x857a69 (0x9a6a bytes) 000289[0] 1788263635.368061 mapwrite 0x838000 thru 0x846aa3 (0xeaa4 bytes) 000290[0] 1788263635.520438 trunc from 0x9007c4 to 0x1e5450 (0x71b373 bytes) 000291[0] 1788263635.562401 read 0x1ce000 thru 0x1cffe0 (0x1fe1 bytes) 000294[0] 1788263635.643123 write 0x49d000 thru 0x4a299f (0x59a0 bytes) 000296[0] 1788263635.694450 trunc from 0x4a29a0 to 0x18d1f7 (0x3157a8 bytes) 000298[1] 1788263635.737899 write 0x5c5000 thru 0x5cb161 (0x6162 bytes) 000300[1] 1788263635.755705 write 0x16a000 thru 0x1727bf (0x87c0 bytes) 000305[1] 1788263635.852891 trunc from 0x5cb162 to 0x200b16 (0x3ca64b bytes) 000308[0] 1788263635.948492 punch from 0x0af9a6 to 0x0b4770 (0x4dcb bytes) 000309[1] 1788263636.021637 write 0x114000 thru 0x12288b (0xe88c bytes) 000310[0] 1788263636.037659 mapwrite 0x47e000 thru 0x48950d (0xb50e bytes) 000311[0] 1788263636.228361 punch from 0x1f2739 to 0x1fbfbd (0x9885 bytes) 000312[0] 1788263636.275351 write 0x5e7000 thru 0x5f6a47 (0xfa48 bytes) 000313[1] 1788263636.300203 punch from 0x0814b1 to 0x084a42 (0x3592 bytes) 000315[0] 1788263636.548173 mapread 0x32b000 thru 0x32dc08 (0x2c09 bytes) 000317[1] 1788263636.612996 trunc from 0x5f6a48 to 0x511f73 (0xe4ad4 bytes) 000318[1] 1788263636.674662 punch from 0x225cec to 0x22fc32 (0x9f47 bytes) 000319[1] 1788263636.710151 trunc from 0x511f74 to 0x792dbd (0x280e4a bytes) truncating to largest ever: 0x9945df 000320[0] 1788263636.775154 trunc from 0x792dbe to 0x9945de (0x201821 bytes) 000321[0] 1788263636.900684 write 0x89a000 thru 0x8a68b7 (0xc8b8 bytes) 000322[0] 1788263637.036892 mapwrite 0x978000 thru 0x97cefe (0x4eff bytes) 000326[0] 1788263637.135055 trunc from 0x9945df to 0x02681b (0x96ddc3 bytes) 000328[0] 1788263637.181373 mapwrite 0x5d4000 thru 0x5e3a9e (0xfa9f bytes) 000331[0] 1788263637.322343 mapread 0x313000 thru 0x313c87 (0xc88 bytes) 000332[1] 1788263637.333434 trunc from 0x5e3a9f to 0x8acd9a (0x2c92fc bytes) 000333[0] 1788263637.375289 mapwrite 0x59d000 thru 0x5a0290 (0x3291 bytes) 000334[1] 1788263637.480424 read 0x354000 thru 0x3637df (0xf7e0 bytes) truncating to largest ever: 0x9bd82d 000335[1] 1788263637.502322 trunc from 0x8acd9b to 0x9bd82c (0x110a92 bytes) 000337[0] 1788263637.576323 trunc from 0x9bd82d to 0x0a97b4 (0x914078 bytes) 000338[1] 1788263637.644821 punch from 0x028f9c to 0x03406a (0xb0cf bytes) 000339[0] 1788263637.660865 trunc from 0x0a97b5 to 0x154000 (0xaa84c bytes) 000342[1] 1788263637.818399 punch from 0x0822fa to 0x08d4fb (0xb202 bytes) 000345[1] 1788263637.918588 punch from 0x03a391 to 0x03a65c (0x2cc bytes) 000346[0] 1788263637.990309 mapwrite 0x130000 thru 0x131fbb (0x1fbc bytes) 000348[1] 1788263638.118594 mapread 0x136000 thru 0x142e74 (0xce75 bytes) 000349[1] 1788263638.181969 read 0x0cd000 thru 0x0cd17d (0x17e bytes) 000353[0] 1788263638.183181 mapwrite 0x9d9000 thru 0x9dc8b2 (0x38b3 bytes) 000355[1] 1788263638.311630 punch from 0x07e513 to 0x0819c9 (0x34b7 bytes) 000356[0] 1788263638.344570 mapwrite 0x867000 thru 0x86d537 (0x6538 bytes) 000357[0] 1788263638.519346 write 0x5cc000 thru 0x5d14b4 (0x54b5 bytes) 000359[1] 1788263638.563368 punch from 0x2f4951 to 0x2f8579 (0x3c29 bytes) 000360[0] 1788263638.608857 trunc from 0x9dc8b3 to 0x4de5fe (0x4fe2b4 bytes) 000361[0] 1788263638.677587 read 0x1ee000 thru 0x1fb5a3 (0xd5a4 bytes) 000362[0] 1788263638.694589 mapwrite 0x5b4000 thru 0x5bc771 (0x8772 bytes) 000363[0] 1788263638.879505 read 0x519000 thru 0x523412 (0xa413 bytes) 000365[0] 1788263638.923959 trunc from 0x5bc772 to 0x22c4f5 (0x39027c bytes) 000367[1] 1788263638.996078 mapwrite 0x02b000 thru 0x033dfd (0x8dfe bytes) 000369[0] 1788263639.124677 trunc from 0x22c4f6 to 0x406aed (0x1da5f8 bytes) 000370[1] 1788263639.165902 mapwrite 0x72f000 thru 0x730983 (0x1984 bytes) 000371[1] 1788263639.479408 read 0x41e000 thru 0x4209d2 (0x29d3 bytes) 000372[1] 1788263639.483326 mapread 0x0d9000 thru 0x0e29b3 (0x99b4 bytes) 000373[0] 1788263639.527279 mapread 0x41f000 thru 0x428d7c (0x9d7d bytes) 000374[0] 1788263639.654217 write 0x474000 thru 0x476d8b (0x2d8c bytes) 000375[1] 1788263639.673258 punch from 0x6c3bcc to 0x6cadc2 (0x71f7 bytes) 000376[1] 1788263639.694335 punch from 0x0a1088 to 0x0ad87c (0xc7f5 bytes) 000377[0] 1788263639.729243 trunc from 0x730984 to 0x872eb9 (0x142536 bytes) 000379[0] 1788263639.838977 read 0x161000 thru 0x1689b2 (0x79b3 bytes) 000380[1] 1788263639.870882 mapwrite 0x2fe000 thru 0x307896 (0x9897 bytes) 000381[1] 1788263640.086735 write 0x279000 thru 0x280892 (0x7893 bytes) 000382[1] 1788263640.091632 punch from 0x2009f2 to 0x204ad9 (0x40e8 bytes) 000383[0] 1788263640.120504 punch from 0x35ff4e to 0x36f6a1 (0xf754 bytes) 000385[0] 1788263640.134828 write 0x25c000 thru 0x25cf6c (0xf6d bytes) 000386[1] 1788263640.138795 mapwrite 0x683000 thru 0x684222 (0x1223 bytes) 000387[0] 1788263640.261521 punch from 0x7669c2 to 0x770080 (0x96bf bytes) 000388[1] 1788263640.276017 read 0x6ed000 thru 0x6f8d13 (0xbd14 bytes) 000390[1] 1788263640.313613 mapread 0x2f8000 thru 0x2f83a8 (0x3a9 bytes) 000391[0] 1788263640.318047 mapread 0x349000 thru 0x3500a9 (0x70aa bytes) 000392[1] 1788263640.336448 mapread 0x34e000 thru 0x3519b6 (0x39b7 bytes) 000394[0] 1788263640.349630 read 0x523000 thru 0x52ebe7 (0xbbe8 bytes) 000396[1] 1788263640.358783 mapread 0x7ee000 thru 0x7f6ded (0x8dee bytes) 000398[0] 1788263640.402274 mapwrite 0x075000 thru 0x083230 (0xe231 bytes) 000399[1] 1788263640.519081 mapwrite 0x33b000 thru 0x344a43 (0x9a44 bytes) 000403[0] 1788263640.667815 read 0x058000 thru 0x067d64 (0xfd65 bytes) 000404[1] 1788263640.697448 mapwrite 0x7aa000 thru 0x7aaf5b (0xf5c bytes) 000405[1] 1788263640.761714 trunc from 0x872eba to 0x8cc7ce (0x59915 bytes) 000408[0] 1788263640.807245 punch from 0x229762 to 0x22ba4d (0x22ec bytes) 000410[1] 1788263640.849894 mapread 0x571000 thru 0x580994 (0xf995 bytes) 000411[1] 1788263640.895572 write 0x2f4000 thru 0x2fee03 (0xae04 bytes) 000412[1] 1788263640.912870 punch from 0x4da056 to 0x4da6f5 (0x6a0 bytes) 000417[1] 1788263640.941153 trunc from 0x8cc7cf to 0x5800fd (0x34c6d1 bytes) 000420[0] 1788263640.980920 mapread 0x3fe000 thru 0x40c856 (0xe857 bytes) 000421[0] 1788263641.040311 mapwrite 0x458000 thru 0x46044c (0x844d bytes) 000422[1] 1788263641.152156 read 0x464000 thru 0x467a97 (0x3a98 bytes) 000423[0] 1788263641.182597 trunc from 0x5800fe to 0x0fe170 (0x481f8d bytes) 000424[0] 1788263641.217974 mapwrite 0x6c2000 thru 0x6ce34b (0xc34c bytes) 000425[1] 1788263641.302361 mapwrite 0x9f1000 thru 0x9f9383 (0x8384 bytes) 000426[0] 1788263641.436117 trunc from 0x9f9384 to 0x771ae6 (0x28789d bytes) 000430[1] 1788263641.552822 mapread 0x179000 thru 0x17c45b (0x345c bytes) 000432[1] 1788263641.609058 punch from 0x1194f8 to 0x1277fe (0xe307 bytes) 000433[1] 1788263641.663301 trunc from 0x771ae7 to 0x25b586 (0x516560 bytes) 000437[1] 1788263641.757871 mapread 0x09e000 thru 0x0a2521 (0x4522 bytes) 000438[1] 1788263641.798754 trunc from 0x25b587 to 0x142f82 (0x118604 bytes) 000439[0] 1788263641.857551 write 0x909000 thru 0x90b10a (0x210b bytes) 000440[0] 1788263641.908377 punch from 0x06656e to 0x067b16 (0x15a9 bytes) 000441[0] 1788263641.914739 trunc from 0x90b10b to 0x23fbfd (0x6cb50d bytes) 000442[0] 1788263642.002769 mapread 0x0b2000 thru 0x0b7029 (0x502a bytes) 000443[0] 1788263642.045767 punch from 0x1ea0f4 to 0x1f026e (0x617b bytes) 000446[0] 1788263642.080556 punch from 0x0d766f to 0x0da2fa (0x2c8c bytes) 000447[0] 1788263642.109169 read 0x0f9000 thru 0x103a96 (0xaa97 bytes) 000448[1] 1788263642.163911 trunc from 0x23fbfe to 0x262c74 (0x23077 bytes) 000449[0] 1788263642.233848 mapwrite 0x294000 thru 0x295e96 (0x1e97 bytes) 000452[1] 1788263642.359308 mapread 0x270000 thru 0x277fc4 (0x7fc5 bytes) 000453[0] 1788263642.462513 mapread 0x101000 thru 0x1089c3 (0x79c4 bytes) 000454[1] 1788263642.535823 write 0x158000 thru 0x165fbf (0xdfc0 bytes) 000456[0] 1788263642.578338 write 0x1f4000 thru 0x1fc37b (0x837c bytes) 000457[1] 1788263642.665945 write 0x907000 thru 0x9083a5 (0x13a6 bytes) 000458[0] 1788263642.725823 mapwrite 0x1ed000 thru 0x1f7ec5 (0xaec6 bytes) 000459[1] 1788263642.913182 trunc from 0x9083a6 to 0x7322b1 (0x1d60f4 bytes) 000460[1] 1788263642.948344 mapread 0x269000 thru 0x277d44 (0xed45 bytes) 000461[0] 1788263642.978275 mapwrite 0x857000 thru 0x85f9f5 (0x89f6 bytes) 000464[0] 1788263643.162277 punch from 0x1a32dc to 0x1a5e18 (0x2b3d bytes) 000467[1] 1788263643.217560 write 0x551000 thru 0x55f267 (0xe268 bytes) 000468[0] 1788263643.308718 read 0x842000 thru 0x850eff (0xef00 bytes) 000472[0] 1788263643.354163 punch from 0x65e640 to 0x661515 (0x2ed6 bytes) 000475[0] 1788263643.395973 write 0x887000 thru 0x896f43 (0xff44 bytes) 000478[0] 1788263643.413305 trunc from 0x896f44 to 0x19642e (0x700b15 bytes) 000480[1] 1788263643.461926 mapread 0x14d000 thru 0x158c1d (0xbc1e bytes) 000481[0] 1788263643.506603 punch from 0x0d5770 to 0x0e1ed9 (0xc76a bytes) 000483[1] 1788263643.533337 mapread 0x0d4000 thru 0x0dedf7 (0xadf8 bytes) 000484[1] 1788263643.583365 read 0x03f000 thru 0x049d29 (0xad2a bytes) 000486[0] 1788263643.607343 write 0x894000 thru 0x898a13 (0x4a14 bytes) 000488[0] 1788263643.637556 write 0x091000 thru 0x099697 (0x8698 bytes) 000489[1] 1788263643.643959 mapread 0x5a7000 thru 0x5abd8d (0x4d8e bytes) 000491[1] 1788263643.800977 trunc from 0x898a14 to 0x034b3b (0x863ed8 bytes) 000493[0] 1788263643.875076 read 0x011000 thru 0x020e96 (0xfe97 bytes) 000494[1] 1788263643.924021 mapread 0x002000 thru 0x003c35 (0x1c36 bytes) 000496[0] 1788263643.938951 trunc from 0x034b3c to 0x52866a (0x4f3b2f bytes) 000498[0] 1788263644.006168 trunc from 0x52866b to 0x19a7cf (0x38de9b bytes) 000499[1] 1788263644.042197 write 0x750000 thru 0x754e97 (0x4e98 bytes) All operations completed A-OK! PASS 16a (53s) == sanityn test 16b: 500 iterations of dual-mount fsx at small size ========================================================== 07:54:09 (1788263649) keep default fallocate mode: 2 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.440623 s, 19.0 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.341957 s, 24.5 MB/s Chance of close/open is 1 in 50 Seed set to 3652 fd 0: /mnt/lustre/f16b.sanityn fd 1: /mnt/lustre2/f16b.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero length punch hole 000007[0] 1788263653.730196 mapwrite 0x85bc6f thru 0x85cd36 (0x10c8 bytes) 000010[1] 1788263653.858972 mapwrite 0x7ba13e thru 0x7bbe73 (0x1d36 bytes) 000012[1] 1788263653.996331 write 0x7b30de thru 0x7b4c64 (0x1b87 bytes) 000018[0] 1788263654.143379 mapwrite 0x8b2388 thru 0x8b2fe4 (0xc5d bytes) 000019[0] 1788263654.279400 punch from 0x703875 to 0x703fb8 (0x744 bytes) truncating to largest ever: 0x7ca83b 000021[0] 1788263654.288533 trunc from 0x8b2fe5 to 0x7ca83a (0xe87aa bytes) 000022[0] 1788263654.334626 read 0x5a5a96 thru 0x5a6be3 (0x114e bytes) 000023[1] 1788263654.346102 mapwrite 0x5e8867 thru 0x5ea45c (0x1bf6 bytes) 000024[1] 1788263654.445301 punch from 0x509927 to 0x50a415 (0xaef bytes) 000025[0] 1788263654.466927 write 0x7a3d02 thru 0x7a509d (0x139c bytes) 000026[1] 1788263654.492193 read 0x3298cf thru 0x32aba8 (0x12da bytes) 000027[0] 1788263654.523127 mapwrite 0x555e8d thru 0x5571b7 (0x132b bytes) 000028[1] 1788263654.648345 punch from 0x601b61 to 0x602c2a (0x10ca bytes) 000029[0] 1788263654.685137 punch from 0x1b3ead to 0x1b4a21 (0xb75 bytes) 000032[0] 1788263654.757148 mapwrite 0x0fa918 thru 0x0fc0f3 (0x17dc bytes) 000033[1] 1788263654.845385 write 0x41442f thru 0x4158f4 (0x14c6 bytes) 000034[1] 1788263654.866443 read 0x4945ca thru 0x496220 (0x1c57 bytes) 000037[0] 1788263654.921376 punch from 0x100be9 to 0x101692 (0xaaa bytes) 000041[1] 1788263654.989496 punch from 0x3f3298 to 0x3f4907 (0x1670 bytes) 000043[1] 1788263655.003632 punch from 0x27c0d5 to 0x27c86b (0x797 bytes) 000044[1] 1788263655.035628 read 0x6de587 thru 0x6e015a (0x1bd4 bytes) 000046[0] 1788263655.042974 read 0x23fc9a thru 0x241456 (0x17bd bytes) 000047[0] 1788263655.070450 write 0x3be3f0 thru 0x3beec4 (0xad5 bytes) 000049[0] 1788263655.120675 read 0x4d7512 thru 0x4d8e99 (0x1988 bytes) 000051[0] 1788263655.139927 mapwrite 0x7235bb thru 0x724589 (0xfcf bytes) 000052[1] 1788263655.198982 read 0x277e1c thru 0x279da5 (0x1f8a bytes) 000053[0] 1788263655.233437 mapwrite 0x75437b thru 0x7555dd (0x1263 bytes) 000055[0] 1788263655.361765 read 0x73b07d thru 0x73c872 (0x17f6 bytes) 000056[1] 1788263655.369328 trunc from 0x7ca83b to 0x1df292 (0x5eb5a8 bytes) 000057[1] 1788263655.454570 mapread 0x127727 thru 0x1282c1 (0xb9b bytes) 000058[0] 1788263655.472735 mapread 0x19a3ae thru 0x19b2ff (0xf52 bytes) 000059[1] 1788263655.516604 trunc from 0x1df293 to 0x02e0b3 (0x1b11df bytes) 000061[0] 1788263655.597662 trunc from 0x02e0b4 to 0x6fe9cf (0x6d091c bytes) 000062[1] 1788263655.645738 mapwrite 0x934c9a thru 0x934d56 (0xbd bytes) 000064[0] 1788263655.794008 punch from 0x7944d2 to 0x7952e6 (0xe15 bytes) truncating to largest ever: 0x8d28c9 000065[0] 1788263655.829806 trunc from 0x934d57 to 0x8d28c8 (0x6248e bytes) 000068[0] 1788263655.884513 write 0x986645 thru 0x98804f (0x1a0b bytes) 000070[0] 1788263655.887440 read 0x2f6ccc thru 0x2f707c (0x3b1 bytes) 000071[1] 1788263655.915252 mapwrite 0x14bb1e thru 0x14bdda (0x2bd bytes) 000074[1] 1788263656.107257 mapwrite 0x4cbda2 thru 0x4cd2e6 (0x1545 bytes) 000075[1] 1788263656.231298 write 0x8494eb thru 0x849cf6 (0x80c bytes) 000078[1] 1788263656.289882 punch from 0x033d30 to 0x034fe0 (0x12b1 bytes) 000079[1] 1788263656.295304 read 0x6e7366 thru 0x6e8803 (0x149e bytes) 000080[0] 1788263656.333863 read 0x557de5 thru 0x5584cc (0x6e8 bytes) 000085[1] 1788263656.401021 write 0x02fb56 thru 0x03111e (0x15c9 bytes) 000086[0] 1788263656.410219 punch from 0x3fd43f to 0x3fdf20 (0xae2 bytes) 000087[1] 1788263656.473441 trunc from 0x988050 to 0x267939 (0x720716 bytes) 000088[0] 1788263656.519575 read 0x024abb thru 0x024dc7 (0x30d bytes) 000092[1] 1788263656.609942 trunc from 0x26793a to 0x002387 (0x2655b2 bytes) 000093[1] 1788263656.641980 mapwrite 0x66c2a4 thru 0x66cc52 (0x9af bytes) 000094[0] 1788263656.748313 read 0x0a031f thru 0x0a0c9b (0x97d bytes) 000095[1] 1788263656.789221 mapread 0x12343e thru 0x124d4f (0x1912 bytes) 000096[0] 1788263656.824101 mapwrite 0x7828d4 thru 0x783b46 (0x1273 bytes) 000097[0] 1788263656.989999 mapread 0x75e242 thru 0x76012e (0x1eed bytes) 000099[1] 1788263657.070077 mapwrite 0x0d63ea thru 0x0d733d (0xf54 bytes) 000103[0] 1788263657.240000 write 0x0f0062 thru 0x0f13bf (0x135e bytes) 000104[1] 1788263657.263887 write 0x889451 thru 0x889614 (0x1c4 bytes) 000106[0] 1788263657.328796 punch from 0x2b5883 to 0x2b5efb (0x679 bytes) 000107[0] 1788263657.354672 mapwrite 0x686242 thru 0x686a24 (0x7e3 bytes) 000109[0] 1788263657.496884 trunc from 0x889615 to 0x20b3d4 (0x67e240 bytes) 000111[1] 1788263657.525858 read 0x141470 thru 0x14283a (0x13cb bytes) 000113[0] 1788263657.625552 mapwrite 0x5e0458 thru 0x5e1805 (0x13ae bytes) 000115[0] 1788263657.762946 write 0x3bcedd thru 0x3bead9 (0x1bfd bytes) 000116[0] 1788263657.775947 mapwrite 0x9fac4d thru 0x9fc677 (0x1a2b bytes) 000117[1] 1788263657.940079 punch from 0x54016f to 0x540c7a (0xb0c bytes) 000119[1] 1788263657.986655 mapwrite 0x64d569 thru 0x64dd20 (0x7b8 bytes) 000120[1] 1788263658.146817 trunc from 0x9fc678 to 0x8b3530 (0x149147 bytes) 000122[0] 1788263658.218184 trunc from 0x8b3531 to 0x636a27 (0x27cb09 bytes) 000123[1] 1788263658.282383 mapwrite 0x5596df thru 0x559c96 (0x5b8 bytes) 000125[0] 1788263658.437127 write 0x08d571 thru 0x08ea46 (0x14d6 bytes) 000126[0] 1788263658.474033 mapwrite 0x492c11 thru 0x493d8f (0x117f bytes) 000127[0] 1788263658.661434 punch from 0x583b2c to 0x583bed (0xc2 bytes) 000128[1] 1788263658.699894 mapwrite 0x4f4baa thru 0x4f50ac (0x503 bytes) 000129[0] 1788263658.798057 mapwrite 0x6576a5 thru 0x658120 (0xa7c bytes) 000131[1] 1788263658.912259 write 0x63a135 thru 0x63b035 (0xf01 bytes) 000132[0] 1788263658.946259 punch from 0x5b6849 to 0x5b763c (0xdf4 bytes) 000133[0] 1788263658.965389 mapwrite 0x9ffa20 thru 0x9fffff (0x5e0 bytes) 000137[1] 1788263659.094504 mapwrite 0x835f18 thru 0x83602b (0x114 bytes) 000138[1] 1788263659.208850 write 0x4460de thru 0x446f7d (0xea0 bytes) 000139[0] 1788263659.259279 trunc from 0xa00000 to 0x3fa27d (0x605d82 bytes) 000140[0] 1788263659.379683 mapwrite 0x53a2ac thru 0x53a99b (0x6f0 bytes) 000141[1] 1788263659.515884 read 0x234d6f thru 0x236813 (0x1aa5 bytes) 000142[1] 1788263659.543904 trunc from 0x53a99c to 0x270636 (0x2ca365 bytes) 000144[0] 1788263659.591945 mapwrite 0x98fee8 thru 0x991713 (0x182c bytes) 000146[1] 1788263659.683952 punch from 0x981eb6 to 0x98351b (0x1666 bytes) 000147[0] 1788263659.711873 read 0x2fca4f thru 0x2fdb36 (0x10e8 bytes) 000149[1] 1788263659.746709 mapwrite 0x44b945 thru 0x44bcc7 (0x383 bytes) 000150[0] 1788263659.813647 mapread 0x5315bf thru 0x532128 (0xb6a bytes) 000151[1] 1788263659.856637 punch from 0x2a1b02 to 0x2a1ddf (0x2de bytes) 000152[1] 1788263659.882186 write 0x16d616 thru 0x16d907 (0x2f2 bytes) 000153[0] 1788263659.887839 trunc from 0x991714 to 0x101c08 (0x88fb0b bytes) 000155[0] 1788263659.981488 mapread 0x0b4e81 thru 0x0b5389 (0x509 bytes) 000156[1] 1788263660.011181 mapwrite 0x371a94 thru 0x372589 (0xaf6 bytes) 000157[0] 1788263660.100799 trunc from 0x37258a to 0x0e1bac (0x2909dd bytes) 000161[0] 1788263660.171141 mapwrite 0x7e0aaf thru 0x7e28e2 (0x1e34 bytes) 000162[1] 1788263660.257845 read 0x4c5133 thru 0x4c64a6 (0x1374 bytes) 000164[1] 1788263660.338767 punch from 0x0f1f19 to 0x0f2fa1 (0x1089 bytes) 000165[0] 1788263660.396704 trunc from 0x7e28e3 to 0x25a4ee (0x5883f4 bytes) 000167[1] 1788263660.477367 write 0x5d2e32 thru 0x5d396c (0xb3b bytes) 000174[0] 1788263660.626993 write 0x4a1ff5 thru 0x4a2a23 (0xa2f bytes) 000175[0] 1788263660.741193 mapread 0x2b6cfc thru 0x2b7e5c (0x1161 bytes) 000177[0] 1788263660.823650 read 0x331388 thru 0x3328ea (0x1563 bytes) 000178[1] 1788263660.853499 write 0x59e6fb thru 0x59e8cc (0x1d2 bytes) 000179[1] 1788263660.900925 write 0x81eac6 thru 0x81ec28 (0x163 bytes) 000180[1] 1788263660.965204 punch from 0x64d799 to 0x64e73d (0xfa5 bytes) 000182[0] 1788263660.977370 write 0x7746b7 thru 0x7760f1 (0x1a3b bytes) 000183[0] 1788263661.059657 punch from 0x4f27fc to 0x4f3707 (0xf0c bytes) 000187[1] 1788263661.148210 write 0x21d0fe thru 0x21e33a (0x123d bytes) 000189[1] 1788263661.206526 read 0x0465fd thru 0x047413 (0xe17 bytes) truncating to largest ever: 0x917da2 000190[1] 1788263661.242750 trunc from 0x81ec29 to 0x917da1 (0xf9179 bytes) 000192[0] 1788263661.300254 mapwrite 0x6af810 thru 0x6afce7 (0x4d8 bytes) 000193[0] 1788263661.416156 trunc from 0x917da2 to 0x708c91 (0x20f110 bytes) 000197[1] 1788263661.495124 read 0x0545a0 thru 0x05471e (0x17f bytes) 000198[0] 1788263661.521875 punch from 0x4b54cf to 0x4b737b (0x1ead bytes) 000200[1] 1788263661.526802 trunc from 0x708c92 to 0x23177e (0x4d7513 bytes) 000201[1] 1788263661.613923 punch from 0x10a70f to 0x10b2c4 (0xbb6 bytes) 000202[1] 1788263661.628871 mapread 0x205954 thru 0x205c14 (0x2c1 bytes) 000204[1] 1788263661.664095 mapwrite 0x6e866c thru 0x6e918c (0xb21 bytes) 000208[0] 1788263661.922351 mapwrite 0x7533cb thru 0x7535d4 (0x20a bytes) 000209[1] 1788263662.229638 write 0x816ce2 thru 0x818b5d (0x1e7c bytes) 000211[0] 1788263662.350089 read 0x7d3ee7 thru 0x7d4000 (0x11a bytes) 000212[0] 1788263662.436956 read 0x0649c8 thru 0x0650af (0x6e8 bytes) 000214[0] 1788263662.491837 mapread 0x6303e2 thru 0x631aef (0x170e bytes) 000215[1] 1788263662.508697 trunc from 0x818b5e to 0x1e6fd8 (0x631b85 bytes) 000217[0] 1788263662.649213 read 0x0f3470 thru 0x0f53ab (0x1f3c bytes) 000218[1] 1788263662.765988 mapread 0x0ab4a0 thru 0x0ad16a (0x1ccb bytes) 000219[1] 1788263662.789163 mapread 0x0d7c73 thru 0x0d813d (0x4cb bytes) 000223[1] 1788263662.825507 write 0x0e3aba thru 0x0e4fd1 (0x1518 bytes) 000226[0] 1788263662.902739 read 0x1824cc thru 0x183d90 (0x18c5 bytes) 000227[1] 1788263662.983609 write 0x8d56e0 thru 0x8d64de (0xdff bytes) 000228[0] 1788263662.996052 punch from 0x138ba5 to 0x13918c (0x5e8 bytes) 000229[0] 1788263663.036929 mapread 0x3a9edd thru 0x3aa16b (0x28f bytes) 000230[0] 1788263663.101959 mapread 0x583d1d thru 0x585a42 (0x1d26 bytes) 000235[0] 1788263663.289956 punch from 0x097672 to 0x09832c (0xcbb bytes) 000236[1] 1788263663.301599 mapwrite 0x64b0d8 thru 0x64bc96 (0xbbf bytes) 000237[1] 1788263663.538369 read 0x48e915 thru 0x48f151 (0x83d bytes) 000238[0] 1788263663.577339 trunc from 0x8d64df to 0x6934cb (0x243013 bytes) 000239[1] 1788263663.655981 trunc from 0x6934cc to 0x3792a4 (0x31a227 bytes) 000241[0] 1788263663.743759 read 0x2959e5 thru 0x297113 (0x172f bytes) 000242[0] 1788263663.876583 write 0x1ef7f1 thru 0x1f0882 (0x1092 bytes) 000244[1] 1788263663.929000 mapread 0x1ff8c6 thru 0x200592 (0xccd bytes) 000246[0] 1788263663.980179 read 0x29e7b7 thru 0x29ecaf (0x4f9 bytes) 000247[0] 1788263664.070820 write 0x8238c7 thru 0x8250a4 (0x17de bytes) 000250[1] 1788263664.126786 trunc from 0x8250a5 to 0x1f8fd5 (0x62c0cf bytes) 000251[1] 1788263664.179610 punch from 0x1739bf to 0x173b72 (0x1b4 bytes) 000254[0] 1788263664.207103 mapread 0x00c923 thru 0x00d467 (0xb45 bytes) 000255[0] 1788263664.244451 read 0x121f76 thru 0x123ccb (0x1d56 bytes) 000257[0] 1788263664.296508 mapwrite 0x703c30 thru 0x704bdc (0xfad bytes) 000258[1] 1788263664.455118 read 0x12bd82 thru 0x12cd52 (0xfd1 bytes) 000259[0] 1788263664.473257 mapread 0x6f4bbd thru 0x6f66cb (0x1b0f bytes) 000260[1] 1788263664.489655 write 0x43610b thru 0x436239 (0x12f bytes) 000261[0] 1788263664.511577 write 0x88d7f5 thru 0x88e83c (0x1048 bytes) 000262[0] 1788263664.539715 punch from 0x174157 to 0x175461 (0x130b bytes) 000266[1] 1788263664.601831 punch from 0x219d61 to 0x21b683 (0x1923 bytes) truncating to largest ever: 0x97afb2 000267[0] 1788263664.646391 trunc from 0x88e83d to 0x97afb1 (0xec775 bytes) 000269[1] 1788263664.718118 mapread 0x28406a thru 0x284779 (0x710 bytes) 000270[0] 1788263664.748251 trunc from 0x97afb2 to 0x2778b7 (0x7036fa bytes) 000274[0] 1788263664.865293 mapread 0x080c8f thru 0x080fb9 (0x32b bytes) 000275[0] 1788263664.882374 read 0x0aeb82 thru 0x0aeead (0x32c bytes) 000276[0] 1788263664.953962 read 0x09fc0a thru 0x0a00da (0x4d1 bytes) 000277[0] 1788263664.977216 write 0x3508fd thru 0x35284e (0x1f52 bytes) 000279[0] 1788263665.001997 mapread 0x1b1b9c thru 0x1b288b (0xcf0 bytes) 000282[0] 1788263665.032339 trunc from 0x35284f to 0x726951 (0x3d4103 bytes) 000283[0] 1788263665.072415 write 0x44ba88 thru 0x44c266 (0x7df bytes) 000284[1] 1788263665.207383 trunc from 0x726952 to 0x627e13 (0xfeb3e bytes) 000285[0] 1788263665.276161 write 0x03bfb8 thru 0x03cead (0xef6 bytes) 000286[0] 1788263665.309304 trunc from 0x627e14 to 0x7f1679 (0x1c9866 bytes) 000287[0] 1788263665.367670 trunc from 0x7f167a to 0x195450 (0x65c229 bytes) 000288[1] 1788263665.387080 mapwrite 0x50487a thru 0x50543a (0xbc1 bytes) 000289[1] 1788263665.492358 trunc from 0x50543b to 0x12ee1e (0x3d661c bytes) 000291[1] 1788263665.586656 mapread 0x01a540 thru 0x01c180 (0x1c41 bytes) 000292[0] 1788263665.625321 mapwrite 0x8840e6 thru 0x88540e (0x1329 bytes) 000295[1] 1788263665.899026 trunc from 0x88540f to 0x95b030 (0xd5c22 bytes) 000297[1] 1788263665.961967 write 0x84947b thru 0x849754 (0x2da bytes) 000298[1] 1788263665.990856 mapread 0x3e9d1f thru 0x3ea270 (0x552 bytes) 000299[1] 1788263666.001446 punch from 0x2f370b to 0x2f55ad (0x1ea3 bytes) 000301[0] 1788263666.034512 punch from 0x22a589 to 0x22bbfb (0x1673 bytes) 000302[1] 1788263666.066161 punch from 0x3094b5 to 0x30a33a (0xe86 bytes) 000303[1] 1788263666.111990 mapread 0x5aa47c thru 0x5ab81c (0x13a1 bytes) 000304[1] 1788263666.134980 mapwrite 0x67c24b thru 0x67d762 (0x1518 bytes) 000305[0] 1788263666.283041 trunc from 0x95b031 to 0x930ff3 (0x2a03d bytes) 000307[1] 1788263666.381620 mapread 0x7859fa thru 0x785ec3 (0x4ca bytes) 000309[1] 1788263666.438262 punch from 0x06015b to 0x0612f1 (0x1197 bytes) 000310[1] 1788263666.456249 mapread 0x607971 thru 0x607e2d (0x4bd bytes) 000311[1] 1788263666.462828 mapwrite 0x690095 thru 0x690f34 (0xea0 bytes) 000312[0] 1788263666.527969 mapwrite 0x8ee6b9 thru 0x8effa8 (0x18f0 bytes) 000313[0] 1788263666.631853 write 0x1e87b9 thru 0x1ea496 (0x1cde bytes) 000315[0] 1788263666.675155 punch from 0x91d9a3 to 0x91e9c7 (0x1025 bytes) 000316[0] 1788263666.696448 mapread 0x2bb2c4 thru 0x2bbbf0 (0x92d bytes) 000318[1] 1788263666.737833 punch from 0x0c3e0a to 0x0c5da7 (0x1f9e bytes) 000320[1] 1788263666.761933 trunc from 0x930ff4 to 0x8cfb4a (0x614a9 bytes) 000322[1] 1788263666.808742 mapwrite 0x7d6a04 thru 0x7d7b66 (0x1163 bytes) 000323[1] 1788263666.916553 mapwrite 0x06677c thru 0x067bbd (0x1442 bytes) 000324[0] 1788263666.980549 punch from 0x402e2d to 0x40327f (0x453 bytes) 000325[1] 1788263667.011961 mapread 0x39e051 thru 0x39f260 (0x1210 bytes) 000326[0] 1788263667.045034 mapread 0x0ae8ae thru 0x0b07a7 (0x1efa bytes) 000327[1] 1788263667.132104 mapwrite 0x660f45 thru 0x661f3e (0xffa bytes) 000330[1] 1788263667.270440 trunc from 0x8cfb4b to 0x2c6cf2 (0x608e58 bytes) 000332[1] 1788263667.309574 trunc from 0x2c6cf3 to 0x403b2f (0x13ce3d bytes) 000333[0] 1788263667.361395 mapwrite 0x3c357e thru 0x3c43fd (0xe80 bytes) 000334[0] 1788263667.529891 write 0x5145ef thru 0x514989 (0x39b bytes) 000335[0] 1788263667.544336 mapread 0x05561b thru 0x0572c9 (0x1caf bytes) 000341[1] 1788263667.590502 write 0x998209 thru 0x999c14 (0x1a0c bytes) 000342[1] 1788263667.631676 mapwrite 0x0a8660 thru 0x0aa5d5 (0x1f76 bytes) 000343[0] 1788263667.799549 write 0x9407bf thru 0x940fe1 (0x823 bytes) 000344[0] 1788263667.837397 mapwrite 0x9afeb3 thru 0x9b1838 (0x1986 bytes) 000346[0] 1788263668.035368 mapread 0x3c104e thru 0x3c2a86 (0x1a39 bytes) 000349[1] 1788263668.133900 punch from 0x1aa375 to 0x1aa8f7 (0x583 bytes) 000351[1] 1788263668.193668 read 0x857f01 thru 0x8594bf (0x15bf bytes) 000352[0] 1788263668.229894 mapwrite 0x8bec1b thru 0x8bf697 (0xa7d bytes) 000353[1] 1788263668.343540 mapwrite 0x1a52d3 thru 0x1a7037 (0x1d65 bytes) 000355[0] 1788263668.519559 read 0x3ae8d7 thru 0x3b0114 (0x183e bytes) 000358[1] 1788263668.579934 mapwrite 0x6d957c thru 0x6da317 (0xd9c bytes) 000359[1] 1788263668.668846 write 0x614e91 thru 0x614f17 (0x87 bytes) 000363[0] 1788263668.719772 mapwrite 0x5a3357 thru 0x5a3a74 (0x71e bytes) 000364[0] 1788263668.848089 punch from 0x089bb6 to 0x08bb46 (0x1f91 bytes) truncating to largest ever: 0x9eae1a 000365[1] 1788263668.893380 trunc from 0x9b1839 to 0x9eae19 (0x395e1 bytes) 000367[0] 1788263669.005812 mapread 0x4b96ec thru 0x4b99bf (0x2d4 bytes) 000368[1] 1788263669.079352 trunc from 0x9eae1a to 0x6c5f55 (0x324ec4 bytes) 000369[1] 1788263669.171087 read 0x0c103a thru 0x0c15d4 (0x59b bytes) 000371[1] 1788263669.212440 mapread 0x08b590 thru 0x08d12d (0x1b9e bytes) 000372[0] 1788263669.241387 write 0x10504f thru 0x105daf (0xd61 bytes) 000373[0] 1788263669.278412 read 0x689cfd thru 0x68afd3 (0x12d7 bytes) 000374[1] 1788263669.321440 trunc from 0x6c5f56 to 0x58643f (0x13fb16 bytes) 000376[1] 1788263669.406328 mapwrite 0x10f718 thru 0x1110c9 (0x19b2 bytes) 000377[1] 1788263669.472092 mapwrite 0x32baec thru 0x32bd39 (0x24e bytes) 000378[0] 1788263669.551204 mapwrite 0x3d8e04 thru 0x3da204 (0x1401 bytes) 000379[1] 1788263669.640513 read 0x1b577e thru 0x1b7625 (0x1ea8 bytes) 000380[1] 1788263669.664435 punch from 0x1b7bf0 to 0x1b9a1a (0x1e2b bytes) 000385[1] 1788263669.729052 punch from 0x2462df to 0x247028 (0xd4a bytes) 000386[1] 1788263669.771809 trunc from 0x586440 to 0x07acbd (0x50b782 bytes) 000389[1] 1788263669.912187 mapread 0x067347 thru 0x068ecc (0x1b86 bytes) 000390[1] 1788263669.934078 write 0x0f59ba thru 0x0f5f2e (0x575 bytes) 000393[1] 1788263669.968209 read 0x089535 thru 0x08a241 (0xd0d bytes) 000396[0] 1788263670.094133 mapwrite 0x8e4eb8 thru 0x8e5393 (0x4dc bytes) 000400[1] 1788263670.293827 trunc from 0x8e5394 to 0x9a4eb3 (0xbfb20 bytes) 000401[1] 1788263670.366968 punch from 0x11f011 to 0x120f02 (0x1ef2 bytes) 000402[1] 1788263670.390050 punch from 0x3677e7 to 0x369288 (0x1aa2 bytes) 000403[0] 1788263670.396984 read 0x977f2e thru 0x979d5e (0x1e31 bytes) 000404[1] 1788263670.420380 mapread 0x61988a thru 0x61ab00 (0x1277 bytes) 000406[0] 1788263670.608689 write 0x720d64 thru 0x722525 (0x17c2 bytes) 000407[0] 1788263670.671064 trunc from 0x9a4eb4 to 0x6845a6 (0x32090d bytes) 000409[0] 1788263670.797425 read 0x15dbeb thru 0x15fbcd (0x1fe3 bytes) 000410[1] 1788263670.807807 mapwrite 0x7dbb16 thru 0x7dd5c2 (0x1aad bytes) 000411[1] 1788263671.101697 write 0x43d660 thru 0x43e470 (0xe11 bytes) 000412[1] 1788263671.123674 mapwrite 0x0bb6b4 thru 0x0bb7d7 (0x124 bytes) 000414[1] 1788263671.161266 mapwrite 0x0c9579 thru 0x0ca86a (0x12f2 bytes) 000419[1] 1788263671.232790 punch from 0x18887a to 0x18a11e (0x18a5 bytes) 000429[0] 1788263671.358293 read 0x4f2d67 thru 0x4f2e54 (0xee bytes) 000431[0] 1788263671.387349 mapread 0x2108b1 thru 0x212843 (0x1f93 bytes) 000433[0] 1788263671.423893 mapwrite 0x581bdd thru 0x583253 (0x1677 bytes) 000435[1] 1788263671.504396 trunc from 0x7dd5c3 to 0x987057 (0x1a9a95 bytes) 000436[1] 1788263671.604995 trunc from 0x987058 to 0x2310a7 (0x755fb0 bytes) 000440[0] 1788263671.668233 read 0x148dbc thru 0x14ace3 (0x1f28 bytes) 000441[0] 1788263671.745022 mapread 0x10f273 thru 0x10fae1 (0x86f bytes) 000443[1] 1788263671.764789 mapread 0x070443 thru 0x0714e7 (0x10a5 bytes) 000447[1] 1788263671.879587 write 0x4a9bc6 thru 0x4ab216 (0x1651 bytes) 000449[0] 1788263671.917354 write 0x205010 thru 0x206bc0 (0x1bb1 bytes) 000451[0] 1788263672.047264 mapread 0x2bfa74 thru 0x2c118d (0x171a bytes) 000452[0] 1788263672.123888 mapread 0x0cc68e thru 0x0cd663 (0xfd6 bytes) 000455[0] 1788263672.230423 mapwrite 0x89f59b thru 0x8a0d64 (0x17ca bytes) 000457[0] 1788263672.567232 mapread 0x49ea4c thru 0x49fc30 (0x11e5 bytes) 000460[1] 1788263672.620620 read 0x35af58 thru 0x35b9cc (0xa75 bytes) 000461[0] 1788263672.650840 mapwrite 0x49961e thru 0x49ac40 (0x1623 bytes) 000462[1] 1788263672.715672 write 0x587892 thru 0x5893b7 (0x1b26 bytes) 000464[1] 1788263672.758682 mapwrite 0x3e16e1 thru 0x3e2fbb (0x18db bytes) 000465[1] 1788263672.818520 read 0x5846b1 thru 0x586395 (0x1ce5 bytes) 000466[0] 1788263672.825706 mapwrite 0x56abb1 thru 0x56bcb3 (0x1103 bytes) 000469[0] 1788263672.946733 punch from 0x38ce79 to 0x38d60c (0x794 bytes) 000470[0] 1788263672.983933 read 0x2a3b1d thru 0x2a50dd (0x15c1 bytes) 000471[1] 1788263672.990761 punch from 0x38eaf7 to 0x38efb6 (0x4c0 bytes) 000472[1] 1788263673.019484 trunc from 0x8a0d65 to 0x8124cf (0x8e895 bytes) 000475[0] 1788263673.068998 read 0x6370c5 thru 0x637a9e (0x9da bytes) 000476[0] 1788263673.095624 read 0x300a07 thru 0x300a8a (0x84 bytes) 000479[1] 1788263673.107431 mapread 0x6ad1e4 thru 0x6adf1c (0xd39 bytes) 000480[1] 1788263673.115832 write 0x8b99ad thru 0x8b9e00 (0x454 bytes) 000481[1] 1788263673.128575 mapread 0x75d203 thru 0x75deeb (0xce9 bytes) 000482[0] 1788263673.143907 mapwrite 0x931098 thru 0x932505 (0x146e bytes) 000483[0] 1788263673.238957 mapread 0x86f9e2 thru 0x86fb9d (0x1bc bytes) 000484[0] 1788263673.254776 write 0x238b71 thru 0x239e6e (0x12fe bytes) 000486[0] 1788263673.295482 mapread 0x84f995 thru 0x85054c (0xbb8 bytes) 000489[0] 1788263673.351444 punch from 0x6100a2 to 0x6117a1 (0x1700 bytes) 000491[1] 1788263673.400279 punch from 0x6b948b to 0x6ba0a5 (0xc1b bytes) 000492[1] 1788263673.425296 punch from 0x84439b to 0x8446af (0x315 bytes) 000493[0] 1788263673.441058 read 0x67471a thru 0x6747b0 (0x97 bytes) 000494[1] 1788263673.459384 trunc from 0x932506 to 0x66c191 (0x2c6374 bytes) 000495[1] 1788263673.527406 read 0x48c8e0 thru 0x48c9a2 (0xc3 bytes) 000496[1] 1788263673.537957 trunc from 0x66c192 to 0x169c64 (0x50252d bytes) 000498[1] 1788263673.576303 write 0x8369de thru 0x836d1b (0x33e bytes) 000499[0] 1788263673.580439 write 0x1b6b2c thru 0x1b8335 (0x180a bytes) 000500[1] 1788263673.621328 punch from 0x2a9c3a to 0x2ab74c (0x1b13 bytes) All operations completed A-OK! PASS 16b (31s) == sanityn test 16c: verify data consistency on ldiskfs with cache disabled (b=17397) ========================================================== 07:54:40 (1788263680) SKIP: sanityn test_16c dio on ldiskfs only SKIP 16c (4s) == sanityn test 16d: Verify DIO and buffer IO with two clients ========================================================== 07:54:44 (1788263684) 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.263543 s, 31.8 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.204726 s, 41.0 MB/s ldlm.namespaces.MGC192.168.201.113@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff911148971800.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff91116186b000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff911148971800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff91116186b000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff911148971800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff91116186b000.lru_size=clear 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 12.5896 s, 8.3 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.17756 s, 25.1 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 3.23701 s, 32.4 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.16562 s, 17.0 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.18378 s, 12.8 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.17498 s, 20.3 MB/s PASS 16d (54s) == sanityn test 16e: Verify size consistency for O_DIRECT write ========================================================== 07:55:38 (1788263738) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.486881 s, 21.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.231603 s, 4.5 MB/s PASS 16e (10s) == sanityn test 16f: rw sequential consistency vs drop_caches ========================================================== 07:55:48 (1788263748) TIMEOUT PASS 16f (29s) == sanityn test 16g: mmap rw sequential consistency vs drop_caches ========================================================== 07:56:17 (1788263777) TIMEOUT PASS 16g (29s) == sanityn test 16h: mmap read after truncate file ======= 07:56:46 (1788263806) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16h (7s) == sanityn test 16i: read after truncate file ============ 07:56:53 (1788263813) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16i (8s) == sanityn test 16j: race dio with buffered i/o ========== 07:57:01 (1788263821) 1+0 records in 1+0 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.19844 s, 11.9 MB/s bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.13144 s, 23.2 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.64056 s, 9.9 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.90475 s, 13.8 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.4069 s, 10.9 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.46691 s, 10.6 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.11302 s, 8.4 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.45255 s, 18.0 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.72463 s, 15.2 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.28461 s, 8.0 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.43183 s, 7.6 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.5482 s, 10.3 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.68148 s, 9.8 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.37627 s, 7.8 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.43405 s, 7.6 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.01196 s, 8.7 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.0874 s, 8.5 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.2906 s, 8.0 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.33895 s, 7.9 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK PASS 16j (40s) == sanityn test 16k: Parallel FSX and drop caches should not panic ========================================================== 07:57:41 (1788263861) + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.1 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.2 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.3 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.4 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.5 -l 67108864 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported truncating to largest ever: 0x1a7062c fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x3c569db truncating to largest ever: 0x32cce75 truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! + fsx 1 done: rc=0 Tue Sep 1 07:58:29 EDT 2026 + fsx 2 done: rc=0 Tue Sep 1 07:58:29 EDT 2026 + fsx 3 done: rc=0 Tue Sep 1 07:58:29 EDT 2026 + fsx 4 done: rc=0 Tue Sep 1 07:58:29 EDT 2026 All operations completed A-OK! + fsx 5 done: rc=0 Tue Sep 1 07:58:29 EDT 2026 PASS 16k (57s) == sanityn test 17: resource creation/LVB creation race ========================================================================= 07:58:38 (1788263918) fail_loc=0x8000030a PASS 17 (13s) == sanityn test 18: mmap sanity check =========================================================================================== 07:58:51 (1788263931) mmap test1: basic mmap operation (PASS, 0.18337s) mmap test2: MAP_PRIVATE not write back (PASS, 0.098843s) mmap test3: concurrent mmap ops on two nodes (PASS, 10.69s) mmap test4: c1 write to f1 from mmapped f2, c2 write to f1 from mmapped f1 (PASS, 4.5684s) mmap test5: read/write file to/from the buffer which mmapped to just this file (PASS, 0.24305s) mmap test6: check mmap write/read content on two nodes (PASS, 0.22053s) mmap test7: file i/o with an unmapped buffer (SKIPPED, 0s) mmap test8: SIGBUS for beyond file size (SKIPPED, 0s) mmap test9: SIGBUS for negative file offset (SKIPPED, 0s) mmap test10: multi-thread mmap access (PASS, 1.09s) mmap test11: multi-thread shared mmap (PASS, 1.1858s) mmap test10: mtime not change for readonly mmap access (PASS, 1.1194s) PASS 18 (29s) == sanityn test 19: test concurrent uncached read races ========================================================================= 07:59:20 (1788263960) oleg113-server: error: get_param: param_path 'osd-*/lustre-OST*/read_cache_enable': No such file or directory pdsh@oleg113-client: oleg113-server: ssh exited with exit code 2 SKIP: sanityn test_19 not cache-capable obdfilter SKIP 19 (4s) == sanityn test 20: test extra readahead page left in cache ============================================================== 07:59:24 (1788263964) PASS 20 (8s) == sanityn test 21: Try to remove mountpoint on another dir ============================================================== 07:59:32 (1788263972) rmdir: removing directory, '/mnt/lustre/d21.sanityn' rmdir: failed to remove '/mnt/lustre/d21.sanityn': Device or resource busy rmdir: removing directory, '/mnt/lustre2/d21.sanityn' Removed mounted directory from another mountpoint, needs to be fixed PASS 21 (7s) == sanityn test 23: others should see updated atime while another read============================================================== 07:59:40 (1788263980) now is 1788263982 starting reads multiop /mnt/lustre/f23.sanityn vor20_c TMPPIPE=/tmp/multiop_open_wait_pipe.7626 new atime is 1788264043 PASS 23 (70s) == sanityn test 24a: lfs df [-ih] [path] test =================================================================================== 08:00:49 (1788264049) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 5120 3764224 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 5120 3764224 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre2 UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 115.3K 403 114.9K 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 114.5K 399 114.1K 1% /mnt/lustre[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.3K 403 114.9K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 399 114.1K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre2 UUID bytes Used Available Use% Mounted on lustre-MDT0000_UUID 2.1G 2.1M 2.1G 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3.6G 5.0M 3.6G 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3.6G 32.0M 3.6G 1% /mnt/lustre[OST:1] filesystem_summary: 7.2G 37.0M 7.2G 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118099 403 117696 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117231 399 116832 1% /mnt/lustre2[OST:1] filesystem_summary: 234914 386 234528 1% /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 5120 3764224 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.3K 403 114.9K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 399 114.1K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 118099 403 117696 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 117231 399 116832 1% /mnt/lustre[OST:1] filesystem_summary: 234914 386 234528 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118099 403 117696 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117231 399 116832 1% /mnt/lustre2[OST:1] filesystem_summary: 234914 386 234528 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 5120 3764224 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 5120 3764224 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre2 PASS 24a (7s) == sanityn test 24b: lfs df should show both filesystems ========================================================================= 08:00:56 (1788264056) PASS 24b (7s) == sanityn test 25a: change ACL on one mountpoint be seen on another ============================================================= 08:01:03 (1788264063) running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] PASS 25a (8s) == sanityn test 25b: change ACL under remote dir on one mountpoint be seen on another ========================================================== 08:01:11 (1788264071) SKIP: sanityn test_25b needs >= 2 MDTs SKIP 25b (4s) == sanityn test 26a: allow mtime to get older ============ 08:01:15 (1788264075) utime: good mknod times 1788264075 <= 1788264076 <= 1788264076 for /mnt/lustre/f26a utime: good mknod times 1788264075 <= 1788264076 <= 1788264076 for /mnt/lustre2/f26a utime: good utime mtimes 100000, atime 200000 utime: updated times for /mnt/lustre2/f26a PASS 26a (8s) == sanityn test 26b: sync mtime between ost and mds ====== 08:01:23 (1788264083) PASS 26b (10s) == sanityn test 26c: set-in-past on open file is not lost on close ========================================================== 08:01:33 (1788264093) multiop /mnt/lustre/f26c.sanityn vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7626 File: /mnt/lustre/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502398 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-09-01 08:01:35.000000000 -0400 Birth: 2026-09-01 08:01:35.000000000 -0400 File: /mnt/lustre2/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502398 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-09-01 08:01:35.000000000 -0400 Birth: 2026-09-01 08:01:35.000000000 -0400 PASS 26c (9s) SKIP: sanityn test_27 skipping excluded test 27 == sanityn test 30: recreate file race =================== 08:01:44 (1788264104) 0 PASS 30 (9s) == sanityn test 31a: voluntary cancel / blocking ast race======================================================================== 08:01:53 (1788264113) fail_loc=0x314 PASS 31a (10s) == sanityn test 31b: voluntary OST cancel / blocking ast race======================================================================== 08:02:03 (1788264123) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg113-server mds-ost sync done. sleep 5 for ZFS MDS Waiting for MDT destroys to complete fail_loc=0x314 fail_loc=0x316 fail_loc=0 fail_loc=0 Filesystem 1K-blocks Used Available Use% Mounted on 192.168.201.113@tcp:/lustre 7542784 39936 7498752 1% /mnt/lustre2 PASS 31b (23s) == sanityn test 31r: open-rename(replace) race =========== 08:02:26 (1788264146) -rw-r--r-- 1 root root 0 Sep 1 08:02 /mnt/lustre/f31r.sanityn.target fail_loc=0x1419 fail_val=3 PASS 31r (11s) == sanityn test 31s: open should not revalidate invalid dentry ========================================================== 08:02:38 (1788264158) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 1205: 43172 Terminated tail -f $tfile PASS 31s (9s) == sanityn test 31t: getattr should not revalidate invalid dentry ========================================================== 08:02:46 (1788264166) PASS 31t (9s) == sanityn test 32b: lockless i/o ======================== 08:02:55 (1788264175) SKIP: sanityn test_32b max_nolock_bytes is removed >= 2.17.53 SKIP 32b (3s) == sanityn test 32c: contention detection ================ 08:02:58 (1788264178) ldlm.namespaces.filter-lustre-OST0000_UUID.contended_locks=2 ldlm.namespaces.filter-lustre-OST0001_UUID.contended_locks=2 ldlm.namespaces.filter-lustre-OST0000_UUID.contention_events=0 ldlm.namespaces.filter-lustre-OST0001_UUID.contention_events=0 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.36963 s, 38.3 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.49705 s, 35.0 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 3.16531 s, 16.6 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 3.3944 s, 15.4 MB/s contention_seconds 4, contention_events 18 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.55345 s, 20.5 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.72317 s, 19.3 MB/s ldlm.namespaces.filter-lustre-OST0000_UUID.contention_events=0 ldlm.namespaces.filter-lustre-OST0001_UUID.contention_events=0 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.82793 s, 28.7 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.8638 s, 28.1 MB/s PASS 32c (35s) SKIP: sanityn test_33a skipping SLOW test 33a == sanityn test 33b: COS: cross create/delete, 2 clients, benchmark under remote dir ========================================================== 08:03:35 (1788264215) SKIP: sanityn test_33b Need two or more clients, have 1 SKIP 33b (4s) == sanityn test 33c: Cancel cross-MDT lock should trigger Sync-on-Lock-Cancel ========================================================== 08:03:38 (1788264218) SKIP: sanityn test_33c needs >= 2 MDTs SKIP 33c (4s) == sanityn test 33d: dependent transactions should trigger COS ========================================================== 08:03:42 (1788264222) SKIP: sanityn test_33d needs >= 2 MDTs SKIP 33d (4s) == sanityn test 33e: independent transactions shouldn't trigger COS ========================================================== 08:03:46 (1788264226) SKIP: sanityn test_33e needs >= 2 MDTs SKIP 33e (3s) == sanityn test 34: no lock timeout under IO ============= 08:03:50 (1788264230) fail_loc=0x512 fail_loc=0x512 lock should not expire writing on client1 reading on client2 success fail_loc=0x511 fail_loc=0x511 lock should expire writing on client1 reading on client2 success oleg113-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff911148971800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff911148971800.ost_server_uuid in FULL state after 0 sec oleg113-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff911148971800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff911148971800.ost_server_uuid in IDLE state after 0 sec oleg113-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff911148971800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff911148971800.ost_server_uuid in FULL state after 0 sec oleg113-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff911148971800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff911148971800.ost_server_uuid in IDLE state after 0 sec oleg113-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff911148971800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff911148971800.ost_server_uuid in FULL state after 0 sec oleg113-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff911148971800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff911148971800.ost_server_uuid in IDLE state after 0 sec PASS 34 (145s) == sanityn test 35: -EINTR cp_ast vs. bl_ast race does not evict client ========================================================== 08:06:15 (1788264375) fail_loc=0x80000317 Race attempt 0 Wait for 50484 50508 for 60 sec... - open/close 812 (time 1788264393.68 total 10.01 last 81.15) - open/close 1632 (time 1788264403.68 total 20.01 last 81.96) - open/close 2380 (time 1788264413.69 total 30.02 last 74.73) - open/close 3151 (time 1788264423.70 total 40.03 last 77.06) - open/close 3766 (time 1788264433.70 total 50.03 last 61.49) total: 4000 open/close in 53.50 seconds: 74.77 ops/second fail_loc=0x0 Filesystem Size Used Avail Use% Mounted on 192.168.201.113@tcp:/lustre 7.2G 39M 7.2G 1% /mnt/lustre 192.168.201.113@tcp:/lustre 7.2G 39M 7.2G 1% /mnt/lustre2 PASS 35 (75s) == sanityn test 36: handle ESTALE/open-unlink correctly == 08:07:30 (1788264450) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg113-server mds-ost sync done. Waiting for MDT destroys to complete 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.60035 s, 32.8 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7626 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(0) *** before(7498752) after_dd(7443456) after(7498752) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.84374 s, 28.4 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7626 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(1) *** before(7498752) after_dd(7443456) after(7494656) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.813712 s, 64.4 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7626 sleep 5 for ZFS MDS Waiting for MDT destroys to complete *** cycle(2) *** before(7494656) after_dd(7441408) after(7492608) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.2365 s, 42.4 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7626 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(3) *** before(7492608) after_dd(7441408) after(7492608) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.58855 s, 33.0 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7626 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(4) *** before(7492608) after_dd(7445504) after(7498752) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.4638 s, 35.8 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7626 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(5) *** before(7498752) after_dd(7441408) after(7498752) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.62957 s, 32.2 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7626 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(6) *** before(7498752) after_dd(7440384) after(7498752) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.6371 s, 32.0 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7626 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(7) *** before(7498752) after_dd(7447552) after(7498752) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.13718 s, 24.5 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7626 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(8) *** before(7498752) after_dd(7443456) after(7498752) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.81818 s, 28.8 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7626 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(9) *** before(7498752) after_dd(7443456) after(7494656) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.35973 s, 38.6 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7626 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(10) *** before(7494656) after_dd(7443456) after(7498752) PASS 36 (345s) == sanityn test 37: check i_size is not updated for directory on close (bug 18695) ======================================================================== 08:13:15 (1788264795) multiop /mnt/lustre/d37.sanityn vD_c TMPPIPE=/tmp/multiop_open_wait_pipe.7626 - create 1500 (time 1788264810.15 total 10.01 last 149.91) - create 2706 (time 1788264820.15 total 20.01 last 120.58) - create 4181 (time 1788264830.15 total 30.01 last 147.44) - create 5869 (time 1788264840.16 total 40.01 last 168.79) - create 7432 (time 1788264850.17 total 50.03 last 156.10) - create 8741 (time 1788264860.17 total 60.03 last 130.84) - create 9981 (time 1788264870.18 total 70.04 last 123.84) total: 10000 create in 70.22 seconds: 142.40 ops/second PASS 37 (84s) == sanityn test 39a: file mtime does not change after rename ========================================================== 08:14:39 (1788264879) repeat after cancel_lru_locks PASS 39a (9s) == sanityn test 39b: file mtime the same on clients with/out lock ========================================================== 08:14:48 (1788264888) repeat after cancel_lru_locks PASS 39b (9s) == sanityn test 39c: check truncate mtime update ================================================================================ 08:14:58 (1788264898) repeat after cancel_lru_locks PASS 39c (9s) == sanityn test 39d: sync write should update mtime ====== 08:15:06 (1788264906) fail_loc=0x411 fail_loc=0 PASS 39d (8s) SKIP: sanityn test_40a skipping ALWAYS excluded test 40a == sanityn test 40b: pdirops: open|create and others ======================================================================== 08:15:16 (1788264916) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40b (20s) == sanityn test 40c: pdirops: link and others ======================================================================== 08:15:36 (1788264936) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40c (19s) == sanityn test 40d: pdirops: unlink and others ======================================================================== 08:15:55 (1788264955) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40d (18s) == sanityn test 40e: pdirops: rename and others ======================================================================== 08:16:14 (1788264974) No conflict No conflict No conflict No conflict No conflict No conflict PASS 40e (18s) == sanityn test 41a: pdirops: create vs mkdir ======================================================================== 08:16:33 (1788264993) mkdir: cannot create directory '/mnt/lustre2/f41a.sanityn': File exists Conflict PASS 41a (12s) == sanityn test 41b: pdirops: create vs create ======================================================================== 08:16:45 (1788265005) open: File exists Conflict PASS 41b (13s) == sanityn test 41c: pdirops: create vs link ======================================================================== 08:16:59 (1788265019) link: cannot create link '/mnt/lustre2/f41c.sanityn' to '/mnt/lustre2/f41c.sanityn-2': File exists Conflict PASS 41c (14s) == sanityn test 41d: pdirops: create vs unlink ======================================================================== 08:17:13 (1788265033) Conflict PASS 41d (13s) == sanityn test 41e: pdirops: create and rename (tgt) ======================================================================== 08:17:26 (1788265046) Conflict PASS 41e (14s) == sanityn test 41f: pdirops: create and rename (src) ======================================================================== 08:17:40 (1788265060) Conflict PASS 41f (14s) == sanityn test 41g: pdirops: create vs getattr ======================================================================== 08:17:54 (1788265074) Conflict PASS 41g (14s) == sanityn test 41h: pdirops: create vs readdir ======================================================================== 08:18:08 (1788265088) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 2427: lctl set_param -n fail_loc=0 2>/dev/null || true: No such file or directory Conflict PASS 41h (14s) == sanityn test 41i: reint_open: create vs create ======== 08:18:23 (1788265103) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 41i (1103s) == sanityn test 42a: pdirops: mkdir vs mkdir ======================================================================== 08:36:46 (1788266206) mkdir: cannot create directory '/mnt/lustre2/f42a.sanityn': File exists Conflict PASS 42a (12s) == sanityn test 42b: pdirops: mkdir vs create ======================================================================== 08:36:58 (1788266218) open: File exists Conflict PASS 42b (16s) == sanityn test 42c: pdirops: mkdir vs link ======================================================================== 08:37:14 (1788266234) link: cannot create link '/mnt/lustre2/f42c.sanityn' to '/mnt/lustre2/f42c.sanityn-2': File exists Conflict PASS 42c (13s) == sanityn test 42d: pdirops: mkdir vs unlink ======================================================================== 08:37:27 (1788266247) Conflict PASS 42d (15s) == sanityn test 42e: pdirops: mkdir and rename (tgt) ======================================================================== 08:37:42 (1788266262) mv: cannot overwrite directory '/mnt/lustre2/f42e.sanityn' with non-directory Conflict PASS 42e (16s) == sanityn test 42f: pdirops: mkdir and rename (src) ======================================================================== 08:37:59 (1788266279) Conflict PASS 42f (15s) == sanityn test 42g: pdirops: mkdir vs getattr ======================================================================== 08:38:14 (1788266294) Conflict PASS 42g (13s) == sanityn test 42h: pdirops: mkdir vs readdir ======================================================================== 08:38:27 (1788266307) Conflict PASS 42h (14s) == sanityn test 43a: rmdir,mkdir doesn't return -EEXIST ======================================================================== 08:38:41 (1788266321) PASS 43a (63s) == sanityn test 43b: pdirops: unlink vs create ======================================================================== 08:39:44 (1788266384) Conflict PASS 43b (14s) == sanityn test 43c: pdirops: unlink vs link ======================================================================== 08:39:58 (1788266398) Conflict PASS 43c (14s) == sanityn test 43d: pdirops: unlink vs unlink ======================================================================== 08:40:12 (1788266412) rm: cannot remove '/mnt/lustre2/f43d.sanityn': No such file or directory Conflict PASS 43d (13s) == sanityn test 43e: pdirops: unlink and rename (tgt) ======================================================================== 08:40:25 (1788266425) Conflict PASS 43e (13s) == sanityn test 43f: pdirops: unlink and rename (src) ======================================================================== 08:40:38 (1788266438) mv: cannot stat '/mnt/lustre2/f43f.sanityn': No such file or directory Conflict PASS 43f (12s) == sanityn test 43g: pdirops: unlink vs getattr ======================================================================== 08:40:51 (1788266451) stat: cannot statx '/mnt/lustre2/f43g.sanityn': No such file or directory Conflict PASS 43g (12s) == sanityn test 43h: pdirops: unlink vs readdir ======================================================================== 08:41:03 (1788266463) Conflict PASS 43h (13s) == sanityn test 43i: pdirops: unlink vs remote mkdir ===== 08:41:16 (1788266476) SKIP: sanityn test_43i needs >= 2 MDTs SKIP 43i (3s) == sanityn test 43j: racy mkdir return EEXIST ======================================================================== 08:41:19 (1788266479) mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists PASS 43j (110s) == sanityn test 43k: unlink vs create ==================== 08:43:09 (1788266589) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 43k (928s) == sanityn test 44a: pdirops: rename tgt vs mkdir ======================================================================== 08:58:37 (1788267517) mkdir: cannot create directory '/mnt/lustre2/f44a.sanityn': File exists Conflict Tue Sep 1 08:58:44 EDT 2026 PASS 44a (11s) == sanityn test 44b: pdirops: rename tgt vs create ======================================================================== 08:58:48 (1788267528) open: File exists Conflict PASS 44b (12s) == sanityn test 44c: pdirops: rename tgt vs link ======================================================================== 08:59:00 (1788267540) link: cannot create link '/mnt/lustre2/f44c.sanityn' to '/mnt/lustre2/f44c.sanityn-3': File exists Conflict PASS 44c (11s) == sanityn test 44d: pdirops: rename tgt vs unlink ======================================================================== 08:59:11 (1788267551) Conflict PASS 44d (11s) == sanityn test 44e: pdirops: rename tgt and rename (tgt) ======================================================================== 08:59:22 (1788267562) Conflict PASS 44e (11s) == sanityn test 44f: pdirops: rename tgt and rename (src) ======================================================================== 08:59:33 (1788267573) Conflict PASS 44f (10s) == sanityn test 44g: pdirops: rename tgt vs getattr ======================================================================== 08:59:44 (1788267584) Conflict PASS 44g (12s) == sanityn test 44h: pdirops: rename tgt vs readdir ======================================================================== 08:59:55 (1788267595) Conflict PASS 44h (11s) == sanityn test 44i: pdirops: rename tgt vs remote mkdir ========================================================== 09:00:06 (1788267606) SKIP: sanityn test_44i needs >= 2 MDTs SKIP 44i (3s) == sanityn test 45a: rename,mkdir doesn't return -EEXIST ======================================================================== 09:00:09 (1788267609) PASS 45a (91s) == sanityn test 45b: pdirops: rename src vs create ======================================================================== 09:01:40 (1788267700) Conflict PASS 45b (9s) == sanityn test 45c: pdirops: rename src vs link ======================================================================== 09:01:49 (1788267709) Conflict PASS 45c (10s) == sanityn test 45d: pdirops: rename src vs unlink ======================================================================== 09:01:59 (1788267719) rm: cannot remove '/mnt/lustre2/f45d.sanityn': No such file or directory Conflict PASS 45d (9s) == sanityn test 45e: pdirops: rename src and rename (tgt) ======================================================================== 09:02:08 (1788267728) Conflict PASS 45e (10s) == sanityn test 45f: pdirops: rename src and rename (src) ======================================================================== 09:02:18 (1788267738) mv: cannot stat '/mnt/lustre2/f45f.sanityn': No such file or directory Conflict PASS 45f (10s) == sanityn test 45g: pdirops: rename src vs getattr ======================================================================== 09:02:28 (1788267748) stat: cannot statx '/mnt/lustre2/f45g.sanityn': No such file or directory Conflict PASS 45g (10s) == sanityn test 45h: pdirops: unlink vs readdir ======================================================================== 09:02:38 (1788267758) Conflict PASS 45h (8s) == sanityn test 45i: pdirops: rename src vs remote mkdir ========================================================== 09:02:46 (1788267766) SKIP: sanityn test_45i needs >= 2 MDTs SKIP 45i (2s) == sanityn test 45j: read vs rename ====================== 09:02:49 (1788267769) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 45j (705s) == sanityn test 46a: pdirops: link vs mkdir ======================================================================== 09:14:33 (1788268473) mkdir: cannot create directory '/mnt/lustre2/f46a.sanityn': File exists Conflict PASS 46a (7s) == sanityn test 46b: pdirops: link vs create ======================================================================== 09:14:40 (1788268480) open: File exists Conflict PASS 46b (6s) == sanityn test 46c: pdirops: link vs link ======================================================================== 09:14:46 (1788268486) link: cannot create link '/mnt/lustre2/f46c.sanityn' to '/mnt/lustre2/f46c.sanityn': File exists Conflict PASS 46c (6s) == sanityn test 46d: pdirops: link vs unlink ======================================================================== 09:14:52 (1788268492) Conflict PASS 46d (5s) == sanityn test 46e: pdirops: link and rename (tgt) ======================================================================== 09:14:57 (1788268497) Conflict PASS 46e (7s) == sanityn test 46f: pdirops: link and rename (src) ======================================================================== 09:15:04 (1788268504) Conflict PASS 46f (7s) == sanityn test 46g: pdirops: link vs getattr ======================================================================== 09:15:11 (1788268511) Conflict PASS 46g (6s) == sanityn test 46h: pdirops: link vs readdir ======================================================================== 09:15:17 (1788268517) Conflict PASS 46h (7s) == sanityn test 46i: pdirops: link vs remote mkdir ======= 09:15:24 (1788268524) SKIP: sanityn test_46i needs >= 2 MDTs SKIP 46i (1s) == sanityn test 47a: pdirops: remote mkdir vs mkdir ====== 09:15:25 (1788268525) SKIP: sanityn test_47a needs >= 2 MDTs SKIP 47a (1s) == sanityn test 47b: pdirops: remote mkdir vs create ===== 09:15:26 (1788268526) SKIP: sanityn test_47b needs >= 2 MDTs SKIP 47b (1s) == sanityn test 47c: pdirops: remote mkdir vs link ======= 09:15:28 (1788268528) SKIP: sanityn test_47c needs >= 2 MDTs SKIP 47c (1s) == sanityn test 47d: pdirops: remote mkdir vs unlink ===== 09:15:29 (1788268529) SKIP: sanityn test_47d needs >= 2 MDTs SKIP 47d (1s) == sanityn test 47e: pdirops: remote mkdir and rename (tgt) ========================================================== 09:15:30 (1788268530) SKIP: sanityn test_47e needs >= 2 MDTs SKIP 47e (1s) == sanityn test 47f: pdirops: remote mkdir and rename (src) ========================================================== 09:15:31 (1788268531) SKIP: sanityn test_47f needs >= 2 MDTs SKIP 47f (2s) == sanityn test 47g: pdirops: remote mkdir vs getattr ==== 09:15:33 (1788268533) SKIP: sanityn test_47g needs >= 2 MDTs SKIP 47g (1s) == sanityn test 50: osc lvb attrs: enqueue vs. CP AST ======================================================================== 09:15:34 (1788268534) 10+0 records in 10+0 records out 10240 bytes (10 kB, 10 KiB) copied, 0.0092043 s, 1.1 MB/s fail_loc=0x410 fail_loc=0x0 PASS 50 (8s) == sanityn test 51a: layout lock: refresh layout should work ========================================================== 09:15:42 (1788268542) 0+1 records in 0+1 records out 158 bytes copied, 0.00493503 s, 32.0 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 3690: kill: (218822) - No such process PASS 51a (5s) == sanityn test 51b: layout lock: glimpse should be able to restart if layout changed ========================================================== 09:15:47 (1788268547) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0118494 s, 86.4 kB/s fail_loc=0x1404 fail_val=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0416572 s, 25.2 MB/s 3145728 PASS 51b (15s) == sanityn test 51c: layout lock: IT_LAYOUT blocked and correct layout can be returned ========================================================== 09:16:02 (1788268562) fail_loc=0x172 Setting layout to have 2 stripes ... 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0139742 s, 73.3 kB/s PASS 51c (8s) == sanityn test 51d: layout lock: losing layout lock should clean up memory map region ========================================================== 09:16:10 (1788268570) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.024586 s, 42.6 MB/s Before revoking layout lock: 1024 KB mapped PASS 51d (4s) == sanityn test 51e: lfs getstripe does not break leases, part 2 ========================================================== 09:16:14 (1788268574) /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 2454 0x996 0x240000400 write lease(2) has applied. /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 2454 0x996 0x240000400 read lease(1) has applied. PASS 51e (5s) == sanityn test 54: rename locking ======================= 09:16:20 (1788268580) llite.lustre-ffff911148971800.xattr_cache=1 llite.lustre-ffff91116186b000.xattr_cache=1 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505015 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-01 09:16:20.000000000 -0400 Modify: 2026-09-01 09:16:20.000000000 -0400 Change: 2026-09-01 09:16:20.000000000 -0400 Birth: 2026-09-01 09:16:21.000000000 -0400 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000154 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505018 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-01 09:16:26.000000000 -0400 Modify: 2026-09-01 09:16:26.000000000 -0400 Change: 2026-09-01 09:16:26.000000000 -0400 Birth: 2026-09-01 09:16:26.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505021 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-01 09:16:32.000000000 -0400 Modify: 2026-09-01 09:16:32.000000000 -0400 Change: 2026-09-01 09:16:32.000000000 -0400 Birth: 2026-09-01 09:16:32.000000000 -0400 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505020 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-01 09:16:32.000000000 -0400 Modify: 2026-09-01 09:16:32.000000000 -0400 Change: 2026-09-01 09:16:32.000000000 -0400 Birth: 2026-09-01 09:16:32.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000154 File: /mnt/lustre/d1 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505023 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-01 09:16:37.000000000 -0400 Modify: 2026-09-01 09:16:38.000000000 -0400 Change: 2026-09-01 09:16:38.000000000 -0400 Birth: 2026-09-01 09:16:38.000000000 -0400 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505024 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-01 09:16:37.000000000 -0400 Modify: 2026-09-01 09:16:38.000000000 -0400 Change: 2026-09-01 09:16:38.000000000 -0400 Birth: 2026-09-01 09:16:38.000000000 -0400 open(O_RDWR|O_CREAT): Is a directory PASS 54 (26s) == sanityn test 55a: rename vs unlink target dir ========= 09:16:46 (1788268606) fail_loc=0x80000156 mv: cannot move '/mnt/lustre/d55a.sanityn/d1/d2' to '/mnt/lustre/d55a.sanityn/d3/d2': No such file or directory PASS 55a (9s) == sanityn test 55b: rename vs unlink source dir ========= 09:16:55 (1788268615) fail_loc=0x80000156 mv: cannot stat '/mnt/lustre/d55b.sanityn/d1/d2': No such file or directory PASS 55b (8s) == sanityn test 55c: rename vs unlink orphan target dir == 09:17:03 (1788268623) fail_loc=0x156 mv: cannot move '/mnt/lustre/d55c.sanityn/d1/d2' to '/mnt/lustre/d55c.sanityn/d3/d2': No such file or directory PASS 55c (14s) == sanityn test 55d: rename file vs link ================= 09:17:17 (1788268637) fail_loc=0x155 mv: '/mnt/lustre/d55d.sanityn/f1' and '/mnt/lustre/d55d.sanityn/d55d.sanityn/f1' are the same file PASS 55d (10s) == sanityn test 55e: rename race AB/BA under the same parent dir ========================================================== 09:17:27 (1788268647) SKIP: sanityn test_55e needs >= 2 MDTs SKIP 55e (2s) == sanityn test 55f: rename: (P1/A -> P2/B) race with (P2/B -> P1/A) ========================================================== 09:17:29 (1788268649) fail_loc=0x155 We survived after P1A_P2B/P2B_P1A race test fail_loc=0x0 PASS 55f (14s) == sanityn test 55g: rename: race with trylock =========== 09:17:43 (1788268663) mdt.lustre-MDT0000.enable_rename_trylock=1 fail_loc=0x155 mv: cannot move '/mnt/lustre/d55g.sanityn/a1/a2' to a subdirectory of itself, '/mnt/lustre/d55g.sanityn/b1/b2/b3/b4/a2' We survived race with trylock test fail_loc=0x0 PASS 55g (15s) == sanityn test 56a: test llverdev with single large stripe ========================================================== 09:17:58 (1788268678) run_llverdev /mnt/lustre/f56a.sanityn -p -s 16000000 -c 4k llverdev: /mnt/lustre/f56a.sanityn is 16777216000000 bytes (15625 GB) in size timestamp: 1788268679 chunksize: 4096 size: 16777216000000 write offset: 16384000000kB 1.19684 MB/s read offset: 16384000000kB inf MB/s llverdev: data verified successfully PASS 56a (55s) == sanityn test 56b: test llverdev and partial verify of wide stripe file ========================================================== 09:18:53 (1788268733) run_llverdev /mnt/lustre/f56b.sanityn -p -s 32000000 -c 4k llverdev: /mnt/lustre/f56b.sanityn is 33554432000000 bytes (31250 GB) in size timestamp: 1788268735 chunksize: 4096 size: 33554432000000 write offset: 32768000000kB 2.54321 MB/s read offset: 32768000000kB inf MB/s llverdev: data verified successfully PASS 56b (54s) == sanityn test 60: Verify data_version behaviour ======== 09:19:47 (1788268787) PASS 60 (3s) == sanityn test 70a: cd directory && rm directory ======== 09:19:51 (1788268791) PASS 70a (4s) == sanityn test 70b: remove files after calling rm_entry ========================================================== 09:19:54 (1788268794) PASS 70b (4s) == sanityn test 71a: correct file map just after write operation is finished ========================================================== 09:19:58 (1788268798) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0128147 s, 5.1 MB/s SKIP: sanityn test_71a ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71a (3s) == sanityn test 71b: check fiemap support for stripecount > 1 ========================================================== 09:20:01 (1788268801) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0161599 s, 4.1 MB/s SKIP: sanityn test_71b ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71b (2s) == sanityn test 71c: check FIEMAP_EXTENT_LAST flag with different extents number ========================================================== 09:20:03 (1788268803) SKIP: sanityn test_71c support only ldiskfs ost SKIP 71c (2s) == sanityn test 71d: fiemap corruption test with fm_extent_count=0 ========================================================== 09:20:05 (1788268805) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.00717279 s, 9.1 MB/s SKIP: sanityn test_71d ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71d (2s) == sanityn test 72: getxattr/setxattr cache should be consistent between nodes ========================================================== 09:20:07 (1788268807) llite.lustre-ffff911148971800.xattr_cache=1 llite.lustre-ffff91116186b000.xattr_cache=1 getfattr: Removing leading '/' from absolute path names user.attr1="value1" getfattr: Removing leading '/' from absolute path names user.attr1="value2" getfattr: Removing leading '/' from absolute path names getfattr: Removing leading '/' from absolute path names b362740ad371eae580e7f85452e8830b - 870e072da1ebfc20ea8c0cd3d6bdf3b6 - PASS 72 (3s) == sanityn test 73: getxattr should not cause xattr lock cancellation ========================================================== 09:20:10 (1788268810) llite.lustre-ffff911148971800.xattr_cache=1 llite.lustre-ffff91116186b000.xattr_cache=1 getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" PASS 73 (3s) == sanityn test 74: flock deadlock: different mounts ======================================================================== 09:20:13 (1788268813) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00983696 s, 104 kB/s 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00510468 s, 201 kB/s 231607: get lock2 231608: get lock1 231608: done 231608: get lock2 231607: done 231607: get lock1 fcntl cmd 7 failed: Resource deadlock avoided 231607: cannot get lock1: Resource deadlock avoided 231608: done 231608: exit rc=0 231607: done 231607: exit rc=0 PASS 74 (7s) == sanityn test 75: osc: upcall after unuse lock============================================================================= 09:20:20 (1788268820) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0346031 s, 60.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0201044 s, 52.2 MB/s fail_loc=0x31d /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 4373: 232188 Killed stat -c %s $DIR1/$tfile fail_loc=0 PASS 75 (8s) == sanityn test 76: Verify MDT open_files listing ======== 09:20:29 (1788268829) opening files: ........ closing files: ........ PASS 76 (35s) == sanityn test 77a: check FIFO NRS policy =============== 09:21:03 (1788268863) ost.OSS.ost.nrs_policies=fifo ost.OSS.ost_create.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_out.nrs_policies=fifo ost.OSS.ost_seq.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.220581 s, 76.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0415549 s, 25.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0339317 s, 30.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0844341 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139631 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109838 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140953 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143535 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13358 s, 7.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135233 s, 7.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126573 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129423 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158975 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156645 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142322 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137742 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176027 s, 6.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff91116186b000.lock_unused_count=3 ldlm.namespaces.lustre-OST0001-osc-ffff91116186b000.lock_unused_count=2 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0118962 s, 88.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0219794 s, 47.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.030762 s, 34.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0439859 s, 23.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0605709 s, 17.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0189994 s, 55.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0203515 s, 51.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0356884 s, 29.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151865 s, 69.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0214076 s, 49.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0118673 s, 88.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128136 s, 81.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0325398 s, 32.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0110828 s, 94.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0180718 s, 58.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155305 s, 67.5 MB/s removed '/mnt/lustre/d77a.sanityn/nrs_r_oleg113-client.virtnet' removed '/mnt/lustre/d77a.sanityn/nrs_w_oleg113-client.virtnet' removed directory '/mnt/lustre/d77a.sanityn' PASS 77a (4s) == sanityn test 77b: check CRR-N NRS policy ============== 09:21:07 (1788268867) ost.OSS.ost.nrs_policies=crrn ost.OSS.ost_create.nrs_policies=crrn ost.OSS.ost_io.nrs_policies=crrn ost.OSS.ost_out.nrs_policies=crrn ost.OSS.ost_seq.nrs_policies=crrn ost.OSS.ost.nrs_crrn_quantum=1 ost.OSS.ost_create.nrs_crrn_quantum=1 ost.OSS.ost_io.nrs_crrn_quantum=1 ost.OSS.ost_out.nrs_crrn_quantum=1 ost.OSS.ost_seq.nrs_crrn_quantum=1 policy: crr-n, crrn_quantum 1 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.191831 s, 87.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0448539 s, 23.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0474271 s, 22.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0438425 s, 23.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0351956 s, 29.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0631614 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.032564 s, 32.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0667766 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0573516 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0312382 s, 33.6 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff91116186b000.lock_unused_count=1 ldlm.namespaces.lustre-OST0000-osc-ffff91116186b000.lock_unused_count=2 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166127 s, 6.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168166 s, 6.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152433 s, 6.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130453 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16705 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161143 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155158 s, 6.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00432678 s, 242 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00359308 s, 292 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00859632 s, 122 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00541136 s, 194 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0589288 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0121907 s, 86.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0031383 s, 334 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0486191 s, 21.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0705868 s, 14.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0324781 s, 32.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0260439 s, 40.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0661698 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0423079 s, 24.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0599926 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0461268 s, 22.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0456513 s, 23.0 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg113-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg113-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost.nrs_crrn_quantum=64 ost.OSS.ost_create.nrs_crrn_quantum=64 ost.OSS.ost_io.nrs_crrn_quantum=64 ost.OSS.ost_out.nrs_crrn_quantum=64 ost.OSS.ost_seq.nrs_crrn_quantum=64 policy: crr-n, crrn_quantum 64 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.193701 s, 86.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0197947 s, 53.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0331499 s, 31.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0383301 s, 27.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0635261 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0712533 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0723173 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0963264 s, 10.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0932593 s, 11.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.091636 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0739268 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0961999 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0752811 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118605 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123795 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130623 s, 8.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff91116186b000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0617675 s, 17.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0298859 s, 35.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01591 s, 65.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0237307 s, 44.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0271087 s, 38.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0270632 s, 38.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0242842 s, 43.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0237617 s, 44.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0249669 s, 42.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0291571 s, 36.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0266911 s, 39.3 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0259802 s, 40.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0342937 s, 30.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0256863 s, 40.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0275891 s, 38.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0370887 s, 28.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0418513 s, 25.1 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg113-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg113-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77b (6s) == sanityn test 77c: check ORR NRS policy ================ 09:21:13 (1788268873) ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_orr_quantum=1 ost.OSS.ost_io.nrs_orr_offset_type=physical ost.OSS.ost_io.nrs_orr_supported=reads policy: orr, orr_quantum 1, orr_offset_type physical, orr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.181018 s, 92.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0297962 s, 35.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0465587 s, 22.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0403814 s, 26.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148199 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152786 s, 6.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143056 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153455 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13455 s, 7.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134001 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150078 s, 7.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162651 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130601 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151328 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169929 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150241 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00952874 s, 110 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00819193 s, 128 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0202822 s, 51.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0200069 s, 52.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167343 s, 6.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.045525 s, 23.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0719204 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0321656 s, 32.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0357291 s, 29.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0705578 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0502943 s, 20.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0374671 s, 28.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00312887 s, 335 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132304 s, 79.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff91116186b000.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff91116186b000.lock_unused_count=7 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00393092 s, 267 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00395796 s, 265 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00319975 s, 328 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg113-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg113-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=writes ost.OSS.ost_io.nrs_orr_quantum=64 policy: orr, orr_quantum 64, orr_offset_type physical, orr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.165122 s, 102 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0385053 s, 27.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0686595 s, 15.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0750023 s, 14.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0929005 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0716513 s, 14.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103259 s, 10.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.084255 s, 12.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0919881 s, 11.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff91116186b000.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0863926 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0859495 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101839 s, 10.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.083462 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0952957 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.088569 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108669 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11568 s, 9.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0211122 s, 49.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0209416 s, 50.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.026704 s, 39.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00594152 s, 176 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0212993 s, 49.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff91116186b000.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00389877 s, 269 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126129 s, 83.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0186902 s, 56.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0311306 s, 33.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144714 s, 72.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0203444 s, 51.5 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00789593 s, 133 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00548456 s, 191 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00447726 s, 234 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0103772 s, 101 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165146 s, 63.5 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg113-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg113-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=reads_and_writes ost.OSS.ost_io.nrs_orr_offset_type=logical policy: orr, orr_quantum 64, orr_offset_type logical, orr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.159967 s, 105 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0405118 s, 25.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.040221 s, 26.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0307748 s, 34.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0334826 s, 31.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0500206 s, 21.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0770212 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0891943 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0760673 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0702271 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102072 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102282 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0738934 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0925675 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125997 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127372 s, 8.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff91116186b000.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff91116186b000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143192 s, 7.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0107864 s, 97.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0337258 s, 31.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162826 s, 64.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00324301 s, 323 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00421606 s, 249 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0259741 s, 40.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.017636 s, 59.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0194479 s, 53.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0192748 s, 54.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0308032 s, 34.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.02316 s, 45.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00903824 s, 116 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0279804 s, 37.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00385411 s, 272 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.024982 s, 42.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0335277 s, 31.3 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg113-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg113-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77c (7s) == sanityn test 77d: check TRR nrs policy ================ 09:21:20 (1788268880) ost.OSS.ost_io.nrs_policies=trr ost.OSS.ost_io.nrs_trr_quantum=1 ost.OSS.ost_io.nrs_trr_offset_type=physical ost.OSS.ost_io.nrs_trr_supported=reads policy: trr, trr_quantum 1, trr_offset_type physical, trr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.249973 s, 67.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0494736 s, 21.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.045703 s, 22.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120608 s, 8.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126131 s, 8.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12207 s, 8.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132215 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138154 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149509 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134357 s, 7.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152675 s, 6.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159848 s, 6.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140349 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185373 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183944 s, 5.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191025 s, 5.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133837 s, 78.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff91116186b000.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff91116186b000.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0422595 s, 24.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.034048 s, 30.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0538607 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00844786 s, 124 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0402399 s, 26.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0041079 s, 255 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0608446 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223187 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0563681 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0578349 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0470629 s, 22.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.047482 s, 22.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0112522 s, 93.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0516007 s, 20.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0113637 s, 92.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158621 s, 66.1 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg113-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg113-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=writes ost.OSS.ost_io.nrs_trr_quantum=64 policy: trr, trr_quantum 64, trr_offset_type physical, trr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.187362 s, 89.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0412879 s, 25.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.030832 s, 34.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0245071 s, 42.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0751972 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125762 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122327 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135692 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126319 s, 8.3 MB/s 1+0 records in 1+0 records out ldlm.namespaces.lustre-OST0000-osc-ffff91116186b000.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff91116186b000.lock_unused_count=3 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161152 s, 6.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15146 s, 6.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144142 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163239 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188434 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195668 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196381 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208798 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0207981 s, 50.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0181908 s, 57.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0429445 s, 24.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151301 s, 69.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.04159 s, 25.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0298156 s, 35.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0240722 s, 43.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167684 s, 62.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0391297 s, 26.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0286332 s, 36.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0299645 s, 35.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0328923 s, 31.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0348998 s, 30.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0354804 s, 29.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0339394 s, 30.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00816462 s, 128 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg113-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg113-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=reads_and_writes ost.OSS.ost_io.nrs_trr_offset_type=logical policy: trr, trr_quantum 64, trr_offset_type logical, trr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.216028 s, 77.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0417229 s, 25.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.033439 s, 31.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0587733 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0725065 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0808873 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0677223 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.089666 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104688 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100182 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0941951 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0872842 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13927 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119 s, 8.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff91116186b000.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff91116186b000.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143676 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141757 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138874 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0272631 s, 38.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0259803 s, 40.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162464 s, 64.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0328919 s, 31.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0193384 s, 54.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0107027 s, 98.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0310314 s, 33.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0247947 s, 42.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0387773 s, 27.0 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0287077 s, 36.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0335885 s, 31.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147334 s, 71.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0304402 s, 34.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0370841 s, 28.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0395115 s, 26.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.035043 s, 29.9 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg113-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg113-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77d (6s) == sanityn test 77e: check TBF NID nrs policy ============ 09:21:26 (1788268886) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_tbf_rule=start localhost nid={0@lo} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start clients nid={192.168.201.13@tcp} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start others nid={*.*.*.*@tcp} rate=50 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.176861 s, 94.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0448866 s, 23.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0291459 s, 36.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0501223 s, 20.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0372403 s, 28.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0395932 s, 26.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0586546 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0580943 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0274185 s, 38.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0425197 s, 24.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0584124 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0752305 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0987216 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0953173 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100714 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10988 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121529 s, 8.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170226 s, 61.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0306932 s, 34.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.018865 s, 55.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0220517 s, 47.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0241644 s, 43.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0247694 s, 42.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.020652 s, 50.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.026576 s, 39.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0239808 s, 43.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0218559 s, 48.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0251428 s, 41.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0359541 s, 29.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0341844 s, 30.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0382947 s, 27.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0317453 s, 33.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0385033 s, 27.2 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg113-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg113-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change localhost rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change clients rate=101 ost.OSS.ost_io.nrs_tbf_rule=change others rate=51 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.173359 s, 96.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0301485 s, 34.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0588075 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0415669 s, 25.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0462754 s, 22.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122738 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119601 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0977091 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108985 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115641 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11968 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110141 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142334 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102484 s, 10.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147173 s, 7.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117155 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143835 s, 7.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff91116186b000.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffff91116186b000.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0246687 s, 42.5 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0533534 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0253969 s, 41.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0354286 s, 29.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0323323 s, 32.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.057624 s, 18.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.051478 s, 20.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0486032 s, 21.6 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0478855 s, 21.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0558373 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0491762 s, 21.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0457014 s, 22.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0766227 s, 13.7 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff91116186b000.lock_unused_count=1 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0599194 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0443463 s, 23.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0543452 s, 19.3 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg113-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg113-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop localhost ost.OSS.ost_io.nrs_tbf_rule=stop clients ost.OSS.ost_io.nrs_tbf_rule=stop others 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.18189 s, 92.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0318804 s, 32.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0392278 s, 26.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0419481 s, 25.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0840056 s, 12.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0833742 s, 12.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106941 s, 9.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0872961 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120366 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0961019 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115296 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103618 s, 10.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123549 s, 8.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110692 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145591 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154636 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138827 s, 7.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00447115 s, 235 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0235295 s, 44.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0294244 s, 35.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0249419 s, 42.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.024977 s, 42.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167348 s, 62.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0308088 s, 34.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0242625 s, 43.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0240193 s, 43.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0384768 s, 27.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0294692 s, 35.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00362451 s, 289 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.04729 s, 22.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00699045 s, 150 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00313351 s, 335 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00704733 s, 149 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff91116186b000.lock_unused_count=4 removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg113-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg113-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.195459 s, 85.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0499202 s, 21.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0444858 s, 23.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0566617 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0767694 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0989735 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122536 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113954 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122391 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144868 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146238 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1363 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142986 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148973 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137779 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15293 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179458 s, 5.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff91116186b000.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159244 s, 65.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152044 s, 69.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00583374 s, 180 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00467303 s, 224 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.019145 s, 54.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0201991 s, 51.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0391724 s, 26.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0313559 s, 33.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0225463 s, 46.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0228753 s, 45.8 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00409691 s, 256 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0324951 s, 32.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0259832 s, 40.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00447186 s, 234 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128231 s, 81.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00390989 s, 268 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg113-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg113-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' PASS 77e (9s) == sanityn test 77f: check TBF JobID nrs policy ========== 09:21:35 (1788268895) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start runas jobid={iozone.500 dd.500 tiotest.500} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start iozone_runas jobid={iozone.500} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg113-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.205824 s, 81.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.032525 s, 32.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0425223 s, 24.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0479793 s, 21.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0387271 s, 27.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0621372 s, 16.9 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0423024 s, 24.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0553945 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.038956 s, 26.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.046758 s, 22.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.091885 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113118 s, 9.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119392 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123159 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0851442 s, 12.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115894 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102802 s, 10.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff91116186b000.lock_unused_count=6 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167053 s, 62.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163202 s, 64.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0202871 s, 51.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145996 s, 71.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0178518 s, 58.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.017528 s, 59.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0349919 s, 30.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0236578 s, 44.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0211071 s, 49.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0353299 s, 29.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0214207 s, 49.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0259027 s, 40.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0266722 s, 39.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0361184 s, 29.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0320487 s, 32.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0319491 s, 32.8 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg113-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg113-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change runas rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change iozone_runas rate=101 ost.OSS.ost_io.nrs_tbf_rule=change dd_runas rate=51 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg113-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.171433 s, 97.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0321248 s, 32.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0294465 s, 35.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.086228 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0986784 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102037 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0858634 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114613 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102399 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10386 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110905 s, 9.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111854 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129027 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118947 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132329 s, 7.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136024 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133903 s, 7.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff91116186b000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff91116186b000.lock_unused_count=2 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150974 s, 69.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00586935 s, 179 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173553 s, 60.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0178836 s, 58.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0238915 s, 43.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0207255 s, 50.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0317831 s, 33.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014677 s, 71.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00467739 s, 224 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00510518 s, 205 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0100307 s, 105 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00418043 s, 251 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00545234 s, 192 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0215884 s, 48.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00241388 s, 434 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00569882 s, 184 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg113-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg113-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop runas ost.OSS.ost_io.nrs_tbf_rule=stop iozone_runas ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg113-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.206553 s, 81.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0275899 s, 38.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0482985 s, 21.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0439484 s, 23.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff91116186b000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0901459 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0811772 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0996112 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.084691 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.093174 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0853037 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0961248 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11925 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110499 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111219 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125907 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117966 s, 8.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130757 s, 8.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff91116186b000.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff91116186b000.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0233094 s, 45.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0250042 s, 41.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0195257 s, 53.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159724 s, 65.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0267241 s, 39.2 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0180543 s, 58.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0185747 s, 56.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00331158 s, 317 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0240216 s, 43.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0237927 s, 44.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0236095 s, 44.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0271937 s, 38.6 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0265385 s, 39.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0232519 s, 45.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00512209 s, 205 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0258443 s, 40.6 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg113-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg113-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_policies=fifo running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg113-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.162639 s, 103 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0266127 s, 39.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0372036 s, 28.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0756469 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0847227 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.094807 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0997549 s, 10.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff91116186b000.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff91116186b000.lock_unused_count=2 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109861 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11617 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129871 s, 8.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113826 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13551 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126463 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13547 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146991 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125294 s, 8.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156328 s, 6.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff91116186b000.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffff91116186b000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139118 s, 75.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0459966 s, 22.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0333844 s, 31.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0345333 s, 30.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.044441 s, 23.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0500671 s, 20.9 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0382234 s, 27.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0026402 s, 397 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0288516 s, 36.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0249505 s, 42.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0245674 s, 42.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0179871 s, 58.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139482 s, 75.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167255 s, 62.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0106439 s, 98.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014743 s, 71.1 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg113-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg113-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' PASS 77f (9s) == sanityn test 77g: Change TBF type directly ============ 09:21:44 (1788268904) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 ost.OSS.ost_io.nrs_policies=fifo PASS 77g (4s) == sanityn test 77h: Wrong policy name should report error, not LBUG ========================================================== 09:21:48 (1788268908) oleg113-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=abc: No such file or directory oleg113-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='abc': No such file or directory pdsh@oleg113-client: oleg113-server: ssh exited with exit code 2 oleg113-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc: Invalid argument oleg113-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc': Invalid argument pdsh@oleg113-client: oleg113-server: ssh exited with exit code 22 oleg113-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf reg abc: Invalid argument oleg113-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf reg abc': Invalid argument pdsh@oleg113-client: oleg113-server: ssh exited with exit code 22 oleg113-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc efg: Invalid argument oleg113-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc efg': Invalid argument pdsh@oleg113-client: oleg113-server: ssh exited with exit code 22 PASS 77h (5s) == sanityn test 77i: Change rank of TBF rule ============= 09:21:53 (1788268913) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start before jobid={jobid} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start after jobid={jobid} rate=1000 rank=default ost.OSS.ost_io.nrs_tbf_rule=start target jobid={jobid} rate=1000 rank=after Move before itself ost.OSS.ost_io.nrs_tbf_rule=change target rank=target Move to higher rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=before Move to lower rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=after Move before default ost.OSS.ost_io.nrs_tbf_rule=change target rank=default ost.OSS.ost_io.nrs_policies=fifo PASS 77i (9s) == sanityn test 77j: check TBF-OPCode NRS policy ========= 09:22:02 (1788268922) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf opcode ost.OSS.ost_io.nrs_tbf_rule=start ost_r opcode={ost_read} rate=5 ost.OSS.ost_io.nrs_tbf_rule=start ost_w opcode={ost_write} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.150273 s, 112 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0282724 s, 37.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0347442 s, 30.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0434276 s, 24.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0374983 s, 28.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0577763 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0790267 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0635315 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.09397 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.080803 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.09077 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0948572 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11463 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114102 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133845 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127018 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121534 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0221765 s, 47.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0263105 s, 39.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0248379 s, 42.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0198175 s, 52.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0221488 s, 47.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0231822 s, 45.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0249279 s, 42.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0250277 s, 41.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.019642 s, 53.4 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0227985 s, 46.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.028251 s, 37.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0263584 s, 39.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0259246 s, 40.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0257303 s, 40.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0299474 s, 35.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0308726 s, 34.0 MB/s removed '/mnt/lustre/d77j.sanityn/nrs_r_oleg113-client.virtnet' removed '/mnt/lustre/d77j.sanityn/nrs_w_oleg113-client.virtnet' removed directory '/mnt/lustre/d77j.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 5 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.96679 s, 21.1 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.4943 s, 5.4 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77j.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_r ost.OSS.ost_io.nrs_tbf_rule=stop ost_w ost.OSS.ost_io.nrs_policies=fifo PASS 77j (47s) == sanityn test 77ja: check TBF-UID/GID NRS policy ======= 09:22:49 (1788268969) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg113-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg113-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg113-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg113-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg113-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.23751 s, 13.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0368117 s, 28.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0510884 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0631025 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0501561 s, 20.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0687066 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0858162 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0745837 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0783185 s, 13.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148157 s, 7.1 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146391 s, 7.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140098 s, 7.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146108 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154316 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155224 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17745 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181019 s, 5.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0254517 s, 41.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0249165 s, 42.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.020078 s, 52.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0271999 s, 38.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0198415 s, 52.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0275928 s, 38.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0247848 s, 42.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0260772 s, 40.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0180542 s, 58.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0244981 s, 42.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0330844 s, 31.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0264618 s, 39.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0297537 s, 35.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0259337 s, 40.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0278333 s, 37.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0350109 s, 29.9 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg113-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg113-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.7325 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.936 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg113-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg113-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg113-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg113-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg113-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.23523 s, 13.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0871566 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0910042 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100324 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0814549 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122403 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0995234 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140768 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12148 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15534 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159488 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193303 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140358 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145321 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182605 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188836 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.226702 s, 4.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0293715 s, 35.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0335238 s, 31.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0229237 s, 45.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0362281 s, 28.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0361228 s, 29.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0346161 s, 30.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0389999 s, 26.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0235838 s, 44.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0348721 s, 30.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0357442 s, 29.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0380291 s, 27.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0379317 s, 27.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0340855 s, 30.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0361273 s, 29.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0481515 s, 21.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0341852 s, 30.7 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg113-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg113-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.7806 s, 5.3 MB/s Write runtime is 20 s, speed is 5.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.951 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77ja (139s) == sanityn test 77jb: check TBF-UID/GID NRS policy on files that don't belong to us ========================================================== 09:25:08 (1788269108) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg113-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg113-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg113-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg113-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg113-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.23806 s, 13.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0623336 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0619385 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0606186 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0599462 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0826925 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0555836 s, 18.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0600414 s, 17.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0808921 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155017 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155125 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170259 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193418 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175501 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194358 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207299 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.217375 s, 4.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0236668 s, 44.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0291558 s, 36.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0261995 s, 40.0 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0261101 s, 40.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0295312 s, 35.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0247943 s, 42.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0244908 s, 42.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.033931 s, 30.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0255612 s, 41.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.029221 s, 35.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0238507 s, 44.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0283563 s, 37.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0267099 s, 39.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0257482 s, 40.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0412015 s, 25.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0313158 s, 33.5 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg113-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg113-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.7565 s, 5.3 MB/s Write runtime is 20 s, speed is 5.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9644 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg113-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg113-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg113-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg113-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg113-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.24812 s, 13.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0633923 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0697497 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0786471 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0847267 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101853 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105485 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0913124 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104845 s, 10.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138233 s, 7.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113306 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142969 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154772 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16537 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139223 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150748 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192148 s, 5.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0262025 s, 40.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0217221 s, 48.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0251918 s, 41.6 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0294442 s, 35.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0343147 s, 30.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0205711 s, 51.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0377161 s, 27.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0382004 s, 27.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0405431 s, 25.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0365741 s, 28.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0411262 s, 25.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0344446 s, 30.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0326786 s, 32.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0340425 s, 30.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0335695 s, 31.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0371894 s, 28.2 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg113-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg113-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.879 s, 5.3 MB/s Write runtime is 20 s, speed is 5.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.2147 s, 5.2 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77jb (145s) == sanityn test 77k: check TBF policy with UID/GID/JobID/OPCode expression ========================================================== 09:27:33 (1788269253) ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg113-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.447986 s, 37.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0389151 s, 26.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0677268 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0738392 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0642591 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164839 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173942 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152708 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185985 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122862 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182335 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.224937 s, 4.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1884 s, 5.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169223 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190371 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.241154 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173567 s, 6.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00754058 s, 139 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0288905 s, 36.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0423919 s, 24.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0194744 s, 53.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0565065 s, 18.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0491729 s, 21.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0324266 s, 32.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00362353 s, 289 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157729 s, 66.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0604759 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0451204 s, 23.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0584225 s, 17.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00510095 s, 206 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0191824 s, 54.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0040063 s, 262 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00220717 s, 475 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg113-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg113-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.68694 s, 18.4 MB/s Write runtime is 7 s, speed is 14.285714 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.84313 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.201.13@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.201.13@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.227332 s, 73.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0369538 s, 28.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0302784 s, 34.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0533226 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0858258 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11259 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0898152 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0885178 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110037 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127036 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125236 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135921 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138877 s, 7.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158336 s, 6.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157776 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198774 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197992 s, 5.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0434978 s, 24.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0414595 s, 25.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0231153 s, 45.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0403004 s, 26.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0520329 s, 20.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.029993 s, 35.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0505609 s, 20.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0334627 s, 31.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0322575 s, 32.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0527994 s, 19.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0344218 s, 30.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0472366 s, 22.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0644759 s, 16.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.040929 s, 25.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0430937 s, 24.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0466488 s, 22.5 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg113-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg113-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.13579 s, 20.4 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.83039 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.201.13@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg113-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.250937 s, 66.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.042765 s, 24.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0485522 s, 21.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125698 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107818 s, 9.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126269 s, 8.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110422 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124038 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151298 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16184 s, 6.5 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12222 s, 8.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164843 s, 6.4 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15977 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151352 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124201 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120949 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16693 s, 6.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0427257 s, 24.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0580544 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0303379 s, 34.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.049118 s, 21.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0322641 s, 32.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0376637 s, 27.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0446906 s, 23.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0704656 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0625485 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0505523 s, 20.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.065404 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0492141 s, 21.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0569041 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0672418 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0907507 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0579857 s, 18.1 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg113-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg113-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.43307 s, 19.3 MB/s Write runtime is 7 s, speed is 14.285714 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.97615 s, 21.1 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg113-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.313224 s, 53.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0536824 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0695225 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0648892 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0750025 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0714883 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0886344 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0851619 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164868 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146505 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180521 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185079 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216819 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183252 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.313608 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.272726 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.296852 s, 3.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0275803 s, 38.0 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.028657 s, 36.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0555676 s, 18.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0465755 s, 22.5 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0524041 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.05378 s, 19.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.065056 s, 16.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0299325 s, 35.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0672189 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0413868 s, 25.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0418091 s, 25.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0444495 s, 23.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0473314 s, 22.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0507499 s, 20.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0828721 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0513669 s, 20.4 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg113-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg113-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.93697 s, 10.6 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.85061 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.54486 s, 18.9 MB/s Write runtime is 7 s, speed is 14.285714 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.82833 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg113-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.20806 s, 80.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0452587 s, 23.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.063505 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0706047 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0369234 s, 28.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0810281 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0774024 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0633636 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0364338 s, 28.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0287239 s, 36.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125156 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117826 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116767 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115586 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14491 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129441 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139525 s, 7.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0223007 s, 47.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0233142 s, 45.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0273141 s, 38.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.023796 s, 44.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0465062 s, 22.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0218648 s, 48.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0332699 s, 31.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0234752 s, 44.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0446117 s, 23.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0430839 s, 24.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0341152 s, 30.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0354885 s, 29.5 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0613993 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0459639 s, 22.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0398992 s, 26.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.053709 s, 19.5 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg113-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg113-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.7017 s, 5.3 MB/s Write runtime is 20 s, speed is 5.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.5313 s, 5.4 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_ug ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg113-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.381081 s, 44.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0460878 s, 22.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0713192 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0802232 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0841919 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110982 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122907 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.214389 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.225846 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222204 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21467 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169356 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.24307 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221232 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169738 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.234865 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213819 s, 4.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0379149 s, 27.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00177548 s, 591 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0609618 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0799707 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0466174 s, 22.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0633394 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0630261 s, 16.6 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.085841 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.044353 s, 23.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0611333 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0281334 s, 37.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0583029 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0262643 s, 39.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0101549 s, 103 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.02542 s, 41.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0175492 s, 59.8 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg113-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg113-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.75582 s, 15.5 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.92993 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg113-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.606549 s, 27.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0543852 s, 19.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.033014 s, 31.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0971655 s, 10.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133965 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0794259 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157646 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156175 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0417377 s, 25.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205087 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0450595 s, 23.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129191 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195308 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.370938 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.347789 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.402237 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.304896 s, 3.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.075263 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0181725 s, 57.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0826547 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0468718 s, 22.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0190176 s, 55.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0959586 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0383109 s, 27.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102401 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00471027 s, 223 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0026278 s, 399 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0261848 s, 40.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0199792 s, 52.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.017575 s, 59.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0124516 s, 84.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162926 s, 64.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00800442 s, 131 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg113-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg113-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0129 s, 10.5 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.87424 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.93726 s, 10.6 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.85557 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_policies=fifo PASS 77k (386s) == sanityn test 77kb: Check different granular TBF type combination: jobid+opcode ========================================================== 09:33:59 (1788269639) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/nrs_r_oleg113-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.364243 s, 46.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0256621 s, 40.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0365963 s, 28.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0394542 s, 26.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11768 s, 8.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121505 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110157 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123365 s, 8.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112391 s, 9.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130214 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126339 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148757 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130665 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125823 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157981 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135351 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134925 s, 7.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0351302 s, 29.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0271926 s, 38.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0275453 s, 38.1 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0279655 s, 37.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0210544 s, 49.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0233027 s, 45.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0233733 s, 44.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0251001 s, 41.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0380537 s, 27.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0287408 s, 36.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0455873 s, 23.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.033893 s, 30.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0338419 s, 31.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0374249 s, 28.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0355323 s, 29.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0381898 s, 27.5 MB/s removed '/mnt/lustre/d77kb.sanityn/nrs_w_oleg113-client.virtnet' removed '/mnt/lustre/d77kb.sanityn/nrs_r_oleg113-client.virtnet' removed directory '/mnt/lustre/d77kb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.40851 s, 19.4 MB/s Write runtime is 7 s, speed is 14.285714 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.83051 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kb (39s) == sanityn test 77kc: Check different granular TBF type combination: nid+opcode ========================================================== 09:34:39 (1788269679) ost.OSS.ost_io.nrs_policies=tbf nid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.201.13@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.201.13@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.27585 s, 60.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0299835 s, 35.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0242731 s, 43.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0875628 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0758883 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104057 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118338 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123375 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11049 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108526 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110089 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117194 s, 8.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107084 s, 9.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130216 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130514 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125771 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122974 s, 8.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130223 s, 80.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0302898 s, 34.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168198 s, 62.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0276269 s, 38.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0263887 s, 39.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0399265 s, 26.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0224093 s, 46.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0270535 s, 38.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0331759 s, 31.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0409219 s, 25.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0317114 s, 33.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0304316 s, 34.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0535895 s, 19.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0328422 s, 31.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0389466 s, 26.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0362544 s, 28.9 MB/s removed '/mnt/lustre/d77kc.sanityn/nrs_w_oleg113-client.virtnet' removed '/mnt/lustre/d77kc.sanityn/nrs_r_oleg113-client.virtnet' removed directory '/mnt/lustre/d77kc.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.3195 s, 19.7 MB/s Write runtime is 7 s, speed is 14.285714 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.81194 s, 10.7 MB/s Read runtime is 10 s, speed is 10.000000 IOPS cleanup_tbf /mnt/lustre/d77kc.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kc (42s) == sanityn test 77kd: Check different granular TBF type combination: nid+jobid ========================================================== 09:35:21 (1788269721) ost.OSS.ost_io.nrs_policies=tbf nid+jobid ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.201.13@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/nrs_r_oleg113-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.444186 s, 37.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0301149 s, 34.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0714254 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113238 s, 9.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0592867 s, 17.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115351 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111682 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0553845 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0799771 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192238 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0925361 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0973951 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151962 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222386 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187844 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.236978 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201759 s, 5.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0367167 s, 28.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0502892 s, 20.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.091933 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00608345 s, 172 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0478974 s, 21.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0105912 s, 99.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0504834 s, 20.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0849864 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0530939 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0541439 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0358353 s, 29.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0201812 s, 52.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.030759 s, 34.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0346952 s, 30.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0618318 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0423081 s, 24.8 MB/s removed '/mnt/lustre/d77kd.sanityn/nrs_w_oleg113-client.virtnet' removed '/mnt/lustre/d77kd.sanityn/nrs_r_oleg113-client.virtnet' removed directory '/mnt/lustre/d77kd.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.62853 s, 18.6 MB/s Write runtime is 7 s, speed is 14.285714 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kd.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.0195 s, 20.9 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kd.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kd (40s) == sanityn test 77ke: Check different granular TBF type combination: jobid+opcode ========================================================== 09:36:01 (1788269761) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/nrs_r_oleg113-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.518946 s, 32.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0255231 s, 41.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0496662 s, 21.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0394882 s, 26.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101628 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131297 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112288 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0928212 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133045 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11387 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120916 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124931 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153305 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118644 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147226 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168347 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125287 s, 8.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0490341 s, 21.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0599401 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0645725 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0636102 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0655606 s, 16.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0720533 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.072155 s, 14.5 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0731938 s, 14.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0797607 s, 13.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0800914 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0795141 s, 13.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.08144 s, 12.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0835812 s, 12.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0990919 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0823556 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0927928 s, 11.3 MB/s removed '/mnt/lustre/d77ke.sanityn/nrs_w_oleg113-client.virtnet' removed '/mnt/lustre/d77ke.sanityn/nrs_r_oleg113-client.virtnet' removed directory '/mnt/lustre/d77ke.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.94905 s, 10.5 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ke.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.78263 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.12588 s, 20.5 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.82459 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77ke (81s) == sanityn test 77kf: Check different granular TBF type combination: uid+gid ========================================================== 09:37:22 (1788269842) ost.OSS.ost_io.nrs_policies=tbf uid+gid ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/nrs_r_oleg113-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.905131 s, 18.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0383414 s, 27.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0550018 s, 19.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0357733 s, 29.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0570339 s, 18.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0511179 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0489234 s, 21.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0519785 s, 20.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0686841 s, 15.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132553 s, 7.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136802 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189372 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209635 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208553 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195749 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202478 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.228491 s, 4.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0461954 s, 22.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0560425 s, 18.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0430393 s, 24.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0579606 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0520102 s, 20.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0438278 s, 23.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0487006 s, 21.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0510886 s, 20.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0516348 s, 20.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0475185 s, 22.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0563145 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0561356 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0730005 s, 14.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.070724 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.058736 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0810238 s, 12.9 MB/s removed '/mnt/lustre/d77kf.sanityn/nrs_w_oleg113-client.virtnet' removed '/mnt/lustre/d77kf.sanityn/nrs_r_oleg113-client.virtnet' removed directory '/mnt/lustre/d77kf.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.6232 s, 5.3 MB/s Write runtime is 20 s, speed is 5.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kf.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.954 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77kf.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kf (67s) == sanityn test 77kg: check different granular TBF type combination: uid+opcode ========================================================== 09:38:29 (1788269909) ost.OSS.ost_io.nrs_policies=tbf uid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg113-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.308606 s, 54.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0568945 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0416331 s, 25.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0607217 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120751 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154374 s, 6.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177717 s, 5.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194017 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190642 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156249 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165268 s, 6.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186172 s, 5.6 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20007 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148238 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189659 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198113 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220542 s, 4.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0388067 s, 27.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0646837 s, 16.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0497579 s, 21.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0517053 s, 20.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0374313 s, 28.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0567646 s, 18.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0411663 s, 25.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0602066 s, 17.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0606509 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0597445 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0457071 s, 22.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0602212 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0505001 s, 20.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0517341 s, 20.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0705702 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0546134 s, 19.2 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg113-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg113-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.31341 s, 19.7 MB/s Write runtime is 7 s, speed is 14.285714 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.82136 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg113-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.185999 s, 90.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0319487 s, 32.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0356683 s, 29.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.02797 s, 37.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0753809 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0792295 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101138 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0678384 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0955746 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0991836 s, 10.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10088 s, 10.4 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105098 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0991438 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127701 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141882 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132357 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144257 s, 7.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00784306 s, 134 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0213489 s, 49.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00969748 s, 108 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0289181 s, 36.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141513 s, 74.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00225561 s, 465 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0217532 s, 48.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00747003 s, 140 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0310884 s, 33.7 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00829068 s, 126 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0644312 s, 16.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0287436 s, 36.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0384959 s, 27.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0392168 s, 26.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0490279 s, 21.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0533451 s, 19.7 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg113-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg113-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.1032 s, 10.4 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.90591 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.94028 s, 10.5 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.82229 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kg (123s) == sanityn test 77kh: Verify that Project ID support for NRS TBF rule ========================================================== 09:40:33 (1788270033) using SAVE_PROJECT_SUPPORTED=0 Stopping clients: oleg113-client.virtnet /mnt/lustre (opts:) Stopping client oleg113-client.virtnet /mnt/lustre opts: Stopping clients: oleg113-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg113-client.virtnet /mnt/lustre2 opts: Stopping /mnt/lustre-mds1 (opts:-f) on oleg113-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg113-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg113-server sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime) proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) devtmpfs on /dev type devtmpfs (rw,nosuid,mode=755) securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime) tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev) devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000) tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755) tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755) cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd) pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime) bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700) cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio) cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma) cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct) cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event) cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb) cgroup on /sys/fs/cgroup/misc type cgroup (rw,nosuid,nodev,noexec,relatime,misc) cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio) cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer) cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset) cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids) cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory) cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices) none on /sys/kernel/tracing type tracefs (rw,relatime) configfs on /sys/kernel/config type configfs (rw,relatime) /dev/nbd0 on / type ext4 (ro,relatime,stripe=32) rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M) debugfs on /sys/kernel/debug type debugfs (rw,relatime) mqueue on /dev/mqueue type mqueue (rw,relatime) none on /mnt type ramfs (rw,relatime) /dev/vda on /home/green/git/lustre-release type squashfs (ro,relatime) none on /var/lib/stateless/writable type tmpfs (rw,relatime) none on /var/cache/man type tmpfs (rw,relatime) none on /var/log type tmpfs (rw,relatime) none on /var/lib/dbus type tmpfs (rw,relatime) none on /tmp type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) none on /var/tmp type tmpfs (rw,relatime) none on /var/lib/NetworkManager type tmpfs (rw,relatime) none on /var/lib/systemd/random-seed type tmpfs (rw,relatime) none on /var/spool type tmpfs (rw,relatime) none on /var/lib/nfs type tmpfs (rw,relatime) none on /var/lib/gssproxy type tmpfs (rw,relatime) none on /var/lib/logrotate type tmpfs (rw,relatime) none on /etc type tmpfs (rw,relatime) none on /var/lib/rsyslog type tmpfs (rw,relatime) none on /root/.ssh type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) 192.168.200.253:/exports/state/oleg113-client.virtnet on /var/lib/stateless/state type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.201.13,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg113-client.virtnet/boot on /boot type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.201.13,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg113-client.virtnet/etc/kdump.conf on /etc/kdump.conf type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.201.13,local_lock=none,addr=192.168.200.253) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=378180k,mode=700) 192.168.200.253:/exports/testreports/69293/testresults/sanityn-zfs-rocky8.10_x86_64-rocky8.10_x86_64 on /tmp/testlogs type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.201.13,local_lock=none,addr=192.168.200.253) /dev/vda on /usr/sbin/mount.lustre type squashfs (ro,relatime) Checking servers environments Checking clients oleg113-client.virtnet environments Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg113-server' oleg113-server: oleg113-server.virtnet: executing load_modules_local oleg113-server: kmemleak disabled oleg113-server: Loading modules from /home/green/git/lustre-release/lustre oleg113-server: detected 4 online CPUs by sysfs oleg113-server: MODOPTS_LIBCFS= oleg113-server: Force libcfs to create 2 CPU partitions oleg113-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Setup mgs, mdt, osts Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg113-server: oleg113-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Started lustre-MDT0000 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg113-server: oleg113-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Started lustre-OST0000 Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg113-server: oleg113-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Started lustre-OST0001 Starting client: oleg113-client.virtnet: -o user_xattr,flock 192.168.201.113@tcp:/lustre /mnt/lustre Starting client oleg113-client.virtnet: -o user_xattr,flock 192.168.201.113@tcp:/lustre /mnt/lustre Started clients oleg113-client.virtnet: 192.168.201.113@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Starting client: oleg113-client.virtnet: -o user_xattr,flock 192.168.201.113@tcp:/lustre /mnt/lustre2 Starting client oleg113-client.virtnet: -o user_xattr,flock 192.168.201.113@tcp:/lustre /mnt/lustre2 Started clients oleg113-client.virtnet: 192.168.201.113@tcp:/lustre on /mnt/lustre2 type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff911158430000.idle_timeout=debug osc.lustre-OST0000-osc-ffff91117235c000.idle_timeout=debug osc.lustre-OST0001-osc-ffff911158430000.idle_timeout=debug osc.lustre-OST0001-osc-ffff91117235c000.idle_timeout=debug disable quota as required ost.OSS.ost_io.nrs_policies=tbf projid ost.OSS.ost_io.nrs_tbf_rule=start ext_proj_rw projid={100} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.352917 s, 47.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0297915 s, 35.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0516445 s, 20.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0846168 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0876374 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123339 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124799 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13962 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14102 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159128 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172206 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137891 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171194 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196267 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174798 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227293 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.226897 s, 4.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0274756 s, 38.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0177822 s, 59.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0318535 s, 32.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0291475 s, 36.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0434272 s, 24.1 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0222476 s, 47.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0348519 s, 30.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0254399 s, 41.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0286715 s, 36.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0270837 s, 38.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.028305 s, 37.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0279703 s, 37.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0324334 s, 32.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0346912 s, 30.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0372311 s, 28.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0400957 s, 26.2 MB/s removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg113-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg113-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.35454 s, 19.6 MB/s Write runtime is 7 s, speed is 14.285714 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.01958 s, 20.9 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_proj_rw ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf projid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w projid={100}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r projid={100}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.31181 s, 53.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0309127 s, 33.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0528955 s, 19.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0260383 s, 40.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0460536 s, 22.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0473904 s, 22.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0509841 s, 20.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0745294 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0628209 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10662 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.091563 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10105 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1267 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197014 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.204679 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.226762 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.217105 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0336827 s, 31.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0313475 s, 33.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00633148 s, 166 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00896485 s, 117 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00256274 s, 409 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0505145 s, 20.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0405675 s, 25.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0552246 s, 19.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.048548 s, 21.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0250089 s, 41.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.032805 s, 32.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00718768 s, 146 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00645557 s, 162 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0273505 s, 38.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164572 s, 63.7 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0328213 s, 31.9 MB/s removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg113-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg113-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.85596 s, 17.9 MB/s Write runtime is 7 s, speed is 14.285714 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.82283 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77kh (142s) == sanityn test 77ki: Add rule with unsupported TBF type should fail for generic TBF ========================================================== 09:42:55 (1788270175) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 oleg113-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_fail projid={100}&opcode={ost_write} rate=20: Invalid argument oleg113-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_fail projid={100}&opcode={ost_write} rate=20': Invalid argument pdsh@oleg113-client: oleg113-server: ssh exited with exit code 22 ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77ki (11s) == sanityn test 77kj: Verify nodemap support for NRS TBF rule ========================================================== 09:43:06 (1788270186) On MGS 192.168.201.113, active = nodemap.active=1 On MGS 192.168.201.113, TBF = ost.OSS.ost_io.nrs_policies=tbf nodemap ost.OSS.ost_io.nrs_tbf_rule=start ext_nm_rw nodemap={TBF} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.342087 s, 49.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0291889 s, 35.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0361683 s, 29.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.034556 s, 30.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0899835 s, 11.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0999239 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111572 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108844 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125155 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116756 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115593 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125494 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136054 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124345 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116708 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189804 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197854 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0276451 s, 37.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0405473 s, 25.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0304465 s, 34.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0605009 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0371234 s, 28.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.035681 s, 29.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0380175 s, 27.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0590078 s, 17.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0476793 s, 22.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0476506 s, 22.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0474678 s, 22.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0477863 s, 21.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0691338 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0534666 s, 19.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0562036 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0623499 s, 16.8 MB/s removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg113-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg113-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.40914 s, 19.4 MB/s Write runtime is 7 s, speed is 14.285714 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.97194 s, 21.1 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete On MGS 192.168.201.113, TBF.id = On MGS 192.168.201.113, active = nodemap.active=0 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.26652 s, 62.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0464042 s, 22.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0796997 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10561 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112111 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119565 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0980423 s, 10.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120649 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131583 s, 8.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162368 s, 6.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126253 s, 8.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121296 s, 8.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148562 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115082 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136581 s, 7.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0958226 s, 10.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff91117235c000.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119127 s, 8.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0194222 s, 54.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131607 s, 79.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140174 s, 74.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0198089 s, 52.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161451 s, 64.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0225822 s, 46.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0352306 s, 29.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0322609 s, 32.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0259751 s, 40.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.020216 s, 51.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0327654 s, 32.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0218984 s, 47.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0280918 s, 37.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0248227 s, 42.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0237012 s, 44.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0291711 s, 35.9 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff91117235c000.lock_unused_count=4 removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg113-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg113-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 1 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.03842 s, 26.0 MB/s Write runtime is 5 s, speed is 20.000000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 1.38144 s, 75.9 MB/s Read runtime is 3 s, speed is 33.333333 IOPS read rate (33.333333) should exceed limit (20 * 1) cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_nm_rw ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf nodemap+opcode oleg113-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_w nodemap={TBF}&opcode={ost_write} rate=20: Invalid argument oleg113-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_w nodemap={TBF}&opcode={ost_write} rate=20': Invalid argument oleg113-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_r nodemap={TBF}&opcode={ost_read} rate=10: Invalid argument oleg113-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_r nodemap={TBF}&opcode={ost_read} rate=10': Invalid argument pdsh@oleg113-client: oleg113-server: ssh exited with exit code 22 On MGS 192.168.201.113, active = nodemap.active=1 On MGS 192.168.201.113, TBF = ost.OSS.ost_io.nrs_tbf_rule=start ext_w nodemap={TBF}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nodemap={TBF}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.292676 s, 57.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0362053 s, 29.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0318076 s, 33.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115861 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12895 s, 8.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146121 s, 7.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132646 s, 7.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118589 s, 8.8 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137477 s, 7.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132176 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145734 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138194 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146024 s, 7.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23813 s, 4.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.243344 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.26616 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.262801 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0506351 s, 20.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0358687 s, 29.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00375193 s, 279 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0366249 s, 28.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0358579 s, 29.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0367424 s, 28.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0271238 s, 38.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0323591 s, 32.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0427729 s, 24.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0413494 s, 25.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00638578 s, 164 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0266678 s, 39.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0498595 s, 21.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.050194 s, 20.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0504779 s, 20.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0678889 s, 15.4 MB/s removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg113-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg113-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.05626 s, 20.7 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.77566 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo On MGS 192.168.201.113, TBF.id = On MGS 192.168.201.113, active = nodemap.active=0 PASS 77kj (134s) == sanityn test 77l: check the output of NRS policies for generic TBF ========================================================== 09:45:20 (1788270320) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_policies=fifo PASS 77l (12s) == sanityn test 77m: check NRS Delay slows write RPC processing ========================================================== 09:45:32 (1788270332) ost.OSS.ost_io.nrs_policies=delay ost.OSS.ost_io.nrs_delay_min=4 ost.OSS.ost_io.nrs_delay_max=4 ost.OSS.ost_io.nrs_delay_pct=100 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg113-client.virtnet] [bs=1M] [count=1] [oflag=direct] [conv=fdatasync] 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 3.35013 s, 313 kB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg113-client.virtnet] [bs=1M] [count=10] [oflag=direct] [conv=fdatasync] 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 40.0224 s, 262 kB/s ost.OSS.ost_io.nrs_policies=fifo PASS 77m (55s) == sanityn test 77n: check wildcard support for TBF JobID NRS policy ========================================================== 09:46:27 (1788270387) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={*.500} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.279319 s, 60.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0381464 s, 27.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0884363 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0733964 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114412 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122517 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141506 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147161 s, 7.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140403 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187645 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13842 s, 7.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183402 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14142 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161901 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196768 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206376 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.244237 s, 4.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff91117235c000.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff91117235c000.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00651534 s, 161 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00786113 s, 133 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0108191 s, 96.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0104916 s, 99.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0534134 s, 19.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0934881 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144313 s, 72.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.027377 s, 38.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0211429 s, 49.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0215391 s, 48.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0235835 s, 44.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0242595 s, 43.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0273755 s, 38.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0293037 s, 35.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0314023 s, 33.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0329769 s, 31.8 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg113-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg113-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77n.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.23246 s, 16.8 MB/s Write runtime is 7 s, speed is 14.285714 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77n.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.09497 s, 20.6 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.*} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.443252 s, 37.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0414086 s, 25.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0667583 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0683687 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142229 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14576 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0982642 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103349 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129175 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158878 s, 6.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143852 s, 7.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207164 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190274 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18157 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190393 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179404 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.214386 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0248965 s, 42.1 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0357838 s, 29.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0649306 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0638656 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0504088 s, 20.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0310422 s, 33.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0473197 s, 22.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0292647 s, 35.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00474808 s, 221 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0538603 s, 19.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0483866 s, 21.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0274193 s, 38.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0713306 s, 14.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0919373 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.055906 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0564167 s, 18.6 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg113-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg113-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.7393 s, 18.3 MB/s Write runtime is 7 s, speed is 14.285714 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.0739 s, 20.7 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_policies=fifo PASS 77n (79s) == sanityn test 77o: Changing rank should not panic ====== 09:47:46 (1788270466) mds.MDS.mdt.nrs_policies=tbf nid mds.MDS.mdt.nrs_tbf_rule=start name nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=start name1 nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=change name1 rank=name mds.MDS.mdt.nrs_tbf_rule=stop name mds.MDS.mdt.nrs_policies=fifo PASS 77o (8s) == sanityn test 77q: Parallel TBF rule definitions should not panic ========================================================== 09:47:54 (1788270474) mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_policies=fifo PASS 77q (77s) == sanityn test 77p: Check validity of rule names for TBF policies ========================================================== 09:49:11 (1788270551) mds.MDS.mdt.nrs_policies=tbf oleg113-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_overflo uid={500} rate=500: File name too long oleg113-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_overflo uid={500} rate=500': File name too long pdsh@oleg113-client: oleg113-server: ssh exited with exit code 36 oleg113-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start uid={500} rate=500: Invalid argument oleg113-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start uid={500} rate=500': Invalid argument pdsh@oleg113-client: oleg113-server: ssh exited with exit code 22 oleg113-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_empty: Invalid argument oleg113-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_empty': Invalid argument pdsh@oleg113-client: oleg113-server: ssh exited with exit code 22 oleg113-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p@spec uid={500} rate=500: Invalid argument oleg113-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p@spec uid={500} rate=500': Invalid argument pdsh@oleg113-client: oleg113-server: ssh exited with exit code 22 oleg113-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p.spec uid={500} rate=500: Invalid argument oleg113-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p.spec uid={500} rate=500': Invalid argument pdsh@oleg113-client: oleg113-server: ssh exited with exit code 22 oleg113-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p~spec uid={500} rate=500: Invalid argument oleg113-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p~spec uid={500} rate=500': Invalid argument pdsh@oleg113-client: oleg113-server: ssh exited with exit code 22 oleg113-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p#spec uid={500} rate=500: Invalid argument oleg113-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p#spec uid={500} rate=500': Invalid argument pdsh@oleg113-client: oleg113-server: ssh exited with exit code 22 oleg113-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p/spec uid={500} rate=500: Invalid argument oleg113-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p/spec uid={500} rate=500': Invalid argument pdsh@oleg113-client: oleg113-server: ssh exited with exit code 22 oleg113-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p^spec uid={500} rate=500: Invalid argument oleg113-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p^spec uid={500} rate=500': Invalid argument pdsh@oleg113-client: oleg113-server: ssh exited with exit code 22 oleg113-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p%spec uid={500} rate=500: Invalid argument oleg113-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p%spec uid={500} rate=500': Invalid argument pdsh@oleg113-client: oleg113-server: ssh exited with exit code 22 oleg113-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p*spec uid={500} rate=500: Invalid argument oleg113-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p*spec uid={500} rate=500': Invalid argument pdsh@oleg113-client: oleg113-server: ssh exited with exit code 22 oleg113-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p;spec uid={500} rate=500: Invalid argument oleg113-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p;spec uid={500} rate=500': Invalid argument pdsh@oleg113-client: oleg113-server: ssh exited with exit code 22 oleg113-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p,spec uid={500} rate=500: Invalid argument oleg113-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p,spec uid={500} rate=500': Invalid argument pdsh@oleg113-client: oleg113-server: ssh exited with exit code 22 oleg113-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p?spec uid={500} rate=500: Invalid argument oleg113-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p?spec uid={500} rate=500': Invalid argument pdsh@oleg113-client: oleg113-server: ssh exited with exit code 22 oleg113-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77pspec uid={500} rate=500: Invalid argument oleg113-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p>spec uid={500} rate=500': Invalid argument pdsh@oleg113-client: oleg113-server: ssh exited with exit code 22 oleg113-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p:spec uid={500} rate=500: Invalid argument oleg113-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p:spec uid={500} rate=500': Invalid argument pdsh@oleg113-client: oleg113-server: ssh exited with exit code 22 oleg113-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p+spec uid={500} rate=500: Invalid argument oleg113-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p+spec uid={500} rate=500': Invalid argument pdsh@oleg113-client: oleg113-server: ssh exited with exit code 22 oleg113-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p=spec uid={500} rate=500: Invalid argument oleg113-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p=spec uid={500} rate=500': Invalid argument pdsh@oleg113-client: oleg113-server: ssh exited with exit code 22 oleg113-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p)spec uid={500} rate=500: Invalid argument oleg113-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p)spec uid={500} rate=500': Invalid argument pdsh@oleg113-client: oleg113-server: ssh exited with exit code 22 oleg113-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p(spec uid={500} rate=500: Invalid argument oleg113-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p(spec uid={500} rate=500': Invalid argument pdsh@oleg113-client: oleg113-server: ssh exited with exit code 22 oleg113-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p{spec uid={500} rate=500: Invalid argument oleg113-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p{spec uid={500} rate=500': Invalid argument pdsh@oleg113-client: oleg113-server: ssh exited with exit code 22 oleg113-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p}spec uid={500} rate=500: Invalid argument oleg113-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p}spec uid={500} rate=500': Invalid argument pdsh@oleg113-client: oleg113-server: ssh exited with exit code 22 oleg113-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p|spec uid={500} rate=500: Invalid argument oleg113-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p|spec uid={500} rate=500': Invalid argument pdsh@oleg113-client: oleg113-server: ssh exited with exit code 22 oleg113-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p[spec uid={500} rate=500: Invalid argument oleg113-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p[spec uid={500} rate=500': Invalid argument pdsh@oleg113-client: oleg113-server: ssh exited with exit code 22 oleg113-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p]spec uid={500} rate=500: Invalid argument oleg113-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p]spec uid={500} rate=500': Invalid argument pdsh@oleg113-client: oleg113-server: ssh exited with exit code 22 oleg113-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p!spec uid={500} rate=500: Invalid argument oleg113-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p!spec uid={500} rate=500': Invalid argument pdsh@oleg113-client: oleg113-server: ssh exited with exit code 22 oleg113-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p&spec uid={500} rate=500: Invalid argument oleg113-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p&spec uid={500} rate=500': Invalid argument pdsh@oleg113-client: oleg113-server: ssh exited with exit code 22 oleg113-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p$spec uid={500} rate=500: Invalid argument oleg113-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p$spec uid={500} rate=500': Invalid argument pdsh@oleg113-client: oleg113-server: ssh exited with exit code 22 oleg113-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p`spec uid={500} rate=500: Invalid argument oleg113-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p`spec uid={500} rate=500': Invalid argument pdsh@oleg113-client: oleg113-server: ssh exited with exit code 22 oleg113-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p\spec uid={500} rate=500: Invalid argument oleg113-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p\spec uid={500} rate=500': Invalid argument pdsh@oleg113-client: oleg113-server: ssh exited with exit code 22 mds.MDS.mdt.nrs_policies=fifo PASS 77p (25s) == sanityn test 77r: Change type of tbf policy at run time ========================================================== 09:49:36 (1788270576) mds.MDS.mdt.nrs_policies=tbf uid mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf gid mds.MDS.mdt.nrs_tbf_rule=start md_rule gid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start md_rule jobid={*.500} rate=1 mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=fifo check the number of requests in queue: fifo: 1 crrn: 0 tbf: 0 delay: 0 fifo: 0 crrn: 0 tbf: 0 delay: 0 cleanup 77r 275654 kill md_thread xargs PASS 77r (50s) == sanityn test 77s: Check TBF LRU shrinker ============== 09:50:26 (1788270626) jobid_name=%e.%p jobid_var=TEST_77s_JOBVAR mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start touch jobid={touch.*} rate=1000 mds.MDS.mdt.nrs_tbf_rule=start unlink jobid={unlink.*} rate=1000 touch jobid={touch.*} 1000, ref 9 touch jobid={touch.*} 1000, ref 0 unlink jobid={unlink.*} 1000, ref 9 unlink jobid={unlink.*} 1000, ref 0 mds.MDS.mdt.nrs_policies=fifo jobid_var=procname_uid jobid_name=%e.%u PASS 77s (46s) == sanityn test 78: Enable policy and specify tunings right away ========================================================== 09:51:12 (1788270672) oleg113-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_orr_quantum=1: No such device oleg113-server: error: set_param: setting 'ost/OSS/*/nrs_orr_quantum'='1': No such device pdsh@oleg113-client: oleg113-server: ssh exited with exit code 19 ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_policies=fifo PASS 78 (9s) == sanityn test 79: xattr: intent error ================== 09:51:21 (1788270681) File: /mnt/lustre/d79.sanityn Size: 16384 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115238826935767 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-01 09:51:23.000000000 -0400 Modify: 2026-09-01 09:51:23.000000000 -0400 Change: 2026-09-01 09:51:23.000000000 -0400 Birth: 2026-09-01 09:51:23.000000000 -0400 fail_val=0 fail_loc=0x80000160 fail_val=0 fail_loc=0x80000131 # file: mnt/lustre/d79.sanityn trusted.name1="value1" PASS 79 (17s) == sanityn test 80a: migrate directory when some children is being opened ========================================================== 09:51:38 (1788270698) SKIP: sanityn test_80a needs >= 2 MDTs SKIP 80a (2s) == sanityn test 80b: Accessing directory during migration ========================================================== 09:51:40 (1788270700) SKIP: sanityn test_80b needs >= 2 MDTs SKIP 80b (3s) == sanityn test 81a: rename and stat under striped directory ========================================================== 09:51:43 (1788270703) SKIP: sanityn test_81a needs >= 2 MDTs SKIP 81a (2s) == sanityn test 81b: rename under striped directory doesn't deadlock ========================================================== 09:51:46 (1788270706) SKIP: sanityn test_81b We need at least 2 MDTs for this test SKIP 81b (2s) == sanityn test 81c: rename revoke LOOKUP lock for remote object ========================================================== 09:51:48 (1788270708) SKIP: sanityn test_81c needs >= 4 MDTs SKIP 81c (3s) == sanityn test 81d: parallel rename file cross-dir on same MDT ========================================================== 09:51:51 (1788270711) - open/close 962 (time 1788270724.97 total 10.00 last 96.15) - open/close 2044 (time 1788270734.97 total 20.01 last 108.17) total: 2500 open/close in 24.29 seconds: 102.92 ops/second snapshot_time 1788270795.681604925 secs.nsecs start_time 1788270713.560589900 secs.nsecs elapsed_time 82.121015025 secs.nsecs open 2505 samples [usecs] 134 36532 2881931 6258746321 close 2504 samples [usecs] 118 12961 780290 600016592 mknod 2501 samples [usecs] 487 36432 2548823 5264178495 unlink 2500 samples [usecs] 647 53760 4460403 43194730073 mkdir 2 samples [usecs] 1186 1190 2376 2822696 rmdir 2 samples [usecs] 6401 13359 19760 219435682 rename 2500 samples [usecs] 362 36877 11370145 103569862673 getattr 3761 samples [usecs] 6 5364 150629 135770605 parallel_rename_file 2500 samples [usecs] 362 36877 11370145 103569862673 crossdir_rename 2500 samples [usecs] 362 36877 11370145 103569862673 readpage 36 samples [usecs] 3281 46358 681451 17090009367 PASS 81d (87s) == sanityn test 82: fsetxattr and fgetxattr on orphan files ========================================================== 09:53:19 (1788270799) multiop /mnt/lustre/f82.sanityn vO_ac TMPPIPE=/tmp/multiop_open_wait_pipe.7626 multiop /mnt/lustre2/f82.sanityn vo_Ac TMPPIPE=/tmp/multiop_open_wait_pipe.7626 PASS 82 (5s) == sanityn test 83: access striped directory while it is being created/unlinked ========================================================== 09:53:23 (1788270803) SKIP: sanityn test_83 needs >= 2 MDTs SKIP 83 (2s) == sanityn test 84: 0-nlink race in lu_object_find() ===== 09:53:25 (1788270805) fail_loc=0x8000060b PASS 84 (10s) == sanityn test 85: Lustre API root cache race =========== 09:53:35 (1788270815) Starting: llapi_root_test /mnt/lustre/f85.sanityn /mnt/lustre2/f85.sanityn Starting test test1 at 1788270816 Finishing test test1 at 1788270819 PASS 85 (7s) == sanityn test 90: open/create and unlink striped directory ========================================================== 09:53:42 (1788270822) SKIP: sanityn test_90 needs >= 2 MDTs SKIP 90 (2s) == sanityn test 91: chmod and unlink striped directory === 09:53:44 (1788270824) SKIP: sanityn test_91 needs >= 2 MDTs SKIP 91 (2s) == sanityn test 92: create remote directory under orphan directory ========================================================== 09:53:47 (1788270827) SKIP: sanityn test_92 needs >= 2 MDTs SKIP 92 (3s) == sanityn test 93: alloc_rr should not allocate on same ost ========================================================== 09:53:49 (1788270829) fail_loc=0x00000163 fail_loc=0x0 /mnt/lustre/f93.sanityn-1/file1 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5880 0x16f8 0x240000400 1 5842 0x16d2 0x280000400 /mnt/lustre2/f93.sanityn-2/file2 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5881 0x16f9 0x240000400 1 5843 0x16d3 0x280000400 rc1=0 and rc2=0 PASS 93 (12s) == sanityn test 94: signal vs CP callback race =========== 09:54:01 (1788270841) osc.lustre-OST0000-osc-ffff911158430000.idle_timeout=0 osc.lustre-OST0000-osc-ffff91117235c000.idle_timeout=0 osc.lustre-OST0001-osc-ffff911158430000.idle_timeout=0 osc.lustre-OST0001-osc-ffff91117235c000.idle_timeout=0 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.177561 s, 46.1 kB/s fail_val=5 fail_loc=0x80000312 fail_val=6 fail_loc=0x80000329 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 3.09368 s, 1.3 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 6389: 292885 Alarm clock dd if=/dev/zero of=$DIR/$tfile conv=notrunc oflag=append bs=4k count=1 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0378288 s, 108 kB/s osc.lustre-OST0000-osc-ffff911158430000.idle_timeout=debug osc.lustre-OST0000-osc-ffff91117235c000.idle_timeout=debug osc.lustre-OST0001-osc-ffff911158430000.idle_timeout=debug osc.lustre-OST0001-osc-ffff91117235c000.idle_timeout=debug PASS 94 (13s) == sanityn test 95a: Check readpage() on a page that was removed from page cache ========================================================== 09:54:14 (1788270854) llite.lustre-ffff911158430000.fast_read=0 llite.lustre-ffff91117235c000.fast_read=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.042423 s, 49.4 MB/s fail_loc=0x80001421 fail_val=7 fail_loc=0x80001422 fail_val=10 llite.lustre-ffff911158430000.fast_read=1 llite.lustre-ffff91117235c000.fast_read=1 PASS 95a (16s) == sanityn test 95b: Check readpage() on a page that is no longer uptodate ========================================================== 09:54:30 (1788270870) llite.lustre-ffff911158430000.fast_read=0 llite.lustre-ffff91117235c000.fast_read=0 1+0 records in 1+0 records out 12288 bytes (12 kB, 12 KiB) copied, 0.00957949 s, 1.3 MB/s fail_loc=0x80001424 fail_val=5 fail_loc=0 llite.lustre-ffff911158430000.fast_read=1 llite.lustre-ffff91117235c000.fast_read=1 PASS 95b (10s) == sanityn test 100a: DoM: glimpse RPCs for stat without IO lock (DoM only file) ========================================================== 09:54:40 (1788270880) SKIP: sanityn test_100a Reserved for glimpse-ahead SKIP 100a (2s) == sanityn test 100b: DoM: no glimpse RPC for stat with IO lock (DoM only file) ========================================================== 09:54:42 (1788270882) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00247585 s, 1.7 MB/s /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK PASS 100b (3s) == sanityn test 100c: DoM: write vs stat without IO lock (combined file) ========================================================== 09:54:45 (1788270885) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0936682 s, 22.4 MB/s /mnt/lustre/d100c.sanityn/dom has type file OK /mnt/lustre/d100c.sanityn/dom has size 2097152 OK PASS 100c (3s) == sanityn test 100d: DoM: write+truncate vs stat without IO lock (combined file) ========================================================== 09:54:49 (1788270889) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0530093 s, 39.6 MB/s /mnt/lustre/d100d.sanityn/dom has type file OK /mnt/lustre/d100d.sanityn/dom has size 4096 OK PASS 100d (3s) == sanityn test 100e: DoM: read on open and file size ==== 09:54:52 (1788270892) 1+0 records in 1+0 records out 12000 bytes (12 kB, 12 KiB) copied, 0.00911434 s, 1.3 MB/s PASS 100e (3s) == sanityn test 101a: Discard DoM data on unlink ========= 09:54:55 (1788270895) /mnt/lustre/f101a.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00297708 s, 1.4 MB/s PASS 101a (3s) == sanityn test 101b: Discard DoM data on rename ========= 09:54:58 (1788270898) /mnt/lustre/f101b.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0161595 s, 253 kB/s PASS 101b (4s) == sanityn test 101c: Discard DoM data on close-unlink === 09:55:02 (1788270902) /mnt/lustre/f101c.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00366012 s, 1.1 MB/s PASS 101c (5s) == sanityn test 102: Test open by handle of unlinked file ========================================================== 09:55:07 (1788270907) file: /mnt/lustre/f102.sanityn fh_bytes: 32 fh_type: 151 fh_data: d1 0b 00 00 02 00 00 00 b8 0f 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 file's parent FID is [0:0x0:0x0] file FID is [0x200000bd1:0xfb8:0x0] access via mount point '/mnt/lustre2' - OK access after unlink - OK check_fhandle_syscalls test Passed! PASS 102 (4s) == sanityn test 103: Test size correctness with lockahead ========================================================== 09:55:11 (1788270911) fail_loc=0x415 fail_loc=0x415 fail_val=2 Incorrect size expected (no glimpse fix): Starting test test23 at 1788270912 lockahead_test: lockahead_test.c:1171: test23: assertion 'sb.st_size == sb2.st_size' failed: size on /mnt/lustre/d103.sanityn/f103.sanityn and /mnt/lustre2/d103.sanityn/f103.sanityn differs: 0 vs 1048576 fail_loc=0 fail_loc=0x214 fail_val=2 Starting test test23 at 1788270915 Finishing test test23 at 1788270915 PASS 103 (9s) == sanityn test 104: Verify that MDS stores atime/mtime/ctime during close ========================================================== 09:55:20 (1788270920) SKIP: sanityn test_104 ldiskfs only test SKIP 104 (1s) == sanityn test 105: Glimpse and lock cancel race ======== 09:55:21 (1788270921) fail_loc=0x416 PASS 105 (19s) == sanityn test 106a: Verify the btime via statx() ======= 09:55:40 (1788270940) SKIP: sanityn test_106a Test only for ldiskfs and statx() supported SKIP 106a (1s) == sanityn test 106b: Glimpse RPCs test for statx ======== 09:55:41 (1788270941) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00475046 s, 216 kB/s File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238826938303 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-01 09:55:42.000000000 -0400 Modify: 2026-09-01 09:55:42.000000000 -0400 Change: 2026-09-01 09:55:42.000000000 -0400 Birth: 2026-09-01 09:55:42.000000000 -0400 /mnt/lustre/f106b.sanityn 144115238826938303 -rw-r--r-- 1 0 0 1788270942 1788270942 1788270942 File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238826938303 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-01 09:55:42.000000000 -0400 Modify: 2026-09-01 09:55:42.000000000 -0400 Change: 2026-09-01 09:55:42.000000000 -0400 Birth: 2026-09-01 09:55:42.000000000 -0400 1788270942 1024 1 PASS 106b (4s) == sanityn test 106c: Verify statx attributes mask ======= 09:55:45 (1788270945) PASS 106c (3s) == sanityn test 107a: Basic grouplock conflict =========== 09:55:48 (1788270948) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.110501 s, 94.9 MB/s /mnt/lustre/f107a.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5890 0x1702 0x240000400 multiop /mnt/lustre/f107a.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7626 multiop /mnt/lustre2/f107a.sanityn vO_G16022000r10g16022000c TMPPIPE=/tmp/multiop_open_wait_pipe.7626 First grouplock blocks second one PASS 107a (5s) == sanityn test 107b: Grouplock is added to the head of waiting list ========================================================== 09:55:53 (1788270953) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.107492 s, 97.5 MB/s /mnt/lustre/f107b.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 5850 0x16da 0x280000400 multiop /mnt/lustre/f107b.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7626 Grouplock blocks IO First grouplock blocks second one Second grouplock blocks IO PASS 107b (8s) == sanityn test 108a: lseek: parallel updates ============ 09:56:01 (1788270961) 1+0 records in 1+0 records out 10000 bytes (10 kB, 9.8 KiB) copied, 0.00196338 s, 5.1 MB/s llite.lustre-ffff911158430000.hybrid_io=0 llite.lustre-ffff91117235c000.hybrid_io=0 fail_val=4 fail_loc=0x80000414 1+0 records in 1+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 4.22851 s, 2.0 MB/s PASS 108a (7s) == sanityn test 109: Race with several mount instances on 1 node ========================================================== 09:56:08 (1788270968) 192.168.201.113@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg113-client.virtnet /mnt/lustre (opts:) 192.168.201.113@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg113-client.virtnet /mnt/lustre2 (opts:) Starting race between client mounts (duration 30) Iteration 0 Starting client: oleg113-client.virtnet: -o user_xattr,flock 192.168.201.113@tcp:/lustre /mnt/lustre Starting client: oleg113-client.virtnet: -o user_xattr,flock 192.168.201.113@tcp:/lustre /mnt/lustre2 192.168.201.113@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.113@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg113-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg113-client.virtnet /mnt/lustre (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg113-server' oleg113-server: oleg113-server.virtnet: executing load_modules_local oleg113-server: kmemleak disabled oleg113-server: Loading modules from /home/green/git/lustre-release/lustre oleg113-server: detected 4 online CPUs by sysfs oleg113-server: MODOPTS_LIBCFS= oleg113-server: Force libcfs to create 2 CPU partitions oleg113-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Iteration 1 Starting client: oleg113-client.virtnet: -o user_xattr,flock 192.168.201.113@tcp:/lustre /mnt/lustre2 Starting client: oleg113-client.virtnet: -o user_xattr,flock 192.168.201.113@tcp:/lustre /mnt/lustre 192.168.201.113@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.113@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg113-client.virtnet /mnt/lustre (opts:) Stopping client oleg113-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg113-server' oleg113-server: oleg113-server.virtnet: executing load_modules_local oleg113-server: kmemleak disabled oleg113-server: Loading modules from /home/green/git/lustre-release/lustre oleg113-server: detected 4 online CPUs by sysfs oleg113-server: MODOPTS_LIBCFS= oleg113-server: Force libcfs to create 2 CPU partitions oleg113-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Iteration 2 Starting client: oleg113-client.virtnet: -o user_xattr,flock 192.168.201.113@tcp:/lustre /mnt/lustre2 Starting client: oleg113-client.virtnet: -o user_xattr,flock 192.168.201.113@tcp:/lustre /mnt/lustre 192.168.201.113@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.113@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg113-client.virtnet /mnt/lustre (opts:) Stopping client oleg113-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg113-server' oleg113-server: oleg113-server.virtnet: executing load_modules_local oleg113-server: kmemleak disabled oleg113-server: Loading modules from /home/green/git/lustre-release/lustre oleg113-server: detected 4 online CPUs by sysfs oleg113-server: MODOPTS_LIBCFS= oleg113-server: Force libcfs to create 2 CPU partitions oleg113-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Starting client: oleg113-client.virtnet: -o user_xattr,flock 192.168.201.113@tcp:/lustre /mnt/lustre Starting client: oleg113-client.virtnet: -o user_xattr,flock 192.168.201.113@tcp:/lustre /mnt/lustre2 PASS 109 (42s) == sanityn test 110: do not grant another lock on resend ========================================================== 09:56:50 (1788271010) fail_loc=0x525 fail_val=3 fail_loc=0x534 File: /mnt/lustre/f110.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238860488707 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-01 09:56:51.000000000 -0400 Modify: 2026-09-01 09:56:51.000000000 -0400 Change: 2026-09-01 09:56:51.000000000 -0400 Birth: 2026-09-01 09:56:51.000000000 -0400 fail_loc=0 fail_val=0 Connected clients: oleg113-client.virtnet oleg113-client.virtnet PASS 110 (166s) == sanityn test 111: A racy rename/link an open file should not cause fs corruption ========================================================== 09:59:36 (1788271176) SKIP: sanityn test_111 needs >= 2 MDTs SKIP 111 (1s) == sanityn test 112: update max-inherit in default LMV === 09:59:37 (1788271177) SKIP: sanityn test_112 We need at least 2 MDTs for this test SKIP 112 (1s) == sanityn test 113: check servers of specified fs ======= 09:59:38 (1788271178) lustre-OST0000-osc-ffff911143995000 active. lustre-OST0001-osc-ffff911143995000 active. lustre-OST0000-osc-ffff911143995000 active. lustre-OST0001-osc-ffff911143995000 active. MGC192.168.201.113@tcp active. PASS 113 (3s) == sanityn test 114: implicit default LMV inherit ======== 09:59:41 (1788271181) SKIP: sanityn test_114 We need at least 2 MDTs for this test SKIP 114 (1s) == sanityn test 115: ldiskfs doesn't check direntry for uniqueness ========================================================== 09:59:42 (1788271182) SKIP: sanityn test_115 ldiskfs only test SKIP 115 (1s) == sanityn test 116: DNE: Set default LMV layout from a remote client ========================================================== 09:59:43 (1788271183) SKIP: sanityn test_116 needs >= 2 MDTs SKIP 116 (1s) == sanityn test 121: trunc append race =================== 09:59:45 (1788271185) 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00418688 s, 4.9 MB/s fail_loc=0x80001436 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0150711 s, 1.4 MB/s 0000000 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0005000 0000 0000 0000 0000 0000 0000 0000 0000 * 000c800 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0011800 PASS 121 (4s) == sanityn test 122: directory size is consistent across mounts ========================================================== 09:59:49 (1788271189) PASS 122 (3s) == sanityn test 200: service remains healthy while able to process request ========================================================== 09:59:52 (1788271192) at_max=10 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=8 fail_loc=0x1e1 fail_val=20 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0869 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0968 s, 0.2 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0878 s, 0.2 kB/s 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0904 s, 0.2 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1035 s, 0.2 kB/s 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1134 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1519 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1429 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1511 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1605 s, 0.1 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1586 s, 0.1 kB/s4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1565 s, 0.1 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2506 s, 0.1 kB/s 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.26 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2382 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2786 s, 0.1 kB/s at_max=600 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=116 oleg113-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff911143995000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff911143995000.ost_server_uuid in FULL state after 0 sec PASS 200 (68s) cleanup: ====================================================== == sanityn test complete, duration 8096 sec ============== 10:01:01 (1788271261) === sanityn: start cleanup 10:01:01 (1788271261) === Stopping clients: oleg113-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg113-client.virtnet /mnt/lustre2 opts: === sanityn: finish cleanup 10:02:27 (1788271347) ===