-----============= acceptance-small: sanityn ============----- Tue Sep 8 09:37:04 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg626-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanityn.*ex': No such file or directory excepting tests: 27 40a skipping tests SLOW=no: 33a === sanityn: start setup 09:37:14 (1788874634) === Starting client oleg626-client.virtnet: -o user_xattr,flock 192.168.206.126@tcp:/lustre /mnt/lustre2 Mount client oleg626-client.virtnet: mount -t lustre -o user_xattr,flock 192.168.206.126@tcp:/lustre /mnt/lustre2 Started clients oleg626-client.virtnet: 192.168.206.126@tcp:/lustre on /mnt/lustre2 type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) oleg626-client.virtnet: executing check_config_client /mnt/lustre oleg626-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg626-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9d9e08e29800.idle_timeout=debug osc.lustre-OST0000-osc-ffff9d9e11804800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9d9e08e29800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9d9e11804800.idle_timeout=debug disable quota as required oleg626-server: oleg626-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanityn: finish setup 09:37:28 (1788874648) === 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146305 s, 71.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7625] == sanityn test 0: do_node_vp() and do_facet_vp() do the right thing ========================================================== 09:37:29 (1788874649) PASS 0 (3s) == sanityn test 1: Check attribute updates on 2 mount points ========================================================== 09:37:32 (1788874652) /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0777 OK /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0666 OK /mnt/lustre/f1.sanityn: absent OK PASS 1 (3s) == sanityn test 2a: check cached attribute updates on 2 mtpt's ================================================================== 09:37:35 (1788874655) -rw-r--r-- 1 root root 0 Sep 8 09:37 /mnt/lustre2/f2a /mnt/lustre/f2a has type file OK /mnt/lustre/f2a has perms 0777 OK PASS 2a (3s) == sanityn test 2b: check cached attribute updates on 2 mtpt's ================================================================== 09:37:38 (1788874658) -rw-r--r-- 1 root root 0 Sep 8 09:37 /mnt/lustre2/f2b /mnt/lustre2/f2b has type file OK /mnt/lustre2/f2b has perms 0777 OK PASS 2b (2s) == sanityn test 2c: check cached attribute updates on 2 mtpt's root ============================================================= 09:37:41 (1788874661) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0777 OK PASS 2c (3s) == sanityn test 2d: check cached attribute updates on 2 mtpt's root ============================================================= 09:37:43 (1788874663) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0755 OK PASS 2d (3s) == sanityn test 2e: check chmod on root is propagated to others ========================================================== 09:37:46 (1788874666) total 1 -rwxrwxrwx 1 root root 0 Sep 8 09:37 f2a -rwxrwxrwx 1 root root 0 Sep 8 09:37 f2b total 1 -rwxrwxrwx 1 root root 0 Sep 8 09:37 f2a -rwxrwxrwx 1 root root 0 Sep 8 09:37 f2b running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre2/f2e.sanityn] [count=1] 1+0 records in 1+0 records out 512 bytes copied, 0.00505464 s, 101 kB/s PASS 2e (2s) == sanityn test 2f: check attr/owner updates on DNE with 2 mtpt's ========================================================== 09:37:48 (1788874668) SKIP: sanityn test_2f needs >= 2 MDTs SKIP 2f (2s) == sanityn test 2g: check blocks update on sync write ==== 09:37:50 (1788874670) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.131922 s, 15.9 MB/s /mnt/lustre/f2g.sanityn has 4103 blocks /mnt/lustre2/f2g.sanityn has 4103 blocks PASS 2g (2s) == sanityn test 3: symlink on one mtpt, readlink on another ===================================================================== 09:37:52 (1788874672) PASS 3 (3s) == sanityn test 4: fstat validation on multiple mount points ==================================================================== 09:37:55 (1788874675) PASS 4 (3s) == sanityn test 5: create a file on one mount, truncate it on the other ========================================================== 09:37:58 (1788874678) /mnt/lustre/f5 has type file OK /mnt/lustre/f5 has size 100 OK PASS 5 (3s) == sanityn test 6: remove of open file on other node ============================================================================ 09:38:01 (1788874681) opening writing unlinking /mnt/lustre2/f6.sanityn accessing (1) seeking (1) accessing (2) fstat... reading comparing data truncating seeking (2) writing again seeking (3) reading again comparing data again closing SUCCESS - goto beer PASS 6 (2s) == sanityn test 7: remove of open directory on other node ======================================================================= 09:38:04 (1788874684) creating directory /mnt/lustre/d7 opening directory unlinking /mnt/lustre/d7 Ok, everything goes well. PASS 7 (3s) == sanityn test 8: remove of open special file on other node ==================================================================== 09:38:06 (1788874686) creating special file /mnt/lustre/f8.sanityn opening file unlinking /mnt/lustre/f8.sanityn Ok, everything goes well. PASS 8 (3s) == sanityn test 9a: append of file with sub-page size on multiple mounts ========================================================== 09:38:09 (1788874689) PASS 9a (4s) == sanityn test 9b: append to striped sparse file ======== 09:38:13 (1788874693) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0246841 s, 42.5 MB/s 3+0 records in 3+0 records out 3 bytes copied, 0.0266772 s, 0.1 kB/s Data read (expecting 'foo'): foo PASS 9b (2s) == sanityn test 10a: write of file with sub-page size on multiple mounts ========================================================== 09:38:15 (1788874695) 1+0 records in 1+0 records out 1 byte copied, 0.00395159 s, 0.3 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0113753 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00857279 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00876132 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00780983 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00770242 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00977876 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00924814 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0101077 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00698636 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00592329 s, 0.2 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0092019 s, 0.1 kB/s PASS 10a (4s) == sanityn test 10b: write of file with sub-page size on multiple mounts ========================================================== 09:38:19 (1788874699) 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.00910298 s, 337 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0184269 s, 222 kB/s 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.000461544 s, 6.7 MB/s PASS 10b (3s) == sanityn test 11: execution of file opened for write should return error ============================================================== 09:38:22 (1788874702) multiop /mnt/lustre/d11/f vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7625 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 298: /mnt/lustre2/d11/f: Text file busy PASS 11 (3s) == sanityn test 12: test lock ordering (link, stat, unlink) ========================================================== 09:38:25 (1788874705) warning: '-runtime' deprecated, use '-t runtime' instead - link 936 (time 1788874716.47 total 10.00 last 93.57) - link 1812 (time 1788874726.47 total 20.01 last 87.58) - link 2598 (time 1788874736.48 total 30.01 last 78.53) - link 3405 (time 1788874746.49 total 40.02 last 80.65) - link 4310 (time 1788874756.49 total 50.02 last 90.49) - link 5330 (time 1788874766.50 total 60.03 last 101.91) - link 6295 (time 1788874776.50 total 70.04 last 96.45) - link 7330 (time 1788874786.51 total 80.04 last 103.41) - link 8440 (time 1788874796.52 total 90.05 last 110.90) total: 9515 link in 100.00 seconds: 95.15 ops/second using seed 3948583243 running for 100 seconds - stat 10000 (time 1788874716 ; total 9 ; last 9) - stat 20000 (time 1788874726 ; total 19 ; last 10) - stat 30000 (time 1788874736 ; total 29 ; last 10) - stat 40000 (time 1788874746 ; total 39 ; last 10) - stat 50000 (time 1788874756 ; total 49 ; last 10) - stat 55723 (time 1788874767 ; total 60 ; last 11) - stat 60000 (time 1788874776 ; total 69 ; last 9) - stat 64853 (time 1788874787 ; total 80 ; last 11) - stat 69975 (time 1788874798 ; total 91 ; last 11) - stat 70000 (time 1788874798 ; total 91 ; last 0) total: 74636 stats in 100 seconds: 746.359985 stats/second - unlinked 0 (time 1788874757 ; total 0 ; last 0) unlink(/mnt/lustre2/lockdir/lockfile9515) error: No such file or directory total: 9514 unlinks in 90 seconds: 105.711113 unlinks/second /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 77: kill: (21399) - No such process /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 78: kill: (21401) - No such process PASS 12 (144s) == sanityn test 13: test directory page revocation ======= 09:40:49 (1788874849) f13 f13 PASS 13 (3s) == sanityn test 14aa: execution of file open for write returns -ETXTBSY ========================================================== 09:40:52 (1788874852) multiop /mnt/lustre/d14aa.sanityn/f14aa.sanityn vOw_c TMPPIPE=/tmp/multiop_open_wait_pipe.7625 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 333: /mnt/lustre2/d14aa.sanityn/f14aa.sanityn: Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5141: 22587 Terminated $MULTIOP_PROG $FILE v$ARGS > $TMPPIPE (wd: ~) PASS 14aa (3s) == sanityn test 14ab: open(RDWR) of executing file returns -ETXTBSY ========================================================== 09:40:55 (1788874855) open(O_RDWR|O_CREAT): Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5141: 23165 Terminated $DIR1/$tdir/sleep 60 (wd: ~) PASS 14ab (3s) == sanityn test 14b: truncate of executing file returns -ETXTBSY ================================================================ 09:40:58 (1788874858) truncate: cannot truncate '/mnt/lustre2/d14b.sanityn/sleep' to length 60: Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 348: 23743 Terminated $DIR1/$tdir/sleep 60 PASS 14b (3s) == sanityn test 14c: open(O_TRUNC) of executing file return -ETXTBSY ============================================================ 09:41:02 (1788874862) cp: cannot create regular file '/mnt/lustre2/d14c.sanityn/sleep': Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 360: 24326 Terminated $DIR1/$tdir/sleep 60 PASS 14c (3s) == sanityn test 14d: chmod of executing file is still possible ================================================================== 09:41:05 (1788874865) chmod /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 371: 24908 Terminated $DIR1/$tdir/sleep 60 PASS 14d (4s) == sanityn test 15: test out-of-space with multiple writers ===================================================================== 09:41:09 (1788874869) PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin: MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky sleep 5 for ZFS MDS Waiting for MDT destroys to complete fallocate on zfs doesn't consume space STRIPECOUNT=2 ORIGFREE=7527424 MAXFREE=800000 SKIP: oos2 test_15 oos2.sh: 7527424KB free > 800000KB, increase MAXFREE (or reduce fs size) sleep 5 for ZFS MDS Waiting for MDT destroys to complete SKIP 15 (33s) == sanityn test 16a: 500 iterations of dual-mount fsx ==== 09:41:42 (1788874902) keep default fallocate mode: 2 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.164456 s, 51.0 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.136227 s, 61.6 MB/s Chance of close/open is 1 in 50 Seed set to 4904 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read skipping zero length punch hole skipping zero size read 000006[0] 1788874904.548457 mapwrite 0x928b09 thru 0x92b6ef (0x2be7 bytes) 000007[1] 1788874904.623477 read 0x90081a thru 0x90846a (0x7c51 bytes) 000008[1] 1788874904.647659 mapread 0x395b60 thru 0x3a4bdf (0xf080 bytes) 000010[1] 1788874904.692341 mapwrite 0x861845 thru 0x866b2f (0x52eb bytes) 000011[1] 1788874904.755499 mapwrite 0x77dcb0 thru 0x78147f (0x37d0 bytes) truncating to largest ever: 0x729c10 000013[0] 1788874904.802462 trunc from 0x92b6f0 to 0x729c0f (0x201ae0 bytes) 000014[0] 1788874904.841866 punch from 0x3dab88 to 0x3e796e (0xcde7 bytes) 000015[0] 1788874904.848249 read 0x7205cd thru 0x7280b3 (0x7ae7 bytes) 000016[1] 1788874904.866900 trunc from 0x729c10 to 0x1e037a (0x549895 bytes) 000020[0] 1788874904.906549 trunc from 0x1e037b to 0x4d579e (0x2f5424 bytes) 000023[1] 1788874904.954065 punch from 0x342848 to 0x34db22 (0xb2db bytes) 000026[0] 1788874905.002948 write 0x7286e0 thru 0x73605c (0xd97d bytes) 000027[0] 1788874905.017939 mapread 0x234deb thru 0x23be85 (0x709b bytes) 000028[0] 1788874905.048100 write 0x916151 thru 0x92282f (0xc6df bytes) 000029[1] 1788874905.066732 punch from 0x7daac8 to 0x7dcbfb (0x2134 bytes) 000030[0] 1788874905.090994 trunc from 0x922830 to 0x60e3ac (0x314483 bytes) 000031[1] 1788874905.125963 read 0x509e63 thru 0x50f694 (0x5832 bytes) 000033[0] 1788874905.156188 trunc from 0x60e3ad to 0x2c39e6 (0x34a9c6 bytes) 000034[0] 1788874905.176875 mapread 0x02e3a1 thru 0x03180e (0x346e bytes) 000038[1] 1788874905.211616 write 0x471f57 thru 0x475675 (0x371f bytes) 000039[1] 1788874905.227321 punch from 0x163816 to 0x16f5af (0xbd9a bytes) 000040[1] 1788874905.239260 mapwrite 0x740192 thru 0x74764a (0x74b9 bytes) 000041[1] 1788874905.322963 write 0x822b30 thru 0x82c71c (0x9bed bytes) 000042[1] 1788874905.335769 read 0x20c859 thru 0x2120dc (0x5884 bytes) 000045[0] 1788874905.346229 mapwrite 0x481aaa thru 0x48d92f (0xbe86 bytes) 000046[1] 1788874905.431416 mapread 0x500e66 thru 0x506685 (0x5820 bytes) truncating to largest ever: 0x7dc8a1 000048[1] 1788874905.488938 trunc from 0x82c71d to 0x7dc8a0 (0x4fe7c bytes) 000049[0] 1788874905.506731 punch from 0x46c14f to 0x471480 (0x5332 bytes) 000050[1] 1788874905.524324 punch from 0x1645b6 to 0x16ed2d (0xa778 bytes) 000052[1] 1788874905.549716 write 0x268d73 thru 0x274a5d (0xbceb bytes) 000053[1] 1788874905.567230 mapread 0x00bc86 thru 0x015fb6 (0xa331 bytes) 000057[1] 1788874905.634139 write 0x96a433 thru 0x973817 (0x93e5 bytes) 000059[1] 1788874905.655369 punch from 0x0d0825 to 0x0d624b (0x5a27 bytes) 000060[0] 1788874905.669645 trunc from 0x973818 to 0x0f3857 (0x87ffc0 bytes) 000063[0] 1788874905.731167 read 0x025e46 thru 0x02ccb5 (0x6e70 bytes) 000064[1] 1788874905.748255 punch from 0x089bcb to 0x08f26c (0x56a2 bytes) truncating to largest ever: 0x81a46a 000065[1] 1788874905.772503 trunc from 0x0f3858 to 0x81a469 (0x726c12 bytes) 000066[0] 1788874905.792869 read 0x36ed7c thru 0x37961e (0xa8a3 bytes) truncating to largest ever: 0x830aed 000067[0] 1788874905.837644 trunc from 0x81a46a to 0x830aec (0x16683 bytes) 000068[0] 1788874905.891101 punch from 0x7923c7 to 0x797a90 (0x56ca bytes) 000069[1] 1788874905.898689 mapread 0x46f73c thru 0x46ffed (0x8b2 bytes) 000071[0] 1788874905.916261 punch from 0x3bb4fb to 0x3c5e30 (0xa936 bytes) 000072[1] 1788874905.928155 read 0x7e0598 thru 0x7e6567 (0x5fd0 bytes) 000073[1] 1788874905.934269 mapwrite 0x33ccc7 thru 0x345f48 (0x9282 bytes) 000074[1] 1788874905.995361 punch from 0x73c10b to 0x73ee88 (0x2d7e bytes) 000075[1] 1788874906.006493 punch from 0x2e9bdf to 0x2ebd73 (0x2195 bytes) 000078[1] 1788874906.031870 punch from 0x386c8b to 0x38882c (0x1ba2 bytes) 000079[1] 1788874906.045392 punch from 0x6dd2f8 to 0x6ebcda (0xe9e3 bytes) 000080[1] 1788874906.051165 mapwrite 0x97ca4e thru 0x988d18 (0xc2cb bytes) 000081[0] 1788874906.122420 mapwrite 0x858ea9 thru 0x864963 (0xbabb bytes) 000082[0] 1788874906.228725 mapwrite 0x664e2d thru 0x6670a7 (0x227b bytes) 000086[0] 1788874906.342650 punch from 0x32e21b to 0x33a9c1 (0xc7a7 bytes) 000087[1] 1788874906.345412 mapread 0x40c7f6 thru 0x418761 (0xbf6c bytes) 000088[0] 1788874906.390466 punch from 0x8bb238 to 0x8c3d53 (0x8b1c bytes) 000091[0] 1788874906.393980 mapread 0x4e5895 thru 0x4f21c7 (0xc933 bytes) 000093[1] 1788874906.457962 mapwrite 0x5e8142 thru 0x5f50fe (0xcfbd bytes) 000095[0] 1788874906.556090 read 0x26aae0 thru 0x2768fb (0xbe1c bytes) 000096[0] 1788874906.578756 punch from 0x45323c to 0x4627ef (0xf5b4 bytes) 000097[0] 1788874906.604438 read 0x5b18dd thru 0x5b54ab (0x3bcf bytes) 000098[1] 1788874906.613693 punch from 0x0dca3f to 0x0e466e (0x7c30 bytes) 000100[1] 1788874906.647887 punch from 0x626fd1 to 0x634b25 (0xdb55 bytes) 000102[1] 1788874906.666645 punch from 0x8a5837 to 0x8aa4c1 (0x4c8b bytes) 000103[0] 1788874906.683570 trunc from 0x988d19 to 0x2b76cb (0x6d164d bytes) 000105[0] 1788874906.723071 read 0x0725a0 thru 0x07511e (0x2b7f bytes) 000106[0] 1788874906.749952 trunc from 0x2b76cc to 0x81c29a (0x564bcf bytes) 000109[0] 1788874906.782403 write 0x658071 thru 0x658ef4 (0xe84 bytes) 000111[0] 1788874906.814539 trunc from 0x81c29b to 0x24bde6 (0x5d04b4 bytes) 000114[0] 1788874906.851900 write 0x09057d thru 0x09bfcd (0xba51 bytes) 000115[0] 1788874906.862007 read 0x23d465 thru 0x242f3c (0x5ad8 bytes) 000116[0] 1788874906.869240 mapwrite 0x96fc7d thru 0x9790ba (0x943e bytes) 000117[1] 1788874906.973356 mapread 0x269595 thru 0x274383 (0xadef bytes) 000118[0] 1788874907.019438 trunc from 0x9790bb to 0x1937b5 (0x7e5905 bytes) 000119[0] 1788874907.047123 read 0x030e17 thru 0x03d42b (0xc615 bytes) 000120[0] 1788874907.087181 mapread 0x057694 thru 0x0642dc (0xcc49 bytes) 000122[0] 1788874907.088515 mapwrite 0x13acfe thru 0x147f6d (0xd270 bytes) 000123[1] 1788874907.129879 mapread 0x03f961 thru 0x04dce9 (0xe389 bytes) 000125[1] 1788874907.197574 punch from 0x031e97 to 0x041d56 (0xfec0 bytes) 000126[0] 1788874907.217002 punch from 0x0c8fe6 to 0x0d4aa3 (0xbabe bytes) 000130[0] 1788874907.239748 mapread 0x1423a4 thru 0x142f21 (0xb7e bytes) 000133[0] 1788874907.253178 mapwrite 0x9dc37a thru 0x9e546a (0x90f1 bytes) 000134[0] 1788874907.382673 write 0x2b7271 thru 0x2b8b7e (0x190e bytes) truncating to largest ever: 0x9753d2 000135[1] 1788874907.385435 trunc from 0x9e546b to 0x9753d1 (0x70099 bytes) 000136[0] 1788874907.437434 mapwrite 0x9f4159 thru 0x9fcef2 (0x8d9a bytes) 000137[1] 1788874907.518851 punch from 0x9f773e to 0x9f9e56 (0x2719 bytes) 000139[1] 1788874907.562870 punch from 0x74e261 to 0x753cb5 (0x5a55 bytes) 000140[1] 1788874907.572341 write 0x486de6 thru 0x494910 (0xdb2b bytes) 000141[1] 1788874907.592158 read 0x022d7e thru 0x027579 (0x47fc bytes) 000142[1] 1788874907.605719 read 0x2ab121 thru 0x2b6b35 (0xba15 bytes) 000144[1] 1788874907.630189 trunc from 0x9fcef3 to 0x00b7f1 (0x9f1701 bytes) 000145[0] 1788874907.652884 mapread 0x008546 thru 0x00b7f1 (0x32ac bytes) 000147[1] 1788874907.721937 write 0x163fb1 thru 0x164455 (0x4a5 bytes) 000149[1] 1788874907.743069 punch from 0x13a68d to 0x140276 (0x5bea bytes) 000150[1] 1788874907.749197 write 0x72101b thru 0x73091c (0xf902 bytes) 000151[1] 1788874907.752288 write 0x9b858e thru 0x9b9b4a (0x15bd bytes) 000152[1] 1788874907.765694 mapread 0x17465b thru 0x17ffc1 (0xb967 bytes) 000153[1] 1788874907.799519 trunc from 0x9b9b4b to 0x75068e (0x2694bc bytes) 000154[1] 1788874907.832246 mapread 0x47a4d3 thru 0x48679c (0xc2ca bytes) 000155[1] 1788874907.851770 mapwrite 0x638fda thru 0x640ecf (0x7ef6 bytes) 000156[1] 1788874907.897429 read 0x58db4f thru 0x58e5ed (0xa9f bytes) 000157[0] 1788874907.901532 mapwrite 0x421e6b thru 0x423606 (0x179c bytes) 000160[0] 1788874907.978192 punch from 0x1cd166 to 0x1d539b (0x8236 bytes) 000162[1] 1788874908.009977 trunc from 0x75068f to 0x5f6a79 (0x159c15 bytes) 000163[1] 1788874908.044657 punch from 0x4d23c7 to 0x4e03e5 (0xe01f bytes) 000165[1] 1788874908.055709 write 0x0c2616 thru 0x0cc346 (0x9d31 bytes) 000168[0] 1788874908.075190 mapread 0x59856d thru 0x5a1c0f (0x96a3 bytes) 000175[1] 1788874908.172848 trunc from 0x5f6a7a to 0x45f61a (0x19745f bytes) 000176[0] 1788874908.192516 trunc from 0x45f61b to 0x463c4c (0x4632 bytes) 000179[0] 1788874908.248928 trunc from 0x463c4d to 0x17424a (0x2efa02 bytes) 000181[0] 1788874908.280338 write 0x6c2f6c thru 0x6c7e45 (0x4eda bytes) 000183[0] 1788874908.304191 read 0x557b80 thru 0x5674aa (0xf92b bytes) 000184[0] 1788874908.310127 punch from 0x27f335 to 0x280fc0 (0x1c8c bytes) 000185[0] 1788874908.313744 read 0x514325 thru 0x5199b5 (0x5691 bytes) 000187[1] 1788874908.327655 punch from 0x5b4a38 to 0x5c46ed (0xfcb6 bytes) 000189[1] 1788874908.366877 write 0x3e3187 thru 0x3e8f15 (0x5d8f bytes) 000191[1] 1788874908.388578 write 0x213900 thru 0x2228b1 (0xefb2 bytes) 000192[1] 1788874908.395131 write 0x0e2f3c thru 0x0e468b (0x1750 bytes) 000194[1] 1788874908.423532 mapread 0x2207dc thru 0x22cb78 (0xc39d bytes) 000197[1] 1788874908.447976 punch from 0x57ee16 to 0x58d53d (0xe728 bytes) 000198[1] 1788874908.451005 trunc from 0x6c7e46 to 0x94808e (0x280249 bytes) 000199[1] 1788874908.466055 mapread 0x3cdac6 thru 0x3d6ed8 (0x9413 bytes) 000200[0] 1788874908.480026 mapwrite 0x78ef4b thru 0x79e504 (0xf5ba bytes) 000201[1] 1788874908.557283 mapwrite 0x5d9977 thru 0x5e77dc (0xde66 bytes) 000203[0] 1788874908.645549 punch from 0x2aba5e to 0x2b5757 (0x9cfa bytes) 000204[0] 1788874908.675060 mapread 0x0a1264 thru 0x0a15d9 (0x376 bytes) 000205[0] 1788874908.695569 trunc from 0x94808f to 0x277960 (0x6d072e bytes) 000206[1] 1788874908.724877 mapread 0x12e25c thru 0x13db83 (0xf928 bytes) 000208[1] 1788874908.780376 mapwrite 0x0c3f7d thru 0x0c65c5 (0x2649 bytes) 000210[1] 1788874908.853912 mapread 0x258180 thru 0x263816 (0xb697 bytes) 000211[1] 1788874908.882375 trunc from 0x277961 to 0x04a305 (0x22d65b bytes) 000212[0] 1788874908.904403 trunc from 0x04a306 to 0x158c2c (0x10e927 bytes) 000214[1] 1788874908.958682 punch from 0x08e55c to 0x091a8b (0x3530 bytes) 000215[0] 1788874908.983098 mapwrite 0x9f4c3f thru 0x9faf51 (0x6313 bytes) 000217[0] 1788874909.060118 mapwrite 0x9ee063 thru 0x9f544d (0x73eb bytes) 000218[1] 1788874909.102332 mapwrite 0x9dfe3d thru 0x9e2975 (0x2b39 bytes) 000219[0] 1788874909.172482 read 0x5c9810 thru 0x5d5946 (0xc137 bytes) 000221[1] 1788874909.198633 punch from 0x777611 to 0x782803 (0xb1f3 bytes) 000222[0] 1788874909.209464 write 0x98f462 thru 0x999677 (0xa216 bytes) 000223[1] 1788874909.234065 mapwrite 0x6a6ccb thru 0x6b26a9 (0xb9df bytes) 000224[0] 1788874909.317665 mapread 0x4ef8e9 thru 0x4f51c6 (0x58de bytes) 000226[0] 1788874909.367080 trunc from 0x9faf52 to 0x3030a7 (0x6f7eaa bytes) 000230[0] 1788874909.418068 read 0x2c1377 thru 0x2cdc6f (0xc8f9 bytes) 000231[0] 1788874909.452221 read 0x2fc8b9 thru 0x300de2 (0x452a bytes) 000233[0] 1788874909.453082 punch from 0x27c09e to 0x286c77 (0xabda bytes) 000234[1] 1788874909.460867 read 0x03528d thru 0x0379ad (0x2721 bytes) 000235[1] 1788874909.522619 mapread 0x06b403 thru 0x0704d8 (0x50d6 bytes) 000237[0] 1788874909.543428 mapwrite 0x224538 thru 0x23291b (0xe3e4 bytes) 000238[0] 1788874909.620708 mapread 0x0651de thru 0x06b9d7 (0x67fa bytes) 000240[0] 1788874909.656499 punch from 0x245e7f to 0x24b692 (0x5814 bytes) 000241[1] 1788874909.675663 read 0x15c6c2 thru 0x15d6ad (0xfec bytes) 000243[0] 1788874909.719644 mapread 0x188c89 thru 0x18940d (0x785 bytes) 000244[0] 1788874909.744695 trunc from 0x3030a8 to 0x94f86b (0x64c7c4 bytes) 000246[0] 1788874909.774088 trunc from 0x94f86c to 0x6bbdf2 (0x293a79 bytes) 000248[1] 1788874909.799351 trunc from 0x6bbdf3 to 0x8931b1 (0x1d73bf bytes) 000250[0] 1788874909.850199 mapwrite 0x9a98ad thru 0x9abb42 (0x2296 bytes) 000251[1] 1788874909.948921 trunc from 0x9abb43 to 0x35aee9 (0x650c59 bytes) 000255[1] 1788874909.996507 mapwrite 0x7b859d thru 0x7bd5e7 (0x504b bytes) 000256[0] 1788874910.103142 mapread 0x4c51c5 thru 0x4ce6ba (0x94f6 bytes) 000258[1] 1788874910.177932 mapwrite 0x4e8469 thru 0x4f5a90 (0xd628 bytes) 000260[0] 1788874910.280708 mapread 0x686cfa thru 0x6875f3 (0x8fa bytes) 000261[0] 1788874910.313315 write 0x56699a thru 0x574d17 (0xe37e bytes) 000262[0] 1788874910.341484 punch from 0x42b845 to 0x42c688 (0xe44 bytes) 000263[0] 1788874910.353340 punch from 0x78c0d0 to 0x799bd3 (0xdb04 bytes) 000264[0] 1788874910.364503 mapread 0x69f891 thru 0x6a780c (0x7f7c bytes) 000266[0] 1788874910.420880 mapwrite 0x3161a8 thru 0x31ae72 (0x4ccb bytes) 000267[1] 1788874910.493855 mapwrite 0x55f109 thru 0x568a81 (0x9979 bytes) 000268[1] 1788874910.579577 read 0x71b949 thru 0x729b4d (0xe205 bytes) 000269[1] 1788874910.598944 mapread 0x366ae2 thru 0x371aae (0xafcd bytes) 000270[0] 1788874910.649699 mapread 0x2695e2 thru 0x26bef1 (0x2910 bytes) 000271[0] 1788874910.667633 trunc from 0x7bd5e8 to 0x347c32 (0x4759b5 bytes) 000272[0] 1788874910.712529 trunc from 0x347c33 to 0x14d45d (0x1fa7d5 bytes) 000273[1] 1788874910.741048 mapwrite 0x59175b thru 0x59dc85 (0xc52b bytes) 000274[0] 1788874910.838011 trunc from 0x59dc86 to 0x3856c2 (0x2185c3 bytes) 000276[0] 1788874910.857454 mapread 0x29b844 thru 0x2a9229 (0xd9e6 bytes) 000277[1] 1788874910.938539 mapwrite 0x4831b4 thru 0x48a224 (0x7071 bytes) 000280[0] 1788874911.017888 punch from 0x195491 to 0x1a3a8f (0xe5ff bytes) 000282[1] 1788874911.067270 trunc from 0x48a225 to 0x8d5204 (0x44afe0 bytes) 000284[1] 1788874911.093065 read 0x3f304e thru 0x3fae77 (0x7e2a bytes) 000286[1] 1788874911.112414 mapwrite 0x724ce6 thru 0x732be7 (0xdf02 bytes) 000287[1] 1788874911.174214 punch from 0x21c772 to 0x21e807 (0x2096 bytes) 000289[1] 1788874911.178255 punch from 0x2b58e1 to 0x2bc52b (0x6c4b bytes) 000290[1] 1788874911.193926 punch from 0x3ac07f to 0x3ba300 (0xe282 bytes) 000291[0] 1788874911.196934 punch from 0x40d96e to 0x416c86 (0x9319 bytes) 000293[1] 1788874911.224029 read 0x71e037 thru 0x720a45 (0x2a0f bytes) 000294[0] 1788874911.239321 punch from 0x81eb12 to 0x82adf6 (0xc2e5 bytes) 000295[1] 1788874911.246471 trunc from 0x8d5205 to 0x16732d (0x76ded7 bytes) 000296[1] 1788874911.265841 read 0x0fa878 thru 0x102cf6 (0x847f bytes) 000297[0] 1788874911.288054 punch from 0x08f2cf to 0x0982cd (0x8fff bytes) 000299[1] 1788874911.317251 trunc from 0x16732e to 0x729819 (0x5c24ec bytes) 000300[0] 1788874911.339313 write 0x0f583d thru 0x0faa05 (0x51c9 bytes) 000302[1] 1788874911.374448 mapread 0x251043 thru 0x25aa93 (0x9a51 bytes) 000303[1] 1788874911.419531 mapread 0x04d8b4 thru 0x05cb59 (0xf2a6 bytes) 000305[0] 1788874911.503528 read 0x5dadd6 thru 0x5dd3ee (0x2619 bytes) 000309[1] 1788874911.518586 punch from 0x445fc9 to 0x450a41 (0xaa79 bytes) 000310[0] 1788874911.529257 mapwrite 0x84a846 thru 0x851b71 (0x732c bytes) 000311[0] 1788874911.600125 punch from 0x666cb8 to 0x669627 (0x2970 bytes) 000314[0] 1788874911.634321 read 0x6beb2d thru 0x6c84e9 (0x99bd bytes) 000315[1] 1788874911.649431 mapwrite 0x49b05e thru 0x49ed11 (0x3cb4 bytes) 000317[1] 1788874911.730333 read 0x437232 thru 0x4402a7 (0x9076 bytes) 000318[1] 1788874911.749240 trunc from 0x851b72 to 0x7fb051 (0x56b20 bytes) 000321[0] 1788874911.780826 read 0x69e33e thru 0x6a144a (0x310d bytes) 000323[0] 1788874911.803820 mapwrite 0x7b3c4e thru 0x7b4d94 (0x1147 bytes) 000324[0] 1788874911.851124 read 0x668f02 thru 0x678de6 (0xfee5 bytes) 000325[0] 1788874911.862583 read 0x39d95c thru 0x3a73f7 (0x9a9c bytes) 000326[0] 1788874911.875548 mapwrite 0x95c46c thru 0x962741 (0x62d6 bytes) 000329[0] 1788874911.949711 read 0x90c0cf thru 0x914e58 (0x8d8a bytes) 000330[0] 1788874911.956721 write 0x646bcf thru 0x647be1 (0x1013 bytes) 000333[1] 1788874911.968978 read 0x1a64c4 thru 0x1a9ffa (0x3b37 bytes) 000336[0] 1788874911.979192 write 0x712b76 thru 0x716a24 (0x3eaf bytes) 000337[0] 1788874911.990088 write 0x314c2a thru 0x3225bc (0xd993 bytes) 000338[0] 1788874912.003108 read 0x1fe8d6 thru 0x2028b1 (0x3fdc bytes) 000339[0] 1788874912.016398 write 0x7981ab thru 0x7a339b (0xb1f1 bytes) 000340[0] 1788874912.032062 write 0x46ba06 thru 0x46c7e0 (0xddb bytes) 000341[0] 1788874912.051449 mapread 0x49e68b thru 0x4a229e (0x3c14 bytes) 000342[1] 1788874912.072647 read 0x10f955 thru 0x11f2ef (0xf99b bytes) 000343[1] 1788874912.091062 trunc from 0x962742 to 0x0671cc (0x8fb575 bytes) 000344[1] 1788874912.124504 read 0x03092d thru 0x035083 (0x4757 bytes) 000345[1] 1788874912.146025 mapread 0x05bdb8 thru 0x0613f5 (0x563e bytes) 000348[0] 1788874912.147636 read 0x011b43 thru 0x0124e0 (0x99e bytes) 000349[1] 1788874912.165593 punch from 0x04d118 to 0x057c46 (0xab2f bytes) 000350[0] 1788874912.184164 mapread 0x02bdb8 thru 0x03082d (0x4a76 bytes) 000352[1] 1788874912.226808 mapread 0x03e7ce thru 0x04b93e (0xd171 bytes) 000354[1] 1788874912.288295 trunc from 0x0671cd to 0x32a6ca (0x2c34fe bytes) 000355[0] 1788874912.315616 mapwrite 0x5ca83d thru 0x5d2577 (0x7d3b bytes) 000357[0] 1788874912.438817 mapwrite 0x9b147f thru 0x9b7b77 (0x66f9 bytes) 000359[1] 1788874912.490928 write 0x26d23c thru 0x27b9c6 (0xe78b bytes) 000360[0] 1788874912.535501 write 0x519745 thru 0x51b97f (0x223b bytes) 000364[1] 1788874912.586693 read 0x5da8f6 thru 0x5daefb (0x606 bytes) 000366[0] 1788874912.607673 mapread 0x105739 thru 0x111975 (0xc23d bytes) 000367[0] 1788874912.671038 punch from 0x27afc8 to 0x27e0dd (0x3116 bytes) 000369[1] 1788874912.701395 read 0x75e1e1 thru 0x75e319 (0x139 bytes) 000370[1] 1788874912.704933 write 0x4d8d9d thru 0x4da594 (0x17f8 bytes) 000371[1] 1788874912.728178 punch from 0x1a6f83 to 0x1b0e55 (0x9ed3 bytes) 000372[0] 1788874912.736972 write 0x674431 thru 0x68194f (0xd51f bytes) 000373[0] 1788874912.763185 trunc from 0x9b7b78 to 0x004f90 (0x9b2be7 bytes) 000375[0] 1788874912.796771 write 0x89e510 thru 0x8ab1ef (0xcce0 bytes) 000377[1] 1788874912.828307 read 0x0d7584 thru 0x0d9dde (0x285b bytes) 000380[0] 1788874912.867135 mapread 0x682840 thru 0x68d404 (0xabc5 bytes) 000382[1] 1788874912.910696 read 0x5d7881 thru 0x5e43e8 (0xcb68 bytes) 000384[1] 1788874912.927169 mapwrite 0x78f74a thru 0x797d09 (0x85c0 bytes) 000386[0] 1788874913.014508 punch from 0x39eafd to 0x3a168a (0x2b8e bytes) 000387[0] 1788874913.030460 punch from 0x4bef32 to 0x4bff58 (0x1027 bytes) 000389[1] 1788874913.057994 read 0x496398 thru 0x49685d (0x4c6 bytes) 000390[1] 1788874913.075050 trunc from 0x8ab1f0 to 0x235430 (0x675dbf bytes) 000391[1] 1788874913.099005 read 0x20c7c4 thru 0x219196 (0xc9d3 bytes) 000392[1] 1788874913.116678 mapread 0x03ff33 thru 0x041d31 (0x1dff bytes) 000394[1] 1788874913.127378 mapread 0x12f66f thru 0x1351b1 (0x5b43 bytes) 000395[1] 1788874913.128717 write 0x8171e8 thru 0x825277 (0xe090 bytes) 000396[1] 1788874913.132573 mapwrite 0x948108 thru 0x94b0bb (0x2fb4 bytes) 000397[0] 1788874913.227592 mapread 0x4dca51 thru 0x4e7cd7 (0xb287 bytes) 000398[0] 1788874913.284360 mapread 0x28ede8 thru 0x291fc5 (0x31de bytes) 000399[0] 1788874913.312940 punch from 0x5a68af to 0x5b4b85 (0xe2d7 bytes) 000401[0] 1788874913.319757 read 0x7127e1 thru 0x71c4d7 (0x9cf7 bytes) 000403[0] 1788874913.335373 mapwrite 0x93edeb thru 0x94021d (0x1433 bytes) 000404[1] 1788874913.406430 punch from 0x698ca7 to 0x6a3272 (0xa5cc bytes) 000406[0] 1788874913.443287 write 0x91cae5 thru 0x92436f (0x788b bytes) 000408[0] 1788874913.477629 trunc from 0x94b0bc to 0x7a60f5 (0x1a4fc6 bytes) 000409[0] 1788874913.513272 read 0x48e49f thru 0x4937eb (0x534d bytes) 000411[0] 1788874913.536844 write 0x2826d8 thru 0x28570a (0x3033 bytes) 000413[0] 1788874913.567134 read 0x138466 thru 0x1453d2 (0xcf6d bytes) 000414[0] 1788874913.590969 mapread 0x749019 thru 0x75666e (0xd656 bytes) 000416[1] 1788874913.638718 write 0x9da102 thru 0x9de7cc (0x46cb bytes) 000420[1] 1788874913.684231 mapwrite 0x8d751a thru 0x8e2a4d (0xb534 bytes) 000421[1] 1788874913.800630 write 0x3b15f4 thru 0x3bce81 (0xb88e bytes) 000422[1] 1788874913.818079 write 0x995bf1 thru 0x9a4774 (0xeb84 bytes) 000423[1] 1788874913.840383 mapwrite 0x81434a thru 0x815cb3 (0x196a bytes) 000424[0] 1788874913.911257 trunc from 0x9de7cd to 0x6ad19d (0x33162f bytes) 000425[1] 1788874913.940459 mapwrite 0x383b5a thru 0x390826 (0xcccd bytes) 000428[1] 1788874914.082064 mapwrite 0x1bd2d0 thru 0x1cc5f5 (0xf326 bytes) 000429[1] 1788874914.191734 read 0x313bf5 thru 0x31ff8d (0xc399 bytes) 000431[1] 1788874914.224764 punch from 0x668a9b to 0x6746d1 (0xbc37 bytes) 000432[0] 1788874914.233046 mapread 0x2b7c9b thru 0x2c1dfa (0xa160 bytes) 000433[1] 1788874914.267550 read 0x0d992c thru 0x0dede1 (0x54b6 bytes) 000436[1] 1788874914.283104 read 0x399fce thru 0x39abb1 (0xbe4 bytes) 000437[0] 1788874914.292058 mapwrite 0x480da0 thru 0x48ff4f (0xf1b0 bytes) 000440[1] 1788874914.358897 mapwrite 0x7c6535 thru 0x7c71df (0xcab bytes) 000444[0] 1788874914.436942 punch from 0x368f3e to 0x36deef (0x4fb2 bytes) 000445[0] 1788874914.453127 trunc from 0x7c71e0 to 0x89991a (0xd273b bytes) 000446[0] 1788874914.466911 write 0x6ac41f thru 0x6bbad9 (0xf6bb bytes) 000447[1] 1788874914.474550 trunc from 0x89991b to 0x3f70ce (0x4a284c bytes) 000448[1] 1788874914.498345 trunc from 0x3f70cf to 0x913198 (0x51c0ca bytes) 000450[0] 1788874914.523380 mapread 0x705481 thru 0x70caa4 (0x7624 bytes) 000451[0] 1788874914.560667 write 0x961799 thru 0x9691af (0x7a17 bytes) 000453[0] 1788874914.586940 write 0x8be3b2 thru 0x8bf022 (0xc71 bytes) 000454[1] 1788874914.608693 trunc from 0x9691b0 to 0x5e2bce (0x3865e1 bytes) 000456[0] 1788874914.643667 write 0x29047f thru 0x296bbd (0x673f bytes) 000460[1] 1788874914.700934 punch from 0x13bc29 to 0x142b3f (0x6f17 bytes) 000462[1] 1788874914.713230 trunc from 0x5e2bcf to 0x8ea071 (0x3074a3 bytes) 000463[0] 1788874914.725856 write 0x494831 thru 0x49532d (0xafd bytes) 000464[1] 1788874914.753517 punch from 0x68147b to 0x681ea2 (0xa28 bytes) 000465[0] 1788874914.782475 punch from 0x6787f1 to 0x67e9fa (0x620a bytes) 000467[0] 1788874914.818933 write 0x873a74 thru 0x87f3bd (0xb94a bytes) 000468[1] 1788874914.847286 mapwrite 0x2f722e thru 0x2ff088 (0x7e5b bytes) 000469[0] 1788874914.940965 mapread 0x7ee07c thru 0x7f90b2 (0xb037 bytes) 000470[0] 1788874914.964831 trunc from 0x8ea072 to 0x538f17 (0x3b115a bytes) 000473[0] 1788874915.005401 read 0x4e5842 thru 0x4e77cd (0x1f8c bytes) 000474[0] 1788874915.022705 mapwrite 0x6bdc0a thru 0x6cd99a (0xfd91 bytes) 000476[0] 1788874915.084369 mapwrite 0x99c33b thru 0x9ac1c4 (0xfe8a bytes) 000477[0] 1788874915.158111 punch from 0x06abb2 to 0x06f41f (0x486e bytes) 000478[1] 1788874915.161978 trunc from 0x9ac1c5 to 0x6263e7 (0x385ddd bytes) 000479[1] 1788874915.195205 trunc from 0x6263e8 to 0x83e23a (0x217e53 bytes) 000480[1] 1788874915.207488 mapread 0x798048 thru 0x79df4e (0x5f07 bytes) 000482[0] 1788874915.239964 punch from 0x1213c9 to 0x126083 (0x4cbb bytes) 000484[0] 1788874915.269173 mapwrite 0x6df140 thru 0x6ee052 (0xef13 bytes) 000488[1] 1788874915.363853 punch from 0x239637 to 0x24159a (0x7f64 bytes) 000489[0] 1788874915.379745 mapwrite 0x670348 thru 0x6752aa (0x4f63 bytes) 000490[0] 1788874915.446325 mapwrite 0x2a5685 thru 0x2b19b8 (0xc334 bytes) 000493[1] 1788874915.547716 mapread 0x1fb190 thru 0x203c4e (0x8abf bytes) 000497[1] 1788874915.589385 write 0x7acbb7 thru 0x7b0794 (0x3bde bytes) 000499[0] 1788874915.615411 read 0x4b1948 thru 0x4b7e19 (0x64d2 bytes) 000500[1] 1788874915.634144 read 0x603d0b thru 0x60f6ec (0xb9e2 bytes) All operations completed A-OK! Chance of close/open is 1 in 50 Seed set to 4915 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000001[1] 1788874916.030403 mapwrite 0x208000 thru 0x21718a (0xf18b bytes) 000002[0] 1788874916.118552 punch from 0x0e6843 to 0x0ebf5e (0x571c bytes) truncating to largest ever: 0x2b98d0 000003[1] 1788874916.129013 trunc from 0x21718b to 0x2b98cf (0xa2745 bytes) 000005[0] 1788874916.164483 mapread 0x085000 thru 0x08fbd5 (0xabd6 bytes) 000009[1] 1788874916.192290 read_OD 0x1a3000 thru 0x1affff (0xd000 bytes) 000010[0] 1788874916.200532 write_OD 0x222000 thru 0x225fff (0x4000 bytes) 000012[1] 1788874916.228452 mapread 0x035000 thru 0x044f95 (0xff96 bytes) 000013[1] 1788874916.249686 punch from 0x1817ea to 0x18cec8 (0xb6df bytes) 000014[0] 1788874916.259502 punch from 0x1ee124 to 0x1ef6ea (0x15c7 bytes) 000015[1] 1788874916.276718 read_OD 0x0ae000 thru 0x0b9fff (0xc000 bytes) 000016[0] 1788874916.296432 mapread 0x1e6000 thru 0x1ef0f0 (0x90f1 bytes) 000017[0] 1788874916.324813 punch from 0x1786a7 to 0x17b781 (0x30db bytes) 000020[1] 1788874916.336942 punch from 0x241211 to 0x241e9a (0xc8a bytes) 000021[1] 1788874916.365706 trunc from 0x2b98d0 to 0x032294 (0x28763b bytes) 000022[1] 1788874916.380702 write_OD 0x3cd000 thru 0x3d3fff (0x7000 bytes) 000024[1] 1788874916.418647 mapread 0x2bf000 thru 0x2bffe7 (0xfe8 bytes) 000026[0] 1788874916.448305 read_OD 0x066000 thru 0x073fff (0xe000 bytes) truncating to largest ever: 0x947a12 000029[1] 1788874916.454539 trunc from 0x3d4000 to 0x947a11 (0x573a12 bytes) 000032[1] 1788874916.503059 read_OD 0x482000 thru 0x48dfff (0xc000 bytes) 000033[1] 1788874916.511862 write_OD 0x292000 thru 0x29efff (0xd000 bytes) 000034[0] 1788874916.555573 mapwrite 0x926000 thru 0x931fa7 (0xbfa8 bytes) 000035[1] 1788874916.614852 write_OD 0x35b000 thru 0x35bfff (0x1000 bytes) 000036[0] 1788874916.650192 mapread 0x62f000 thru 0x6397c3 (0xa7c4 bytes) 000037[1] 1788874916.663444 mapwrite 0x18a000 thru 0x18b215 (0x1216 bytes) 000039[1] 1788874916.729691 write_OD 0x89d000 thru 0x8a0fff (0x4000 bytes) 000040[1] 1788874916.761032 write_OD 0x77a000 thru 0x77cfff (0x3000 bytes) 000043[0] 1788874916.789274 punch from 0x58a9d7 to 0x58bb83 (0x11ad bytes) 000044[1] 1788874916.807975 mapread 0x5ea000 thru 0x5f9af8 (0xfaf9 bytes) 000045[1] 1788874916.834860 punch from 0x7bd16d to 0x7caf1c (0xddb0 bytes) 000048[0] 1788874916.860215 punch from 0x93ca0e to 0x94760d (0xac00 bytes) 000051[1] 1788874916.890915 punch from 0x224685 to 0x2291e6 (0x4b62 bytes) 000052[0] 1788874916.900133 mapread 0x49c000 thru 0x4a4b40 (0x8b41 bytes) 000054[1] 1788874916.932475 trunc from 0x947a12 to 0x2c8298 (0x67f779 bytes) 000057[0] 1788874916.966337 write_OD 0x778000 thru 0x785fff (0xe000 bytes) 000058[1] 1788874917.008247 trunc from 0x786000 to 0x3ebb92 (0x39a46d bytes) 000059[1] 1788874917.037958 trunc from 0x3ebb93 to 0x8176a6 (0x42bb14 bytes) 000064[0] 1788874917.073758 write_OD 0x5b3000 thru 0x5bafff (0x8000 bytes) 000065[1] 1788874917.105308 mapread 0x1af000 thru 0x1be18b (0xf18c bytes) 000069[0] 1788874917.131932 punch from 0x7a26ca to 0x7a80be (0x59f5 bytes) 000070[1] 1788874917.145770 mapread 0x039000 thru 0x0441d7 (0xb1d8 bytes) 000072[0] 1788874917.173441 mapwrite 0x016000 thru 0x01783c (0x183d bytes) 000075[0] 1788874917.251060 mapwrite 0x435000 thru 0x43fcc5 (0xacc6 bytes) 000076[0] 1788874917.299689 punch from 0x111d63 to 0x119ecd (0x816b bytes) 000077[1] 1788874917.320052 mapwrite 0x8be000 thru 0x8cb8f8 (0xd8f9 bytes) 000079[1] 1788874917.419266 trunc from 0x8cb8f9 to 0x33e385 (0x58d573 bytes) 000080[1] 1788874917.447961 trunc from 0x33e386 to 0x49eb48 (0x1607c3 bytes) 000081[0] 1788874917.472861 punch from 0x21e33f to 0x222d08 (0x49ca bytes) 000086[1] 1788874917.535585 punch from 0x4662f7 to 0x46949f (0x31a9 bytes) 000089[0] 1788874917.579036 mapread 0x284000 thru 0x28ae75 (0x6e76 bytes) 000090[1] 1788874917.609889 trunc from 0x49eb49 to 0x3ce3e0 (0xd0768 bytes) 000091[0] 1788874917.633351 mapread 0x336000 thru 0x33c56a (0x656b bytes) 000092[0] 1788874917.656942 mapwrite 0x87a000 thru 0x889c85 (0xfc86 bytes) 000094[0] 1788874917.744591 trunc from 0x889c86 to 0x7846b5 (0x1055d0 bytes) 000096[0] 1788874917.768405 trunc from 0x7846b6 to 0x037d76 (0x74c93f bytes) 000097[0] 1788874917.786508 read_OD 0x001000 thru 0x009fff (0x9000 bytes) 000098[1] 1788874917.795436 mapwrite 0x2f9000 thru 0x2fb6a2 (0x26a3 bytes) 000102[1] 1788874917.876015 read_OD 0x002000 thru 0x00cfff (0xb000 bytes) 000104[1] 1788874917.890586 mapwrite 0x89e000 thru 0x8a9b3e (0xbb3f bytes) 000107[0] 1788874918.011949 trunc from 0x8a9b3f to 0x00458e (0x8a55b0 bytes) 000111[1] 1788874918.078984 trunc from 0x00458f to 0x0d2c5c (0xce6ce bytes) 000113[1] 1788874918.110859 mapread 0x097000 thru 0x099a27 (0x2a28 bytes) 000114[0] 1788874918.133194 punch from 0x09ff30 to 0x0a193e (0x1a0f bytes) 000117[1] 1788874918.190468 mapread 0x099000 thru 0x0a4a44 (0xba45 bytes) 000118[1] 1788874918.249754 read_OD 0x05c000 thru 0x065fff (0xa000 bytes) 000119[1] 1788874918.254209 mapwrite 0x222000 thru 0x225344 (0x3345 bytes) 000121[1] 1788874918.357948 write_OD 0x5d7000 thru 0x5dbfff (0x5000 bytes) 000124[0] 1788874918.405450 mapread 0x406000 thru 0x40f23f (0x9240 bytes) 000126[1] 1788874918.451400 trunc from 0x5dc000 to 0x6e9588 (0x10d589 bytes) 000127[0] 1788874918.482405 punch from 0x57e227 to 0x5888a2 (0xa67c bytes) 000128[0] 1788874918.498034 read_OD 0x041000 thru 0x049fff (0x9000 bytes) 000129[1] 1788874918.511824 write_OD 0x78c000 thru 0x78ffff (0x4000 bytes) 000131[0] 1788874918.555121 mapread 0x6e5000 thru 0x6f4eef (0xfef0 bytes) 000133[0] 1788874918.578430 mapwrite 0x5bb000 thru 0x5c472c (0x972d bytes) 000134[1] 1788874918.656065 punch from 0x77f3df to 0x780ed2 (0x1af4 bytes) 000135[0] 1788874918.675347 write_OD 0x9a4000 thru 0x9a9fff (0x6000 bytes) 000136[0] 1788874918.714309 mapread 0x7e0000 thru 0x7e2b45 (0x2b46 bytes) 000138[1] 1788874918.730130 read_OD 0x632000 thru 0x639fff (0x8000 bytes) 000139[0] 1788874918.736685 mapwrite 0x1ec000 thru 0x1f6895 (0xa896 bytes) 000140[1] 1788874918.815965 mapread 0x512000 thru 0x5192d2 (0x72d3 bytes) 000142[1] 1788874918.843926 read_OD 0x2d3000 thru 0x2dbfff (0x9000 bytes) 000144[0] 1788874918.848530 punch from 0x87dbd4 to 0x889292 (0xb6bf bytes) 000145[1] 1788874918.859393 mapwrite 0x307000 thru 0x3073d3 (0x3d4 bytes) 000146[0] 1788874918.896218 read_OD 0x046000 thru 0x052fff (0xd000 bytes) 000149[1] 1788874918.913745 mapread 0x4e1000 thru 0x4e99b1 (0x89b2 bytes) 000150[1] 1788874918.924019 mapwrite 0x9e2000 thru 0x9ed73b (0xb73c bytes) 000151[0] 1788874918.983247 write_OD 0x8e1000 thru 0x8eefff (0xe000 bytes) 000154[0] 1788874919.026225 read_OD 0x893000 thru 0x893fff (0x1000 bytes) 000155[1] 1788874919.033092 write_OD 0x89b000 thru 0x89dfff (0x3000 bytes) 000156[0] 1788874919.068869 write_OD 0x982000 thru 0x98afff (0x9000 bytes) 000158[0] 1788874919.095910 read_OD 0x73c000 thru 0x74afff (0xf000 bytes) 000159[1] 1788874919.099912 read_OD 0x086000 thru 0x08dfff (0x8000 bytes) 000161[0] 1788874919.108391 mapwrite 0x005000 thru 0x00f672 (0xa673 bytes) 000162[1] 1788874919.170547 trunc from 0x9ed73c to 0x224cdc (0x7c8a5f bytes) 000163[0] 1788874919.209142 read_OD 0x061000 thru 0x069fff (0x9000 bytes) 000164[0] 1788874919.235929 write_OD 0x10f000 thru 0x118fff (0xa000 bytes) 000165[0] 1788874919.277242 read_OD 0x0f7000 thru 0x0f7fff (0x1000 bytes) 000167[0] 1788874919.283993 mapwrite 0x508000 thru 0x513ebc (0xbebd bytes) 000169[0] 1788874919.385228 mapread 0x4f7000 thru 0x50664d (0xf64e bytes) 000170[1] 1788874919.386008 trunc from 0x513ebd to 0x5c6fde (0xb3122 bytes) 000172[1] 1788874919.416381 read_OD 0x5b4000 thru 0x5c2fff (0xf000 bytes) 000176[0] 1788874919.437971 punch from 0x5141a7 to 0x51c782 (0x85dc bytes) 000182[0] 1788874919.511464 mapwrite 0x333000 thru 0x342924 (0xf925 bytes) 000185[1] 1788874919.619001 read_OD 0x5c2000 thru 0x5c5fff (0x4000 bytes) 000189[0] 1788874919.636126 mapread 0x162000 thru 0x16768a (0x568b bytes) 000190[0] 1788874919.662625 punch from 0x064d52 to 0x067d8c (0x303b bytes) 000192[1] 1788874919.684867 read_OD 0x14a000 thru 0x157fff (0xe000 bytes) 000193[1] 1788874919.704169 mapwrite 0x700000 thru 0x7006bb (0x6bc bytes) 000196[1] 1788874919.797291 write_OD 0x592000 thru 0x596fff (0x5000 bytes) 000198[1] 1788874919.829602 punch from 0x0ebb57 to 0x0f7cdf (0xc189 bytes) 000199[1] 1788874919.841004 mapwrite 0x111000 thru 0x115e47 (0x4e48 bytes) 000200[0] 1788874919.910471 mapwrite 0x8e5000 thru 0x8f3a02 (0xea03 bytes) 000202[1] 1788874920.018648 write_OD 0x3ad000 thru 0x3adfff (0x1000 bytes) 000203[0] 1788874920.044072 trunc from 0x8f3a03 to 0x62b1ac (0x2c8856 bytes) 000204[0] 1788874920.077175 mapwrite 0x9d0000 thru 0x9d1652 (0x1653 bytes) 000205[0] 1788874920.152242 punch from 0x8d30a7 to 0x8d7252 (0x41ac bytes) 000208[1] 1788874920.176488 mapwrite 0x4bb000 thru 0x4ca748 (0xf749 bytes) 000210[1] 1788874920.234117 trunc from 0x9d1653 to 0x524365 (0x4ad2ed bytes) 000211[0] 1788874920.247261 mapwrite 0x2e2000 thru 0x2eab45 (0x8b46 bytes) 000212[0] 1788874920.346400 read_OD 0x18c000 thru 0x196fff (0xb000 bytes) 000213[0] 1788874920.360455 write_OD 0x2f2000 thru 0x2f3fff (0x2000 bytes) 000214[1] 1788874920.395085 write_OD 0x02a000 thru 0x030fff (0x7000 bytes) 000216[0] 1788874920.427059 punch from 0x522364 to 0x5227d2 (0x46f bytes) 000217[1] 1788874920.435364 read_OD 0x072000 thru 0x074fff (0x3000 bytes) 000219[0] 1788874920.451101 read_OD 0x0fb000 thru 0x103fff (0x9000 bytes) 000220[1] 1788874920.455423 mapread 0x1cc000 thru 0x1d51b5 (0x91b6 bytes) 000221[0] 1788874920.474531 read_OD 0x0b9000 thru 0x0bffff (0x7000 bytes) 000222[1] 1788874920.480745 mapwrite 0x040000 thru 0x041fa6 (0x1fa7 bytes) 000223[1] 1788874920.534868 trunc from 0x524366 to 0x4dcb0f (0x47856 bytes) 000224[1] 1788874920.549479 mapread 0x421000 thru 0x42e57f (0xd580 bytes) 000228[1] 1788874920.592467 write_OD 0x233000 thru 0x233fff (0x1000 bytes) 000229[1] 1788874920.620324 punch from 0x011500 to 0x01ac52 (0x9753 bytes) 000230[1] 1788874920.639358 trunc from 0x4dcb10 to 0x7ff1bd (0x3226ae bytes) 000231[1] 1788874920.661428 punch from 0x293cb4 to 0x2a1e7e (0xe1cb bytes) 000232[0] 1788874920.668827 mapwrite 0x335000 thru 0x341b2c (0xcb2d bytes) truncating to largest ever: 0x9a8907 000234[1] 1788874920.741587 trunc from 0x7ff1be to 0x9a8906 (0x1a9749 bytes) 000235[1] 1788874920.762369 write_OD 0x6aa000 thru 0x6b4fff (0xb000 bytes) 000236[1] 1788874920.804082 trunc from 0x9a8907 to 0x02dfc7 (0x97a93f bytes) 000238[1] 1788874920.846173 mapwrite 0x2ac000 thru 0x2b0397 (0x4398 bytes) 000239[0] 1788874920.903466 trunc from 0x2b0398 to 0x05fccb (0x2506cc bytes) 000241[0] 1788874920.938515 read_OD 0x037000 thru 0x03bfff (0x5000 bytes) 000242[1] 1788874920.951058 mapread 0x053000 thru 0x056bb4 (0x3bb5 bytes) 000243[1] 1788874920.973076 mapwrite 0x66a000 thru 0x6782fb (0xe2fc bytes) 000244[1] 1788874921.038448 punch from 0x06caf9 to 0x0751f8 (0x8700 bytes) 000246[1] 1788874921.043279 mapread 0x2f2000 thru 0x2f4cb9 (0x2cba bytes) 000252[1] 1788874921.080348 trunc from 0x6782fc to 0x93dde1 (0x2c5ae6 bytes) 000255[1] 1788874921.093739 mapread 0x8d4000 thru 0x8d5b7e (0x1b7f bytes) truncating to largest ever: 0x9fea43 000258[0] 1788874921.111426 trunc from 0x93dde2 to 0x9fea42 (0xc0c61 bytes) 000259[0] 1788874921.137696 write_OD 0x79e000 thru 0x79ffff (0x2000 bytes) 000260[0] 1788874921.183666 write_OD 0x862000 thru 0x864fff (0x3000 bytes) 000261[1] 1788874921.221460 trunc from 0x9fea43 to 0x10272d (0x8fc315 bytes) 000263[0] 1788874921.261230 write_OD 0x7da000 thru 0x7dafff (0x1000 bytes) 000265[0] 1788874921.304233 mapread 0x47b000 thru 0x48041f (0x5420 bytes) 000267[1] 1788874921.316557 read_OD 0x2e4000 thru 0x2e6fff (0x3000 bytes) 000272[1] 1788874921.322101 read_OD 0x74c000 thru 0x74efff (0x3000 bytes) 000274[0] 1788874921.327147 write_OD 0x05d000 thru 0x061fff (0x5000 bytes) 000275[0] 1788874921.366862 mapread 0x1cc000 thru 0x1db68b (0xf68c bytes) 000277[1] 1788874921.393194 punch from 0x662375 to 0x6628e3 (0x56f bytes) 000278[0] 1788874921.418465 mapread 0x086000 thru 0x0897ae (0x37af bytes) 000279[1] 1788874921.447337 punch from 0x16ad13 to 0x174422 (0x9710 bytes) 000281[1] 1788874921.465932 punch from 0x61ca3a to 0x6271ae (0xa775 bytes) 000282[0] 1788874921.470212 mapwrite 0x0fa000 thru 0x0fb984 (0x1985 bytes) 000283[1] 1788874921.513477 punch from 0x1b662a to 0x1c19fb (0xb3d2 bytes) 000287[0] 1788874921.555582 mapread 0x3a6000 thru 0x3b1fb9 (0xbfba bytes) 000288[0] 1788874921.588454 trunc from 0x7db000 to 0x839250 (0x5e251 bytes) 000290[1] 1788874921.613642 read_OD 0x5c6000 thru 0x5cbfff (0x6000 bytes) 000293[1] 1788874921.642411 read_OD 0x102000 thru 0x10afff (0x9000 bytes) 000294[0] 1788874921.647749 trunc from 0x839251 to 0x185aa9 (0x6b37a7 bytes) 000298[0] 1788874921.692344 punch from 0x03d8d1 to 0x04b1e1 (0xd911 bytes) 000300[0] 1788874921.717825 read_OD 0x063000 thru 0x069fff (0x7000 bytes) 000303[1] 1788874921.731336 trunc from 0x185aaa to 0x1b0d78 (0x2b2cf bytes) 000304[0] 1788874921.760264 punch from 0x1409d7 to 0x150714 (0xfd3e bytes) 000307[0] 1788874921.805543 mapread 0x074000 thru 0x077efa (0x3efb bytes) 000310[1] 1788874921.831118 write_OD 0x0d6000 thru 0x0e1fff (0xc000 bytes) 000314[1] 1788874921.870962 trunc from 0x1b0d79 to 0x6cb66e (0x51a8f6 bytes) 000315[1] 1788874921.892062 punch from 0x3a09b7 to 0x3aeabf (0xe109 bytes) 000316[0] 1788874921.912310 read_OD 0x68e000 thru 0x694fff (0x7000 bytes) 000317[1] 1788874921.929517 mapwrite 0x407000 thru 0x407d17 (0xd18 bytes) 000318[1] 1788874921.986683 read_OD 0x53c000 thru 0x53efff (0x3000 bytes) 000319[1] 1788874922.006295 mapwrite 0x0ff000 thru 0x10e7d5 (0xf7d6 bytes) 000320[0] 1788874922.101574 mapread 0x13c000 thru 0x14bf43 (0xff44 bytes) 000325[1] 1788874922.135889 mapwrite 0x8da000 thru 0x8e69a1 (0xc9a2 bytes) 000326[1] 1788874922.235627 mapread 0x1a7000 thru 0x1b30db (0xc0dc bytes) 000328[1] 1788874922.283092 mapread 0x4a9000 thru 0x4b2b5a (0x9b5b bytes) 000330[0] 1788874922.302530 write_OD 0x9ea000 thru 0x9f4fff (0xb000 bytes) 000331[0] 1788874922.343231 punch from 0x64387e to 0x646091 (0x2814 bytes) 000333[0] 1788874922.364337 trunc from 0x9f5000 to 0x011608 (0x9e39f7 bytes) 000335[1] 1788874922.399001 write_OD 0x967000 thru 0x969fff (0x3000 bytes) 000337[0] 1788874922.441315 read_OD 0x7aa000 thru 0x7b6fff (0xd000 bytes) 000339[0] 1788874922.447908 mapread 0x433000 thru 0x43374f (0x750 bytes) 000341[0] 1788874922.452801 write_OD 0x205000 thru 0x20cfff (0x8000 bytes) 000345[1] 1788874922.505434 mapread 0x7cd000 thru 0x7d4cf2 (0x7cf3 bytes) 000348[0] 1788874922.524176 mapread 0x94d000 thru 0x956d6b (0x9d6c bytes) 000350[0] 1788874922.537915 punch from 0x1fad75 to 0x1fc4f8 (0x1784 bytes) 000351[1] 1788874922.561004 write_OD 0x7f1000 thru 0x7f1fff (0x1000 bytes) 000352[1] 1788874922.611755 trunc from 0x96a000 to 0x13f299 (0x82ad66 bytes) 000354[0] 1788874922.650621 mapread 0x07e000 thru 0x08d394 (0xf395 bytes) 000355[0] 1788874922.689854 punch from 0x074c88 to 0x07d082 (0x83fb bytes) 000358[0] 1788874922.718743 punch from 0x01ae49 to 0x022305 (0x74bd bytes) 000359[0] 1788874922.722580 punch from 0x02710c to 0x02f2ed (0x81e2 bytes) 000360[1] 1788874922.732341 read_OD 0x001000 thru 0x006fff (0x6000 bytes) 000363[1] 1788874922.750643 trunc from 0x13f29a to 0x0197aa (0x125aef bytes) 000365[0] 1788874922.775516 punch from 0x008fcf to 0x00d946 (0x4978 bytes) 000367[0] 1788874922.783652 trunc from 0x0197ab to 0x68185b (0x6680b1 bytes) 000368[0] 1788874922.800886 mapwrite 0x17a000 thru 0x187458 (0xd459 bytes) 000369[0] 1788874922.865294 mapwrite 0x1b5000 thru 0x1b5814 (0x815 bytes) 000370[1] 1788874922.895219 write_OD 0x586000 thru 0x589fff (0x4000 bytes) 000375[1] 1788874922.927629 mapread 0x2c0000 thru 0x2c1a00 (0x1a01 bytes) 000376[1] 1788874922.971119 trunc from 0x68185c to 0x38ad94 (0x2f6ac7 bytes) 000377[1] 1788874922.996420 trunc from 0x38ad95 to 0x7bd0a0 (0x43230c bytes) 000378[0] 1788874923.019402 punch from 0x78c581 to 0x79a8cc (0xe34c bytes) 000379[0] 1788874923.043211 punch from 0x553915 to 0x55c15c (0x8848 bytes) 000380[0] 1788874923.046353 mapwrite 0x539000 thru 0x53c078 (0x3079 bytes) 000385[1] 1788874923.123726 mapwrite 0x0a4000 thru 0x0a8599 (0x459a bytes) 000387[0] 1788874923.203686 mapwrite 0x251000 thru 0x258083 (0x7084 bytes) 000388[0] 1788874923.267174 read_OD 0x717000 thru 0x719fff (0x3000 bytes) 000389[1] 1788874923.282421 punch from 0x4385c6 to 0x43c3c6 (0x3e01 bytes) 000392[0] 1788874923.299578 write_OD 0x627000 thru 0x630fff (0xa000 bytes) 000393[1] 1788874923.330866 write_OD 0x187000 thru 0x189fff (0x3000 bytes) 000396[0] 1788874923.358845 punch from 0x5f9b83 to 0x5fddd4 (0x4252 bytes) 000397[1] 1788874923.369361 punch from 0x78c12e to 0x7933a9 (0x727c bytes) 000398[0] 1788874923.376350 trunc from 0x7bd0a1 to 0x4b2ce6 (0x30a3ba bytes) 000402[0] 1788874923.432466 read_OD 0x456000 thru 0x45cfff (0x7000 bytes) 000404[0] 1788874923.452757 trunc from 0x4b2ce7 to 0x2c46c0 (0x1ee626 bytes) 000405[0] 1788874923.469764 mapread 0x27a000 thru 0x284d5f (0xad60 bytes) 000407[1] 1788874923.515567 trunc from 0x2c46c1 to 0x93be1f (0x67775f bytes) 000408[1] 1788874923.549070 mapwrite 0x8e3000 thru 0x8efbc4 (0xcbc5 bytes) 000409[0] 1788874923.655363 mapread 0x08b000 thru 0x08bc15 (0xc16 bytes) 000412[1] 1788874923.686086 mapread 0x607000 thru 0x616aed (0xfaee bytes) 000414[1] 1788874923.749127 read_OD 0x5e1000 thru 0x5e7fff (0x7000 bytes) 000415[1] 1788874923.753650 trunc from 0x93be20 to 0x1c51d8 (0x776c47 bytes) 000417[0] 1788874923.799534 mapwrite 0x46c000 thru 0x475b8f (0x9b90 bytes) 000419[0] 1788874923.901105 trunc from 0x475b90 to 0x8b965b (0x443acc bytes) 000420[0] 1788874923.931979 mapread 0x656000 thru 0x657615 (0x1616 bytes) 000421[0] 1788874923.952374 write_OD 0x7e0000 thru 0x7eefff (0xf000 bytes) 000423[0] 1788874924.002292 read_OD 0x28e000 thru 0x29afff (0xd000 bytes) 000424[1] 1788874924.013178 punch from 0x438f42 to 0x43afa6 (0x2065 bytes) 000428[0] 1788874924.045063 mapread 0x03b000 thru 0x04ab8f (0xfb90 bytes) 000429[1] 1788874924.079085 mapwrite 0x840000 thru 0x8487de (0x87df bytes) 000432[0] 1788874924.150215 mapread 0x47e000 thru 0x48cd04 (0xed05 bytes) 000433[0] 1788874924.180398 read_OD 0x813000 thru 0x81cfff (0xa000 bytes) 000437[1] 1788874924.186729 write_OD 0x368000 thru 0x36ffff (0x8000 bytes) 000438[1] 1788874924.215336 mapwrite 0x2a0000 thru 0x2ae41d (0xe41e bytes) 000440[1] 1788874924.288225 write_OD 0x3b1000 thru 0x3bffff (0xf000 bytes) 000442[0] 1788874924.322150 mapread 0x7bc000 thru 0x7bedcc (0x2dcd bytes) 000446[0] 1788874924.337576 read_OD 0x0b1000 thru 0x0b9fff (0x9000 bytes) 000447[1] 1788874924.343739 read_OD 0x3e1000 thru 0x3edfff (0xd000 bytes) 000448[0] 1788874924.352436 write_OD 0x1c5000 thru 0x1c8fff (0x4000 bytes) 000450[1] 1788874924.386046 punch from 0x299362 to 0x2a5994 (0xc633 bytes) 000452[1] 1788874924.407313 mapread 0x2b5000 thru 0x2bdc55 (0x8c56 bytes) 000456[1] 1788874924.441404 read_OD 0x882000 thru 0x890fff (0xf000 bytes) 000457[0] 1788874924.456037 mapwrite 0x2de000 thru 0x2ed065 (0xf066 bytes) 000459[0] 1788874924.576060 punch from 0x83aca5 to 0x842b22 (0x7e7e bytes) 000460[1] 1788874924.591457 punch from 0x4ff93a to 0x507f6f (0x8636 bytes) 000461[0] 1788874924.600232 mapread 0x15e000 thru 0x1626ba (0x46bb bytes) 000464[1] 1788874924.636663 read_OD 0x7bc000 thru 0x7c8fff (0xd000 bytes) 000466[1] 1788874924.646252 write_OD 0x5fc000 thru 0x5fcfff (0x1000 bytes) 000468[1] 1788874924.686298 trunc from 0x8b965c to 0x33e81c (0x57ae3f bytes) 000469[1] 1788874924.720903 write_OD 0x310000 thru 0x319fff (0xa000 bytes) 000470[0] 1788874924.749291 trunc from 0x33e81d to 0x3c975d (0x8af41 bytes) 000471[0] 1788874924.771656 punch from 0x20161a to 0x20206d (0xa54 bytes) 000472[0] 1788874924.777624 mapwrite 0x969000 thru 0x9730c5 (0xa0c6 bytes) 000475[0] 1788874924.853108 write_OD 0x4d6000 thru 0x4e1fff (0xc000 bytes) 000476[1] 1788874924.888937 punch from 0x142d41 to 0x145635 (0x28f5 bytes) 000477[1] 1788874924.915036 mapread 0x106000 thru 0x107a2e (0x1a2f bytes) 000480[0] 1788874924.937111 write_OD 0x02b000 thru 0x035fff (0xb000 bytes) 000481[1] 1788874924.967927 write_OD 0x266000 thru 0x26dfff (0x8000 bytes) 000485[0] 1788874924.997656 mapwrite 0x7cd000 thru 0x7d6511 (0x9512 bytes) 000486[1] 1788874925.051577 punch from 0x0a8505 to 0x0aae3d (0x2939 bytes) 000487[1] 1788874925.061516 read_OD 0x6e4000 thru 0x6f2fff (0xf000 bytes) 000488[0] 1788874925.073387 punch from 0x0c416e to 0x0d222e (0xe0c1 bytes) 000489[0] 1788874925.083563 read_OD 0x2a9000 thru 0x2aafff (0x2000 bytes) 000491[1] 1788874925.095119 punch from 0x3b350e to 0x3b4eee (0x19e1 bytes) 000492[0] 1788874925.117346 read_OD 0x363000 thru 0x36dfff (0xb000 bytes) 000493[0] 1788874925.139032 read_OD 0x214000 thru 0x221fff (0xe000 bytes) 000494[1] 1788874925.148155 mapwrite 0x5f4000 thru 0x6010a0 (0xd0a1 bytes) 000496[0] 1788874925.226622 mapread 0x87b000 thru 0x88269f (0x76a0 bytes) 000498[0] 1788874925.261356 mapwrite 0x269000 thru 0x277fcd (0xefce bytes) All operations completed A-OK! PASS 16a (25s) == sanityn test 16b: 500 iterations of dual-mount fsx at small size ========================================================== 09:42:07 (1788874927) keep default fallocate mode: 2 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.169494 s, 49.5 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.185087 s, 45.3 MB/s Chance of close/open is 1 in 50 Seed set to 4929 fd 0: /mnt/lustre/f16b.sanityn fd 1: /mnt/lustre2/f16b.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read skipping zero length punch hole skipping zero size read truncating to largest ever: 0x2674e4 000010[1] 1788874930.151619 trunc from 00000000 to 0x2674e3 (0x2674e4 bytes) 000011[0] 1788874930.167124 mapwrite 0x1d5356 thru 0x1d5a21 (0x6cc bytes) 000012[1] 1788874930.230900 mapread 0x01f075 thru 0x01f6dd (0x669 bytes) 000014[1] 1788874930.258312 read 0x16e172 thru 0x16f9b7 (0x1846 bytes) 000015[0] 1788874930.275751 read 0x10052e thru 0x100d94 (0x867 bytes) 000016[0] 1788874930.279000 mapread 0x1eb715 thru 0x1ec313 (0xbff bytes) 000019[1] 1788874930.307643 mapread 0x22612c thru 0x2272b5 (0x118a bytes) 000020[0] 1788874930.339194 read 0x01c461 thru 0x01d9bd (0x155d bytes) 000021[1] 1788874930.384698 mapread 0x1d8767 thru 0x1d9da0 (0x163a bytes) 000022[1] 1788874930.404718 punch from 0x25143f to 0x252ab3 (0x1675 bytes) 000023[1] 1788874930.419234 write 0x61a875 thru 0x61c4c2 (0x1c4e bytes) 000024[1] 1788874930.420903 read 0x0d48fb thru 0x0d58b3 (0xfb9 bytes) 000026[1] 1788874930.427426 mapread 0x2ee137 thru 0x2ef612 (0x14dc bytes) 000029[1] 1788874930.462084 mapwrite 0x328153 thru 0x329f17 (0x1dc5 bytes) truncating to largest ever: 0x31db21 000030[0] 1788874930.518136 trunc from 0x61c4c3 to 0x31db20 (0x2fe9a2 bytes) 000031[0] 1788874930.562114 read 0x2e1e8b thru 0x2e2020 (0x196 bytes) 000036[1] 1788874930.634340 punch from 0x194486 to 0x1951be (0xd39 bytes) 000038[0] 1788874930.667101 write 0x7650a8 thru 0x766b4e (0x1aa7 bytes) 000039[1] 1788874930.682228 punch from 0x1ba0b6 to 0x1ba5ce (0x519 bytes) 000040[1] 1788874930.697473 mapread 0x3fce91 thru 0x3fd8cc (0xa3c bytes) 000041[0] 1788874930.730904 read 0x44a15b thru 0x44aeca (0xd70 bytes) 000044[1] 1788874930.761104 mapread 0x10ed8a thru 0x1103df (0x1656 bytes) 000045[0] 1788874930.784447 write 0x0e728f thru 0x0e733c (0xae bytes) truncating to largest ever: 0x7dab12 000046[1] 1788874930.795024 trunc from 0x766b4f to 0x7dab11 (0x73fc3 bytes) 000055[0] 1788874930.879365 read 0x2e99a2 thru 0x2eb94a (0x1fa9 bytes) 000056[0] 1788874930.895340 mapwrite 0x8a783c thru 0x8a90aa (0x186f bytes) 000057[1] 1788874930.955475 mapread 0x53084c thru 0x531427 (0xbdc bytes) 000061[1] 1788874930.995571 punch from 0x0f26f7 to 0x0f3853 (0x115d bytes) 000062[1] 1788874931.014757 punch from 0x490124 to 0x4908e4 (0x7c1 bytes) 000063[1] 1788874931.022414 mapwrite 0x08120b thru 0x082875 (0x166b bytes) 000064[1] 1788874931.073967 write 0x67deee thru 0x67e580 (0x693 bytes) 000066[0] 1788874931.093666 read 0x357b5c thru 0x358e54 (0x12f9 bytes) 000071[1] 1788874931.146688 write 0x2ff8f3 thru 0x30044e (0xb5c bytes) 000072[1] 1788874931.170662 mapwrite 0x8e22ec thru 0x8e2792 (0x4a7 bytes) truncating to largest ever: 0x9341c8 000074[0] 1788874931.227419 trunc from 0x8e2793 to 0x9341c7 (0x51a35 bytes) 000075[0] 1788874931.251834 trunc from 0x9341c8 to 0x267a7f (0x6cc748 bytes) 000076[0] 1788874931.278162 mapread 0x21cd23 thru 0x21eb38 (0x1e16 bytes) 000077[1] 1788874931.298893 mapwrite 0x9e045b thru 0x9e163f (0x11e5 bytes) 000078[1] 1788874931.370368 read 0x8a45f0 thru 0x8a5a23 (0x1434 bytes) 000079[1] 1788874931.374019 punch from 0x0e6875 to 0x0e6a4e (0x1da bytes) 000082[1] 1788874931.430963 punch from 0x5e7a67 to 0x5e863f (0xbd9 bytes) 000083[0] 1788874931.433830 write 0x20f3d8 thru 0x21044b (0x1074 bytes) 000084[1] 1788874931.457742 mapread 0x592b99 thru 0x5945cd (0x1a35 bytes) 000085[1] 1788874931.495547 read 0x4cf650 thru 0x4d0dfc (0x17ad bytes) 000086[0] 1788874931.509285 mapread 0x9a7217 thru 0x9a7a5f (0x849 bytes) 000087[1] 1788874931.528184 mapread 0x6af147 thru 0x6aff7c (0xe36 bytes) 000088[0] 1788874931.545956 mapwrite 0x8409b2 thru 0x842099 (0x16e8 bytes) 000089[1] 1788874931.603480 trunc from 0x9e1640 to 0x57eec6 (0x462779 bytes) 000090[1] 1788874931.636334 read 0x3261d5 thru 0x3271b3 (0xfdf bytes) 000091[0] 1788874931.658157 punch from 0x225d6b to 0x226d69 (0xfff bytes) 000093[1] 1788874931.690076 mapread 0x331f7d thru 0x3323ea (0x46e bytes) 000095[0] 1788874931.720477 mapwrite 0x67f9e6 thru 0x680f40 (0x155b bytes) 000096[1] 1788874931.772077 trunc from 0x680f41 to 0x097d86 (0x5e91ba bytes) 000097[1] 1788874931.813579 mapwrite 0x3e808a thru 0x3e889c (0x813 bytes) 000098[0] 1788874931.856348 write 0x0b1656 thru 0x0b2c3b (0x15e6 bytes) 000102[0] 1788874931.909404 mapread 0x046126 thru 0x046770 (0x64b bytes) 000103[0] 1788874931.924773 punch from 0x1a626f to 0x1a80c3 (0x1e55 bytes) 000104[1] 1788874931.934951 punch from 0x0a4894 to 0x0a4fe8 (0x755 bytes) 000106[0] 1788874931.955965 read 0x3a0f90 thru 0x3a0fad (0x1e bytes) 000109[1] 1788874932.000230 mapwrite 0x0874af thru 0x088ccf (0x1821 bytes) 000110[1] 1788874932.040360 mapread 0x23829c thru 0x2390f3 (0xe58 bytes) 000112[0] 1788874932.059896 read 0x0cd641 thru 0x0cedd9 (0x1799 bytes) 000114[0] 1788874932.119975 mapread 0x2c004a thru 0x2c095b (0x912 bytes) 000115[0] 1788874932.130371 write 0x6efbcd thru 0x6f15b0 (0x19e4 bytes) truncating to largest ever: 0x9649f9 000116[1] 1788874932.148422 trunc from 0x6f15b1 to 0x9649f8 (0x273448 bytes) 000119[0] 1788874932.203752 mapread 0x04d0f7 thru 0x04d4a8 (0x3b2 bytes) 000120[1] 1788874932.243436 mapread 0x1879ba thru 0x18815f (0x7a6 bytes) 000122[0] 1788874932.260284 punch from 0x371758 to 0x373086 (0x192f bytes) 000125[1] 1788874932.301364 punch from 0x952fb0 to 0x9540d3 (0x1124 bytes) 000127[1] 1788874932.327636 punch from 0x6c4000 to 0x6c5da6 (0x1da7 bytes) 000130[0] 1788874932.355705 write 0x2926a5 thru 0x293bff (0x155b bytes) 000131[1] 1788874932.374360 punch from 0x527921 to 0x5288ee (0xfce bytes) 000132[0] 1788874932.387863 punch from 0x0040a3 to 0x005024 (0xf82 bytes) 000133[0] 1788874932.401424 write 0x7afdcc thru 0x7b0fd2 (0x1207 bytes) 000134[0] 1788874932.442381 write 0x11a6ef thru 0x11bf73 (0x1885 bytes) 000135[0] 1788874932.463757 punch from 0x00d921 to 0x00db6e (0x24e bytes) 000136[0] 1788874932.478498 punch from 0x01e78b to 0x01f5eb (0xe61 bytes) 000141[0] 1788874932.524329 read 0x95a04c thru 0x95a895 (0x84a bytes) 000143[1] 1788874932.550095 write 0x794938 thru 0x7965ab (0x1c74 bytes) 000144[0] 1788874932.574019 write 0x077b1f thru 0x079389 (0x186b bytes) 000145[0] 1788874932.591612 mapwrite 0x3d7d7a thru 0x3d8e12 (0x1099 bytes) 000146[1] 1788874932.652293 read 0x5f6343 thru 0x5f68c5 (0x583 bytes) 000147[0] 1788874932.661918 mapread 0x24e6c1 thru 0x24eb16 (0x456 bytes) 000148[1] 1788874932.676358 write 0x203047 thru 0x203973 (0x92d bytes) 000149[1] 1788874932.697404 mapread 0x903b74 thru 0x905181 (0x160e bytes) 000150[1] 1788874932.717726 write 0x04a783 thru 0x04c248 (0x1ac6 bytes) 000153[1] 1788874932.741492 punch from 0x451682 to 0x451ec5 (0x844 bytes) 000154[1] 1788874932.756480 read 0x1bdf6c thru 0x1bfe36 (0x1ecb bytes) 000155[0] 1788874932.759684 trunc from 0x9649f9 to 0x7beff0 (0x1a5a08 bytes) 000158[0] 1788874932.810710 trunc from 0x7beff1 to 0x003e6f (0x7bb181 bytes) 000159[1] 1788874932.822318 punch from 0x0025ac to 0x0038bf (0x1314 bytes) 000160[0] 1788874932.844059 write 0x35d55b thru 0x35e666 (0x110c bytes) 000161[0] 1788874932.862092 read 0x2e7ad7 thru 0x2e7ed8 (0x402 bytes) 000162[1] 1788874932.892124 mapread 0x35414e thru 0x3553c2 (0x1275 bytes) 000164[1] 1788874932.924264 punch from 0x0ad2c6 to 0x0adda1 (0xadc bytes) 000172[1] 1788874933.022153 read 0x18326a thru 0x1839e3 (0x77a bytes) 000173[1] 1788874933.053609 mapwrite 0x686c1b thru 0x686d85 (0x16b bytes) 000174[1] 1788874933.126183 read 0x443643 thru 0x444a85 (0x1443 bytes) 000176[0] 1788874933.136853 mapread 0x5d9920 thru 0x5db845 (0x1f26 bytes) 000177[0] 1788874933.165314 mapwrite 0x354c4b thru 0x354c69 (0x1f bytes) 000178[0] 1788874933.234049 read 0x3de3b8 thru 0x3df476 (0x10bf bytes) 000179[0] 1788874933.234656 mapread 0x55e03f thru 0x55ff59 (0x1f1b bytes) 000180[1] 1788874933.250658 mapwrite 0x93bd71 thru 0x93db13 (0x1da3 bytes) 000183[0] 1788874933.330682 read 0x094c37 thru 0x095de9 (0x11b3 bytes) 000185[1] 1788874933.350464 punch from 0x3dc7dd to 0x3ddc6a (0x148e bytes) 000186[0] 1788874933.380265 read 0x750023 thru 0x750a4a (0xa28 bytes) 000189[1] 1788874933.403443 mapwrite 0x4fc898 thru 0x4fd8f7 (0x1060 bytes) 000190[1] 1788874933.453208 trunc from 0x93db14 to 0x2f4d1f (0x648df4 bytes) 000191[1] 1788874933.476091 mapread 0x24aad9 thru 0x24c4b4 (0x19dc bytes) 000193[0] 1788874933.511705 mapwrite 0x60f92f thru 0x61176d (0x1e3f bytes) 000195[0] 1788874933.568302 punch from 0x33c93c to 0x33d658 (0xd1d bytes) 000199[1] 1788874933.588925 trunc from 0x61176e to 0x4fe336 (0x113437 bytes) 000200[0] 1788874933.611887 punch from 0x443bec to 0x4453e1 (0x17f6 bytes) 000203[0] 1788874933.659290 punch from 0x0fb33e to 0x0fbff1 (0xcb4 bytes) 000204[0] 1788874933.687110 trunc from 0x4fe337 to 0x5ed1c5 (0xeee8f bytes) 000205[0] 1788874933.706498 trunc from 0x5ed1c6 to 0x67c2e2 (0x8f11d bytes) 000206[1] 1788874933.717593 mapwrite 0x81ec1a thru 0x81f96a (0xd51 bytes) 000208[0] 1788874933.776516 mapwrite 0x5e02d6 thru 0x5e0d3d (0xa68 bytes) 000209[1] 1788874933.827562 punch from 0x7db1ed to 0x7dce8a (0x1c9e bytes) 000211[1] 1788874933.852136 punch from 0x75a68a to 0x75b475 (0xdec bytes) 000212[0] 1788874933.862374 mapread 0x1a1cce thru 0x1a2b54 (0xe87 bytes) 000213[0] 1788874933.884091 write 0x899697 thru 0x89a2d8 (0xc42 bytes) 000214[0] 1788874933.895393 write 0x397186 thru 0x3978f0 (0x76b bytes) 000216[1] 1788874933.919703 mapread 0x0d2226 thru 0x0d269d (0x478 bytes) 000217[1] 1788874933.938394 write 0x94f76c thru 0x950e16 (0x16ab bytes) 000219[1] 1788874933.954671 mapwrite 0x7a74e1 thru 0x7a8874 (0x1394 bytes) 000220[1] 1788874933.976387 read 0x014202 thru 0x014b28 (0x927 bytes) 000223[1] 1788874933.997518 write 0x79e9c8 thru 0x79f35b (0x994 bytes) 000225[1] 1788874934.016460 write 0x536f4d thru 0x538544 (0x15f8 bytes) 000226[0] 1788874934.035919 read 0x32d125 thru 0x32df21 (0xdfd bytes) 000227[0] 1788874934.058540 trunc from 0x950e17 to 0x4737b9 (0x4dd65d bytes) 000228[0] 1788874934.084695 mapread 0x04eec8 thru 0x05029f (0x13d8 bytes) 000229[0] 1788874934.106372 punch from 0x33133e to 0x331e6d (0xb30 bytes) 000230[1] 1788874934.116035 trunc from 0x4737ba to 0x2c5b47 (0x1adc72 bytes) 000231[0] 1788874934.143688 write 0x4b6eb6 thru 0x4b7610 (0x75b bytes) 000233[1] 1788874934.162371 read 0x220697 thru 0x2224b9 (0x1e23 bytes) 000235[0] 1788874934.216538 trunc from 0x4b7611 to 0x2b3244 (0x2043cc bytes) 000238[1] 1788874934.263982 trunc from 0x2b3245 to 0x21c487 (0x96dbd bytes) 000239[1] 1788874934.289282 write 0x0d13c5 thru 0x0d151f (0x15b bytes) 000240[0] 1788874934.293362 mapread 0x011fae thru 0x012f2c (0xf7f bytes) 000241[0] 1788874934.311257 write 0x2ec2e7 thru 0x2edd34 (0x1a4e bytes) 000243[0] 1788874934.333215 read 0x05734d thru 0x058ca5 (0x1959 bytes) 000245[0] 1788874934.383252 mapread 0x0414dd thru 0x041784 (0x2a8 bytes) 000247[0] 1788874934.404544 punch from 0x17e516 to 0x17e523 (0xe bytes) 000249[1] 1788874934.422878 trunc from 0x2edd35 to 0x80a25e (0x51c52a bytes) 000251[1] 1788874934.458687 mapread 0x6ac39e thru 0x6ac6a7 (0x30a bytes) 000252[0] 1788874934.488486 trunc from 0x80a25f to 0x6299df (0x1e087f bytes) 000255[1] 1788874934.547901 write 0x0bb095 thru 0x0bb1d2 (0x13e bytes) 000256[1] 1788874934.567816 mapread 0x56772e thru 0x5696d9 (0x1fac bytes) 000259[0] 1788874934.610406 mapread 0x098936 thru 0x099ffb (0x16c6 bytes) 000261[1] 1788874934.643543 write 0x832e54 thru 0x8348b1 (0x1a5e bytes) 000263[1] 1788874934.658646 mapread 0x4dc723 thru 0x4dd83d (0x111b bytes) 000264[0] 1788874934.670074 mapread 0x0e6603 thru 0x0e66aa (0xa8 bytes) 000267[1] 1788874934.693078 punch from 0x68cc71 to 0x68d515 (0x8a5 bytes) 000269[1] 1788874934.710032 read 0x6ae5b2 thru 0x6af2a1 (0xcf0 bytes) 000270[1] 1788874934.719934 punch from 0x05eb3c to 0x0607b8 (0x1c7d bytes) 000271[0] 1788874934.738395 write 0x47029e thru 0x470a41 (0x7a4 bytes) 000272[0] 1788874934.782328 punch from 0x14e3e0 to 0x14f70d (0x132e bytes) 000274[0] 1788874934.805027 trunc from 0x8348b2 to 0x301f55 (0x53295c bytes) 000276[0] 1788874934.834570 mapwrite 0x6ef430 thru 0x6f112f (0x1d00 bytes) 000278[0] 1788874934.884963 mapread 0x6876aa thru 0x688e31 (0x1788 bytes) 000279[1] 1788874934.894016 write 0x5f74aa thru 0x5f74c1 (0x18 bytes) 000283[0] 1788874934.932600 mapread 0x4c8787 thru 0x4c962d (0xea7 bytes) 000284[0] 1788874934.954408 punch from 0x30effe to 0x30f566 (0x569 bytes) 000285[0] 1788874934.958306 read 0x0fdd78 thru 0x0fed59 (0xfe2 bytes) 000287[0] 1788874934.974720 mapread 0x223c80 thru 0x225636 (0x19b7 bytes) 000289[1] 1788874935.004867 mapread 0x201b0e thru 0x203709 (0x1bfc bytes) 000292[0] 1788874935.030097 write 0x491df4 thru 0x492128 (0x335 bytes) 000293[0] 1788874935.048613 mapread 0x43d15b thru 0x43d39a (0x240 bytes) 000294[0] 1788874935.051452 punch from 0x035171 to 0x035918 (0x7a8 bytes) 000295[1] 1788874935.058011 write 0x68a4c9 thru 0x68bcdf (0x1817 bytes) 000296[0] 1788874935.071098 trunc from 0x6f1130 to 0x1ca4d6 (0x526c59 bytes) 000299[0] 1788874935.084258 mapread 0x026ec6 thru 0x02725e (0x399 bytes) 000305[0] 1788874935.120334 read 0x045dd2 thru 0x047343 (0x1572 bytes) 000308[0] 1788874935.142279 trunc from 0x1ca4d7 to 0x60ce2f (0x442959 bytes) 000312[1] 1788874935.179102 write 0x6d5899 thru 0x6d61c7 (0x92f bytes) 000313[1] 1788874935.192073 mapwrite 0x7b3401 thru 0x7b4afa (0x16fa bytes) 000314[0] 1788874935.250917 mapread 0x3ecad3 thru 0x3edc52 (0x1180 bytes) 000316[1] 1788874935.303073 read 0x09436b thru 0x094434 (0xca bytes) 000320[0] 1788874935.344981 read 0x2b7d58 thru 0x2b89b2 (0xc5b bytes) 000321[1] 1788874935.345552 trunc from 0x7b4afb to 0x7a921a (0xb8e0 bytes) 000322[0] 1788874935.368884 read 0x1762d4 thru 0x17720d (0xf3a bytes) 000325[1] 1788874935.403259 mapread 0x5dec2e thru 0x5dece7 (0xba bytes) 000329[0] 1788874935.429325 trunc from 0x7a921b to 0x1054dd (0x6a3d3d bytes) 000334[1] 1788874935.475142 mapread 0x0753f1 thru 0x077301 (0x1f11 bytes) 000335[1] 1788874935.492342 mapread 0x0f1fbc thru 0x0f21bf (0x204 bytes) 000338[1] 1788874935.506685 mapwrite 0x64333d thru 0x644164 (0xe28 bytes) 000344[1] 1788874935.578019 write 0x5bb823 thru 0x5bc5ae (0xd8c bytes) 000345[1] 1788874935.593236 trunc from 0x644165 to 0x5e3efa (0x6026a bytes) 000347[1] 1788874935.619433 read 0x3ec4e9 thru 0x3ec64c (0x164 bytes) 000349[1] 1788874935.632610 read 0x4a53c8 thru 0x4a5f58 (0xb91 bytes) 000350[0] 1788874935.641557 mapwrite 0x7ef1b3 thru 0x7f0a15 (0x1863 bytes) 000353[1] 1788874935.696322 mapread 0x653715 thru 0x654140 (0xa2c bytes) 000354[0] 1788874935.711598 mapwrite 0x8a39bb thru 0x8a53c4 (0x1a0a bytes) 000356[1] 1788874935.777127 mapwrite 0x76f2df thru 0x7704a2 (0x11c4 bytes) 000357[1] 1788874935.825194 write 0x45e070 thru 0x45f2a2 (0x1233 bytes) truncating to largest ever: 0x9e584e 000358[1] 1788874935.839500 trunc from 0x8a53c5 to 0x9e584d (0x140489 bytes) skipping zero size write 000360[1] 1788874935.880234 mapread 0x2b666e thru 0x2b7429 (0xdbc bytes) 000361[0] 1788874935.894198 mapread 0x7b4c9d thru 0x7b607d (0x13e1 bytes) 000364[0] 1788874935.922020 mapwrite 0x0fb9c3 thru 0x0fcba6 (0x11e4 bytes) 000365[0] 1788874935.950584 read 0x315266 thru 0x315d78 (0xb13 bytes) 000366[0] 1788874935.962557 write 0x0fcf46 thru 0x0fed43 (0x1dfe bytes) 000368[1] 1788874935.966388 write 0x1b8640 thru 0x1b93bd (0xd7e bytes) 000372[0] 1788874936.004069 mapread 0x377e8c thru 0x3789b0 (0xb25 bytes) 000373[0] 1788874936.016312 write 0x4918a0 thru 0x491b65 (0x2c6 bytes) 000374[0] 1788874936.022576 punch from 0x019b4f to 0x01b78a (0x1c3c bytes) 000375[1] 1788874936.028326 read 0x86c74e thru 0x86d61c (0xecf bytes) 000377[1] 1788874936.037665 mapwrite 0x2342a3 thru 0x235ec2 (0x1c20 bytes) 000378[0] 1788874936.070748 mapwrite 0x16156d thru 0x162747 (0x11db bytes) 000380[1] 1788874936.110260 read 0x164a85 thru 0x166365 (0x18e1 bytes) 000381[0] 1788874936.119003 write 0x9f50db thru 0x9f525f (0x185 bytes) 000382[1] 1788874936.132336 write 0x307410 thru 0x3088f9 (0x14ea bytes) 000388[0] 1788874936.188420 write 0x3e1d9c thru 0x3e2f73 (0x11d8 bytes) 000389[1] 1788874936.202219 mapwrite 0x529053 thru 0x5296f4 (0x6a2 bytes) 000390[0] 1788874936.233913 read 0x5bbcda thru 0x5bd199 (0x14c0 bytes) 000393[1] 1788874936.248811 mapread 0x9ac083 thru 0x9ae01b (0x1f99 bytes) 000394[1] 1788874936.268541 trunc from 0x9f5260 to 0x289ab8 (0x76b7a7 bytes) 000396[0] 1788874936.303739 write 0x25aebc thru 0x25c136 (0x127b bytes) 000398[1] 1788874936.324771 mapread 0x0f2fa4 thru 0x0f4b72 (0x1bcf bytes) 000399[1] 1788874936.357616 punch from 0x2694e9 to 0x26b09f (0x1bb7 bytes) 000401[0] 1788874936.369830 punch from 0x16b625 to 0x16d221 (0x1bfd bytes) 000403[1] 1788874936.391667 punch from 0x122053 to 0x122af3 (0xaa1 bytes) 000405[1] 1788874936.405582 punch from 0x0fb4c6 to 0x0fbdfd (0x938 bytes) 000406[1] 1788874936.414207 write 0x3c2a37 thru 0x3c3127 (0x6f1 bytes) 000409[0] 1788874936.431428 punch from 0x3997c5 to 0x399df4 (0x630 bytes) 000411[0] 1788874936.448575 punch from 0x26b150 to 0x26ba6f (0x920 bytes) 000412[1] 1788874936.456046 mapwrite 0x659c44 thru 0x65acca (0x1087 bytes) 000413[1] 1788874936.493528 write 0x98b1ba thru 0x98d123 (0x1f6a bytes) 000415[1] 1788874936.509034 mapwrite 0x06249c thru 0x0634b0 (0x1015 bytes) 000416[0] 1788874936.538430 mapread 0x95258f thru 0x953f65 (0x19d7 bytes) 000418[1] 1788874936.554113 mapwrite 0x458615 thru 0x459f23 (0x190f bytes) 000419[0] 1788874936.588273 mapwrite 0x5e1fe7 thru 0x5e27d5 (0x7ef bytes) 000420[1] 1788874936.632444 punch from 0x5aa105 to 0x5aa844 (0x740 bytes) 000421[0] 1788874936.642249 mapwrite 0x27bb15 thru 0x27d0ae (0x159a bytes) 000423[0] 1788874936.687938 write 0x6c3bd4 thru 0x6c5297 (0x16c4 bytes) 000425[0] 1788874936.709910 punch from 0x71dc89 to 0x71ea85 (0xdfd bytes) 000426[1] 1788874936.721200 punch from 0x5a5fde to 0x5a661f (0x642 bytes) 000428[0] 1788874936.747200 read 0x63a7c0 thru 0x63b13c (0x97d bytes) 000430[1] 1788874936.753477 read 0x675936 thru 0x677422 (0x1aed bytes) 000431[1] 1788874936.761483 trunc from 0x98d124 to 0x81e6bd (0x16ea66 bytes) 000432[1] 1788874936.778666 trunc from 0x81e6be to 0x620a43 (0x1fdc7a bytes) 000434[1] 1788874936.793750 punch from 0x192b5b to 0x193d3e (0x11e4 bytes) 000435[1] 1788874936.796420 trunc from 0x620a44 to 0x35ff3d (0x2c0b06 bytes) 000439[1] 1788874936.830109 mapwrite 0x3cb644 thru 0x3cbc87 (0x644 bytes) 000440[1] 1788874936.874234 mapwrite 0x431b59 thru 0x431e38 (0x2e0 bytes) 000442[0] 1788874936.925668 read 0x20f83a thru 0x20f9c1 (0x188 bytes) 000443[0] 1788874936.934956 write 0x7295b7 thru 0x72aeab (0x18f5 bytes) 000447[0] 1788874936.961347 read 0x071818 thru 0x0735e4 (0x1dcd bytes) 000449[1] 1788874936.998177 mapwrite 0x0d9583 thru 0x0db359 (0x1dd7 bytes) 000451[0] 1788874937.061933 write 0x916765 thru 0x9169b2 (0x24e bytes) 000452[1] 1788874937.072869 mapwrite 0x442430 thru 0x44403e (0x1c0f bytes) 000454[0] 1788874937.118336 write 0x3243ac thru 0x324f9a (0xbef bytes) 000455[0] 1788874937.128968 write 0x4fa444 thru 0x4fbcd4 (0x1891 bytes) 000456[0] 1788874937.145158 write 0x54cf74 thru 0x54dc0b (0xc98 bytes) 000457[1] 1788874937.157136 write 0x13d44f thru 0x13dabc (0x66e bytes) 000462[0] 1788874937.203841 trunc from 0x9169b3 to 0x1eac6a (0x72bd48 bytes) 000463[0] 1788874937.223080 mapwrite 0x75b1d0 thru 0x75cc29 (0x1a5a bytes) 000464[1] 1788874937.266465 write 0x55446a thru 0x555f42 (0x1ad9 bytes) 000465[1] 1788874937.285877 write 0x1b6918 thru 0x1b7612 (0xcfb bytes) 000466[1] 1788874937.294513 read 0x756750 thru 0x757ffe (0x18af bytes) 000467[1] 1788874937.305188 mapwrite 0x2f5318 thru 0x2f72dd (0x1fc6 bytes) 000469[0] 1788874937.362144 mapwrite 0x2eeaf6 thru 0x2f0448 (0x1953 bytes) 000470[1] 1788874937.419236 read 0x3d3a23 thru 0x3d5997 (0x1f75 bytes) 000471[0] 1788874937.438228 write 0x609719 thru 0x60b3e3 (0x1ccb bytes) 000472[0] 1788874937.459027 mapread 0x518c11 thru 0x518c80 (0x70 bytes) 000473[1] 1788874937.462513 mapread 0x66d958 thru 0x66de77 (0x520 bytes) 000474[0] 1788874937.480814 trunc from 0x75cc2a to 0x797d7d (0x3b154 bytes) 000479[0] 1788874937.518526 trunc from 0x797d7e to 0x071a74 (0x726309 bytes) 000481[1] 1788874937.545238 mapwrite 0x57dcdc thru 0x57e414 (0x739 bytes) 000484[0] 1788874937.619344 trunc from 0x57e415 to 0x28176e (0x2fcca6 bytes) 000486[1] 1788874937.651382 mapread 0x01114a thru 0x012c3c (0x1af3 bytes) 000487[0] 1788874937.668728 write 0x232a8e thru 0x233af8 (0x106b bytes) 000488[1] 1788874937.680426 mapwrite 0x897c5b thru 0x898457 (0x7fd bytes) 000489[1] 1788874937.728274 punch from 0x840eae to 0x841649 (0x79c bytes) 000490[0] 1788874937.734197 punch from 0x07cfe7 to 0x07e258 (0x1272 bytes) 000491[0] 1788874937.754568 mapread 0x72b1bc thru 0x72b4c8 (0x30d bytes) 000492[0] 1788874937.770845 mapread 0x2bf1d3 thru 0x2bf687 (0x4b5 bytes) 000493[0] 1788874937.785924 read 0x7c74fc thru 0x7c8045 (0xb4a bytes) 000495[1] 1788874937.789626 punch from 0x72c8ed to 0x72cbeb (0x2ff bytes) 000496[0] 1788874937.808550 read 0x20675f thru 0x207f90 (0x1832 bytes) 000497[1] 1788874937.820668 mapread 0x68cdef thru 0x68e70c (0x191e bytes) 000498[0] 1788874937.852513 mapwrite 0x7a13b7 thru 0x7a2cce (0x1918 bytes) 000499[1] 1788874937.888922 punch from 0x46ed89 to 0x46fccb (0xf43 bytes) All operations completed A-OK! PASS 16b (12s) == sanityn test 16c: verify data consistency on ldiskfs with cache disabled (b=17397) ========================================================== 09:42:19 (1788874939) SKIP: sanityn test_16c dio on ldiskfs only SKIP 16c (2s) == sanityn test 16d: Verify DIO and buffer IO with two clients ========================================================== 09:42:21 (1788874941) 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.165239 s, 50.8 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.17016 s, 49.3 MB/s ldlm.namespaces.MGC192.168.206.126@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff9d9e08e29800.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff9d9e11804800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff9d9e08e29800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff9d9e11804800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff9d9e08e29800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff9d9e11804800.lru_size=clear 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.41343 s, 23.8 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 1.19822 s, 87.5 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 0.983991 s, 107 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 2.1219 s, 49.4 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 3.28841 s, 31.9 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 1.56307 s, 67.1 MB/s PASS 16d (19s) == sanityn test 16e: Verify size consistency for O_DIRECT write ========================================================== 09:42:40 (1788874960) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.100512 s, 104 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.060385 s, 17.4 MB/s PASS 16e (3s) == sanityn test 16f: rw sequential consistency vs drop_caches ========================================================== 09:42:43 (1788874963) TIMEOUT PASS 16f (23s) == sanityn test 16g: mmap rw sequential consistency vs drop_caches ========================================================== 09:43:06 (1788874986) TIMEOUT PASS 16g (23s) == sanityn test 16h: mmap read after truncate file ======= 09:43:29 (1788875009) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16h (3s) == sanityn test 16i: read after truncate file ============ 09:43:32 (1788875012) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16i (3s) == sanityn test 16j: race dio with buffered i/o ========== 09:43:35 (1788875015) 1+0 records in 1+0 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.480791 s, 54.5 MB/s bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.480598 s, 54.5 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.845251 s, 31.0 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.630777 s, 41.6 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.939347 s, 27.9 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.00735 s, 26.0 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.27389 s, 20.6 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.821613 s, 31.9 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.84312 s, 31.1 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.03153 s, 25.4 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.09445 s, 24.0 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.783338 s, 33.5 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.827238 s, 31.7 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.992929 s, 26.4 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.08496 s, 24.2 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.975268 s, 26.9 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.08846 s, 24.1 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.20539 s, 21.7 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.24683 s, 21.0 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK PASS 16j (14s) == sanityn test 16k: Parallel FSX and drop caches should not panic ========================================================== 09:43:49 (1788875029) + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.1 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.2 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.3 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.4 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.5 -l 67108864 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: fsx: Operation not supported test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported truncating to largest ever: 0x1a7062c truncating to largest ever: 0x1a7062c fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported truncating to largest ever: 0x232e1d3 fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x32cce75 truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! + fsx 1 done: rc=0 Tue Sep 8 09:44:02 EDT 2026 + fsx 2 done: rc=0 Tue Sep 8 09:44:02 EDT 2026 + fsx 3 done: rc=0 Tue Sep 8 09:44:02 EDT 2026 + fsx 4 done: rc=0 Tue Sep 8 09:44:02 EDT 2026 All operations completed A-OK! + fsx 5 done: rc=0 Tue Sep 8 09:44:02 EDT 2026 PASS 16k (17s) == sanityn test 17: resource creation/LVB creation race ========================================================================= 09:44:06 (1788875046) fail_loc=0x8000030a PASS 17 (5s) == sanityn test 18: mmap sanity check =========================================================================================== 09:44:11 (1788875051) mmap test1: basic mmap operation (PASS, 0.051977s) mmap test2: MAP_PRIVATE not write back (PASS, 0.042581s) mmap test3: concurrent mmap ops on two nodes (PASS, 6.7939s) mmap test4: c1 write to f1 from mmapped f2, c2 write to f1 from mmapped f1 (PASS, 4.164s) mmap test5: read/write file to/from the buffer which mmapped to just this file (PASS, 0.064776s) mmap test6: check mmap write/read content on two nodes (PASS, 0.072401s) mmap test7: file i/o with an unmapped buffer (SKIPPED, 0s) mmap test8: SIGBUS for beyond file size (SKIPPED, 0s) mmap test9: SIGBUS for negative file offset (SKIPPED, 0s) mmap test10: multi-thread mmap access (PASS, 0.47968s) mmap test11: multi-thread shared mmap (PASS, 0.49289s) mmap test10: mtime not change for readonly mmap access (PASS, 1.042s) PASS 18 (17s) == sanityn test 19: test concurrent uncached read races ========================================================================= 09:44:28 (1788875068) oleg626-server: error: get_param: param_path 'osd-*/lustre-OST*/read_cache_enable': No such file or directory pdsh@oleg626-client: oleg626-server: ssh exited with exit code 2 SKIP: sanityn test_19 not cache-capable obdfilter SKIP 19 (2s) == sanityn test 20: test extra readahead page left in cache ============================================================== 09:44:30 (1788875070) PASS 20 (2s) == sanityn test 21: Try to remove mountpoint on another dir ============================================================== 09:44:32 (1788875072) rmdir: removing directory, '/mnt/lustre/d21.sanityn' rmdir: failed to remove '/mnt/lustre/d21.sanityn': Device or resource busy rmdir: removing directory, '/mnt/lustre2/d21.sanityn' Removed mounted directory from another mountpoint, needs to be fixed PASS 21 (3s) == sanityn test 23: others should see updated atime while another read============================================================== 09:44:35 (1788875075) now is 1788875076 starting reads multiop /mnt/lustre/f23.sanityn vor20_c TMPPIPE=/tmp/multiop_open_wait_pipe.7625 new atime is 1788875137 PASS 23 (64s) == sanityn test 24a: lfs df [-ih] [path] test =================================================================================== 09:45:39 (1788875139) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2048 2206592 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 6144 3763200 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2048 2206592 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 6144 3763200 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre2 UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.9K 387 524.5K 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 115.3K 403 114.9K 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 114.5K 400 114.1K 1% /mnt/lustre[OST:1] filesystem_summary: 229.4K 387 229.0K 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.9K 387 524.5K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.3K 403 114.9K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 400 114.1K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.4K 387 229.0K 1% /mnt/lustre2 UUID bytes Used Available Use% Mounted on lustre-MDT0000_UUID 2.1G 2.0M 2.1G 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3.6G 6.0M 3.6G 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3.6G 32.0M 3.6G 1% /mnt/lustre[OST:1] filesystem_summary: 7.2G 38.0M 7.2G 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537448 387 537061 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118067 403 117664 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117232 400 116832 1% /mnt/lustre2[OST:1] filesystem_summary: 234883 387 234496 1% /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2048 2206592 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 6144 3763200 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.9K 387 524.5K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.3K 403 114.9K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 400 114.1K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.4K 387 229.0K 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537448 387 537061 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 118067 403 117664 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 117232 400 116832 1% /mnt/lustre[OST:1] filesystem_summary: 234883 387 234496 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537448 387 537061 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118067 403 117664 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117232 400 116832 1% /mnt/lustre2[OST:1] filesystem_summary: 234883 387 234496 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2048 2206592 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 6144 3763200 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2048 2206592 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 6144 3763200 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre2 PASS 24a (3s) == sanityn test 24b: lfs df should show both filesystems ========================================================================= 09:45:42 (1788875142) PASS 24b (2s) == sanityn test 25a: change ACL on one mountpoint be seen on another ============================================================= 09:45:44 (1788875144) running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] PASS 25a (3s) == sanityn test 25b: change ACL under remote dir on one mountpoint be seen on another ========================================================== 09:45:47 (1788875147) SKIP: sanityn test_25b needs >= 2 MDTs SKIP 25b (2s) == sanityn test 26a: allow mtime to get older ============ 09:45:49 (1788875149) utime: good mknod times 1788875148 <= 1788875149 <= 1788875149 for /mnt/lustre/f26a utime: good mknod times 1788875148 <= 1788875149 <= 1788875149 for /mnt/lustre2/f26a utime: good utime mtimes 100000, atime 200000 utime: updated times for /mnt/lustre2/f26a PASS 26a (3s) == sanityn test 26b: sync mtime between ost and mds ====== 09:45:52 (1788875152) PASS 26b (5s) == sanityn test 26c: set-in-past on open file is not lost on close ========================================================== 09:45:57 (1788875157) multiop /mnt/lustre/f26c.sanityn vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7625 File: /mnt/lustre/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502398 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-09-08 09:45:57.000000000 -0400 Birth: 2026-09-08 09:45:57.000000000 -0400 File: /mnt/lustre2/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502398 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-09-08 09:45:57.000000000 -0400 Birth: 2026-09-08 09:45:57.000000000 -0400 PASS 26c (3s) SKIP: sanityn test_27 skipping excluded test 27 == sanityn test 30: recreate file race =================== 09:46:01 (1788875161) 0 PASS 30 (4s) == sanityn test 31a: voluntary cancel / blocking ast race======================================================================== 09:46:06 (1788875166) fail_loc=0x314 PASS 31a (4s) == sanityn test 31b: voluntary OST cancel / blocking ast race======================================================================== 09:46:09 (1788875169) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg626-server mds-ost sync done. sleep 5 for ZFS MDS Waiting for MDT destroys to complete fail_loc=0x314 fail_loc=0x316 fail_loc=0 fail_loc=0 Filesystem 1K-blocks Used Available Use% Mounted on 192.168.206.126@tcp:/lustre 7542784 40960 7497728 1% /mnt/lustre2 PASS 31b (13s) == sanityn test 31r: open-rename(replace) race =========== 09:46:22 (1788875182) -rw-r--r-- 1 root root 0 Sep 8 09:46 /mnt/lustre/f31r.sanityn.target fail_loc=0x1419 fail_val=3 PASS 31r (6s) == sanityn test 31s: open should not revalidate invalid dentry ========================================================== 09:46:28 (1788875188) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 1205: 43067 Terminated tail -f $tfile PASS 31s (3s) == sanityn test 31t: getattr should not revalidate invalid dentry ========================================================== 09:46:31 (1788875191) PASS 31t (4s) == sanityn test 32b: lockless i/o ======================== 09:46:35 (1788875195) SKIP: sanityn test_32b max_nolock_bytes is removed >= 2.17.53 SKIP 32b (1s) == sanityn test 32c: contention detection ================ 09:46:37 (1788875197) ldlm.namespaces.filter-lustre-OST0000_UUID.contended_locks=2 ldlm.namespaces.filter-lustre-OST0001_UUID.contended_locks=2 ldlm.namespaces.filter-lustre-OST0000_UUID.contention_events=0 ldlm.namespaces.filter-lustre-OST0001_UUID.contention_events=0 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.691059 s, 75.9 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.584181 s, 89.7 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.826226 s, 63.5 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.959131 s, 54.7 MB/s contention_seconds 4, contention_events 20 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.853019 s, 61.5 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.945842 s, 55.4 MB/s ldlm.namespaces.filter-lustre-OST0000_UUID.contention_events=0 ldlm.namespaces.filter-lustre-OST0001_UUID.contention_events=0 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.770879 s, 68.0 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.773227 s, 67.8 MB/s PASS 32c (13s) SKIP: sanityn test_33a skipping SLOW test 33a == sanityn test 33b: COS: cross create/delete, 2 clients, benchmark under remote dir ========================================================== 09:46:50 (1788875210) SKIP: sanityn test_33b Need two or more clients, have 1 SKIP 33b (2s) == sanityn test 33c: Cancel cross-MDT lock should trigger Sync-on-Lock-Cancel ========================================================== 09:46:52 (1788875212) SKIP: sanityn test_33c needs >= 2 MDTs SKIP 33c (2s) == sanityn test 33d: dependent transactions should trigger COS ========================================================== 09:46:54 (1788875214) SKIP: sanityn test_33d needs >= 2 MDTs SKIP 33d (2s) == sanityn test 33e: independent transactions shouldn't trigger COS ========================================================== 09:46:56 (1788875216) SKIP: sanityn test_33e needs >= 2 MDTs SKIP 33e (1s) == sanityn test 34: no lock timeout under IO ============= 09:46:57 (1788875217) fail_loc=0x512 fail_loc=0x512 lock should not expire writing on client1 reading on client2 success fail_loc=0x511 fail_loc=0x511 lock should expire writing on client1 reading on client2 success oleg626-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9d9e08e29800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9d9e08e29800.ost_server_uuid in FULL state after 0 sec oleg626-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9d9e08e29800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9d9e08e29800.ost_server_uuid in IDLE state after 0 sec oleg626-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9d9e08e29800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9d9e08e29800.ost_server_uuid in FULL state after 0 sec oleg626-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9d9e08e29800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9d9e08e29800.ost_server_uuid in IDLE state after 0 sec oleg626-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9d9e08e29800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9d9e08e29800.ost_server_uuid in FULL state after 0 sec oleg626-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9d9e08e29800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9d9e08e29800.ost_server_uuid in IDLE state after 0 sec PASS 34 (97s) == sanityn test 35: -EINTR cp_ast vs. bl_ast race does not evict client ========================================================== 09:48:34 (1788875314) fail_loc=0x80000317 Race attempt 0 Wait for 50352 50487 for 60 sec... - open/close 2103 (time 1788875327.20 total 10.00 last 210.29) total: 4000 open/close in 18.85 seconds: 212.19 ops/second fail_loc=0x0 Filesystem Size Used Avail Use% Mounted on 192.168.206.126@tcp:/lustre 7.2G 41M 7.2G 1% /mnt/lustre 192.168.206.126@tcp:/lustre 7.2G 41M 7.2G 1% /mnt/lustre2 PASS 35 (66s) == sanityn test 36: handle ESTALE/open-unlink correctly == 09:49:40 (1788875380) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg626-server mds-ost sync done. Waiting for MDT destroys to complete 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.528488 s, 99.2 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7625 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(0) *** before(7496704) after_dd(7445504) after(7496704) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.557997 s, 94.0 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7625 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(1) *** before(7496704) after_dd(7441408) after(7492608) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.496654 s, 106 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7625 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(2) *** before(7492608) after_dd(7445504) after(7496704) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.489427 s, 107 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7625 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(3) *** before(7496704) after_dd(7445504) after(7496704) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.542152 s, 96.7 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7625 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(4) *** before(7496704) after_dd(7443456) after(7494656) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.510247 s, 103 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7625 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(5) *** before(7494656) after_dd(7445504) after(7496704) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.463657 s, 113 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7625 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(6) *** before(7496704) after_dd(7445504) after(7496704) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.469177 s, 112 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7625 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(7) *** before(7496704) after_dd(7445504) after(7496704) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.495925 s, 106 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7625 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(8) *** before(7496704) after_dd(7445504) after(7496704) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.456415 s, 115 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7625 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(9) *** before(7496704) after_dd(7445504) after(7496704) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.483534 s, 108 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7625 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(10) *** before(7496704) after_dd(7445504) after(7496704) PASS 36 (242s) == sanityn test 37: check i_size is not updated for directory on close (bug 18695) ======================================================================== 09:53:42 (1788875622) multiop /mnt/lustre/d37.sanityn vD_c TMPPIPE=/tmp/multiop_open_wait_pipe.7625 - create 5108 (time 1788875633.93 total 10.00 last 510.79) total: 10000 create in 18.48 seconds: 541.13 ops/second PASS 37 (22s) == sanityn test 39a: file mtime does not change after rename ========================================================== 09:54:04 (1788875644) repeat after cancel_lru_locks PASS 39a (3s) == sanityn test 39b: file mtime the same on clients with/out lock ========================================================== 09:54:07 (1788875647) repeat after cancel_lru_locks PASS 39b (3s) == sanityn test 39c: check truncate mtime update ================================================================================ 09:54:10 (1788875650) repeat after cancel_lru_locks PASS 39c (3s) == sanityn test 39d: sync write should update mtime ====== 09:54:13 (1788875653) fail_loc=0x411 fail_loc=0 PASS 39d (2s) SKIP: sanityn test_40a skipping ALWAYS excluded test 40a == sanityn test 40b: pdirops: open|create and others ======================================================================== 09:54:16 (1788875656) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40b (8s) == sanityn test 40c: pdirops: link and others ======================================================================== 09:54:24 (1788875664) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40c (8s) == sanityn test 40d: pdirops: unlink and others ======================================================================== 09:54:32 (1788875672) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40d (9s) == sanityn test 40e: pdirops: rename and others ======================================================================== 09:54:41 (1788875681) No conflict No conflict No conflict No conflict No conflict No conflict PASS 40e (7s) == sanityn test 41a: pdirops: create vs mkdir ======================================================================== 09:54:48 (1788875688) mkdir: cannot create directory '/mnt/lustre2/f41a.sanityn': File exists Conflict PASS 41a (6s) == sanityn test 41b: pdirops: create vs create ======================================================================== 09:54:54 (1788875694) open: File exists Conflict PASS 41b (5s) == sanityn test 41c: pdirops: create vs link ======================================================================== 09:54:59 (1788875699) link: cannot create link '/mnt/lustre2/f41c.sanityn' to '/mnt/lustre2/f41c.sanityn-2': File exists Conflict PASS 41c (5s) == sanityn test 41d: pdirops: create vs unlink ======================================================================== 09:55:04 (1788875704) Conflict PASS 41d (6s) == sanityn test 41e: pdirops: create and rename (tgt) ======================================================================== 09:55:10 (1788875710) Conflict PASS 41e (5s) == sanityn test 41f: pdirops: create and rename (src) ======================================================================== 09:55:16 (1788875716) Conflict PASS 41f (5s) == sanityn test 41g: pdirops: create vs getattr ======================================================================== 09:55:21 (1788875721) Conflict PASS 41g (5s) == sanityn test 41h: pdirops: create vs readdir ======================================================================== 09:55:26 (1788875726) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 2427: lctl set_param -n fail_loc=0 2>/dev/null || true: No such file or directory Conflict PASS 41h (5s) == sanityn test 41i: reint_open: create vs create ======== 09:55:31 (1788875731) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 41i (718s) == sanityn test 42a: pdirops: mkdir vs mkdir ======================================================================== 10:07:29 (1788876449) mkdir: cannot create directory '/mnt/lustre2/f42a.sanityn': File exists Conflict PASS 42a (5s) == sanityn test 42b: pdirops: mkdir vs create ======================================================================== 10:07:34 (1788876454) open: File exists Conflict PASS 42b (5s) == sanityn test 42c: pdirops: mkdir vs link ======================================================================== 10:07:39 (1788876459) link: cannot create link '/mnt/lustre2/f42c.sanityn' to '/mnt/lustre2/f42c.sanityn-2': File exists Conflict PASS 42c (6s) == sanityn test 42d: pdirops: mkdir vs unlink ======================================================================== 10:07:45 (1788876465) Conflict PASS 42d (5s) == sanityn test 42e: pdirops: mkdir and rename (tgt) ======================================================================== 10:07:50 (1788876470) mv: cannot overwrite directory '/mnt/lustre2/f42e.sanityn' with non-directory Conflict PASS 42e (5s) == sanityn test 42f: pdirops: mkdir and rename (src) ======================================================================== 10:07:55 (1788876475) Conflict PASS 42f (4s) == sanityn test 42g: pdirops: mkdir vs getattr ======================================================================== 10:07:59 (1788876479) Conflict PASS 42g (5s) == sanityn test 42h: pdirops: mkdir vs readdir ======================================================================== 10:08:04 (1788876484) Conflict PASS 42h (5s) == sanityn test 43a: rmdir,mkdir doesn't return -EEXIST ======================================================================== 10:08:09 (1788876489) PASS 43a (17s) == sanityn test 43b: pdirops: unlink vs create ======================================================================== 10:08:26 (1788876506) Conflict PASS 43b (5s) == sanityn test 43c: pdirops: unlink vs link ======================================================================== 10:08:31 (1788876511) Conflict PASS 43c (5s) == sanityn test 43d: pdirops: unlink vs unlink ======================================================================== 10:08:36 (1788876516) rm: cannot remove '/mnt/lustre2/f43d.sanityn': No such file or directory Conflict PASS 43d (5s) == sanityn test 43e: pdirops: unlink and rename (tgt) ======================================================================== 10:08:41 (1788876521) Conflict PASS 43e (5s) == sanityn test 43f: pdirops: unlink and rename (src) ======================================================================== 10:08:46 (1788876526) mv: cannot stat '/mnt/lustre2/f43f.sanityn': No such file or directory Conflict PASS 43f (5s) == sanityn test 43g: pdirops: unlink vs getattr ======================================================================== 10:08:51 (1788876531) stat: cannot statx '/mnt/lustre2/f43g.sanityn': No such file or directory Conflict PASS 43g (5s) == sanityn test 43h: pdirops: unlink vs readdir ======================================================================== 10:08:56 (1788876536) Conflict PASS 43h (5s) == sanityn test 43i: pdirops: unlink vs remote mkdir ===== 10:09:01 (1788876541) SKIP: sanityn test_43i needs >= 2 MDTs SKIP 43i (1s) == sanityn test 43j: racy mkdir return EEXIST ======================================================================== 10:09:02 (1788876542) mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists PASS 43j (35s) == sanityn test 43k: unlink vs create ==================== 10:09:37 (1788876577) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 43k (414s) == sanityn test 44a: pdirops: rename tgt vs mkdir ======================================================================== 10:16:32 (1788876992) mkdir: cannot create directory '/mnt/lustre2/f44a.sanityn': File exists Conflict Tue Sep 8 10:16:35 EDT 2026 PASS 44a (5s) == sanityn test 44b: pdirops: rename tgt vs create ======================================================================== 10:16:36 (1788876996) open: File exists Conflict PASS 44b (5s) == sanityn test 44c: pdirops: rename tgt vs link ======================================================================== 10:16:41 (1788877001) link: cannot create link '/mnt/lustre2/f44c.sanityn' to '/mnt/lustre2/f44c.sanityn-3': File exists Conflict PASS 44c (5s) == sanityn test 44d: pdirops: rename tgt vs unlink ======================================================================== 10:16:46 (1788877006) Conflict PASS 44d (5s) == sanityn test 44e: pdirops: rename tgt and rename (tgt) ======================================================================== 10:16:51 (1788877011) Conflict PASS 44e (5s) == sanityn test 44f: pdirops: rename tgt and rename (src) ======================================================================== 10:16:56 (1788877016) Conflict PASS 44f (5s) == sanityn test 44g: pdirops: rename tgt vs getattr ======================================================================== 10:17:01 (1788877021) Conflict PASS 44g (5s) == sanityn test 44h: pdirops: rename tgt vs readdir ======================================================================== 10:17:06 (1788877026) Conflict PASS 44h (5s) == sanityn test 44i: pdirops: rename tgt vs remote mkdir ========================================================== 10:17:11 (1788877031) SKIP: sanityn test_44i needs >= 2 MDTs SKIP 44i (1s) == sanityn test 45a: rename,mkdir doesn't return -EEXIST ======================================================================== 10:17:12 (1788877032) PASS 45a (24s) == sanityn test 45b: pdirops: rename src vs create ======================================================================== 10:17:36 (1788877056) Conflict PASS 45b (5s) == sanityn test 45c: pdirops: rename src vs link ======================================================================== 10:17:41 (1788877061) Conflict PASS 45c (5s) == sanityn test 45d: pdirops: rename src vs unlink ======================================================================== 10:17:46 (1788877066) rm: cannot remove '/mnt/lustre2/f45d.sanityn': No such file or directory Conflict PASS 45d (5s) == sanityn test 45e: pdirops: rename src and rename (tgt) ======================================================================== 10:17:51 (1788877071) Conflict PASS 45e (5s) == sanityn test 45f: pdirops: rename src and rename (src) ======================================================================== 10:17:56 (1788877076) mv: cannot stat '/mnt/lustre2/f45f.sanityn': No such file or directory Conflict PASS 45f (5s) == sanityn test 45g: pdirops: rename src vs getattr ======================================================================== 10:18:01 (1788877081) stat: cannot statx '/mnt/lustre2/f45g.sanityn': No such file or directory Conflict PASS 45g (6s) == sanityn test 45h: pdirops: unlink vs readdir ======================================================================== 10:18:07 (1788877087) Conflict PASS 45h (4s) == sanityn test 45i: pdirops: rename src vs remote mkdir ========================================================== 10:18:11 (1788877091) SKIP: sanityn test_45i needs >= 2 MDTs SKIP 45i (2s) == sanityn test 45j: read vs rename ====================== 10:18:13 (1788877093) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 45j (415s) == sanityn test 46a: pdirops: link vs mkdir ======================================================================== 10:25:09 (1788877509) mkdir: cannot create directory '/mnt/lustre2/f46a.sanityn': File exists Conflict PASS 46a (4s) == sanityn test 46b: pdirops: link vs create ======================================================================== 10:25:13 (1788877513) open: File exists Conflict PASS 46b (5s) == sanityn test 46c: pdirops: link vs link ======================================================================== 10:25:18 (1788877518) link: cannot create link '/mnt/lustre2/f46c.sanityn' to '/mnt/lustre2/f46c.sanityn': File exists Conflict PASS 46c (5s) == sanityn test 46d: pdirops: link vs unlink ======================================================================== 10:25:23 (1788877523) Conflict PASS 46d (5s) == sanityn test 46e: pdirops: link and rename (tgt) ======================================================================== 10:25:28 (1788877528) Conflict PASS 46e (5s) == sanityn test 46f: pdirops: link and rename (src) ======================================================================== 10:25:33 (1788877533) Conflict PASS 46f (5s) == sanityn test 46g: pdirops: link vs getattr ======================================================================== 10:25:38 (1788877538) Conflict PASS 46g (6s) == sanityn test 46h: pdirops: link vs readdir ======================================================================== 10:25:44 (1788877544) Conflict PASS 46h (5s) == sanityn test 46i: pdirops: link vs remote mkdir ======= 10:25:49 (1788877549) SKIP: sanityn test_46i needs >= 2 MDTs SKIP 46i (1s) == sanityn test 47a: pdirops: remote mkdir vs mkdir ====== 10:25:50 (1788877550) SKIP: sanityn test_47a needs >= 2 MDTs SKIP 47a (1s) == sanityn test 47b: pdirops: remote mkdir vs create ===== 10:25:51 (1788877551) SKIP: sanityn test_47b needs >= 2 MDTs SKIP 47b (1s) == sanityn test 47c: pdirops: remote mkdir vs link ======= 10:25:52 (1788877552) SKIP: sanityn test_47c needs >= 2 MDTs SKIP 47c (1s) == sanityn test 47d: pdirops: remote mkdir vs unlink ===== 10:25:53 (1788877553) SKIP: sanityn test_47d needs >= 2 MDTs SKIP 47d (1s) == sanityn test 47e: pdirops: remote mkdir and rename (tgt) ========================================================== 10:25:54 (1788877554) SKIP: sanityn test_47e needs >= 2 MDTs SKIP 47e (1s) == sanityn test 47f: pdirops: remote mkdir and rename (src) ========================================================== 10:25:55 (1788877555) SKIP: sanityn test_47f needs >= 2 MDTs SKIP 47f (1s) == sanityn test 47g: pdirops: remote mkdir vs getattr ==== 10:25:56 (1788877556) SKIP: sanityn test_47g needs >= 2 MDTs SKIP 47g (1s) == sanityn test 50: osc lvb attrs: enqueue vs. CP AST ======================================================================== 10:25:57 (1788877557) 10+0 records in 10+0 records out 10240 bytes (10 kB, 10 KiB) copied, 0.00630349 s, 1.6 MB/s fail_loc=0x410 fail_loc=0x0 PASS 50 (8s) == sanityn test 51a: layout lock: refresh layout should work ========================================================== 10:26:05 (1788877565) 0+1 records in 0+1 records out 158 bytes copied, 0.0038184 s, 41.4 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 3690: kill: (218786) - No such process PASS 51a (4s) == sanityn test 51b: layout lock: glimpse should be able to restart if layout changed ========================================================== 10:26:09 (1788877569) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.002818 s, 363 kB/s fail_loc=0x1404 fail_val=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0182249 s, 57.5 MB/s 3145728 PASS 51b (14s) == sanityn test 51c: layout lock: IT_LAYOUT blocked and correct layout can be returned ========================================================== 10:26:23 (1788877583) fail_loc=0x172 Setting layout to have 2 stripes ... 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00388824 s, 263 kB/s PASS 51c (7s) == sanityn test 51d: layout lock: losing layout lock should clean up memory map region ========================================================== 10:26:30 (1788877590) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126539 s, 82.9 MB/s Before revoking layout lock: 1024 KB mapped PASS 51d (3s) == sanityn test 51e: lfs getstripe does not break leases, part 2 ========================================================== 10:26:33 (1788877593) /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 2452 0x994 0x280000400 write lease(2) has applied. /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 2452 0x994 0x280000400 read lease(1) has applied. PASS 51e (4s) == sanityn test 54: rename locking ======================= 10:26:37 (1788877597) llite.lustre-ffff9d9e08e29800.xattr_cache=1 llite.lustre-ffff9d9e11804800.xattr_cache=1 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505015 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-08 10:26:38.000000000 -0400 Modify: 2026-09-08 10:26:38.000000000 -0400 Change: 2026-09-08 10:26:38.000000000 -0400 Birth: 2026-09-08 10:26:37.000000000 -0400 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000154 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505018 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-08 10:26:43.000000000 -0400 Modify: 2026-09-08 10:26:44.000000000 -0400 Change: 2026-09-08 10:26:44.000000000 -0400 Birth: 2026-09-08 10:26:43.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505021 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-08 10:26:49.000000000 -0400 Modify: 2026-09-08 10:26:49.000000000 -0400 Change: 2026-09-08 10:26:49.000000000 -0400 Birth: 2026-09-08 10:26:48.000000000 -0400 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505020 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-08 10:26:49.000000000 -0400 Modify: 2026-09-08 10:26:49.000000000 -0400 Change: 2026-09-08 10:26:49.000000000 -0400 Birth: 2026-09-08 10:26:48.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000154 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505023 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-08 10:26:54.000000000 -0400 Modify: 2026-09-08 10:26:55.000000000 -0400 Change: 2026-09-08 10:26:55.000000000 -0400 Birth: 2026-09-08 10:26:54.000000000 -0400 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505024 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-08 10:26:54.000000000 -0400 Modify: 2026-09-08 10:26:55.000000000 -0400 Change: 2026-09-08 10:26:55.000000000 -0400 Birth: 2026-09-08 10:26:54.000000000 -0400 PASS 54 (24s) == sanityn test 55a: rename vs unlink target dir ========= 10:27:01 (1788877621) fail_loc=0x80000156 mv: cannot move '/mnt/lustre/d55a.sanityn/d1/d2' to '/mnt/lustre/d55a.sanityn/d3/d2': No such file or directory PASS 55a (8s) == sanityn test 55b: rename vs unlink source dir ========= 10:27:09 (1788877629) fail_loc=0x80000156 mv: cannot stat '/mnt/lustre/d55b.sanityn/d1/d2': No such file or directory PASS 55b (8s) == sanityn test 55c: rename vs unlink orphan target dir == 10:27:17 (1788877637) fail_loc=0x156 mv: cannot move '/mnt/lustre/d55c.sanityn/d1/d2' to '/mnt/lustre/d55c.sanityn/d3/d2': No such file or directory PASS 55c (12s) == sanityn test 55d: rename file vs link ================= 10:27:30 (1788877650) fail_loc=0x155 mv: '/mnt/lustre/d55d.sanityn/f1' and '/mnt/lustre/d55d.sanityn/d55d.sanityn/f1' are the same file PASS 55d (9s) == sanityn test 55e: rename race AB/BA under the same parent dir ========================================================== 10:27:39 (1788877659) SKIP: sanityn test_55e needs >= 2 MDTs SKIP 55e (1s) == sanityn test 55f: rename: (P1/A -> P2/B) race with (P2/B -> P1/A) ========================================================== 10:27:40 (1788877660) fail_loc=0x155 We survived after P1A_P2B/P2B_P1A race test fail_loc=0x0 PASS 55f (13s) == sanityn test 55g: rename: race with trylock =========== 10:27:53 (1788877673) mdt.lustre-MDT0000.enable_rename_trylock=1 fail_loc=0x155 mv: cannot move '/mnt/lustre/d55g.sanityn/a1/a2' to a subdirectory of itself, '/mnt/lustre/d55g.sanityn/b1/b2/b3/b4/a2' We survived race with trylock test fail_loc=0x0 PASS 55g (14s) == sanityn test 56a: test llverdev with single large stripe ========================================================== 10:28:07 (1788877687) run_llverdev /mnt/lustre/f56a.sanityn -p -s 16000000 -c 4k llverdev: /mnt/lustre/f56a.sanityn is 16777216000000 bytes (15625 GB) in size timestamp: 1788877687 chunksize: 4096 size: 16777216000000 write offset: 16384000000kB 2.65387 MB/s read offset: 16384000000kB inf MB/s llverdev: data verified successfully PASS 56a (25s) == sanityn test 56b: test llverdev and partial verify of wide stripe file ========================================================== 10:28:32 (1788877712) run_llverdev /mnt/lustre/f56b.sanityn -p -s 32000000 -c 4k llverdev: /mnt/lustre/f56b.sanityn is 33554432000000 bytes (31250 GB) in size timestamp: 1788877714 chunksize: 4096 size: 33554432000000 write offset: 32768000000kB 3.93788 MB/s read offset: 32768000000kB inf MB/s llverdev: data verified successfully PASS 56b (35s) == sanityn test 60: Verify data_version behaviour ======== 10:29:07 (1788877747) PASS 60 (3s) == sanityn test 70a: cd directory && rm directory ======== 10:29:10 (1788877750) PASS 70a (2s) == sanityn test 70b: remove files after calling rm_entry ========================================================== 10:29:12 (1788877752) PASS 70b (2s) == sanityn test 71a: correct file map just after write operation is finished ========================================================== 10:29:14 (1788877754) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.00622635 s, 10.5 MB/s SKIP: sanityn test_71a ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71a (2s) == sanityn test 71b: check fiemap support for stripecount > 1 ========================================================== 10:29:16 (1788877756) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.00352572 s, 18.6 MB/s SKIP: sanityn test_71b ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71b (1s) == sanityn test 71c: check FIEMAP_EXTENT_LAST flag with different extents number ========================================================== 10:29:17 (1788877757) SKIP: sanityn test_71c support only ldiskfs ost SKIP 71c (1s) == sanityn test 71d: fiemap corruption test with fm_extent_count=0 ========================================================== 10:29:18 (1788877758) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0039331 s, 16.7 MB/s SKIP: sanityn test_71d ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71d (1s) == sanityn test 72: getxattr/setxattr cache should be consistent between nodes ========================================================== 10:29:19 (1788877759) llite.lustre-ffff9d9e08e29800.xattr_cache=1 llite.lustre-ffff9d9e11804800.xattr_cache=1 getfattr: Removing leading '/' from absolute path names user.attr1="value1" getfattr: Removing leading '/' from absolute path names user.attr1="value2" getfattr: Removing leading '/' from absolute path names getfattr: Removing leading '/' from absolute path names b362740ad371eae580e7f85452e8830b - 870e072da1ebfc20ea8c0cd3d6bdf3b6 - PASS 72 (2s) == sanityn test 73: getxattr should not cause xattr lock cancellation ========================================================== 10:29:21 (1788877761) llite.lustre-ffff9d9e08e29800.xattr_cache=1 llite.lustre-ffff9d9e11804800.xattr_cache=1 getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" PASS 73 (2s) == sanityn test 74: flock deadlock: different mounts ======================================================================== 10:29:23 (1788877763) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00257327 s, 398 kB/s 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00248817 s, 412 kB/s 231551: get lock2 231552: get lock1 231552: done 231552: get lock2 231551: done 231551: get lock1 fcntl cmd 7 failed: Resource deadlock avoided 231552: cannot get lock2: Resource deadlock avoided 231552: exit rc=0 fcntl cmd 7 failed: Resource deadlock avoided 231551: cannot get lock1: Resource deadlock avoided 231551: done 231551: exit rc=0 PASS 74 (6s) == sanityn test 75: osc: upcall after unuse lock============================================================================= 10:29:29 (1788877769) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0184541 s, 114 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0119797 s, 87.5 MB/s fail_loc=0x31d /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 4373: 232132 Killed stat -c %s $DIR1/$tfile fail_loc=0 PASS 75 (7s) == sanityn test 76: Verify MDT open_files listing ======== 10:29:36 (1788877776) opening files: ........ closing files: ........ PASS 76 (19s) == sanityn test 77a: check FIFO NRS policy =============== 10:29:55 (1788877795) ost.OSS.ost.nrs_policies=fifo ost.OSS.ost_create.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_out.nrs_policies=fifo ost.OSS.ost_seq.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.116197 s, 144 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.025257 s, 41.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0269869 s, 38.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0555734 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.048919 s, 21.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0623053 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0769007 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0722517 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0827825 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0837697 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0970819 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0886972 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100175 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0929426 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105207 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0930736 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0854507 s, 12.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9d9e11804800.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff9d9e11804800.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014084 s, 74.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155355 s, 67.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.017908 s, 58.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162644 s, 64.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0225378 s, 46.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00516593 s, 203 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0186071 s, 56.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148138 s, 70.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144452 s, 72.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.018083 s, 58.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01323 s, 79.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00955691 s, 110 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132025 s, 79.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0197921 s, 53.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00381196 s, 275 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9d9e11804800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142846 s, 73.4 MB/s removed '/mnt/lustre/d77a.sanityn/nrs_w_oleg626-client.virtnet' removed '/mnt/lustre/d77a.sanityn/nrs_r_oleg626-client.virtnet' removed directory '/mnt/lustre/d77a.sanityn' PASS 77a (2s) == sanityn test 77b: check CRR-N NRS policy ============== 10:29:57 (1788877797) ost.OSS.ost.nrs_policies=crrn ost.OSS.ost_create.nrs_policies=crrn ost.OSS.ost_io.nrs_policies=crrn ost.OSS.ost_out.nrs_policies=crrn ost.OSS.ost_seq.nrs_policies=crrn ost.OSS.ost.nrs_crrn_quantum=1 ost.OSS.ost_create.nrs_crrn_quantum=1 ost.OSS.ost_io.nrs_crrn_quantum=1 ost.OSS.ost_out.nrs_crrn_quantum=1 ost.OSS.ost_seq.nrs_crrn_quantum=1 policy: crr-n, crrn_quantum 1 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.136416 s, 123 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0289199 s, 36.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0291055 s, 36.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0792379 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.068635 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.093577 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100182 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0864591 s, 12.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9d9e11804800.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff9d9e11804800.lock_unused_count=4 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103689 s, 10.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105545 s, 9.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0895452 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102647 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0985847 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118746 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11267 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115709 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118019 s, 8.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9d9e11804800.lock_unused_count=3 ldlm.namespaces.lustre-OST0001-osc-ffff9d9e11804800.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0231765 s, 45.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0214129 s, 49.0 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0222368 s, 47.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0244744 s, 42.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0201835 s, 52.0 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0176173 s, 59.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0215403 s, 48.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0241932 s, 43.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00890807 s, 118 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.025926 s, 40.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141531 s, 74.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0199386 s, 52.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00712794 s, 147 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0199686 s, 52.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149266 s, 70.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155613 s, 67.4 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg626-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg626-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost.nrs_crrn_quantum=64 ost.OSS.ost_create.nrs_crrn_quantum=64 ost.OSS.ost_io.nrs_crrn_quantum=64 ost.OSS.ost_out.nrs_crrn_quantum=64 ost.OSS.ost_seq.nrs_crrn_quantum=64 policy: crr-n, crrn_quantum 64 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.13044 s, 129 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0214729 s, 48.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0375837 s, 27.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105112 s, 10.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117849 s, 8.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121542 s, 8.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115673 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125298 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0991035 s, 10.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105199 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114034 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117046 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0962393 s, 10.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9d9e11804800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107098 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125551 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121286 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119222 s, 8.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0204534 s, 51.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168093 s, 62.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0174599 s, 60.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0226927 s, 46.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0271336 s, 38.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0343068 s, 30.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0239991 s, 43.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0194656 s, 53.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0239137 s, 43.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0297007 s, 35.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144782 s, 72.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166605 s, 62.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0100082 s, 105 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00900492 s, 116 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0262047 s, 40.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9d9e11804800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160821 s, 65.2 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg626-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg626-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77b (4s) == sanityn test 77c: check ORR NRS policy ================ 10:30:01 (1788877801) ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_orr_quantum=1 ost.OSS.ost_io.nrs_orr_offset_type=physical ost.OSS.ost_io.nrs_orr_supported=reads policy: orr, orr_quantum 1, orr_offset_type physical, orr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.176896 s, 94.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0292292 s, 35.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0264896 s, 39.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0906983 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104039 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101421 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105967 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118503 s, 8.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101196 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117564 s, 8.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0945993 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112865 s, 9.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9d9e11804800.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffff9d9e11804800.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0984677 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120341 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117287 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134771 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117656 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012164 s, 86.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00276472 s, 379 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00359586 s, 292 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00378067 s, 277 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00403951 s, 260 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00383559 s, 273 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0301036 s, 34.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0210153 s, 49.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00291644 s, 360 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0293459 s, 35.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00254602 s, 412 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0355055 s, 29.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0257083 s, 40.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0423977 s, 24.7 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0361803 s, 29.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0335665 s, 31.2 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg626-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg626-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=writes ost.OSS.ost_io.nrs_orr_quantum=64 policy: orr, orr_quantum 64, orr_offset_type physical, orr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.12642 s, 133 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0491528 s, 21.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0738553 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.063268 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0705427 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0873693 s, 12.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0759518 s, 13.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0994134 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.081793 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0920786 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0900135 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0804036 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0996866 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106579 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119257 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0905478 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115193 s, 9.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9d9e11804800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9d9e11804800.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0180522 s, 58.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147287 s, 71.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173974 s, 60.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148756 s, 70.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0215632 s, 48.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9d9e11804800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00349482 s, 300 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.018524 s, 56.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0188578 s, 55.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00895972 s, 117 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00413906 s, 253 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00421628 s, 249 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0265734 s, 39.5 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0290374 s, 36.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0171771 s, 61.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0261041 s, 40.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140776 s, 74.5 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg626-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg626-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=reads_and_writes ost.OSS.ost_io.nrs_orr_offset_type=logical policy: orr, orr_quantum 64, orr_offset_type logical, orr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.139516 s, 120 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0269254 s, 38.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0347075 s, 30.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0758265 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0958516 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103205 s, 10.2 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1168 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113917 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115597 s, 9.1 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9d9e11804800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120991 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118157 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153193 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137815 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140211 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15634 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14927 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150592 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0175354 s, 59.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154888 s, 67.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0177365 s, 59.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0206849 s, 50.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0216266 s, 48.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0203509 s, 51.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0285046 s, 36.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161057 s, 65.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0206263 s, 50.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0100611 s, 104 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00857938 s, 122 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0117731 s, 89.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0172526 s, 60.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139352 s, 75.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00727913 s, 144 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00607394 s, 173 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg626-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg626-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77c (5s) == sanityn test 77d: check TRR nrs policy ================ 10:30:06 (1788877806) ost.OSS.ost_io.nrs_policies=trr ost.OSS.ost_io.nrs_trr_quantum=1 ost.OSS.ost_io.nrs_trr_offset_type=physical ost.OSS.ost_io.nrs_trr_supported=reads policy: trr, trr_quantum 1, trr_offset_type physical, trr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.199953 s, 83.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0248578 s, 42.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0212422 s, 49.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0679026 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0632879 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0758822 s, 13.8 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9d9e11804800.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0545989 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0987463 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116933 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111023 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104481 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107666 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125213 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101685 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127677 s, 8.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9d9e11804800.lock_unused_count=5 ldlm.namespaces.lustre-OST0001-osc-ffff9d9e11804800.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127978 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131361 s, 8.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157762 s, 66.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155807 s, 67.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0121675 s, 86.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0204953 s, 51.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0203047 s, 51.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167912 s, 62.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0273715 s, 38.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163498 s, 64.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0200407 s, 52.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0206063 s, 50.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0227116 s, 46.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0196825 s, 53.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0318557 s, 32.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0225204 s, 46.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0231382 s, 45.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0237721 s, 44.1 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg626-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg626-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=writes ost.OSS.ost_io.nrs_trr_quantum=64 policy: trr, trr_quantum 64, trr_offset_type physical, trr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.140034 s, 120 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0292511 s, 35.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0265465 s, 39.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0647155 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100443 s, 10.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106943 s, 9.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101715 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108129 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117858 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117138 s, 9.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132253 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125601 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149728 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131339 s, 8.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00374077 s, 280 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0229563 s, 45.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0190168 s, 55.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0361203 s, 29.0 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139669 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0294037 s, 35.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0240203 s, 43.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173156 s, 6.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0214312 s, 48.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171152 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0256743 s, 40.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00423393 s, 248 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157668 s, 66.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0273 s, 38.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9d9e11804800.lock_unused_count=8 ldlm.namespaces.lustre-OST0001-osc-ffff9d9e11804800.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00357562 s, 293 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00699861 s, 150 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0342259 s, 30.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.011621 s, 90.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00905794 s, 116 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg626-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg626-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=reads_and_writes ost.OSS.ost_io.nrs_trr_offset_type=logical policy: trr, trr_quantum 64, trr_offset_type logical, trr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.131741 s, 127 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0182898 s, 57.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0382407 s, 27.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0515479 s, 20.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100783 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0751748 s, 13.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11187 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13064 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.022174 s, 47.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139803 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130571 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0370965 s, 28.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0321335 s, 32.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116463 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.030601 s, 34.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116764 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0394179 s, 26.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0453743 s, 23.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0209408 s, 50.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00294068 s, 357 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142154 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15584 s, 6.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0407679 s, 25.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0285807 s, 36.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0350177 s, 29.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0339766 s, 30.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152158 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122328 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0410199 s, 25.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0573882 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15843 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0104323 s, 101 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00333257 s, 315 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg626-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg626-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77d (5s) == sanityn test 77e: check TBF NID nrs policy ============ 10:30:11 (1788877811) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_tbf_rule=start localhost nid={0@lo} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start clients nid={192.168.206.26@tcp} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start others nid={*.*.*.*@tcp} rate=50 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.161109 s, 104 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0251969 s, 41.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0622056 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.079349 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0841029 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0784081 s, 13.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0778189 s, 13.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0637459 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0679202 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0589489 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0744966 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0697747 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0877471 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.079862 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0846316 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.083357 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0778639 s, 13.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0115143 s, 91.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00921305 s, 114 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.011798 s, 88.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01199 s, 87.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0192374 s, 54.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160787 s, 65.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132757 s, 79.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0249353 s, 42.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0171622 s, 61.1 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.017087 s, 61.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00759819 s, 138 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0180123 s, 58.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0182183 s, 57.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0204592 s, 51.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00319166 s, 329 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00391143 s, 268 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg626-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg626-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change localhost rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change clients rate=101 ost.OSS.ost_io.nrs_tbf_rule=change others rate=51 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.128222 s, 131 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.023607 s, 44.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0324418 s, 32.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0459763 s, 22.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0718186 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0750345 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0717653 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.081462 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0832075 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.072518 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0996164 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0955437 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0938545 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0903309 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0919303 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0910503 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103606 s, 10.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145635 s, 72.0 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0175409 s, 59.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144207 s, 72.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164624 s, 63.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0179766 s, 58.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156426 s, 67.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015368 s, 68.2 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0206368 s, 50.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0217975 s, 48.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0113871 s, 92.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0192386 s, 54.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00462216 s, 227 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0185366 s, 56.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0226795 s, 46.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129448 s, 81.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00615604 s, 170 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg626-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg626-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop localhost ost.OSS.ost_io.nrs_tbf_rule=stop clients ost.OSS.ost_io.nrs_tbf_rule=stop others 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.144078 s, 116 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0306456 s, 34.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0804278 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0931903 s, 11.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0726292 s, 14.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0944071 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111228 s, 9.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108191 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120268 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141234 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15173 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136432 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136374 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148567 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173606 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166263 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185239 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.017503 s, 59.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.027783 s, 37.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0258092 s, 40.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0246807 s, 42.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0384021 s, 27.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.048183 s, 21.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0281638 s, 37.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00500934 s, 209 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.034482 s, 30.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0362696 s, 28.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9d9e11804800.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0298856 s, 35.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0345603 s, 30.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0362501 s, 28.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166454 s, 63.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0418014 s, 25.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00192678 s, 544 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg626-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg626-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.125456 s, 134 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0419516 s, 25.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0349056 s, 30.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.057851 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0563158 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0779123 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0759769 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.085586 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0827388 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102187 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0942954 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0836058 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0881076 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109686 s, 9.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9d9e11804800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9d9e11804800.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106411 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0870456 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0939646 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142922 s, 73.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00956127 s, 110 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0198347 s, 52.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0189244 s, 55.4 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0121324 s, 86.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0188816 s, 55.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00376154 s, 279 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0210462 s, 49.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125431 s, 83.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150252 s, 69.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015744 s, 66.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0201454 s, 52.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0171818 s, 61.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00901854 s, 116 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130907 s, 80.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0318307 s, 32.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9d9e11804800.lock_unused_count=1 removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg626-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg626-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' PASS 77e (8s) == sanityn test 77f: check TBF JobID nrs policy ========== 10:30:19 (1788877819) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start runas jobid={iozone.500 dd.500 tiotest.500} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start iozone_runas jobid={iozone.500} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg626-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.140207 s, 120 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0187832 s, 55.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0195139 s, 53.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0532321 s, 19.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0515325 s, 20.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.057613 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0414679 s, 25.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0514049 s, 20.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0460384 s, 22.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0606673 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.072977 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.074942 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0802337 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0983961 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0949304 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103416 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0932836 s, 11.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134549 s, 77.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0109547 s, 95.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142532 s, 73.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0178234 s, 58.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143295 s, 73.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144277 s, 72.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00399039 s, 263 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016289 s, 64.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156065 s, 67.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0202801 s, 51.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00722159 s, 145 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00609305 s, 172 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00483067 s, 217 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00487096 s, 215 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00198125 s, 529 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00204102 s, 514 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg626-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg626-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change runas rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change iozone_runas rate=101 ost.OSS.ost_io.nrs_tbf_rule=change dd_runas rate=51 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg626-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.116731 s, 144 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0408889 s, 25.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0682547 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0583255 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0983356 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100332 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104587 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0979096 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0954322 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0994286 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102619 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0995528 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109901 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0906855 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106281 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0700541 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0864119 s, 12.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.011911 s, 88.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141377 s, 74.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153171 s, 68.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0190421 s, 55.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0114217 s, 91.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137635 s, 76.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0103183 s, 102 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146549 s, 71.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0182917 s, 57.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155787 s, 67.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0036386 s, 288 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00386465 s, 271 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0264821 s, 39.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00441159 s, 238 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00464289 s, 226 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0253058 s, 41.4 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg626-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg626-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop runas ost.OSS.ost_io.nrs_tbf_rule=stop iozone_runas ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg626-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.145576 s, 115 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0429465 s, 24.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0662178 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.046881 s, 22.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0764104 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0772269 s, 13.6 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9d9e11804800.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.081031 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0446888 s, 23.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0756426 s, 13.9 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0740958 s, 14.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0848468 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0720211 s, 14.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132176 s, 7.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0305394 s, 34.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0348878 s, 30.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0361921 s, 29.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107858 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0349339 s, 30.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151385 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0275004 s, 38.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157268 s, 6.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0221285 s, 47.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0290632 s, 36.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9d9e11804800.lock_unused_count=5 ldlm.namespaces.lustre-OST0001-osc-ffff9d9e11804800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0485534 s, 21.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0282326 s, 37.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0352615 s, 29.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139342 s, 75.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178129 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0344408 s, 30.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0230042 s, 45.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0113998 s, 92.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137536 s, 76.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0340113 s, 30.8 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg626-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg626-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_policies=fifo running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg626-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.123609 s, 136 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0229911 s, 45.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0571108 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0647866 s, 16.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0456958 s, 22.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0611584 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0672185 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103493 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10016 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0892253 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0810105 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0904865 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107796 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104686 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113125 s, 9.3 MB/s 1+0 records in 1+0 records out ldlm.namespaces.lustre-OST0000-osc-ffff9d9e11804800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9d9e11804800.lock_unused_count=6 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103091 s, 10.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109636 s, 9.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137034 s, 76.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0180861 s, 58.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0200111 s, 52.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0202905 s, 51.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130755 s, 80.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0234648 s, 44.7 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0204089 s, 51.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183392 s, 57.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.022156 s, 47.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0223826 s, 46.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00498544 s, 210 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00902136 s, 116 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126561 s, 82.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0085418 s, 123 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00886593 s, 118 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00480937 s, 218 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg626-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg626-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' PASS 77f (7s) == sanityn test 77g: Change TBF type directly ============ 10:30:26 (1788877826) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 ost.OSS.ost_io.nrs_policies=fifo PASS 77g (3s) == sanityn test 77h: Wrong policy name should report error, not LBUG ========================================================== 10:30:29 (1788877829) oleg626-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=abc: No such file or directory oleg626-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='abc': No such file or directory pdsh@oleg626-client: oleg626-server: ssh exited with exit code 2 oleg626-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc: Invalid argument oleg626-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc': Invalid argument pdsh@oleg626-client: oleg626-server: ssh exited with exit code 22 oleg626-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf reg abc: Invalid argument oleg626-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf reg abc': Invalid argument pdsh@oleg626-client: oleg626-server: ssh exited with exit code 22 oleg626-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc efg: Invalid argument oleg626-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc efg': Invalid argument pdsh@oleg626-client: oleg626-server: ssh exited with exit code 22 PASS 77h (4s) == sanityn test 77i: Change rank of TBF rule ============= 10:30:33 (1788877833) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start before jobid={jobid} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start after jobid={jobid} rate=1000 rank=default ost.OSS.ost_io.nrs_tbf_rule=start target jobid={jobid} rate=1000 rank=after Move before itself ost.OSS.ost_io.nrs_tbf_rule=change target rank=target Move to higher rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=before Move to lower rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=after Move before default ost.OSS.ost_io.nrs_tbf_rule=change target rank=default ost.OSS.ost_io.nrs_policies=fifo PASS 77i (6s) == sanityn test 77j: check TBF-OPCode NRS policy ========= 10:30:40 (1788877840) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf opcode ost.OSS.ost_io.nrs_tbf_rule=start ost_r opcode={ost_read} rate=5 ost.OSS.ost_io.nrs_tbf_rule=start ost_w opcode={ost_write} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.113248 s, 148 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183181 s, 57.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0297301 s, 35.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0571679 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0580066 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0301834 s, 34.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0629709 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0395418 s, 26.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0495438 s, 21.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0669986 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0820245 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.085198 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.086593 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0969239 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0722782 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0844131 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0915496 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166851 s, 62.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136058 s, 77.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0121734 s, 86.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0222833 s, 47.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0203634 s, 51.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146902 s, 71.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0206626 s, 50.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0249026 s, 42.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.018549 s, 56.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0196094 s, 53.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.019616 s, 53.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.023148 s, 45.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0221434 s, 47.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0241634 s, 43.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0238561 s, 44.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.025012 s, 41.9 MB/s removed '/mnt/lustre/d77j.sanityn/nrs_w_oleg626-client.virtnet' removed '/mnt/lustre/d77j.sanityn/nrs_r_oleg626-client.virtnet' removed directory '/mnt/lustre/d77j.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 5 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.96342 s, 21.1 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.4361 s, 5.4 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77j.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_r ost.OSS.ost_io.nrs_tbf_rule=stop ost_w ost.OSS.ost_io.nrs_policies=fifo PASS 77j (49s) == sanityn test 77ja: check TBF-UID/GID NRS policy ======= 10:31:28 (1788877888) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg626-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg626-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg626-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg626-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg626-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.22877 s, 13.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0728646 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0795192 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107505 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118384 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125713 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0963675 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0931116 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.096384 s, 10.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154832 s, 6.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154915 s, 6.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160269 s, 6.5 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153215 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156484 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152318 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179737 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16512 s, 6.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0112242 s, 93.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0214681 s, 48.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0175884 s, 59.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158824 s, 66.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0235517 s, 44.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0201358 s, 52.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0204317 s, 51.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0280586 s, 37.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016641 s, 63.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00328435 s, 319 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0215339 s, 48.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.021516 s, 48.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00487605 s, 215 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0237973 s, 44.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00539378 s, 194 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0053055 s, 198 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg626-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg626-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8252 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9458 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg626-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg626-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg626-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg626-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg626-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.22219 s, 13.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0449125 s, 23.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0753259 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0657724 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0808686 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0800007 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0728019 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0924283 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0835653 s, 12.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159314 s, 6.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159357 s, 6.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160929 s, 6.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160037 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147951 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137385 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183848 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170321 s, 6.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149399 s, 70.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00376872 s, 278 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142474 s, 73.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146942 s, 71.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.020731 s, 50.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0176228 s, 59.5 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162014 s, 64.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00438823 s, 239 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0197846 s, 53.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.020974 s, 50.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.020318 s, 51.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0211208 s, 49.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00378821 s, 277 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00923234 s, 114 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00389353 s, 269 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00899516 s, 117 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg626-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg626-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8334 s, 5.3 MB/s Write runtime is 20 s, speed is 5.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9454 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77ja (137s) == sanityn test 77jb: check TBF-UID/GID NRS policy on files that don't belong to us ========================================================== 10:33:45 (1788878025) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg626-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg626-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg626-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg626-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg626-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.22347 s, 13.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0727958 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0627537 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0533272 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0616996 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0697292 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0694672 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0704963 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0602393 s, 17.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164074 s, 6.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164111 s, 6.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168589 s, 6.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164428 s, 6.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175922 s, 6.0 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170552 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166393 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183235 s, 5.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0186903 s, 56.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0203807 s, 51.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0210952 s, 49.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161322 s, 65.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0182517 s, 57.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0192561 s, 54.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0175913 s, 59.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00394487 s, 266 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00426466 s, 246 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.02296 s, 45.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0177221 s, 59.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00511597 s, 205 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0116289 s, 90.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00630548 s, 166 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00614175 s, 171 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00262202 s, 400 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg626-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg626-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8319 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9598 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg626-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg626-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg626-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg626-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg626-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.22601 s, 13.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0464104 s, 22.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0659428 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0963704 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0787039 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0872412 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0940925 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107477 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0948 s, 11.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154661 s, 6.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154665 s, 6.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16436 s, 6.4 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155096 s, 6.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159602 s, 6.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144734 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186274 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167954 s, 6.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016173 s, 64.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141999 s, 73.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0209315 s, 50.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0203451 s, 51.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0190132 s, 55.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0246941 s, 42.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160478 s, 65.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.022817 s, 46.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0233087 s, 45.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00535656 s, 196 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00612493 s, 171 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0247931 s, 42.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0207966 s, 50.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00427376 s, 245 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0171487 s, 61.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0101103 s, 104 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg626-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg626-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8301 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9559 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77jb (134s) == sanityn test 77k: check TBF policy with UID/GID/JobID/OPCode expression ========================================================== 10:35:59 (1788878159) ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg626-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.247536 s, 67.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0265289 s, 39.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0515836 s, 20.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0514294 s, 20.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0651941 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0770247 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0643185 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0929782 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1224 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110976 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109103 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128014 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121987 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111582 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116901 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126148 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142404 s, 7.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169242 s, 62.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170065 s, 61.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155058 s, 67.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167124 s, 62.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173578 s, 60.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146216 s, 71.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00980991 s, 107 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0208069 s, 50.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183532 s, 57.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0105562 s, 99.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0235682 s, 44.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0218917 s, 47.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00605805 s, 173 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00418531 s, 251 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0039119 s, 268 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00480476 s, 218 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg626-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg626-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.94226 s, 21.2 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.72965 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.206.26@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.206.26@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.236755 s, 70.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0293994 s, 35.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0257054 s, 40.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0946711 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0959138 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0925098 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0870146 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0976926 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.092129 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.095662 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0935478 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0875455 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0885817 s, 11.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.261461 s, 4.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.258845 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.303295 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.32191 s, 3.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0107033 s, 98.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162401 s, 64.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139223 s, 75.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0213651 s, 49.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.019987 s, 52.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00439907 s, 238 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00742431 s, 141 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0323965 s, 32.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0244277 s, 42.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0261913 s, 40.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0202352 s, 51.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0276696 s, 37.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00817251 s, 128 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0279001 s, 37.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142389 s, 73.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00557432 s, 188 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg626-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg626-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.92575 s, 21.3 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.72676 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.206.26@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg626-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.235769 s, 71.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.029746 s, 35.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0608872 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.07516 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0634045 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.070266 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0894295 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0681744 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0673741 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0931297 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.091324 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0857479 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101681 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0933978 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0874984 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0712978 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0994575 s, 10.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155491 s, 67.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0245451 s, 42.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013591 s, 77.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0302074 s, 34.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0213569 s, 49.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0311202 s, 33.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0347832 s, 30.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00856881 s, 122 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0297896 s, 35.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0289195 s, 36.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0301972 s, 34.7 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125311 s, 83.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0102209 s, 103 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00821077 s, 128 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0363517 s, 28.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00537614 s, 195 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg626-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg626-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.92617 s, 21.3 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.87623 s, 21.5 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg626-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.434456 s, 38.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0344995 s, 30.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.04666 s, 22.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.034377 s, 30.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0436084 s, 24.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0444511 s, 23.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.028149 s, 37.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0620152 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.043168 s, 24.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0388866 s, 27.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0732657 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0945541 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0827832 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.371546 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.35751 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.417511 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.414698 s, 2.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0118049 s, 88.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0178641 s, 58.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0180905 s, 58.0 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169051 s, 62.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160752 s, 65.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01871 s, 56.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00480398 s, 218 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00602097 s, 174 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142872 s, 73.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0242489 s, 43.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00982292 s, 107 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0190196 s, 55.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00577484 s, 182 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149773 s, 70.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00321369 s, 326 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00304415 s, 344 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg626-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg626-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.79282 s, 10.7 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73203 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.90597 s, 21.4 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.72664 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg626-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.84023 s, 20.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0233387 s, 44.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0495272 s, 21.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0400488 s, 26.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.05201 s, 20.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.05207 s, 20.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0641908 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0613433 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0497095 s, 21.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159724 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174063 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184567 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181395 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182312 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193077 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207033 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20912 s, 5.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153164 s, 68.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0191466 s, 54.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0185684 s, 56.5 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157299 s, 66.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166723 s, 62.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0116094 s, 90.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.018363 s, 57.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.025275 s, 41.5 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0185526 s, 56.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.017527 s, 59.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0230992 s, 45.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0194313 s, 54.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.033377 s, 31.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0244275 s, 42.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0217785 s, 48.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0265074 s, 39.6 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg626-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg626-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.6648 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.4256 s, 5.4 MB/s Read runtime is 20 s, speed is 5.000000 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_ug ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg626-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.239424 s, 70.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0397773 s, 26.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0344877 s, 30.4 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0430344 s, 24.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0981128 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119176 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110694 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111681 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116638 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13236 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140253 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16007 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15469 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146538 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151827 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147145 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141503 s, 7.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0181073 s, 57.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.019313 s, 54.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.024924 s, 42.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.029118 s, 36.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0248861 s, 42.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0233279 s, 44.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0212422 s, 49.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0262208 s, 40.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00677606 s, 155 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148909 s, 70.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0320359 s, 32.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0215267 s, 48.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.021683 s, 48.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0278662 s, 37.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0273482 s, 38.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0110048 s, 95.3 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg626-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg626-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.92384 s, 21.3 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73175 s, 10.8 MB/s Read runtime is 10 s, speed is 10.000000 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg626-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.447914 s, 37.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0351622 s, 29.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0479956 s, 21.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0618651 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0566823 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0416345 s, 25.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0472072 s, 22.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0540495 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0850317 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0803718 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110779 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0986702 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10823 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.302739 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.303661 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.364503 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.346476 s, 3.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0193503 s, 54.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0188563 s, 55.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0185667 s, 56.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0230534 s, 45.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.022611 s, 46.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0177454 s, 59.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0044827 s, 234 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0120456 s, 87.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137553 s, 76.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0196115 s, 53.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0214813 s, 48.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0185605 s, 56.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0231847 s, 45.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0292259 s, 35.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00359663 s, 292 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00332701 s, 315 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg626-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg626-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.77966 s, 10.7 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.72928 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.76091 s, 10.7 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73044 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_policies=fifo PASS 77k (359s) == sanityn test 77kb: Check different granular TBF type combination: jobid+opcode ========================================================== 10:41:58 (1788878518) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/nrs_r_oleg626-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.249526 s, 67.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0249271 s, 42.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0305801 s, 34.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0505439 s, 20.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.068055 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0553657 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0633399 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0762339 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.07132 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0593818 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101447 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0932756 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109869 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106494 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109887 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104487 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108027 s, 9.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0194799 s, 53.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183946 s, 57.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152649 s, 68.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00881358 s, 119 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151976 s, 69.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0206309 s, 50.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0251645 s, 41.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0240951 s, 43.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00788303 s, 133 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.02064 s, 50.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0227628 s, 46.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00511293 s, 205 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0043516 s, 241 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0047937 s, 219 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00394635 s, 266 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00441779 s, 237 MB/s removed '/mnt/lustre/d77kb.sanityn/nrs_w_oleg626-client.virtnet' removed '/mnt/lustre/d77kb.sanityn/nrs_r_oleg626-client.virtnet' removed directory '/mnt/lustre/d77kb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.92529 s, 21.3 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.72848 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kb (36s) == sanityn test 77kc: Check different granular TBF type combination: nid+opcode ========================================================== 10:42:34 (1788878554) ost.OSS.ost_io.nrs_policies=tbf nid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.206.26@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.206.26@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.241696 s, 69.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0268925 s, 39.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0423617 s, 24.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0680578 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0732872 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0934159 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110167 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117555 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137258 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111634 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122517 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141149 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143374 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.263194 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.251148 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.293557 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.295415 s, 3.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153719 s, 68.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0174944 s, 59.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0193187 s, 54.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00763904 s, 137 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154136 s, 68.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0179016 s, 58.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00384317 s, 273 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0222739 s, 47.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137565 s, 76.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0228342 s, 45.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00389698 s, 269 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.022924 s, 45.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0308369 s, 34.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0107301 s, 97.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00914526 s, 115 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015043 s, 69.7 MB/s removed '/mnt/lustre/d77kc.sanityn/nrs_w_oleg626-client.virtnet' removed '/mnt/lustre/d77kc.sanityn/nrs_r_oleg626-client.virtnet' removed directory '/mnt/lustre/d77kc.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.93083 s, 21.3 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.7275 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kc.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kc (37s) == sanityn test 77kd: Check different granular TBF type combination: nid+jobid ========================================================== 10:43:11 (1788878591) ost.OSS.ost_io.nrs_policies=tbf nid+jobid ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.206.26@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/nrs_r_oleg626-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.236698 s, 70.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0320252 s, 32.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0278975 s, 37.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0825284 s, 12.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0912233 s, 11.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114409 s, 9.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110237 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104965 s, 10.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1217 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118924 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12686 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129546 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120239 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121252 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131837 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115794 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116047 s, 9.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0176881 s, 59.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0211008 s, 49.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0174625 s, 60.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0216074 s, 48.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0182099 s, 57.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0255114 s, 41.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0214604 s, 48.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0083151 s, 126 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0240586 s, 43.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150296 s, 69.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0211984 s, 49.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00815514 s, 129 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0040226 s, 261 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0268125 s, 39.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00455293 s, 230 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136453 s, 76.8 MB/s removed '/mnt/lustre/d77kd.sanityn/nrs_w_oleg626-client.virtnet' removed '/mnt/lustre/d77kd.sanityn/nrs_r_oleg626-client.virtnet' removed directory '/mnt/lustre/d77kd.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.92154 s, 21.3 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kd.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.96719 s, 21.1 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kd.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kd (36s) == sanityn test 77ke: Check different granular TBF type combination: jobid+opcode ========================================================== 10:43:47 (1788878627) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/nrs_r_oleg626-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.446606 s, 37.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0279406 s, 37.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0570341 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0449447 s, 23.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0650932 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0943145 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0882285 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0857358 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0946221 s, 11.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113898 s, 9.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137729 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138436 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162115 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146164 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189016 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188778 s, 5.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0111518 s, 94.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161742 s, 64.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0197866 s, 53.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159398 s, 65.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165435 s, 63.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128087 s, 81.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0240039 s, 43.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0115284 s, 91.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0239519 s, 43.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0253403 s, 41.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0251574 s, 41.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0108675 s, 96.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00641017 s, 164 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012453 s, 84.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00949515 s, 110 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0061837 s, 170 MB/s removed '/mnt/lustre/d77ke.sanityn/nrs_w_oleg626-client.virtnet' removed '/mnt/lustre/d77ke.sanityn/nrs_r_oleg626-client.virtnet' removed directory '/mnt/lustre/d77ke.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.79374 s, 10.7 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ke.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73967 s, 10.8 MB/s Read runtime is 10 s, speed is 10.000000 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.93312 s, 21.3 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.74033 s, 10.8 MB/s Read runtime is 10 s, speed is 10.000000 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77ke (76s) == sanityn test 77kf: Check different granular TBF type combination: uid+gid ========================================================== 10:45:03 (1788878703) ost.OSS.ost_io.nrs_policies=tbf uid+gid ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/nrs_r_oleg626-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.840813 s, 20.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0778718 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0862118 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0854096 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.097586 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0844785 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.092054 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0859556 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0736853 s, 14.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157299 s, 6.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157264 s, 6.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143197 s, 7.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144305 s, 7.3 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179104 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168385 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175253 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167813 s, 6.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0213662 s, 49.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155259 s, 67.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151258 s, 69.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169533 s, 61.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0245733 s, 42.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0256501 s, 40.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.023726 s, 44.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0212727 s, 49.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0187846 s, 55.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0172657 s, 60.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0333875 s, 31.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136801 s, 76.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0186222 s, 56.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.036401 s, 28.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00822964 s, 127 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00963597 s, 109 MB/s removed '/mnt/lustre/d77kf.sanityn/nrs_w_oleg626-client.virtnet' removed '/mnt/lustre/d77kf.sanityn/nrs_r_oleg626-client.virtnet' removed directory '/mnt/lustre/d77kf.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.6617 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kf.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9609 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77kf.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kf (67s) == sanityn test 77kg: check different granular TBF type combination: uid+opcode ========================================================== 10:46:10 (1788878770) ost.OSS.ost_io.nrs_policies=tbf uid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg626-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.334883 s, 50.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0441133 s, 23.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0375319 s, 27.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0900355 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0858419 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0949829 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104015 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0977136 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114033 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102373 s, 10.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141458 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107822 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113205 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132469 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141552 s, 7.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113649 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122825 s, 8.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0181816 s, 57.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0219305 s, 47.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168831 s, 62.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.026937 s, 38.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0300456 s, 34.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00380056 s, 276 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0252291 s, 41.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141472 s, 74.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0218753 s, 47.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.026566 s, 39.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0194999 s, 53.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00990209 s, 106 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0282109 s, 37.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00430207 s, 244 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00788896 s, 133 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130141 s, 80.6 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg626-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg626-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.96693 s, 21.1 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73343 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg626-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.471763 s, 35.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0313644 s, 33.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0452788 s, 23.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0975304 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.090386 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101914 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120576 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115379 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0984835 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127282 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134883 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131392 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120046 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127979 s, 8.2 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127095 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131604 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160869 s, 6.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0187628 s, 55.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0234719 s, 44.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170214 s, 61.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0171855 s, 61.0 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0186421 s, 56.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0192355 s, 54.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0196381 s, 53.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0203783 s, 51.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0229851 s, 45.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0211308 s, 49.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0239752 s, 43.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0220592 s, 47.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0233724 s, 44.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0235479 s, 44.5 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0260663 s, 40.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0281474 s, 37.3 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg626-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg626-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.85942 s, 10.6 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.74662 s, 10.8 MB/s Read runtime is 10 s, speed is 10.000000 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.77103 s, 10.7 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73239 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kg (123s) == sanityn test 77kh: Verify that Project ID support for NRS TBF rule ========================================================== 10:48:13 (1788878893) using SAVE_PROJECT_SUPPORTED=0 Stopping clients: oleg626-client.virtnet /mnt/lustre (opts:) Stopping client oleg626-client.virtnet /mnt/lustre opts: Stopping clients: oleg626-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg626-client.virtnet /mnt/lustre2 opts: Stopping /mnt/lustre-mds1 (opts:-f) on oleg626-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg626-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg626-server sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime) proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) devtmpfs on /dev type devtmpfs (rw,nosuid,mode=755) securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime) tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev) devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000) tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755) tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755) cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd) pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime) bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700) cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer) cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset) cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio) cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct) cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices) cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio) cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb) cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event) cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory) cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma) cgroup on /sys/fs/cgroup/misc type cgroup (rw,nosuid,nodev,noexec,relatime,misc) cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids) none on /sys/kernel/tracing type tracefs (rw,relatime) configfs on /sys/kernel/config type configfs (rw,relatime) /dev/nbd0 on / type ext4 (ro,relatime,stripe=32) rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) mqueue on /dev/mqueue type mqueue (rw,relatime) hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M) debugfs on /sys/kernel/debug type debugfs (rw,relatime) none on /mnt type ramfs (rw,relatime) /dev/vda on /home/green/git/lustre-release type squashfs (ro,relatime) none on /var/lib/stateless/writable type tmpfs (rw,relatime) none on /var/cache/man type tmpfs (rw,relatime) none on /var/log type tmpfs (rw,relatime) none on /var/lib/dbus type tmpfs (rw,relatime) none on /tmp type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) none on /var/tmp type tmpfs (rw,relatime) none on /var/lib/NetworkManager type tmpfs (rw,relatime) none on /var/lib/systemd/random-seed type tmpfs (rw,relatime) none on /var/spool type tmpfs (rw,relatime) none on /var/lib/nfs type tmpfs (rw,relatime) none on /var/lib/gssproxy type tmpfs (rw,relatime) none on /var/lib/logrotate type tmpfs (rw,relatime) none on /etc type tmpfs (rw,relatime) none on /var/lib/rsyslog type tmpfs (rw,relatime) none on /root/.ssh type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) 192.168.200.253:/exports/state/oleg626-client.virtnet on /var/lib/stateless/state type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.206.26,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg626-client.virtnet/boot on /boot type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.206.26,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg626-client.virtnet/etc/kdump.conf on /etc/kdump.conf type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.206.26,local_lock=none,addr=192.168.200.253) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=378180k,mode=700) 192.168.200.253:/exports/testreports/69818/testresults/sanityn-zfs-rocky8.10_x86_64-rocky8.10_x86_64 on /tmp/testlogs type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.206.26,local_lock=none,addr=192.168.200.253) /dev/vda on /usr/sbin/mount.lustre type squashfs (ro,relatime) Checking servers environments Checking clients oleg626-client.virtnet environments Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg626-server' oleg626-server: oleg626-server.virtnet: executing load_modules_local oleg626-server: kmemleak disabled oleg626-server: Loading modules from /home/green/git/lustre-release/lustre oleg626-server: detected 4 online CPUs by sysfs oleg626-server: MODOPTS_LIBCFS= oleg626-server: Force libcfs to create 2 CPU partitions oleg626-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Setup mgs, mdt, osts Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg626-server: oleg626-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg626-client: oleg626-server: ssh exited with exit code 1 Started lustre-MDT0000 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg626-server: oleg626-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg626-client: oleg626-server: ssh exited with exit code 1 Started lustre-OST0000 Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg626-server: oleg626-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg626-client: oleg626-server: ssh exited with exit code 1 Started lustre-OST0001 Starting client: oleg626-client.virtnet: -o user_xattr,flock 192.168.206.126@tcp:/lustre /mnt/lustre Starting client oleg626-client.virtnet: -o user_xattr,flock 192.168.206.126@tcp:/lustre /mnt/lustre Started clients oleg626-client.virtnet: 192.168.206.126@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Starting client: oleg626-client.virtnet: -o user_xattr,flock 192.168.206.126@tcp:/lustre /mnt/lustre2 Starting client oleg626-client.virtnet: -o user_xattr,flock 192.168.206.126@tcp:/lustre /mnt/lustre2 Started clients oleg626-client.virtnet: 192.168.206.126@tcp:/lustre on /mnt/lustre2 type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9d9e090ec000.idle_timeout=debug osc.lustre-OST0000-osc-ffff9d9e369bb000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9d9e090ec000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9d9e369bb000.idle_timeout=debug disable quota as required ost.OSS.ost_io.nrs_policies=tbf projid ost.OSS.ost_io.nrs_tbf_rule=start ext_proj_rw projid={100} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.252367 s, 66.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0308045 s, 34.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0295606 s, 35.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0893822 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107966 s, 9.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106208 s, 9.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0860239 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0891918 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111138 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102372 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.09838 s, 10.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107229 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123953 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.254007 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.247495 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.282029 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.30224 s, 3.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0200387 s, 52.3 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170271 s, 61.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0229412 s, 45.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0195174 s, 53.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0188035 s, 55.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0278928 s, 37.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00575836 s, 182 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00426516 s, 246 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0299399 s, 35.0 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0180709 s, 58.0 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0267397 s, 39.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0114415 s, 91.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00481173 s, 218 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0178027 s, 58.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00391029 s, 268 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151774 s, 69.1 MB/s removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg626-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg626-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.95961 s, 21.1 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.93583 s, 21.2 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_proj_rw ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf projid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w projid={100}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r projid={100}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.248849 s, 67.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0282605 s, 37.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0726258 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0790385 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0772942 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0943512 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0817853 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0740422 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0784391 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102521 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0868124 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119469 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140125 s, 7.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21874 s, 4.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.246523 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.275262 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.3019 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143546 s, 73.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138045 s, 76.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152454 s, 68.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0197613 s, 53.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0191529 s, 54.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0222116 s, 47.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0207336 s, 50.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00439152 s, 239 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0255785 s, 41.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01209 s, 86.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0215629 s, 48.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0184221 s, 56.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00605536 s, 173 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0192488 s, 54.5 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00702333 s, 149 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142888 s, 73.4 MB/s removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg626-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg626-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.93662 s, 21.2 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.72967 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77kh (128s) == sanityn test 77ki: Add rule with unsupported TBF type should fail for generic TBF ========================================================== 10:50:21 (1788879021) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 oleg626-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_fail projid={100}&opcode={ost_write} rate=20: Invalid argument oleg626-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_fail projid={100}&opcode={ost_write} rate=20': Invalid argument pdsh@oleg626-client: oleg626-server: ssh exited with exit code 22 ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77ki (8s) == sanityn test 77kj: Verify nodemap support for NRS TBF rule ========================================================== 10:50:30 (1788879030) On MGS 192.168.206.126, active = nodemap.active=1 On MGS 192.168.206.126, TBF = ost.OSS.ost_io.nrs_policies=tbf nodemap ost.OSS.ost_io.nrs_tbf_rule=start ext_nm_rw nodemap={TBF} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.252505 s, 66.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0371184 s, 28.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0630706 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108946 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0865401 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122616 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136606 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12724 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136055 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143923 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135739 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134097 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141624 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.252794 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.240595 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.31548 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.295794 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0233257 s, 45.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0229089 s, 45.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0241858 s, 43.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0193015 s, 54.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0218661 s, 48.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0274849 s, 38.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0228234 s, 45.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0204722 s, 51.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0212086 s, 49.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.03212 s, 32.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0269194 s, 39.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0261478 s, 40.1 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0301162 s, 34.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0273346 s, 38.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0316418 s, 33.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0290374 s, 36.1 MB/s removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg626-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg626-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.9247 s, 21.3 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.96673 s, 21.1 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete On MGS 192.168.206.126, TBF.id = On MGS 192.168.206.126, active = nodemap.active=0 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.213545 s, 78.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0239748 s, 43.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0215107 s, 48.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0747516 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113835 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109036 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112582 s, 9.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12192 s, 8.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0998913 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126511 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104477 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110334 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119444 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0990571 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115101 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120202 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110223 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0122622 s, 85.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.017083 s, 61.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016509 s, 63.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0182894 s, 57.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016894 s, 62.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9d9e090ec000.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0182264 s, 57.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00436382 s, 240 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0224135 s, 46.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0229386 s, 45.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00382459 s, 274 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160341 s, 65.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0221331 s, 47.4 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.028361 s, 37.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0122136 s, 85.9 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0251189 s, 41.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0201709 s, 52.0 MB/s removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg626-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg626-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 1 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 1.8854 s, 55.6 MB/s Write runtime is 3 s, speed is 33.333333 IOPS write rate (33.333333) should exceed limit (20 * 1) Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 0.522094 s, 201 MB/s Read runtime is 1 s, speed is 100.000000 IOPS read rate (100.000000) should exceed limit (20 * 1) cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_nm_rw ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf nodemap+opcode oleg626-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_w nodemap={TBF}&opcode={ost_write} rate=20: Invalid argument oleg626-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_w nodemap={TBF}&opcode={ost_write} rate=20': Invalid argument oleg626-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_r nodemap={TBF}&opcode={ost_read} rate=10: Invalid argument oleg626-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_r nodemap={TBF}&opcode={ost_read} rate=10': Invalid argument pdsh@oleg626-client: oleg626-server: ssh exited with exit code 22 On MGS 192.168.206.126, active = nodemap.active=1 On MGS 192.168.206.126, TBF = ost.OSS.ost_io.nrs_tbf_rule=start ext_w nodemap={TBF}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nodemap={TBF}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.254347 s, 66.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0202893 s, 51.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0355122 s, 29.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0521396 s, 20.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0578202 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0540708 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0546602 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0702799 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.068926 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0974609 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0718552 s, 14.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10703 s, 9.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0898008 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0951197 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104418 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0969018 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163034 s, 6.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146684 s, 71.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173678 s, 60.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149611 s, 70.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157053 s, 66.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164907 s, 63.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142595 s, 73.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0188856 s, 55.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0120795 s, 86.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00371531 s, 282 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.022441 s, 46.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00391138 s, 268 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0097409 s, 108 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0202782 s, 51.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00836146 s, 125 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00429733 s, 244 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00883515 s, 119 MB/s removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg626-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg626-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.94102 s, 21.2 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.74715 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo On MGS 192.168.206.126, TBF.id = On MGS 192.168.206.126, active = nodemap.active=0 PASS 77kj (119s) == sanityn test 77l: check the output of NRS policies for generic TBF ========================================================== 10:52:28 (1788879148) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_policies=fifo PASS 77l (3s) == sanityn test 77m: check NRS Delay slows write RPC processing ========================================================== 10:52:31 (1788879151) ost.OSS.ost_io.nrs_policies=delay ost.OSS.ost_io.nrs_delay_min=4 ost.OSS.ost_io.nrs_delay_max=4 ost.OSS.ost_io.nrs_delay_pct=100 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg626-client.virtnet] [bs=1M] [count=1] [oflag=direct] [conv=fdatasync] 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 3.37122 s, 311 kB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg626-client.virtnet] [bs=1M] [count=10] [oflag=direct] [conv=fdatasync] 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 40.015 s, 262 kB/s ost.OSS.ost_io.nrs_policies=fifo PASS 77m (51s) == sanityn test 77n: check wildcard support for TBF JobID NRS policy ========================================================== 10:53:22 (1788879202) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={*.500} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.119536 s, 140 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0536143 s, 19.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0880508 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0854161 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0999286 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0798871 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0871172 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100939 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0962594 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0978951 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102534 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0998066 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0965261 s, 10.9 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110991 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0946914 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100769 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110576 s, 9.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9d9e090ec000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9d9e090ec000.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143176 s, 73.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0110494 s, 94.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0184719 s, 56.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164961 s, 63.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0215951 s, 48.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00641881 s, 163 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0224969 s, 46.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0062764 s, 167 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0186753 s, 56.1 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014142 s, 74.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136233 s, 77.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157267 s, 66.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00325818 s, 322 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0179968 s, 58.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00799644 s, 131 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00440448 s, 238 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg626-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg626-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77n.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.96687 s, 21.1 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77n.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.92854 s, 21.3 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.*} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.132647 s, 126 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0263462 s, 39.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0492283 s, 21.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0480176 s, 21.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.092394 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0897393 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111711 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106971 s, 9.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.09553 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0975982 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122481 s, 8.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0941834 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11604 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.093399 s, 11.2 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105801 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105121 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115195 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0236306 s, 44.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0272755 s, 38.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0196658 s, 53.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0263203 s, 39.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0271083 s, 38.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0299161 s, 35.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0296711 s, 35.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0258779 s, 40.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0280883 s, 37.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0237365 s, 44.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0319328 s, 32.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0278446 s, 37.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0236721 s, 44.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0247872 s, 42.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0258091 s, 40.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0290267 s, 36.1 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg626-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg626-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.94971 s, 21.2 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.97038 s, 21.1 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_policies=fifo PASS 77n (65s) == sanityn test 77o: Changing rank should not panic ====== 10:54:27 (1788879267) mds.MDS.mdt.nrs_policies=tbf nid mds.MDS.mdt.nrs_tbf_rule=start name nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=start name1 nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=change name1 rank=name mds.MDS.mdt.nrs_tbf_rule=stop name mds.MDS.mdt.nrs_policies=fifo PASS 77o (4s) == sanityn test 77q: Parallel TBF rule definitions should not panic ========================================================== 10:54:31 (1788879271) mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_policies=fifo PASS 77q (40s) == sanityn test 77p: Check validity of rule names for TBF policies ========================================================== 10:55:11 (1788879311) mds.MDS.mdt.nrs_policies=tbf oleg626-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_overflo uid={500} rate=500: File name too long oleg626-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_overflo uid={500} rate=500': File name too long pdsh@oleg626-client: oleg626-server: ssh exited with exit code 36 oleg626-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start uid={500} rate=500: Invalid argument oleg626-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start uid={500} rate=500': Invalid argument pdsh@oleg626-client: oleg626-server: ssh exited with exit code 22 oleg626-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_empty: Invalid argument oleg626-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_empty': Invalid argument pdsh@oleg626-client: oleg626-server: ssh exited with exit code 22 oleg626-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p@spec uid={500} rate=500: Invalid argument oleg626-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p@spec uid={500} rate=500': Invalid argument pdsh@oleg626-client: oleg626-server: ssh exited with exit code 22 oleg626-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p.spec uid={500} rate=500: Invalid argument oleg626-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p.spec uid={500} rate=500': Invalid argument pdsh@oleg626-client: oleg626-server: ssh exited with exit code 22 oleg626-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p~spec uid={500} rate=500: Invalid argument oleg626-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p~spec uid={500} rate=500': Invalid argument pdsh@oleg626-client: oleg626-server: ssh exited with exit code 22 oleg626-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p#spec uid={500} rate=500: Invalid argument oleg626-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p#spec uid={500} rate=500': Invalid argument pdsh@oleg626-client: oleg626-server: ssh exited with exit code 22 oleg626-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p/spec uid={500} rate=500: Invalid argument oleg626-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p/spec uid={500} rate=500': Invalid argument pdsh@oleg626-client: oleg626-server: ssh exited with exit code 22 oleg626-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p^spec uid={500} rate=500: Invalid argument oleg626-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p^spec uid={500} rate=500': Invalid argument pdsh@oleg626-client: oleg626-server: ssh exited with exit code 22 oleg626-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p%spec uid={500} rate=500: Invalid argument oleg626-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p%spec uid={500} rate=500': Invalid argument pdsh@oleg626-client: oleg626-server: ssh exited with exit code 22 oleg626-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p*spec uid={500} rate=500: Invalid argument oleg626-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p*spec uid={500} rate=500': Invalid argument pdsh@oleg626-client: oleg626-server: ssh exited with exit code 22 oleg626-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p;spec uid={500} rate=500: Invalid argument oleg626-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p;spec uid={500} rate=500': Invalid argument pdsh@oleg626-client: oleg626-server: ssh exited with exit code 22 oleg626-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p,spec uid={500} rate=500: Invalid argument oleg626-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p,spec uid={500} rate=500': Invalid argument pdsh@oleg626-client: oleg626-server: ssh exited with exit code 22 oleg626-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p?spec uid={500} rate=500: Invalid argument oleg626-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p?spec uid={500} rate=500': Invalid argument pdsh@oleg626-client: oleg626-server: ssh exited with exit code 22 oleg626-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77pspec uid={500} rate=500: Invalid argument oleg626-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p>spec uid={500} rate=500': Invalid argument pdsh@oleg626-client: oleg626-server: ssh exited with exit code 22 oleg626-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p:spec uid={500} rate=500: Invalid argument oleg626-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p:spec uid={500} rate=500': Invalid argument pdsh@oleg626-client: oleg626-server: ssh exited with exit code 22 oleg626-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p+spec uid={500} rate=500: Invalid argument oleg626-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p+spec uid={500} rate=500': Invalid argument pdsh@oleg626-client: oleg626-server: ssh exited with exit code 22 oleg626-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p=spec uid={500} rate=500: Invalid argument oleg626-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p=spec uid={500} rate=500': Invalid argument pdsh@oleg626-client: oleg626-server: ssh exited with exit code 22 oleg626-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p)spec uid={500} rate=500: Invalid argument oleg626-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p)spec uid={500} rate=500': Invalid argument pdsh@oleg626-client: oleg626-server: ssh exited with exit code 22 oleg626-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p(spec uid={500} rate=500: Invalid argument oleg626-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p(spec uid={500} rate=500': Invalid argument pdsh@oleg626-client: oleg626-server: ssh exited with exit code 22 oleg626-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p{spec uid={500} rate=500: Invalid argument oleg626-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p{spec uid={500} rate=500': Invalid argument pdsh@oleg626-client: oleg626-server: ssh exited with exit code 22 oleg626-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p}spec uid={500} rate=500: Invalid argument oleg626-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p}spec uid={500} rate=500': Invalid argument pdsh@oleg626-client: oleg626-server: ssh exited with exit code 22 oleg626-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p|spec uid={500} rate=500: Invalid argument oleg626-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p|spec uid={500} rate=500': Invalid argument pdsh@oleg626-client: oleg626-server: ssh exited with exit code 22 oleg626-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p[spec uid={500} rate=500: Invalid argument oleg626-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p[spec uid={500} rate=500': Invalid argument pdsh@oleg626-client: oleg626-server: ssh exited with exit code 22 oleg626-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p]spec uid={500} rate=500: Invalid argument oleg626-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p]spec uid={500} rate=500': Invalid argument pdsh@oleg626-client: oleg626-server: ssh exited with exit code 22 oleg626-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p!spec uid={500} rate=500: Invalid argument oleg626-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p!spec uid={500} rate=500': Invalid argument pdsh@oleg626-client: oleg626-server: ssh exited with exit code 22 oleg626-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p&spec uid={500} rate=500: Invalid argument oleg626-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p&spec uid={500} rate=500': Invalid argument pdsh@oleg626-client: oleg626-server: ssh exited with exit code 22 oleg626-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p$spec uid={500} rate=500: Invalid argument oleg626-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p$spec uid={500} rate=500': Invalid argument pdsh@oleg626-client: oleg626-server: ssh exited with exit code 22 oleg626-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p`spec uid={500} rate=500: Invalid argument oleg626-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p`spec uid={500} rate=500': Invalid argument pdsh@oleg626-client: oleg626-server: ssh exited with exit code 22 oleg626-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p\spec uid={500} rate=500: Invalid argument oleg626-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p\spec uid={500} rate=500': Invalid argument pdsh@oleg626-client: oleg626-server: ssh exited with exit code 22 mds.MDS.mdt.nrs_policies=fifo PASS 77p (14s) == sanityn test 77r: Change type of tbf policy at run time ========================================================== 10:55:25 (1788879325) mds.MDS.mdt.nrs_policies=tbf uid mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf gid mds.MDS.mdt.nrs_tbf_rule=start md_rule gid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start md_rule jobid={*.500} rate=1 mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=fifo check the number of requests in queue: fifo: 0 crrn: 0 tbf: 0 delay: 0 fifo: 0 crrn: 0 tbf: 0 delay: 0 cleanup 77r 277140 kill md_thread xargs PASS 77r (42s) == sanityn test 77s: Check TBF LRU shrinker ============== 10:56:07 (1788879367) jobid_name=%e.%p jobid_var=TEST_77s_JOBVAR mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start touch jobid={touch.*} rate=1000 mds.MDS.mdt.nrs_tbf_rule=start unlink jobid={unlink.*} rate=1000 touch jobid={touch.*} 1000, ref 10 touch jobid={touch.*} 1000, ref 0 unlink jobid={unlink.*} 1000, ref 10 unlink jobid={unlink.*} 1000, ref 0 mds.MDS.mdt.nrs_policies=fifo jobid_var=procname_uid jobid_name=%e.%u PASS 77s (16s) == sanityn test 78: Enable policy and specify tunings right away ========================================================== 10:56:23 (1788879383) oleg626-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_orr_quantum=1: No such device oleg626-server: error: set_param: setting 'ost/OSS/*/nrs_orr_quantum'='1': No such device pdsh@oleg626-client: oleg626-server: ssh exited with exit code 19 ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_policies=fifo PASS 78 (4s) == sanityn test 79: xattr: intent error ================== 10:56:27 (1788879387) File: /mnt/lustre/d79.sanityn Size: 16384 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115238826936347 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-08 10:56:27.000000000 -0400 Modify: 2026-09-08 10:56:27.000000000 -0400 Change: 2026-09-08 10:56:27.000000000 -0400 Birth: 2026-09-08 10:56:27.000000000 -0400 fail_val=0 fail_loc=0x80000160 fail_val=0 fail_loc=0x80000131 # file: mnt/lustre/d79.sanityn trusted.name1="value1" PASS 79 (12s) == sanityn test 80a: migrate directory when some children is being opened ========================================================== 10:56:39 (1788879399) SKIP: sanityn test_80a needs >= 2 MDTs SKIP 80a (2s) == sanityn test 80b: Accessing directory during migration ========================================================== 10:56:41 (1788879401) SKIP: sanityn test_80b needs >= 2 MDTs SKIP 80b (1s) == sanityn test 81a: rename and stat under striped directory ========================================================== 10:56:42 (1788879402) SKIP: sanityn test_81a needs >= 2 MDTs SKIP 81a (1s) == sanityn test 81b: rename under striped directory doesn't deadlock ========================================================== 10:56:43 (1788879403) SKIP: sanityn test_81b We need at least 2 MDTs for this test SKIP 81b (1s) == sanityn test 81c: rename revoke LOOKUP lock for remote object ========================================================== 10:56:44 (1788879404) SKIP: sanityn test_81c needs >= 4 MDTs SKIP 81c (1s) == sanityn test 81d: parallel rename file cross-dir on same MDT ========================================================== 10:56:45 (1788879405) total: 2500 open/close in 7.86 seconds: 318.14 ops/second snapshot_time 1788879436.087028932 secs.nsecs start_time 1788879406.188067469 secs.nsecs elapsed_time 29.898961463 secs.nsecs open 2505 samples [usecs] 59 4750 1246165 756158555 close 2504 samples [usecs] 69 577 341718 54632474 mknod 2501 samples [usecs] 225 4648 1107617 609172005 unlink 2500 samples [usecs] 378 81745 3005043 20148079377 mkdir 2 samples [usecs] 636 860 1496 1144096 rmdir 2 samples [usecs] 3990 4338 8328 34738344 rename 2500 samples [usecs] 297 12256 3419109 7174401367 getattr 3760 samples [usecs] 8 5912 114081 94272745 parallel_rename_file 2500 samples [usecs] 297 12256 3419109 7174401367 crossdir_rename 2500 samples [usecs] 297 12256 3419109 7174401367 readpage 29 samples [usecs] 2570 78579 414040 12254590852 PASS 81d (33s) == sanityn test 82: fsetxattr and fgetxattr on orphan files ========================================================== 10:57:18 (1788879438) multiop /mnt/lustre/f82.sanityn vO_ac TMPPIPE=/tmp/multiop_open_wait_pipe.7625 multiop /mnt/lustre2/f82.sanityn vo_Ac TMPPIPE=/tmp/multiop_open_wait_pipe.7625 PASS 82 (3s) == sanityn test 83: access striped directory while it is being created/unlinked ========================================================== 10:57:21 (1788879441) SKIP: sanityn test_83 needs >= 2 MDTs SKIP 83 (2s) == sanityn test 84: 0-nlink race in lu_object_find() ===== 10:57:23 (1788879443) fail_loc=0x8000060b PASS 84 (8s) == sanityn test 85: Lustre API root cache race =========== 10:57:31 (1788879451) Starting: llapi_root_test /mnt/lustre/f85.sanityn /mnt/lustre2/f85.sanityn Starting test test1 at 1788879451 Finishing test test1 at 1788879452 PASS 85 (3s) == sanityn test 90: open/create and unlink striped directory ========================================================== 10:57:34 (1788879454) SKIP: sanityn test_90 needs >= 2 MDTs SKIP 90 (1s) == sanityn test 91: chmod and unlink striped directory === 10:57:35 (1788879455) SKIP: sanityn test_91 needs >= 2 MDTs SKIP 91 (2s) == sanityn test 92: create remote directory under orphan directory ========================================================== 10:57:37 (1788879457) SKIP: sanityn test_92 needs >= 2 MDTs SKIP 92 (1s) == sanityn test 93: alloc_rr should not allocate on same ost ========================================================== 10:57:38 (1788879458) fail_loc=0x00000163 fail_loc=0x0 /mnt/lustre/f93.sanityn-1/file1 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 6540 0x198c 0x240000400 1 6468 0x1944 0x280000400 /mnt/lustre2/f93.sanityn-2/file2 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 6469 0x1945 0x280000400 0 6541 0x198d 0x240000400 rc1=0 and rc2=0 PASS 93 (10s) == sanityn test 94: signal vs CP callback race =========== 10:57:48 (1788879468) osc.lustre-OST0000-osc-ffff9d9e090ec000.idle_timeout=0 osc.lustre-OST0000-osc-ffff9d9e369bb000.idle_timeout=0 osc.lustre-OST0001-osc-ffff9d9e090ec000.idle_timeout=0 osc.lustre-OST0001-osc-ffff9d9e369bb000.idle_timeout=0 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0458679 s, 179 kB/s fail_val=5 fail_loc=0x80000312 fail_val=6 fail_loc=0x80000329 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 3.08204 s, 1.3 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 6389: 298110 Alarm clock dd if=/dev/zero of=$DIR/$tfile conv=notrunc oflag=append bs=4k count=1 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.021641 s, 189 kB/s osc.lustre-OST0000-osc-ffff9d9e090ec000.idle_timeout=debug osc.lustre-OST0000-osc-ffff9d9e369bb000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9d9e090ec000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9d9e369bb000.idle_timeout=debug PASS 94 (11s) == sanityn test 95a: Check readpage() on a page that was removed from page cache ========================================================== 10:57:59 (1788879479) llite.lustre-ffff9d9e090ec000.fast_read=0 llite.lustre-ffff9d9e369bb000.fast_read=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0215131 s, 97.5 MB/s fail_loc=0x80001421 fail_val=7 fail_loc=0x80001422 fail_val=10 llite.lustre-ffff9d9e090ec000.fast_read=1 llite.lustre-ffff9d9e369bb000.fast_read=1 PASS 95a (15s) == sanityn test 95b: Check readpage() on a page that is no longer uptodate ========================================================== 10:58:15 (1788879495) llite.lustre-ffff9d9e090ec000.fast_read=0 llite.lustre-ffff9d9e369bb000.fast_read=0 1+0 records in 1+0 records out 12288 bytes (12 kB, 12 KiB) copied, 0.00200936 s, 6.1 MB/s fail_loc=0x80001424 fail_val=5 fail_loc=0 llite.lustre-ffff9d9e090ec000.fast_read=1 llite.lustre-ffff9d9e369bb000.fast_read=1 PASS 95b (9s) == sanityn test 100a: DoM: glimpse RPCs for stat without IO lock (DoM only file) ========================================================== 10:58:23 (1788879503) SKIP: sanityn test_100a Reserved for glimpse-ahead SKIP 100a (2s) == sanityn test 100b: DoM: no glimpse RPC for stat with IO lock (DoM only file) ========================================================== 10:58:25 (1788879505) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00181828 s, 2.3 MB/s /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK PASS 100b (2s) == sanityn test 100c: DoM: write vs stat without IO lock (combined file) ========================================================== 10:58:27 (1788879507) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0796347 s, 26.3 MB/s /mnt/lustre/d100c.sanityn/dom has type file OK /mnt/lustre/d100c.sanityn/dom has size 2097152 OK PASS 100c (3s) == sanityn test 100d: DoM: write+truncate vs stat without IO lock (combined file) ========================================================== 10:58:30 (1788879510) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0509736 s, 41.1 MB/s /mnt/lustre/d100d.sanityn/dom has type file OK /mnt/lustre/d100d.sanityn/dom has size 4096 OK PASS 100d (3s) == sanityn test 100e: DoM: read on open and file size ==== 10:58:33 (1788879513) 1+0 records in 1+0 records out 12000 bytes (12 kB, 12 KiB) copied, 0.00278278 s, 4.3 MB/s PASS 100e (3s) == sanityn test 101a: Discard DoM data on unlink ========= 10:58:36 (1788879516) /mnt/lustre/f101a.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00221968 s, 1.8 MB/s PASS 101a (3s) == sanityn test 101b: Discard DoM data on rename ========= 10:58:39 (1788879519) /mnt/lustre/f101b.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00252388 s, 1.6 MB/s PASS 101b (2s) == sanityn test 101c: Discard DoM data on close-unlink === 10:58:41 (1788879521) /mnt/lustre/f101c.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00152411 s, 2.7 MB/s PASS 101c (4s) == sanityn test 102: Test open by handle of unlinked file ========================================================== 10:58:45 (1788879525) file: /mnt/lustre/f102.sanityn fh_bytes: 32 fh_type: 151 fh_data: d1 0b 00 00 02 00 00 00 fc 11 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 file's parent FID is [0:0x0:0x0] file FID is [0x200000bd1:0x11fc:0x0] access via mount point '/mnt/lustre2' - OK access after unlink - OK check_fhandle_syscalls test Passed! PASS 102 (2s) == sanityn test 103: Test size correctness with lockahead ========================================================== 10:58:47 (1788879527) fail_loc=0x415 fail_loc=0x415 fail_val=2 Incorrect size expected (no glimpse fix): Starting test test23 at 1788879528 lockahead_test: lockahead_test.c:1171: test23: assertion 'sb.st_size == sb2.st_size' failed: size on /mnt/lustre/d103.sanityn/f103.sanityn and /mnt/lustre2/d103.sanityn/f103.sanityn differs: 0 vs 1048576 fail_loc=0 fail_loc=0x214 fail_val=2 Starting test test23 at 1788879531 Finishing test test23 at 1788879531 PASS 103 (8s) == sanityn test 104: Verify that MDS stores atime/mtime/ctime during close ========================================================== 10:58:55 (1788879535) SKIP: sanityn test_104 ldiskfs only test SKIP 104 (2s) == sanityn test 105: Glimpse and lock cancel race ======== 10:58:57 (1788879537) fail_loc=0x416 PASS 105 (23s) == sanityn test 106a: Verify the btime via statx() ======= 10:59:20 (1788879560) SKIP: sanityn test_106a Test only for ldiskfs and statx() supported SKIP 106a (1s) == sanityn test 106b: Glimpse RPCs test for statx ======== 10:59:21 (1788879561) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00372451 s, 275 kB/s File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238826938883 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-08 10:59:22.000000000 -0400 Modify: 2026-09-08 10:59:22.000000000 -0400 Change: 2026-09-08 10:59:22.000000000 -0400 Birth: 2026-09-08 10:59:22.000000000 -0400 /mnt/lustre/f106b.sanityn 144115238826938883 -rw-r--r-- 1 0 0 1788879562 1788879562 1788879562 File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238826938883 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-08 10:59:22.000000000 -0400 Modify: 2026-09-08 10:59:22.000000000 -0400 Change: 2026-09-08 10:59:22.000000000 -0400 Birth: 2026-09-08 10:59:22.000000000 -0400 1788879562 1024 1 PASS 106b (3s) == sanityn test 106c: Verify statx attributes mask ======= 10:59:24 (1788879564) PASS 106c (3s) == sanityn test 107a: Basic grouplock conflict =========== 10:59:27 (1788879567) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.113128 s, 92.7 MB/s /mnt/lustre/f107a.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 6476 0x194c 0x280000400 multiop /mnt/lustre/f107a.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7625 multiop /mnt/lustre2/f107a.sanityn vO_G16022000r10g16022000c TMPPIPE=/tmp/multiop_open_wait_pipe.7625 First grouplock blocks second one PASS 107a (5s) == sanityn test 107b: Grouplock is added to the head of waiting list ========================================================== 10:59:32 (1788879572) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.08276 s, 127 MB/s /mnt/lustre/f107b.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 6550 0x1996 0x240000400 multiop /mnt/lustre/f107b.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7625 Grouplock blocks IO First grouplock blocks second one Second grouplock blocks IO PASS 107b (9s) == sanityn test 108a: lseek: parallel updates ============ 10:59:41 (1788879581) 1+0 records in 1+0 records out 10000 bytes (10 kB, 9.8 KiB) copied, 0.00267567 s, 3.7 MB/s llite.lustre-ffff9d9e090ec000.hybrid_io=0 llite.lustre-ffff9d9e369bb000.hybrid_io=0 fail_val=4 fail_loc=0x80000414 1+0 records in 1+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 4.34179 s, 1.9 MB/s PASS 108a (8s) == sanityn test 109: Race with several mount instances on 1 node ========================================================== 10:59:49 (1788879589) 192.168.206.126@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg626-client.virtnet /mnt/lustre (opts:) 192.168.206.126@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg626-client.virtnet /mnt/lustre2 (opts:) Starting race between client mounts (duration 30) Iteration 0 Starting client: oleg626-client.virtnet: -o user_xattr,flock 192.168.206.126@tcp:/lustre /mnt/lustre2 Starting client: oleg626-client.virtnet: -o user_xattr,flock 192.168.206.126@tcp:/lustre /mnt/lustre 192.168.206.126@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 192.168.206.126@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg626-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg626-client.virtnet /mnt/lustre (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg626-server' oleg626-server: oleg626-server.virtnet: executing load_modules_local oleg626-server: kmemleak disabled oleg626-server: Loading modules from /home/green/git/lustre-release/lustre oleg626-server: detected 4 online CPUs by sysfs oleg626-server: MODOPTS_LIBCFS= oleg626-server: Force libcfs to create 2 CPU partitions oleg626-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Iteration 1 Starting client: oleg626-client.virtnet: -o user_xattr,flock 192.168.206.126@tcp:/lustre /mnt/lustre Starting client: oleg626-client.virtnet: -o user_xattr,flock 192.168.206.126@tcp:/lustre /mnt/lustre2 192.168.206.126@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 192.168.206.126@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg626-client.virtnet /mnt/lustre (opts:) Stopping client oleg626-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg626-server' oleg626-server: oleg626-server.virtnet: executing load_modules_local oleg626-server: kmemleak disabled oleg626-server: Loading modules from /home/green/git/lustre-release/lustre oleg626-server: detected 4 online CPUs by sysfs oleg626-server: MODOPTS_LIBCFS= oleg626-server: Force libcfs to create 2 CPU partitions oleg626-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Iteration 2 Starting client: oleg626-client.virtnet: -o user_xattr,flock 192.168.206.126@tcp:/lustre /mnt/lustre Starting client: oleg626-client.virtnet: -o user_xattr,flock 192.168.206.126@tcp:/lustre /mnt/lustre2 192.168.206.126@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 192.168.206.126@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg626-client.virtnet /mnt/lustre (opts:) Stopping client oleg626-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg626-server' oleg626-server: oleg626-server.virtnet: executing load_modules_local oleg626-server: kmemleak disabled oleg626-server: Loading modules from /home/green/git/lustre-release/lustre oleg626-server: detected 4 online CPUs by sysfs oleg626-server: MODOPTS_LIBCFS= oleg626-server: Force libcfs to create 2 CPU partitions oleg626-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Starting client: oleg626-client.virtnet: -o user_xattr,flock 192.168.206.126@tcp:/lustre /mnt/lustre Starting client: oleg626-client.virtnet: -o user_xattr,flock 192.168.206.126@tcp:/lustre /mnt/lustre2 PASS 109 (39s) == sanityn test 110: do not grant another lock on resend ========================================================== 11:00:28 (1788879628) fail_loc=0x525 fail_val=3 fail_loc=0x534 File: /mnt/lustre/f110.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238860488707 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-08 11:00:28.000000000 -0400 Modify: 2026-09-08 11:00:28.000000000 -0400 Change: 2026-09-08 11:00:28.000000000 -0400 Birth: 2026-09-08 11:00:28.000000000 -0400 fail_loc=0 fail_val=0 Connected clients: oleg626-client.virtnet oleg626-client.virtnet PASS 110 (170s) == sanityn test 111: A racy rename/link an open file should not cause fs corruption ========================================================== 11:03:18 (1788879798) SKIP: sanityn test_111 needs >= 2 MDTs SKIP 111 (1s) == sanityn test 112: update max-inherit in default LMV === 11:03:19 (1788879799) SKIP: sanityn test_112 We need at least 2 MDTs for this test SKIP 112 (1s) == sanityn test 113: check servers of specified fs ======= 11:03:21 (1788879801) lustre-OST0000-osc-ffff9d9e1166e000 active. lustre-OST0001-osc-ffff9d9e1166e000 active. lustre-OST0000-osc-ffff9d9e1166e000 active. lustre-OST0001-osc-ffff9d9e1166e000 active. MGC192.168.206.126@tcp active. PASS 113 (2s) == sanityn test 114: implicit default LMV inherit ======== 11:03:23 (1788879803) SKIP: sanityn test_114 We need at least 2 MDTs for this test SKIP 114 (1s) == sanityn test 115: ldiskfs doesn't check direntry for uniqueness ========================================================== 11:03:24 (1788879804) SKIP: sanityn test_115 ldiskfs only test SKIP 115 (1s) == sanityn test 116: DNE: Set default LMV layout from a remote client ========================================================== 11:03:25 (1788879805) SKIP: sanityn test_116 needs >= 2 MDTs SKIP 116 (1s) == sanityn test 121: trunc append race =================== 11:03:26 (1788879806) 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00451206 s, 4.5 MB/s fail_loc=0x80001436 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.013555 s, 1.5 MB/s 0000000 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0005000 0000 0000 0000 0000 0000 0000 0000 0000 * 000c800 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0011800 PASS 121 (5s) == sanityn test 122: directory size is consistent across mounts ========================================================== 11:03:31 (1788879811) PASS 122 (3s) == sanityn test 200: service remains healthy while able to process request ========================================================== 11:03:34 (1788879814) at_max=10 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=9 fail_loc=0x1e1 fail_val=20 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1046 s, 0.2 kB/s 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1022 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1185 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1177 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1106 s, 0.2 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.113 s, 0.2 kB/s 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.108 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1717 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1769 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1767 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1831 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1867 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1931 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1825 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2897 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2944 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2995 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.289 s, 0.1 kB/s at_max=600 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=116 oleg626-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9d9e040f8000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9d9e040f8000.ost_server_uuid in IDLE state after 0 sec PASS 200 (69s) cleanup: ====================================================== == sanityn test complete, duration 5259 sec ============== 11:04:44 (1788879884) === sanityn: start cleanup 11:04:45 (1788879885) === Stopping clients: oleg626-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg626-client.virtnet /mnt/lustre2 opts: === sanityn: finish cleanup 11:06:07 (1788879967) ===