-----============= acceptance-small: sanityn ============----- Sat Sep 5 19:32:47 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg336-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanityn.*ex': No such file or directory excepting tests: 27 40a skipping tests SLOW=no: 33a === sanityn: start setup 19:33:06 (1788651186) === Starting client oleg336-client.virtnet: -o user_xattr,flock 192.168.203.136@tcp:/lustre /mnt/lustre2 Mount client oleg336-client.virtnet: mount -t lustre -o user_xattr,flock 192.168.203.136@tcp:/lustre /mnt/lustre2 Started clients oleg336-client.virtnet: 192.168.203.136@tcp:/lustre on /mnt/lustre2 type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) oleg336-client.virtnet: executing check_config_client /mnt/lustre oleg336-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg336-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff88adc7b35000.idle_timeout=debug osc.lustre-OST0000-osc-ffff88add8dd7800.idle_timeout=debug osc.lustre-OST0001-osc-ffff88adc7b35000.idle_timeout=debug osc.lustre-OST0001-osc-ffff88add8dd7800.idle_timeout=debug disable quota as required oleg336-server: oleg336-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanityn: finish setup 19:33:37 (1788651217) === 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0251649 s, 41.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7631] == sanityn test 0: do_node_vp() and do_facet_vp() do the right thing ========================================================== 19:33:39 (1788651219) PASS 0 (7s) == sanityn test 1: Check attribute updates on 2 mount points ========================================================== 19:33:46 (1788651226) /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0777 OK /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0666 OK /mnt/lustre/f1.sanityn: absent OK PASS 1 (6s) == sanityn test 2a: check cached attribute updates on 2 mtpt's ================================================================== 19:33:52 (1788651232) -rw-r--r-- 1 root root 0 Sep 5 19:33 /mnt/lustre2/f2a /mnt/lustre/f2a has type file OK /mnt/lustre/f2a has perms 0777 OK PASS 2a (6s) == sanityn test 2b: check cached attribute updates on 2 mtpt's ================================================================== 19:33:58 (1788651238) -rw-r--r-- 1 root root 0 Sep 5 19:33 /mnt/lustre2/f2b /mnt/lustre2/f2b has type file OK /mnt/lustre2/f2b has perms 0777 OK PASS 2b (6s) == sanityn test 2c: check cached attribute updates on 2 mtpt's root ============================================================= 19:34:04 (1788651244) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0777 OK PASS 2c (6s) == sanityn test 2d: check cached attribute updates on 2 mtpt's root ============================================================= 19:34:10 (1788651250) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0755 OK PASS 2d (6s) == sanityn test 2e: check chmod on root is propagated to others ========================================================== 19:34:16 (1788651256) total 1 -rwxrwxrwx 1 root root 0 Sep 5 19:33 f2a -rwxrwxrwx 1 root root 0 Sep 5 19:33 f2b total 1 -rwxrwxrwx 1 root root 0 Sep 5 19:33 f2a -rwxrwxrwx 1 root root 0 Sep 5 19:33 f2b running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre2/f2e.sanityn] [count=1] 1+0 records in 1+0 records out 512 bytes copied, 0.0105233 s, 48.7 kB/s PASS 2e (6s) == sanityn test 2f: check attr/owner updates on DNE with 2 mtpt's ========================================================== 19:34:22 (1788651262) SKIP: sanityn test_2f needs >= 2 MDTs SKIP 2f (3s) == sanityn test 2g: check blocks update on sync write ==== 19:34:25 (1788651265) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.328073 s, 6.4 MB/s /mnt/lustre/f2g.sanityn has 4103 blocks /mnt/lustre2/f2g.sanityn has 4103 blocks PASS 2g (5s) == sanityn test 3: symlink on one mtpt, readlink on another ===================================================================== 19:34:31 (1788651271) PASS 3 (5s) == sanityn test 4: fstat validation on multiple mount points ==================================================================== 19:34:36 (1788651276) PASS 4 (7s) == sanityn test 5: create a file on one mount, truncate it on the other ========================================================== 19:34:43 (1788651283) /mnt/lustre/f5 has type file OK /mnt/lustre/f5 has size 100 OK PASS 5 (6s) == sanityn test 6: remove of open file on other node ============================================================================ 19:34:49 (1788651289) opening writing unlinking /mnt/lustre2/f6.sanityn accessing (1) seeking (1) accessing (2) fstat... reading comparing data truncating seeking (2) writing again seeking (3) reading again comparing data again closing SUCCESS - goto beer PASS 6 (6s) == sanityn test 7: remove of open directory on other node ======================================================================= 19:34:55 (1788651295) creating directory /mnt/lustre/d7 opening directory unlinking /mnt/lustre/d7 Ok, everything goes well. PASS 7 (6s) == sanityn test 8: remove of open special file on other node ==================================================================== 19:35:01 (1788651301) creating special file /mnt/lustre/f8.sanityn opening file unlinking /mnt/lustre/f8.sanityn Ok, everything goes well. PASS 8 (6s) == sanityn test 9a: append of file with sub-page size on multiple mounts ========================================================== 19:35:07 (1788651307) PASS 9a (7s) == sanityn test 9b: append to striped sparse file ======== 19:35:15 (1788651315) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0545028 s, 19.2 MB/s 3+0 records in 3+0 records out 3 bytes copied, 0.0609632 s, 0.0 kB/s Data read (expecting 'foo'): foo PASS 9b (6s) == sanityn test 10a: write of file with sub-page size on multiple mounts ========================================================== 19:35:21 (1788651321) 1+0 records in 1+0 records out 1 byte copied, 0.00591434 s, 0.2 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0146563 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.02444 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0307115 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0292222 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0106805 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0134847 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.017304 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0175056 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0125704 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0161796 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0129553 s, 0.1 kB/s PASS 10a (8s) == sanityn test 10b: write of file with sub-page size on multiple mounts ========================================================== 19:35:29 (1788651329) 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.0260093 s, 118 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0427748 s, 95.8 kB/s 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.00052586 s, 5.8 MB/s PASS 10b (8s) == sanityn test 11: execution of file opened for write should return error ============================================================== 19:35:37 (1788651337) multiop /mnt/lustre/d11/f vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7631 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 298: /mnt/lustre2/d11/f: Text file busy PASS 11 (9s) == sanityn test 12: test lock ordering (link, stat, unlink) ========================================================== 19:35:46 (1788651346) warning: '-runtime' deprecated, use '-t runtime' instead - link 353 (time 1788651358.83 total 10.02 last 35.24) - link 686 (time 1788651368.83 total 20.02 last 33.29) - link 1069 (time 1788651378.86 total 30.04 last 38.21) - link 1427 (time 1788651388.86 total 40.05 last 35.79) - link 1724 (time 1788651398.87 total 50.06 last 29.66) - link 2046 (time 1788651408.90 total 60.09 last 32.11) - link 2398 (time 1788651418.92 total 70.11 last 35.12) - link 2776 (time 1788651428.94 total 80.13 last 37.74) - link 3092 (time 1788651438.96 total 90.15 last 31.53) total: 3358 link in 100.01 seconds: 33.58 ops/second using seed 2528208830 running for 100 seconds - stat 4797 (time 1788651360 ; total 11 ; last 11) - stat 9793 (time 1788651371 ; total 22 ; last 11) - stat 10000 (time 1788651371 ; total 22 ; last 0) - stat 15137 (time 1788651382 ; total 33 ; last 11) - stat 20000 (time 1788651391 ; total 42 ; last 9) - stat 24552 (time 1788651402 ; total 53 ; last 11) - stat 27162 (time 1788651413 ; total 64 ; last 11) - stat 29935 (time 1788651424 ; total 75 ; last 11) - stat 30000 (time 1788651424 ; total 75 ; last 0) - stat 32662 (time 1788651435 ; total 86 ; last 11) - stat 35407 (time 1788651446 ; total 97 ; last 11) total: 36160 stats in 100 seconds: 361.600006 stats/second - unlinked 0 (time 1788651399 ; total 0 ; last 0) unlink(/mnt/lustre2/lockdir/lockfile3358) error: No such file or directory total: 3357 unlinks in 91 seconds: 36.890110 unlinks/second /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 77: kill: (21428) - No such process /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 78: kill: (21430) - No such process PASS 12 (150s) == sanityn test 13: test directory page revocation ======= 19:38:16 (1788651496) f13 f13 PASS 13 (10s) == sanityn test 14aa: execution of file open for write returns -ETXTBSY ========================================================== 19:38:26 (1788651506) multiop /mnt/lustre/d14aa.sanityn/f14aa.sanityn vOw_c TMPPIPE=/tmp/multiop_open_wait_pipe.7631 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 333: /mnt/lustre2/d14aa.sanityn/f14aa.sanityn: Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 22616 Terminated $MULTIOP_PROG $FILE v$ARGS > $TMPPIPE (wd: ~) PASS 14aa (7s) == sanityn test 14ab: open(RDWR) of executing file returns -ETXTBSY ========================================================== 19:38:33 (1788651513) open(O_RDWR|O_CREAT): Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 23196 Terminated $DIR1/$tdir/sleep 60 (wd: ~) PASS 14ab (8s) == sanityn test 14b: truncate of executing file returns -ETXTBSY ================================================================ 19:38:41 (1788651521) truncate: cannot truncate '/mnt/lustre2/d14b.sanityn/sleep' to length 60: Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 348: 23774 Terminated $DIR1/$tdir/sleep 60 PASS 14b (7s) == sanityn test 14c: open(O_TRUNC) of executing file return -ETXTBSY ============================================================ 19:38:48 (1788651528) cp: cannot create regular file '/mnt/lustre2/d14c.sanityn/sleep': Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 360: 24357 Terminated $DIR1/$tdir/sleep 60 PASS 14c (6s) == sanityn test 14d: chmod of executing file is still possible ================================================================== 19:38:55 (1788651535) chmod /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 371: 24939 Terminated $DIR1/$tdir/sleep 60 PASS 14d (8s) == sanityn test 15: test out-of-space with multiple writers ===================================================================== 19:39:03 (1788651543) PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin: MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky sleep 5 for ZFS MDS Waiting for MDT destroys to complete fallocate on zfs doesn't consume space STRIPECOUNT=2 ORIGFREE=7531520 MAXFREE=800000 SKIP: oos2 test_15 oos2.sh: 7531520KB free > 800000KB, increase MAXFREE (or reduce fs size) sleep 5 for ZFS MDS Waiting for MDT destroys to complete SKIP 15 (51s) == sanityn test 16a: 500 iterations of dual-mount fsx ==== 19:39:54 (1788651594) keep default fallocate mode: 2 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.587118 s, 14.3 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.396547 s, 21.2 MB/s Chance of close/open is 1 in 50 Seed set to 1598 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read skipping zero length punch hole skipping zero size read skipping zero size read skipping zero size read truncating to largest ever: 0x90a535 000012[0] 1788651599.728618 trunc from 00000000 to 0x90a534 (0x90a535 bytes) 000015[1] 1788651599.810382 mapread 0x218962 thru 0x21c674 (0x3d13 bytes) 000016[1] 1788651599.869964 punch from 0x76a59f to 0x777617 (0xd079 bytes) 000018[0] 1788651599.886703 write 0x496e72 thru 0x4a30b8 (0xc247 bytes) 000019[0] 1788651599.942301 trunc from 0x90a535 to 0x05f877 (0x8aacbd bytes) 000021[1] 1788651600.048604 mapread 0x05cf51 thru 0x05f877 (0x2927 bytes) 000023[0] 1788651600.146876 read 0x0241b6 thru 0x02fefe (0xbd49 bytes) 000024[0] 1788651600.165456 mapread 0x0563f0 thru 0x058a0b (0x261c bytes) 000026[1] 1788651600.201165 punch from 0x00a7b2 to 0x010389 (0x5bd8 bytes) 000027[0] 1788651600.255620 write 0x48090b thru 0x490048 (0xf73e bytes) 000031[1] 1788651600.420126 write 0x13f020 thru 0x14bcd7 (0xccb8 bytes) 000033[0] 1788651600.476756 mapwrite 0x32410b thru 0x32fad7 (0xb9cd bytes) 000034[0] 1788651600.755426 write 0x4598d5 thru 0x464234 (0xa960 bytes) 000036[1] 1788651600.779593 mapwrite 0x9efc4d thru 0x9ffab9 (0xfe6d bytes) 000045[1] 1788651601.112091 mapread 0x3b46b1 thru 0x3be97f (0xa2cf bytes) 000046[0] 1788651601.140580 write 0x1ccffa thru 0x1db665 (0xe66c bytes) 000047[1] 1788651601.205383 trunc from 0x9ffaba to 0x8ae546 (0x151573 bytes) 000050[0] 1788651601.299408 write 0x9dc0db thru 0x9e0917 (0x483d bytes) 000052[1] 1788651601.398645 trunc from 0x9e0918 to 0x3f5f4d (0x5ea9ca bytes) 000053[0] 1788651601.520294 write 0x2deb39 thru 0x2e0cfc (0x21c4 bytes) 000054[0] 1788651601.616265 punch from 0x1ed3a4 to 0x1eec25 (0x1882 bytes) 000059[0] 1788651601.799476 mapread 0x248953 thru 0x248f1b (0x5c9 bytes) 000062[0] 1788651601.904514 punch from 0x0b4884 to 0x0be64d (0x9dca bytes) 000063[1] 1788651601.937396 trunc from 0x3f5f4e to 0x409d1d (0x13dd0 bytes) 000067[1] 1788651602.056712 mapwrite 0x4ca2d1 thru 0x4cd020 (0x2d50 bytes) 000069[1] 1788651602.253414 mapwrite 0x06e00d thru 0x06f338 (0x132c bytes) 000072[1] 1788651602.390062 mapwrite 0x05802d thru 0x05b949 (0x391d bytes) 000073[0] 1788651602.458010 mapwrite 0x61c67b thru 0x61d33c (0xcc2 bytes) 000075[0] 1788651602.683520 write 0x825880 thru 0x82594c (0xcd bytes) 000076[0] 1788651602.686079 mapread 0x46902b thru 0x46e7a8 (0x577e bytes) 000078[0] 1788651602.757507 write 0x96c711 thru 0x975d73 (0x9663 bytes) 000079[1] 1788651602.759308 punch from 0x00b9ad to 0x00d538 (0x1b8c bytes) 000080[0] 1788651602.807329 read 0x69037f thru 0x694875 (0x44f7 bytes) 000084[1] 1788651602.966739 trunc from 0x975d74 to 0x78afbe (0x1eadb5 bytes) 000085[0] 1788651603.020022 punch from 0x50ca6b to 0x51abf8 (0xe18e bytes) 000086[0] 1788651603.060156 punch from 0x1e4301 to 0x1e7594 (0x3294 bytes) 000087[1] 1788651603.085582 punch from 0x4393a6 to 0x441c02 (0x885d bytes) 000090[0] 1788651603.170423 write 0x35677f thru 0x363919 (0xd19b bytes) 000091[0] 1788651603.217298 read 0x201090 thru 0x20f594 (0xe505 bytes) 000094[1] 1788651603.290629 read 0x218d10 thru 0x22036e (0x765f bytes) 000095[1] 1788651603.353555 write 0x0fcbaf thru 0x0fecc8 (0x211a bytes) 000096[1] 1788651603.392311 write 0x204ad0 thru 0x212642 (0xdb73 bytes) 000097[0] 1788651603.415012 write 0x273ae9 thru 0x279691 (0x5ba9 bytes) truncating to largest ever: 0x9ee47b 000098[0] 1788651603.458746 trunc from 0x78afbf to 0x9ee47a (0x2634bc bytes) 000100[1] 1788651603.533938 read 0x3d6ce0 thru 0x3e0f1c (0xa23d bytes) 000101[1] 1788651603.584554 trunc from 0x9ee47b to 0x9e1e37 (0xc643 bytes) 000102[1] 1788651603.662048 punch from 0x28a855 to 0x292f32 (0x86de bytes) 000103[1] 1788651603.670722 read 0x3a8718 thru 0x3ac0c2 (0x39ab bytes) 000108[1] 1788651603.827835 write 0x0d2214 thru 0x0e0b3a (0xe927 bytes) 000109[0] 1788651603.873050 trunc from 0x9e1e38 to 0x3d40fc (0x60dd3b bytes) 000112[1] 1788651604.017102 write 0x5e5835 thru 0x5f2f8a (0xd756 bytes) 000113[1] 1788651604.040435 write 0x58f617 thru 0x5957cf (0x61b9 bytes) 000116[1] 1788651604.088442 write 0x79febc thru 0x7a42db (0x4420 bytes) 000117[1] 1788651604.127009 trunc from 0x7a42dc to 0x60448b (0x19fe50 bytes) 000118[0] 1788651604.178938 read 0x548886 thru 0x551c25 (0x93a0 bytes) 000119[0] 1788651604.248171 trunc from 0x60448c to 0x4f5d95 (0x10e6f6 bytes) 000122[1] 1788651604.305031 read 0x4f0df8 thru 0x4f4f59 (0x4162 bytes) 000123[0] 1788651604.340640 write 0x6016d0 thru 0x6079d9 (0x630a bytes) 000125[1] 1788651604.419771 read 0x0ec6c7 thru 0x0edc22 (0x155c bytes) 000126[1] 1788651604.451211 read 0x3441ef thru 0x351969 (0xd77b bytes) 000127[1] 1788651604.578675 write 0x54c495 thru 0x55006c (0x3bd8 bytes) 000130[0] 1788651604.754589 mapread 0x075970 thru 0x084363 (0xe9f4 bytes) 000131[1] 1788651604.859888 mapwrite 0x6143b6 thru 0x61b456 (0x70a1 bytes) 000132[0] 1788651605.064321 mapread 0x4ff2df thru 0x504e9e (0x5bc0 bytes) 000133[0] 1788651605.162177 trunc from 0x61b457 to 0x7187a7 (0xfd351 bytes) 000134[1] 1788651605.226153 read 0x3a4ee5 thru 0x3b248f (0xd5ab bytes) 000135[0] 1788651605.273894 write 0x55a969 thru 0x55dcde (0x3376 bytes) 000137[1] 1788651605.315547 mapwrite 0x86b5f4 thru 0x879267 (0xdc74 bytes) 000140[0] 1788651605.547211 mapread 0x22ed13 thru 0x23857b (0x9869 bytes) 000141[1] 1788651605.616791 mapread 0x27e1bc thru 0x2899df (0xb824 bytes) 000142[0] 1788651605.670673 mapread 0x0c06b7 thru 0x0ca5ae (0x9ef8 bytes) 000143[1] 1788651605.717854 mapread 0x8452ce thru 0x8458a3 (0x5d6 bytes) 000144[0] 1788651605.722996 write 0x4b17bf thru 0x4b79b3 (0x61f5 bytes) 000147[0] 1788651605.763596 read 0x46805a thru 0x471d74 (0x9d1b bytes) 000149[1] 1788651605.792353 punch from 0x0c2557 to 0x0d0341 (0xddeb bytes) 000150[1] 1788651605.830720 mapwrite 0x830942 thru 0x837688 (0x6d47 bytes) 000151[0] 1788651605.970138 write 0x6091a0 thru 0x60d7ee (0x464f bytes) 000155[1] 1788651606.043575 mapwrite 0x3917d5 thru 0x39ec15 (0xd441 bytes) 000157[1] 1788651606.224639 trunc from 0x879268 to 0x16da76 (0x70b7f1 bytes) 000158[1] 1788651606.328219 write 0x143398 thru 0x14a997 (0x7600 bytes) 000160[1] 1788651606.417528 mapread 0x0d0559 thru 0x0d4d06 (0x47ae bytes) 000161[0] 1788651606.476357 mapread 0x16c7a8 thru 0x16da76 (0x12cf bytes) 000164[1] 1788651606.626976 write 0x6186b9 thru 0x61f407 (0x6d4f bytes) 000166[1] 1788651606.685551 mapread 0x160f63 thru 0x16e68a (0xd728 bytes) 000167[1] 1788651606.741246 read 0x318596 thru 0x3277e9 (0xf254 bytes) 000168[0] 1788651606.849960 mapread 0x5b532f thru 0x5bf91d (0xa5ef bytes) 000169[0] 1788651606.911787 trunc from 0x61f408 to 0x2182a5 (0x407162 bytes) 000170[1] 1788651606.964552 trunc from 0x2182a6 to 0x5949f5 (0x37c750 bytes) 000171[1] 1788651607.013763 read 0x41f17d thru 0x425685 (0x6509 bytes) 000174[1] 1788651607.083221 trunc from 0x5949f6 to 0x61ce9b (0x884a6 bytes) 000175[1] 1788651607.120286 punch from 0x573143 to 0x57a602 (0x74c0 bytes) 000176[0] 1788651607.126485 mapread 0x579a84 thru 0x588004 (0xe581 bytes) 000177[1] 1788651607.197939 read 0x4b9524 thru 0x4bfdd8 (0x68b5 bytes) 000179[1] 1788651607.224373 read 0x379c6f thru 0x37afd8 (0x136a bytes) 000180[0] 1788651607.233384 trunc from 0x61ce9c to 0x0a395d (0x57953e bytes) 000181[0] 1788651607.351576 write 0x4389d0 thru 0x447523 (0xeb54 bytes) 000184[1] 1788651607.368003 punch from 0x3ece78 to 0x3f815b (0xb2e4 bytes) 000186[0] 1788651607.451378 punch from 0x11cbbb to 0x1204a1 (0x38e7 bytes) 000188[1] 1788651607.562486 trunc from 0x447524 to 0x400325 (0x471fe bytes) 000189[1] 1788651607.675485 read 0x0774ba thru 0x086676 (0xf1bd bytes) 000195[1] 1788651607.852569 mapwrite 0x161008 thru 0x1634a7 (0x24a0 bytes) 000196[1] 1788651607.965690 trunc from 0x400326 to 0x328383 (0xd7fa2 bytes) 000197[0] 1788651608.035616 write 0x968e44 thru 0x96ef11 (0x60ce bytes) 000199[0] 1788651608.137889 mapread 0x3ba3a0 thru 0x3c7b93 (0xd7f4 bytes) 000200[0] 1788651608.248863 write 0x228fc2 thru 0x2356f3 (0xc732 bytes) 000201[1] 1788651608.276448 mapwrite 0x7b1dba thru 0x7ba4ee (0x8735 bytes) 000202[1] 1788651608.674168 mapwrite 0x97ac90 thru 0x98310f (0x8480 bytes) 000204[0] 1788651608.849539 write 0x7dd5c5 thru 0x7e21bc (0x4bf8 bytes) 000205[0] 1788651608.932689 read 0x405bd3 thru 0x40c928 (0x6d56 bytes) 000206[1] 1788651608.985742 mapwrite 0x6a8d0a thru 0x6ac56a (0x3861 bytes) 000207[1] 1788651609.147826 read 0x2ade91 thru 0x2b2f5e (0x50ce bytes) 000208[1] 1788651609.194539 write 0x25feb0 thru 0x26684c (0x699d bytes) 000209[0] 1788651609.207960 read 0x42ad42 thru 0x43a4c5 (0xf784 bytes) 000211[1] 1788651609.258305 write 0x32a4a8 thru 0x32a9aa (0x503 bytes) 000213[0] 1788651609.286942 punch from 0x790111 to 0x795f6a (0x5e5a bytes) 000214[0] 1788651609.323089 read 0x510469 thru 0x51df38 (0xdad0 bytes) 000215[1] 1788651609.346730 punch from 0x38c723 to 0x38ca66 (0x344 bytes) 000216[0] 1788651609.374025 read 0x3affc8 thru 0x3b5f36 (0x5f6f bytes) 000217[1] 1788651609.392111 mapwrite 0x7bf0a3 thru 0x7bfac1 (0xa1f bytes) 000218[1] 1788651609.473458 mapwrite 0x8ecd3c thru 0x8fbebf (0xf184 bytes) 000220[1] 1788651609.631978 read 0x15c53c thru 0x167589 (0xb04e bytes) 000222[0] 1788651609.672695 read 0x839ab9 thru 0x83a5e5 (0xb2d bytes) 000223[0] 1788651609.699283 write 0x103393 thru 0x11224d (0xeebb bytes) 000225[1] 1788651609.771751 mapwrite 0x289a7c thru 0x293289 (0x980e bytes) 000227[0] 1788651609.960435 trunc from 0x983110 to 0x4c54ad (0x4bdc62 bytes) 000228[1] 1788651610.102285 write 0x55c8ed thru 0x55cdef (0x503 bytes) 000229[0] 1788651610.171751 mapwrite 0x91f52c thru 0x92ac9c (0xb771 bytes) 000233[0] 1788651610.475413 read 0x9242cf thru 0x92ac41 (0x6973 bytes) 000235[0] 1788651610.476539 punch from 0x4d20b6 to 0x4dd630 (0xb57b bytes) 000237[1] 1788651610.508911 mapwrite 0x2508ac thru 0x257ab0 (0x7205 bytes) 000238[1] 1788651610.629883 trunc from 0x92ac9d to 0x0cfb12 (0x85b18a bytes) 000239[1] 1788651610.669968 read 0x038a8b thru 0x04374d (0xacc3 bytes) 000240[1] 1788651610.711627 write 0x219dc5 thru 0x21bba8 (0x1de4 bytes) 000241[1] 1788651610.713703 read 0x1f92b9 thru 0x206637 (0xd37f bytes) 000242[1] 1788651610.764023 write 0x729d1c thru 0x7333a6 (0x968b bytes) 000243[0] 1788651610.768190 trunc from 0x7333a7 to 0x71c965 (0x16a41 bytes) 000249[0] 1788651610.873696 write 0x939711 thru 0x946d06 (0xd5f6 bytes) 000250[0] 1788651610.876991 trunc from 0x946d07 to 0x0b2dce (0x893f38 bytes) 000253[0] 1788651610.927472 mapread 0x04d38e thru 0x05580e (0x8481 bytes) 000255[1] 1788651610.969377 mapwrite 0x38c555 thru 0x38fbe6 (0x3692 bytes) 000258[1] 1788651611.177030 read 0x1ef270 thru 0x1f9174 (0x9f05 bytes) 000259[1] 1788651611.270486 punch from 0x0f4d29 to 0x0fd8cb (0x8ba3 bytes) 000263[1] 1788651611.398673 mapwrite 0x5b9dfc thru 0x5bf392 (0x5597 bytes) 000265[1] 1788651611.529870 write 0x33ec0a thru 0x34db85 (0xef7c bytes) 000266[1] 1788651611.575145 write 0x12fc1d thru 0x13a2c5 (0xa6a9 bytes) 000268[0] 1788651611.609367 read 0x51efcb thru 0x529a57 (0xaa8d bytes) 000269[0] 1788651611.671661 write 0x0dcd8a thru 0x0df478 (0x26ef bytes) 000271[1] 1788651611.745259 mapread 0x5ae57b thru 0x5b4849 (0x62cf bytes) 000272[0] 1788651611.850133 read 0x1df54e thru 0x1e1bf9 (0x26ac bytes) 000273[0] 1788651611.877082 write 0x9215c5 thru 0x925153 (0x3b8f bytes) 000274[1] 1788651611.881308 write 0x7776aa thru 0x77c5a8 (0x4eff bytes) 000275[1] 1788651611.923089 punch from 0x7c9782 to 0x7ce51f (0x4d9e bytes) 000276[0] 1788651611.968059 write 0x2ca1cd thru 0x2cd82d (0x3661 bytes) 000277[0] 1788651612.002719 mapwrite 0x83f9ae thru 0x84f280 (0xf8d3 bytes) 000278[1] 1788651612.156376 mapwrite 0x35f6f3 thru 0x36d78b (0xe099 bytes) 000279[0] 1788651612.430673 read 0x66f971 thru 0x675b44 (0x61d4 bytes) 000280[0] 1788651612.464139 read 0x8294e7 thru 0x835aa6 (0xc5c0 bytes) 000282[1] 1788651612.515593 read 0x045320 thru 0x04fd96 (0xaa77 bytes) 000285[0] 1788651612.528146 mapread 0x12a0d8 thru 0x130122 (0x604b bytes) 000286[0] 1788651612.558400 read 0x87362d thru 0x87b478 (0x7e4c bytes) 000288[1] 1788651612.569886 mapread 0x20255d thru 0x20493d (0x23e1 bytes) 000289[1] 1788651612.610702 read 0x008199 thru 0x009ae3 (0x194b bytes) 000293[1] 1788651612.619906 trunc from 0x925154 to 0x23f391 (0x6e5dc2 bytes) 000294[0] 1788651612.692983 trunc from 0x23f392 to 0x7e5ddc (0x5a6a4b bytes) 000295[0] 1788651612.800852 write 0x053636 thru 0x061b92 (0xe55d bytes) 000296[0] 1788651612.821496 mapwrite 0x499ae7 thru 0x49e82e (0x4d48 bytes) 000299[1] 1788651612.937259 mapwrite 0x7f9d18 thru 0x809278 (0xf561 bytes) 000301[1] 1788651613.122781 mapread 0x1f89ab thru 0x1fc42f (0x3a85 bytes) 000302[0] 1788651613.171374 read 0x0e6262 thru 0x0eda1d (0x77bc bytes) 000303[1] 1788651613.213948 trunc from 0x809279 to 0x14a9e2 (0x6be896 bytes) 000304[1] 1788651613.307518 write 0x1245c1 thru 0x1304e1 (0xbf21 bytes) 000306[0] 1788651613.335878 write 0x48ea85 thru 0x4970af (0x862b bytes) 000307[1] 1788651613.378252 mapwrite 0x9f205e thru 0x9f35f0 (0x1593 bytes) 000309[0] 1788651613.585362 write 0x1e36f7 thru 0x1f2bfa (0xf504 bytes) 000311[0] 1788651613.654498 mapwrite 0x24466d thru 0x24833f (0x3cd3 bytes) 000312[0] 1788651613.755325 trunc from 0x9f35f1 to 0x2784c9 (0x77b127 bytes) 000318[0] 1788651613.902650 read 0x09b0cd thru 0x09d91b (0x284f bytes) 000320[0] 1788651614.007254 read 0x174e61 thru 0x17a8f2 (0x5a92 bytes) 000322[0] 1788651614.071756 punch from 0x0500b9 to 0x05e923 (0xe86b bytes) 000323[0] 1788651614.103119 trunc from 0x2784ca to 0x80fb9e (0x5976d5 bytes) 000324[1] 1788651614.155703 write 0x9ba770 thru 0x9c6e75 (0xc706 bytes) 000325[1] 1788651614.202442 punch from 0x0067b6 to 0x01637b (0xfbc6 bytes) 000326[0] 1788651614.229323 read 0x94e521 thru 0x954ace (0x65ae bytes) 000327[1] 1788651614.299661 punch from 0x627854 to 0x633b08 (0xc2b5 bytes) 000329[1] 1788651614.358930 read 0x0e2c96 thru 0x0e5896 (0x2c01 bytes) 000330[0] 1788651614.369682 trunc from 0x9c6e76 to 0x1613ae (0x865ac7 bytes) 000334[1] 1788651614.516405 punch from 0x099db5 to 0x0a44e0 (0xa72c bytes) 000335[0] 1788651614.577576 read 0x142811 thru 0x14e878 (0xc068 bytes) 000338[0] 1788651614.717180 trunc from 0x1613af to 0x1437ad (0x1dc01 bytes) 000339[0] 1788651614.838266 punch from 0x00f790 to 0x010fa7 (0x1818 bytes) 000341[1] 1788651614.889931 punch from 0x01cdfb to 0x020110 (0x3316 bytes) 000343[0] 1788651614.943399 mapwrite 0x525112 thru 0x52618b (0x107a bytes) 000345[1] 1788651615.142772 mapread 0x238c52 thru 0x2477d0 (0xeb7f bytes) 000350[0] 1788651615.292362 mapread 0x176c52 thru 0x176fd9 (0x388 bytes) 000351[1] 1788651615.312206 mapwrite 0x7f0316 thru 0x7fa09a (0x9d85 bytes) 000354[0] 1788651615.479365 mapwrite 0x1fdd9e thru 0x1fe41e (0x681 bytes) 000355[1] 1788651615.614866 read 0x407bd0 thru 0x409108 (0x1539 bytes) 000357[1] 1788651615.657556 punch from 0x6d13be to 0x6ddbf3 (0xc836 bytes) 000358[0] 1788651615.699924 read 0x08bd5d thru 0x08d3de (0x1682 bytes) 000360[0] 1788651615.741392 trunc from 0x7fa09b to 0x6ab78a (0x14e910 bytes) 000361[1] 1788651615.791731 write 0x549e42 thru 0x5520ec (0x82ab bytes) 000362[1] 1788651615.850697 punch from 0x08a66a to 0x09635b (0xbcf2 bytes) 000363[0] 1788651615.874389 trunc from 0x6ab78b to 0x304442 (0x3a7348 bytes) 000365[0] 1788651615.991994 write 0x77e0d5 thru 0x77f9a6 (0x18d2 bytes) 000366[1] 1788651616.039443 mapread 0x1d5149 thru 0x1ded30 (0x9be8 bytes) 000369[1] 1788651616.105267 mapwrite 0x189bf8 thru 0x1930d4 (0x94dd bytes) 000372[1] 1788651616.290920 mapwrite 0x63960c thru 0x641b9f (0x8594 bytes) 000373[1] 1788651616.470694 read 0x3bd987 thru 0x3c1092 (0x370c bytes) 000376[0] 1788651616.508531 write 0x8e087d thru 0x8ee8bc (0xe040 bytes) 000377[0] 1788651616.538923 mapwrite 0x95aa70 thru 0x968bbe (0xe14f bytes) 000378[0] 1788651616.673853 mapwrite 0x3c6dad thru 0x3ca00a (0x325e bytes) 000380[1] 1788651616.785559 mapread 0x463a93 thru 0x465b86 (0x20f4 bytes) 000381[1] 1788651616.859929 mapwrite 0x41943f thru 0x41ae34 (0x19f6 bytes) 000382[0] 1788651616.995706 write 0x1af2d6 thru 0x1bc29c (0xcfc7 bytes) 000383[0] 1788651617.031323 read 0x29cfff thru 0x2a88cd (0xb8cf bytes) 000385[1] 1788651617.138378 mapwrite 0x388482 thru 0x391e77 (0x99f6 bytes) 000386[1] 1788651617.328849 trunc from 0x968bbf to 0x07ee25 (0x8e9d99 bytes) 000387[0] 1788651617.366728 read 0x014829 thru 0x01f6c8 (0xaea0 bytes) 000389[0] 1788651617.477173 punch from 0x0052a4 to 0x0149ee (0xf74b bytes) 000390[1] 1788651617.550428 trunc from 0x07ee26 to 0x3846c2 (0x30589d bytes) 000392[0] 1788651617.669271 write 0x7d2fb1 thru 0x7dd6c8 (0xa718 bytes) 000394[0] 1788651617.718014 trunc from 0x7dd6c9 to 0x66131c (0x17c3ac bytes) 000395[1] 1788651617.775982 trunc from 0x66131d to 0x787044 (0x125d28 bytes) 000396[1] 1788651617.838523 mapread 0x729bb4 thru 0x72fb85 (0x5fd2 bytes) 000398[1] 1788651617.952502 read 0x45faaa thru 0x46d5a7 (0xdafe bytes) 000399[1] 1788651617.967756 mapread 0x4c3e6c thru 0x4cd158 (0x92ed bytes) 000400[1] 1788651617.992420 read 0x454d4b thru 0x4614a7 (0xc75d bytes) 000403[1] 1788651618.001753 read 0x1cdafd thru 0x1cf1df (0x16e3 bytes) 000404[1] 1788651618.027157 read 0x32dd99 thru 0x337ae7 (0x9d4f bytes) 000405[1] 1788651618.035706 read 0x1dc838 thru 0x1e88cc (0xc095 bytes) 000407[0] 1788651618.109907 punch from 0x19e48e to 0x1ab6ab (0xd21e bytes) 000408[0] 1788651618.155612 read 0x1c0d59 thru 0x1c4d66 (0x400e bytes) 000409[0] 1788651618.194730 trunc from 0x787045 to 0x15822c (0x62ee18 bytes) 000412[0] 1788651618.343944 read 0x001570 thru 0x00e522 (0xcfb3 bytes) 000415[0] 1788651618.465316 punch from 0x111614 to 0x11fd65 (0xe752 bytes) 000418[1] 1788651618.529699 write 0x179992 thru 0x183445 (0x9ab4 bytes) 000420[1] 1788651618.593039 read 0x0f3396 thru 0x0ffaf2 (0xc75d bytes) 000422[1] 1788651618.667045 read 0x033262 thru 0x0419f5 (0xe794 bytes) 000423[0] 1788651618.680458 mapread 0x0fc96e thru 0x1006ff (0x3d92 bytes) 000424[1] 1788651618.759640 write 0x2b59a8 thru 0x2be5b4 (0x8c0d bytes) 000425[1] 1788651618.813475 mapread 0x180bb0 thru 0x18aa1d (0x9e6e bytes) 000426[0] 1788651618.900988 punch from 0x21ad96 to 0x21eb9a (0x3e05 bytes) 000428[1] 1788651619.016011 trunc from 0x2be5b5 to 0x6cd0bb (0x40eb07 bytes) 000431[0] 1788651619.150829 mapwrite 0x1bb2ec thru 0x1c1508 (0x621d bytes) 000432[1] 1788651619.368440 read 0x667f80 thru 0x67045a (0x84db bytes) 000433[1] 1788651619.399879 mapwrite 0x49f968 thru 0x49faf8 (0x191 bytes) 000434[1] 1788651619.503850 write 0x1eb291 thru 0x1faa52 (0xf7c2 bytes) 000435[1] 1788651619.527543 read 0x0478a8 thru 0x04dd25 (0x647e bytes) 000442[0] 1788651619.641881 mapread 0x69d134 thru 0x6abcb3 (0xeb80 bytes) 000443[1] 1788651619.723043 read 0x67b3ff thru 0x67fae8 (0x46ea bytes) 000444[0] 1788651619.744631 mapwrite 0x41b37a thru 0x41d704 (0x238b bytes) 000445[1] 1788651619.872016 read 0x30e777 thru 0x31df20 (0xf7aa bytes) 000447[1] 1788651619.942548 read 0x04e872 thru 0x05b198 (0xc927 bytes) 000448[0] 1788651620.000791 mapwrite 0x76f29c thru 0x76f561 (0x2c6 bytes) 000452[1] 1788651620.251303 mapwrite 0x5d6946 thru 0x5d9c17 (0x32d2 bytes) 000453[1] 1788651620.393512 write 0x84835f thru 0x84b778 (0x341a bytes) 000455[0] 1788651620.440650 write 0x4bf762 thru 0x4c60af (0x694e bytes) 000456[1] 1788651620.490939 trunc from 0x84b779 to 0x710b93 (0x13abe5 bytes) 000457[1] 1788651620.546061 trunc from 0x710b94 to 0x35d017 (0x3b3b7c bytes) 000458[0] 1788651620.566844 write 0x213acf thru 0x21600d (0x253f bytes) 000460[0] 1788651620.628031 read 0x1f248b thru 0x1ffe84 (0xd9fa bytes) 000464[1] 1788651620.707318 read 0x0e3357 thru 0x0f0f9e (0xdc48 bytes) 000465[0] 1788651620.792742 trunc from 0x35d018 to 0x7f2114 (0x4950fd bytes) 000466[1] 1788651620.861618 mapwrite 0x11e1d8 thru 0x1284b8 (0xa2e1 bytes) 000467[1] 1788651621.045784 mapwrite 0x8e1845 thru 0x8e6b7c (0x5338 bytes) 000469[0] 1788651621.273120 punch from 0x5a4957 to 0x5b30f1 (0xe79b bytes) 000470[0] 1788651621.343650 mapwrite 0x78831f thru 0x793a0a (0xb6ec bytes) 000471[1] 1788651621.593704 punch from 0x531a08 to 0x5417dd (0xfdd6 bytes) 000472[1] 1788651621.640564 mapread 0x3c2129 thru 0x3c6cf9 (0x4bd1 bytes) 000473[1] 1788651621.689726 mapread 0x40afb4 thru 0x41096b (0x59b8 bytes) 000475[0] 1788651621.780426 punch from 0x3cff9e to 0x3df3b1 (0xf414 bytes) 000479[1] 1788651621.882651 trunc from 0x8e6b7d to 0x7e2b74 (0x104008 bytes) 000480[1] 1788651621.925886 write 0x8c9939 thru 0x8d19f2 (0x80ba bytes) 000481[1] 1788651621.933376 punch from 0x041290 to 0x0476a6 (0x6417 bytes) 000485[1] 1788651622.013713 write 0x5fd907 thru 0x5ff051 (0x174b bytes) 000488[1] 1788651622.051694 write 0x3b9e1c thru 0x3c5a85 (0xbc6a bytes) 000490[0] 1788651622.105990 trunc from 0x8d19f3 to 0x139ca5 (0x797d4d bytes) 000491[0] 1788651622.214293 trunc from 0x139ca6 to 0x499e4c (0x3601a7 bytes) 000493[0] 1788651622.274883 read 0x48909e thru 0x48f6b9 (0x661c bytes) 000496[1] 1788651622.326498 mapwrite 0x229f96 thru 0x231e71 (0x7edc bytes) 000497[0] 1788651622.481631 mapread 0x39558b thru 0x397d53 (0x27c9 bytes) All operations completed A-OK! Chance of close/open is 1 in 50 Seed set to 1622 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000001[1] 1788651623.309059 mapwrite 0x6b2000 thru 0x6b6600 (0x4601 bytes) truncating to largest ever: 0x81accf 000003[1] 1788651623.430546 trunc from 0x6b6601 to 0x81acce (0x1646ce bytes) 000006[0] 1788651623.478539 read_OD 0x165000 thru 0x172fff (0xe000 bytes) 000007[1] 1788651623.502174 trunc from 0x81accf to 0x24bedb (0x5cedf3 bytes) truncating to largest ever: 0x98edf9 000010[1] 1788651623.593436 trunc from 0x24bedc to 0x98edf8 (0x742f1d bytes) 000012[1] 1788651623.628358 read 0x579000 thru 0x5819ec (0x89ed bytes) 000014[1] 1788651623.677378 write 0x13d000 thru 0x141336 (0x4337 bytes) 000016[1] 1788651623.696384 punch from 0x931d24 to 0x940985 (0xec62 bytes) 000018[1] 1788651623.700408 trunc from 0x98edf9 to 0x1829a7 (0x80c451 bytes) 000020[1] 1788651623.739765 trunc from 0x1829a8 to 0x1aad89 (0x283e2 bytes) 000021[1] 1788651623.775245 write 0x1aa000 thru 0x1ab336 (0x1337 bytes) 000022[0] 1788651623.780000 trunc from 0x1ab337 to 0x675faa (0x4cac74 bytes) 000023[0] 1788651623.830408 mapread 0x353000 thru 0x35d263 (0xa264 bytes) 000025[1] 1788651623.875992 punch from 0x6550b4 to 0x655f7b (0xec8 bytes) 000027[1] 1788651623.942570 mapread 0x05c000 thru 0x063a1f (0x7a20 bytes) 000028[0] 1788651623.965291 mapread 0x145000 thru 0x14c058 (0x7059 bytes) 000029[0] 1788651624.010950 punch from 0x0a132f to 0x0a75a1 (0x6273 bytes) 000030[0] 1788651624.033290 write_OD 0x991000 thru 0x99ffff (0xf000 bytes) 000031[0] 1788651624.108055 trunc from 0x9a0000 to 0x6572bf (0x348d40 bytes) 000032[1] 1788651624.175605 mapwrite 0x138000 thru 0x13eb91 (0x6b92 bytes) 000034[0] 1788651624.316287 read_OD 0x33a000 thru 0x33dfff (0x4000 bytes) 000036[0] 1788651624.361958 punch from 0x356043 to 0x359ca4 (0x3c62 bytes) 000038[1] 1788651624.399937 mapwrite 0x09b000 thru 0x09c414 (0x1415 bytes) 000039[1] 1788651624.515328 punch from 0x53b257 to 0x54abed (0xf997 bytes) 000040[1] 1788651624.538602 read 0x33d000 thru 0x34c501 (0xf502 bytes) 000041[0] 1788651624.567554 mapread 0x53f000 thru 0x54a6f5 (0xb6f6 bytes) truncating to largest ever: 0x9b6cfb 000042[1] 1788651624.618349 trunc from 0x6572c0 to 0x9b6cfa (0x35fa3b bytes) 000044[1] 1788651624.687165 read 0x805000 thru 0x8120d0 (0xd0d1 bytes) 000045[1] 1788651624.703287 trunc from 0x9b6cfb to 0x6d4025 (0x2e2cd5 bytes) 000046[1] 1788651624.735049 trunc from 0x6d4026 to 0x86fdb2 (0x19bd8d bytes) 000047[1] 1788651624.779259 read 0x71c000 thru 0x71d26e (0x126f bytes) 000049[0] 1788651624.807281 mapread 0x81b000 thru 0x81ea31 (0x3a32 bytes) 000052[1] 1788651624.899991 write 0x226000 thru 0x231067 (0xb068 bytes) 000053[1] 1788651624.956411 trunc from 0x86fdb3 to 0x7935b8 (0xdc7fa bytes) 000055[0] 1788651625.005958 mapwrite 0x12e000 thru 0x1302dd (0x22de bytes) 000056[0] 1788651625.165904 write_OD 0x110000 thru 0x114fff (0x5000 bytes) 000057[1] 1788651625.262607 trunc from 0x7935b9 to 0x5ffad3 (0x193ae5 bytes) 000059[0] 1788651625.350672 write_OD 0x140000 thru 0x145fff (0x6000 bytes) 000061[1] 1788651625.477536 trunc from 0x5ffad4 to 0x119ada (0x4e5ff9 bytes) 000062[1] 1788651625.554943 read 0x0cc000 thru 0x0d4d2e (0x8d2f bytes) 000063[0] 1788651625.617403 write_OD 0x8f7000 thru 0x8f9fff (0x3000 bytes) 000064[1] 1788651625.787159 mapread 0x1bc000 thru 0x1c8593 (0xc594 bytes) 000065[0] 1788651625.901536 read_OD 0x0a6000 thru 0x0b2fff (0xd000 bytes) 000071[0] 1788651625.920437 read_OD 0x032000 thru 0x03afff (0x9000 bytes) 000072[1] 1788651625.929681 trunc from 0x8fa000 to 0x71655e (0x1e3aa1 bytes) 000073[0] 1788651625.994645 write_OD 0x5d1000 thru 0x5d1fff (0x1000 bytes) 000075[1] 1788651626.052751 punch from 0x2d5c70 to 0x2e52a3 (0xf634 bytes) 000076[0] 1788651626.069395 mapread 0x32f000 thru 0x337748 (0x8749 bytes) 000079[0] 1788651626.114015 mapread 0x36e000 thru 0x378abc (0xaabd bytes) 000081[0] 1788651626.132574 punch from 0x21fe9b to 0x21ff89 (0xef bytes) 000085[0] 1788651626.186247 trunc from 0x71655f to 0x6cdbcd (0x48991 bytes) 000088[0] 1788651626.255472 mapread 0x112000 thru 0x11f586 (0xd587 bytes) 000089[1] 1788651626.310556 punch from 0x27dad8 to 0x28b286 (0xd7af bytes) 000091[1] 1788651626.330369 write 0x3c8000 thru 0x3c855e (0x55f bytes) 000093[1] 1788651626.357283 trunc from 0x6cdbce to 0x2981a8 (0x435a25 bytes) 000094[0] 1788651626.388429 trunc from 0x2981a9 to 0x8af5dd (0x617435 bytes) 000095[0] 1788651626.421453 read_OD 0x3ed000 thru 0x3effff (0x3000 bytes) 000098[0] 1788651626.445606 read_OD 0x6be000 thru 0x6c5fff (0x8000 bytes) 000099[1] 1788651626.454376 write 0x5a0000 thru 0x5a7b2b (0x7b2c bytes) 000100[1] 1788651626.479456 write 0x10b000 thru 0x115223 (0xa224 bytes) 000101[0] 1788651626.495900 read_OD 0x521000 thru 0x52afff (0xa000 bytes) 000103[0] 1788651626.530300 read_OD 0x40b000 thru 0x40cfff (0x2000 bytes) 000105[0] 1788651626.550052 mapread 0x461000 thru 0x46a44b (0x944c bytes) 000106[1] 1788651626.586621 trunc from 0x8af5de to 0x0e9090 (0x7c654d bytes) 000107[0] 1788651626.683672 mapread 0x0ad000 thru 0x0b083f (0x3840 bytes) 000109[1] 1788651626.768366 write 0x2f2000 thru 0x2f4b93 (0x2b94 bytes) 000112[0] 1788651626.857370 trunc from 0x2f4b94 to 0x805906 (0x510d73 bytes) 000116[1] 1788651626.987017 mapwrite 0x398000 thru 0x3990e5 (0x10e6 bytes) 000117[0] 1788651627.089694 mapwrite 0x469000 thru 0x46b778 (0x2779 bytes) 000118[0] 1788651627.225464 write_OD 0x9eb000 thru 0x9f1fff (0x7000 bytes) 000119[1] 1788651627.314234 read 0x7ae000 thru 0x7b3ee2 (0x5ee3 bytes) 000120[1] 1788651627.335083 mapwrite 0x59b000 thru 0x5aa786 (0xf787 bytes) 000121[0] 1788651627.563110 trunc from 0x9f2000 to 0x2efe61 (0x70219e bytes) 000122[0] 1788651627.679309 punch from 0x14c642 to 0x15013c (0x3afb bytes) 000127[0] 1788651627.788583 read_OD 0x265000 thru 0x26efff (0xa000 bytes) 000129[0] 1788651627.828360 trunc from 0x2efe62 to 0x533ad8 (0x243c77 bytes) 000131[0] 1788651627.878399 write_OD 0x391000 thru 0x399fff (0x9000 bytes) 000132[1] 1788651627.934620 read 0x395000 thru 0x39f13f (0xa140 bytes) 000134[0] 1788651627.947328 read_OD 0x42b000 thru 0x42bfff (0x1000 bytes) 000135[0] 1788651627.952188 trunc from 0x533ad9 to 0x25fa17 (0x2d40c1 bytes) 000136[1] 1788651627.975773 trunc from 0x25fa18 to 0x3ba1b4 (0x15a79d bytes) 000137[1] 1788651628.010136 trunc from 0x3ba1b5 to 0x559ac8 (0x19f914 bytes) 000140[0] 1788651628.102740 mapread 0x25b000 thru 0x2616f9 (0x66fa bytes) 000141[0] 1788651628.158525 write_OD 0x979000 thru 0x97bfff (0x3000 bytes) 000143[0] 1788651628.236527 read_OD 0x45e000 thru 0x46bfff (0xe000 bytes) 000144[1] 1788651628.250686 mapwrite 0x662000 thru 0x670412 (0xe413 bytes) 000145[1] 1788651628.334725 mapwrite 0x774000 thru 0x781484 (0xd485 bytes) 000147[1] 1788651628.429284 mapread 0x0b5000 thru 0x0be67a (0x967b bytes) 000149[1] 1788651628.467195 punch from 0x4351ae to 0x43f76c (0xa5bf bytes) 000152[0] 1788651628.493726 mapread 0x61d000 thru 0x622bba (0x5bbb bytes) 000153[0] 1788651628.544705 read_OD 0x0fb000 thru 0x100fff (0x6000 bytes) 000155[0] 1788651628.557594 mapread 0x2e7000 thru 0x2e7e5f (0xe60 bytes) 000156[0] 1788651628.563697 mapread 0x0b7000 thru 0x0c1fb1 (0xafb2 bytes) 000158[0] 1788651628.588028 mapread 0x027000 thru 0x036ce5 (0xfce6 bytes) 000160[1] 1788651628.610843 punch from 0x529269 to 0x5311f5 (0x7f8d bytes) 000161[0] 1788651628.628515 write_OD 0x665000 thru 0x66afff (0x6000 bytes) 000162[0] 1788651628.697281 write_OD 0x3b7000 thru 0x3c1fff (0xb000 bytes) 000163[0] 1788651628.775078 mapread 0x7cc000 thru 0x7cd292 (0x1293 bytes) 000165[1] 1788651628.793412 punch from 0x2c24fc to 0x2cf804 (0xd309 bytes) 000166[0] 1788651628.830850 mapread 0x0ef000 thru 0x0fa189 (0xb18a bytes) 000168[0] 1788651628.915027 mapread 0x559000 thru 0x5609fe (0x79ff bytes) 000169[0] 1788651628.948608 punch from 0x1be732 to 0x1ce148 (0xfa17 bytes) 000172[0] 1788651629.018152 trunc from 0x97c000 to 0x35c0fa (0x61ff05 bytes) 000174[1] 1788651629.044440 write 0x2d0000 thru 0x2d28d0 (0x28d1 bytes) 000176[0] 1788651629.090471 read_OD 0x22a000 thru 0x22efff (0x5000 bytes) 000179[0] 1788651629.141139 write_OD 0x293000 thru 0x29dfff (0xb000 bytes) 000180[1] 1788651629.225191 mapread 0x240000 thru 0x247162 (0x7163 bytes) 000181[0] 1788651629.258395 mapread 0x309000 thru 0x30eea1 (0x5ea2 bytes) 000183[1] 1788651629.341599 mapwrite 0x7ba000 thru 0x7bf1c9 (0x51ca bytes) 000184[1] 1788651629.493023 trunc from 0x7bf1ca to 0x233ff0 (0x58b1d9 bytes) 000185[1] 1788651629.517745 trunc from 0x233ff1 to 0x8b0b0d (0x67cb1d bytes) 000186[0] 1788651629.547608 punch from 0x46d655 to 0x47c07d (0xea29 bytes) 000193[0] 1788651629.638904 punch from 0x372663 to 0x37db04 (0xb4a2 bytes) 000194[0] 1788651629.656675 mapwrite 0x64b000 thru 0x64e20d (0x320e bytes) 000195[0] 1788651629.841433 read_OD 0x423000 thru 0x42ffff (0xd000 bytes) 000199[1] 1788651629.877685 write 0x16f000 thru 0x17e0b1 (0xf0b2 bytes) 000204[0] 1788651629.951534 trunc from 0x8b0b0e to 0x4c3d85 (0x3ecd88 bytes) 000205[0] 1788651630.001017 read_OD 0x14a000 thru 0x14ffff (0x6000 bytes) 000207[1] 1788651630.039204 punch from 0x03ef57 to 0x04a667 (0xb711 bytes) 000208[0] 1788651630.057926 mapread 0x1b0000 thru 0x1b1230 (0x1231 bytes) 000209[1] 1788651630.094448 write 0x765000 thru 0x76a22a (0x522b bytes) 000212[1] 1788651630.127363 punch from 0x16d123 to 0x16e3b3 (0x1291 bytes) 000213[0] 1788651630.148921 write_OD 0x24d000 thru 0x251fff (0x5000 bytes) 000214[1] 1788651630.250636 mapwrite 0x841000 thru 0x84581f (0x4820 bytes) 000215[0] 1788651630.359622 read_OD 0x545000 thru 0x54dfff (0x9000 bytes) 000219[0] 1788651630.392672 trunc from 0x845820 to 0x82df7c (0x178a3 bytes) 000220[1] 1788651630.446524 write 0x0a8000 thru 0x0a8bde (0xbdf bytes) 000221[1] 1788651630.485018 mapread 0x80d000 thru 0x81b5bb (0xe5bc bytes) 000223[0] 1788651630.529164 mapwrite 0x884000 thru 0x890848 (0xc849 bytes) 000225[0] 1788651630.698891 write_OD 0x255000 thru 0x255fff (0x1000 bytes) 000226[1] 1788651630.780344 punch from 0x59bd3b to 0x5a70fd (0xb3c3 bytes) 000227[1] 1788651630.805874 mapread 0x13b000 thru 0x1444c9 (0x94ca bytes) 000228[0] 1788651630.833641 mapread 0x0da000 thru 0x0dc8bd (0x28be bytes) 000229[0] 1788651630.895190 punch from 0x382e1b to 0x388da8 (0x5f8e bytes) 000230[0] 1788651630.928934 mapwrite 0x5b5000 thru 0x5bab05 (0x5b06 bytes) 000231[0] 1788651631.127766 write_OD 0x6d5000 thru 0x6e0fff (0xc000 bytes) 000232[0] 1788651631.259467 mapwrite 0x3c7000 thru 0x3c749a (0x49b bytes) 000233[1] 1788651631.390590 punch from 0x61e2d3 to 0x62541f (0x714d bytes) 000235[1] 1788651631.436588 write 0x03e000 thru 0x04a164 (0xc165 bytes) 000237[0] 1788651631.445315 write_OD 0x7f0000 thru 0x7fdfff (0xe000 bytes) 000238[1] 1788651631.508315 punch from 0x3b04f6 to 0x3b2968 (0x2473 bytes) 000239[1] 1788651631.548034 read 0x33d000 thru 0x34b92f (0xe930 bytes) 000240[1] 1788651631.560934 mapwrite 0x37e000 thru 0x386170 (0x8171 bytes) 000241[0] 1788651631.658441 punch from 0x525618 to 0x525819 (0x202 bytes) 000242[0] 1788651631.682661 read_OD 0x3a0000 thru 0x3a6fff (0x7000 bytes) 000243[1] 1788651631.706068 mapwrite 0x6aa000 thru 0x6b7663 (0xd664 bytes) 000245[0] 1788651631.867672 trunc from 0x890849 to 0x70b9a5 (0x184ea3 bytes) 000246[0] 1788651631.930447 mapwrite 0x696000 thru 0x6976ef (0x16f0 bytes) 000247[0] 1788651631.995223 trunc from 0x70b9a6 to 0x5b3a66 (0x157f3f bytes) 000249[1] 1788651632.036610 mapread 0x55b000 thru 0x55be8e (0xe8f bytes) 000250[1] 1788651632.060189 mapread 0x518000 thru 0x52376f (0xb770 bytes) 000252[0] 1788651632.088980 write_OD 0x088000 thru 0x095fff (0xe000 bytes) 000253[1] 1788651632.208515 read 0x3fb000 thru 0x402ef3 (0x7ef4 bytes) 000254[1] 1788651632.242072 punch from 0x55d5ed to 0x5681a0 (0xabb4 bytes) 000256[1] 1788651632.309536 mapwrite 0x119000 thru 0x11f906 (0x6907 bytes) 000258[0] 1788651632.400126 mapwrite 0x840000 thru 0x8451a3 (0x51a4 bytes) 000259[1] 1788651632.562051 read 0x4c0000 thru 0x4c975c (0x975d bytes) 000260[1] 1788651632.598168 read 0x3d6000 thru 0x3e1032 (0xb033 bytes) 000261[0] 1788651632.612702 trunc from 0x8451a4 to 0x680b36 (0x1c466d bytes) 000264[1] 1788651632.653749 trunc from 0x680b37 to 0x2ab453 (0x3d56e3 bytes) 000265[0] 1788651632.714774 mapread 0x06a000 thru 0x06b002 (0x1003 bytes) 000266[1] 1788651632.759642 read 0x047000 thru 0x047e85 (0xe86 bytes) 000268[0] 1788651632.812191 write_OD 0x76a000 thru 0x76bfff (0x2000 bytes) 000269[1] 1788651632.869353 write 0x2bd000 thru 0x2beeda (0x1edb bytes) 000270[0] 1788651632.962340 write_OD 0x22c000 thru 0x232fff (0x7000 bytes) 000273[1] 1788651633.024385 punch from 0x212882 to 0x21c260 (0x99df bytes) 000274[1] 1788651633.052000 mapwrite 0x99f000 thru 0x9a4580 (0x5581 bytes) 000275[1] 1788651633.203613 punch from 0x4d599f to 0x4e24a3 (0xcb05 bytes) 000278[1] 1788651633.227170 mapwrite 0x0a0000 thru 0x0a1b2a (0x1b2b bytes) 000281[1] 1788651633.336234 read 0x217000 thru 0x21c545 (0x5546 bytes) 000282[0] 1788651633.349253 read_OD 0x8da000 thru 0x8e2fff (0x9000 bytes) 000284[1] 1788651633.397648 mapread 0x27c000 thru 0x286213 (0xa214 bytes) 000285[0] 1788651633.436619 write_OD 0x77e000 thru 0x78bfff (0xe000 bytes) 000287[1] 1788651633.544691 read 0x854000 thru 0x8580d9 (0x40da bytes) 000289[0] 1788651633.566066 mapwrite 0x347000 thru 0x34a87b (0x387c bytes) 000292[1] 1788651633.691947 mapwrite 0x589000 thru 0x59183a (0x883b bytes) 000293[1] 1788651633.837645 mapwrite 0x443000 thru 0x443c21 (0xc22 bytes) 000295[1] 1788651633.940404 mapwrite 0x4f4000 thru 0x4f9fd5 (0x5fd6 bytes) 000296[0] 1788651634.021401 mapwrite 0x374000 thru 0x375fba (0x1fbb bytes) 000297[1] 1788651634.132649 punch from 0x54022a to 0x54d7bf (0xd596 bytes) 000298[1] 1788651634.172071 mapwrite 0x980000 thru 0x986fb3 (0x6fb4 bytes) 000299[1] 1788651634.238595 read 0x356000 thru 0x362874 (0xc875 bytes) 000302[0] 1788651634.281731 read_OD 0x2de000 thru 0x2e9fff (0xc000 bytes) 000304[1] 1788651634.321240 write 0x1af000 thru 0x1b4dd2 (0x5dd3 bytes) 000307[1] 1788651634.373210 mapread 0x3a4000 thru 0x3aedd5 (0xadd6 bytes) 000309[0] 1788651634.417998 trunc from 0x9a4581 to 0x134738 (0x86fe48 bytes) 000311[0] 1788651634.491666 punch from 0x0c6d90 to 0x0d2592 (0xb803 bytes) 000312[1] 1788651634.496480 write 0x3cf000 thru 0x3d32c4 (0x42c5 bytes) 000313[0] 1788651634.525747 trunc from 0x3d32c5 to 0x8be1c2 (0x4eaefe bytes) 000314[0] 1788651634.575729 punch from 0x67c65b to 0x67ebf8 (0x259e bytes) 000315[0] 1788651634.604741 punch from 0x6322d5 to 0x6388c8 (0x65f4 bytes) 000316[0] 1788651634.616538 trunc from 0x8be1c3 to 0x48d1ad (0x431015 bytes) 000318[0] 1788651634.685988 mapread 0x470000 thru 0x471ce9 (0x1cea bytes) 000320[1] 1788651634.741190 mapwrite 0x690000 thru 0x692b6d (0x2b6e bytes) 000323[1] 1788651634.895532 punch from 0x06b579 to 0x07146a (0x5ef2 bytes) 000324[0] 1788651634.905431 mapwrite 0x353000 thru 0x35fd4a (0xcd4b bytes) 000326[1] 1788651635.012907 read 0x41d000 thru 0x42301c (0x601d bytes) 000327[0] 1788651635.037238 punch from 0x504020 to 0x513cad (0xfc8e bytes) 000328[0] 1788651635.068276 mapread 0x319000 thru 0x323ec6 (0xaec7 bytes) 000329[0] 1788651635.095674 trunc from 0x692b6e to 0x6ff6f7 (0x6cb8a bytes) 000332[0] 1788651635.156431 mapread 0x62d000 thru 0x630cb7 (0x3cb8 bytes) 000334[1] 1788651635.228305 mapwrite 0x866000 thru 0x874d29 (0xed2a bytes) 000335[1] 1788651635.413519 write 0x0e4000 thru 0x0f0902 (0xc903 bytes) 000336[1] 1788651635.445723 punch from 0x186636 to 0x18c464 (0x5e2f bytes) 000337[1] 1788651635.464123 mapwrite 0x9a0000 thru 0x9ad881 (0xd882 bytes) 000340[1] 1788651635.658543 punch from 0x3f3e90 to 0x3f97b6 (0x5927 bytes) 000341[1] 1788651635.701612 punch from 0x322187 to 0x324809 (0x2683 bytes) 000342[0] 1788651635.726944 mapread 0x683000 thru 0x687a89 (0x4a8a bytes) 000344[0] 1788651635.763258 punch from 0x359616 to 0x35d996 (0x4381 bytes) 000345[0] 1788651635.780169 read_OD 0x224000 thru 0x22efff (0xb000 bytes) 000346[0] 1788651635.805619 mapwrite 0x285000 thru 0x290597 (0xb598 bytes) 000348[1] 1788651635.934727 read 0x91a000 thru 0x927c5f (0xdc60 bytes) 000352[1] 1788651635.972547 write 0x13f000 thru 0x14bd4c (0xcd4d bytes) 000353[1] 1788651636.019540 mapwrite 0x4fe000 thru 0x506f82 (0x8f83 bytes) 000356[0] 1788651636.112534 mapwrite 0x1e8000 thru 0x1ef84e (0x784f bytes) 000357[0] 1788651636.234486 read_OD 0x8f5000 thru 0x8fffff (0xb000 bytes) 000358[1] 1788651636.258620 punch from 0x2698b1 to 0x2718e3 (0x8033 bytes) 000359[0] 1788651636.279710 trunc from 0x9ad882 to 0x0b32e2 (0x8fa59f bytes) 000360[1] 1788651636.345347 read 0x064000 thru 0x071bb4 (0xdbb5 bytes) 000362[0] 1788651636.414229 write_OD 0x5be000 thru 0x5c0fff (0x3000 bytes) 000365[1] 1788651636.627893 write 0x265000 thru 0x2724f0 (0xd4f1 bytes) 000367[1] 1788651636.711918 trunc from 0x5c1000 to 0x331026 (0x28ffd9 bytes) 000369[1] 1788651636.789597 mapread 0x0e0000 thru 0x0e21e4 (0x21e5 bytes) 000371[0] 1788651636.887662 mapread 0x01f000 thru 0x02d8c4 (0xe8c5 bytes) 000374[0] 1788651636.968755 read_OD 0x0d8000 thru 0x0d8fff (0x1000 bytes) 000375[0] 1788651636.974239 mapread 0x02b000 thru 0x031bce (0x6bcf bytes) 000376[1] 1788651636.975349 mapwrite 0x3d4000 thru 0x3e1939 (0xd93a bytes) 000377[1] 1788651637.116211 write 0x455000 thru 0x462ff4 (0xdff5 bytes) 000379[0] 1788651637.179882 write_OD 0x0c7000 thru 0x0c7fff (0x1000 bytes) 000381[0] 1788651637.268445 read_OD 0x270000 thru 0x27cfff (0xd000 bytes) 000383[1] 1788651637.289658 mapwrite 0x0fd000 thru 0x10b665 (0xe666 bytes) 000384[0] 1788651637.493409 write_OD 0x1b2000 thru 0x1bafff (0x9000 bytes) 000385[0] 1788651637.617462 write_OD 0x611000 thru 0x61ffff (0xf000 bytes) 000386[0] 1788651637.705687 read_OD 0x3d5000 thru 0x3dffff (0xb000 bytes) 000387[0] 1788651637.731347 punch from 0x4831c9 to 0x48ddb9 (0xabf1 bytes) 000390[1] 1788651637.772953 punch from 0x1d6fd1 to 0x1e4429 (0xd459 bytes) 000393[0] 1788651637.850396 read_OD 0x163000 thru 0x16dfff (0xb000 bytes) truncating to largest ever: 0x9ca2af 000394[0] 1788651637.894551 trunc from 0x620000 to 0x9ca2ae (0x3aa2af bytes) 000395[0] 1788651637.944374 mapread 0x36f000 thru 0x376b6d (0x7b6e bytes) 000396[1] 1788651638.046730 read 0x611000 thru 0x613a6b (0x2a6c bytes) 000397[1] 1788651638.089216 read 0x677000 thru 0x67d7a2 (0x67a3 bytes) 000398[0] 1788651638.108765 trunc from 0x9ca2af to 0x8cde55 (0xfc459 bytes) 000401[1] 1788651638.201597 punch from 0x845caf to 0x852ada (0xce2c bytes) 000404[0] 1788651638.279450 punch from 0x734e79 to 0x73d8b1 (0x8a39 bytes) 000410[1] 1788651638.341158 trunc from 0x8cde56 to 0x1fcbeb (0x6d126a bytes) 000411[1] 1788651638.426508 write 0x998000 thru 0x9a48e8 (0xc8e9 bytes) 000412[1] 1788651638.471678 mapwrite 0x876000 thru 0x88431a (0xe31b bytes) 000414[1] 1788651638.608111 trunc from 0x9a48e9 to 0x194eb4 (0x80fa34 bytes) 000415[0] 1788651638.637623 mapwrite 0x1fb000 thru 0x204a81 (0x9a82 bytes) 000417[1] 1788651638.747393 write 0x988000 thru 0x990428 (0x8429 bytes) 000419[0] 1788651638.761644 trunc from 0x990429 to 0x941f60 (0x4e4c8 bytes) 000420[1] 1788651638.798787 mapread 0x43b000 thru 0x44ab6a (0xfb6b bytes) 000421[1] 1788651638.854742 mapwrite 0x657000 thru 0x664810 (0xd811 bytes) 000422[0] 1788651639.000749 trunc from 0x941f61 to 0x006e64 (0x93b0fc bytes) 000424[1] 1788651639.085341 punch from 0x003491 to 0x006e64 (0x39d4 bytes) 000427[0] 1788651639.190389 mapread 0x004000 thru 0x00638c (0x238d bytes) 000429[1] 1788651639.251521 trunc from 0x006e65 to 0x6b597e (0x6aeb1a bytes) 000431[0] 1788651639.303620 read_OD 0x53b000 thru 0x547fff (0xd000 bytes) 000432[0] 1788651639.340706 read_OD 0x425000 thru 0x42bfff (0x7000 bytes) 000435[0] 1788651639.363788 mapwrite 0x1da000 thru 0x1da9c8 (0x9c9 bytes) 000436[1] 1788651639.454543 read 0x64e000 thru 0x654b19 (0x6b1a bytes) 000437[1] 1788651639.491397 mapread 0x136000 thru 0x143d3e (0xdd3f bytes) 000439[0] 1788651639.536227 write_OD 0x8f2000 thru 0x8f9fff (0x8000 bytes) 000441[1] 1788651639.598312 punch from 0x6d63d4 to 0x6d83aa (0x1fd7 bytes) 000444[0] 1788651639.634707 mapread 0x29b000 thru 0x2a1e8d (0x6e8e bytes) 000446[0] 1788651639.696610 mapwrite 0x85a000 thru 0x85daa4 (0x3aa5 bytes) 000448[1] 1788651639.849947 mapread 0x84f000 thru 0x856f84 (0x7f85 bytes) 000449[0] 1788651639.901610 write_OD 0x0f1000 thru 0x0f6fff (0x6000 bytes) 000450[1] 1788651639.961581 mapwrite 0x71c000 thru 0x724edb (0x8edc bytes) 000454[0] 1788651640.171551 punch from 0x014ff3 to 0x018c6e (0x3c7c bytes) 000459[1] 1788651640.257294 punch from 0x85fba5 to 0x86e529 (0xe985 bytes) 000463[0] 1788651640.305440 mapwrite 0x447000 thru 0x448612 (0x1613 bytes) 000464[0] 1788651640.433598 mapread 0x114000 thru 0x11e212 (0xa213 bytes) 000467[0] 1788651640.487854 read_OD 0x676000 thru 0x67efff (0x9000 bytes) 000469[0] 1788651640.516444 write_OD 0x041000 thru 0x043fff (0x3000 bytes) 000473[1] 1788651640.577513 trunc from 0x8fa000 to 0x62b9b6 (0x2ce649 bytes) 000474[0] 1788651640.633645 trunc from 0x62b9b7 to 0x0553f8 (0x5d65be bytes) 000476[0] 1788651640.685315 punch from 0x04f6f6 to 0x0553f8 (0x5d03 bytes) 000477[1] 1788651640.705935 mapread 0x00c000 thru 0x01269d (0x669e bytes) 000480[1] 1788651640.761527 mapwrite 0x146000 thru 0x154395 (0xe396 bytes) 000481[1] 1788651640.880523 read 0x135000 thru 0x138157 (0x3158 bytes) 000482[1] 1788651640.906429 read 0x0d5000 thru 0x0d7e0b (0x2e0c bytes) 000483[0] 1788651640.950302 read_OD 0x03f000 thru 0x04dfff (0xf000 bytes) 000485[0] 1788651641.026094 mapread 0x149000 thru 0x14cbc9 (0x3bca bytes) 000486[1] 1788651641.064700 punch from 0x0ade92 to 0x0b7c3b (0x9daa bytes) 000488[0] 1788651641.105760 write_OD 0x648000 thru 0x650fff (0x9000 bytes) 000489[1] 1788651641.170917 trunc from 0x651000 to 0x370f65 (0x2e009a bytes) 000490[0] 1788651641.212615 trunc from 0x370f66 to 0x5f436a (0x283405 bytes) 000494[1] 1788651641.325560 trunc from 0x5f436b to 0x4040cb (0x1f029f bytes) 000495[1] 1788651641.410975 mapwrite 0x88b000 thru 0x88ce35 (0x1e36 bytes) 000496[1] 1788651641.655516 trunc from 0x88ce36 to 0x30c4ff (0x580936 bytes) 000497[1] 1788651641.698877 mapwrite 0x345000 thru 0x34c2cd (0x72ce bytes) 000500[1] 1788651641.905519 mapwrite 0x4ca000 thru 0x4d762d (0xd62e bytes) All operations completed A-OK! PASS 16a (54s) == sanityn test 16b: 500 iterations of dual-mount fsx at small size ========================================================== 19:40:48 (1788651648) keep default fallocate mode: 2 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.472226 s, 17.8 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.421174 s, 19.9 MB/s Chance of close/open is 1 in 50 Seed set to 1652 fd 0: /mnt/lustre/f16b.sanityn fd 1: /mnt/lustre2/f16b.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero length punch hole skipping zero size read 000003[0] 1788651653.322409 mapwrite 0x6c7223 thru 0x6c7f44 (0xd22 bytes) 000005[1] 1788651653.543449 punch from 0x06b177 to 0x06b317 (0x1a1 bytes) 000006[0] 1788651653.640930 mapwrite 0x2b13f0 thru 0x2b1768 (0x379 bytes) truncating to largest ever: 0x28c1b6 000007[1] 1788651653.844359 trunc from 0x6c7f45 to 0x28c1b5 (0x43bd8f bytes) 000008[0] 1788651653.887760 trunc from 0x28c1b6 to 0x21d5b6 (0x6ebff bytes) 000009[0] 1788651653.955501 mapread 0x0aa51e thru 0x0abbe7 (0x16ca bytes) 000011[1] 1788651654.031467 write 0x21f0c9 thru 0x22006d (0xfa5 bytes) 000012[0] 1788651654.095159 mapread 0x1d5da2 thru 0x1d6583 (0x7e2 bytes) 000013[0] 1788651654.171679 read 0x11c02f thru 0x11ce42 (0xe14 bytes) 000014[0] 1788651654.202283 mapread 0x21c169 thru 0x21e01a (0x1eb2 bytes) 000015[0] 1788651654.220780 mapwrite 0x4707a6 thru 0x47224e (0x1aa9 bytes) 000016[1] 1788651654.458055 mapread 0x3e79f8 thru 0x3e939b (0x19a4 bytes) 000019[1] 1788651654.605483 mapwrite 0x603407 thru 0x604b31 (0x172b bytes) 000020[1] 1788651654.815969 write 0x5c35ca thru 0x5c5331 (0x1d68 bytes) 000022[0] 1788651654.957305 read 0x213d0c thru 0x214bec (0xee1 bytes) truncating to largest ever: 0x76a89b 000025[1] 1788651655.043558 trunc from 0x604b32 to 0x76a89a (0x165d69 bytes) truncating to largest ever: 0x9ef14d 000026[1] 1788651655.095517 trunc from 0x76a89b to 0x9ef14c (0x2848b2 bytes) 000027[1] 1788651655.131604 write 0x79d9f7 thru 0x79f814 (0x1e1e bytes) 000028[0] 1788651655.145548 punch from 0x58f6ec to 0x58f7e7 (0xfc bytes) 000030[1] 1788651655.267223 read 0x59e349 thru 0x59f274 (0xf2c bytes) 000032[0] 1788651655.359135 write 0x6e0cc2 thru 0x6e2b21 (0x1e60 bytes) 000033[1] 1788651655.417280 trunc from 0x9ef14d to 0x052d65 (0x99c3e7 bytes) 000035[1] 1788651655.513401 punch from 0x034a24 to 0x034c2b (0x208 bytes) 000038[0] 1788651655.551043 write 0x0fa8d3 thru 0x0fc6d8 (0x1e06 bytes) 000040[0] 1788651655.598636 punch from 0x0b29ae to 0x0b3764 (0xdb7 bytes) 000041[1] 1788651655.628572 trunc from 0x0fc6d9 to 0x228793 (0x12c0bb bytes) 000042[0] 1788651655.683302 write 0x9ed0ec thru 0x9eec68 (0x1b7d bytes) 000043[1] 1788651655.726641 mapwrite 0x8ccb57 thru 0x8ceb30 (0x1fda bytes) 000044[0] 1788651655.921510 write 0x32bc22 thru 0x32d55b (0x193a bytes) 000046[1] 1788651656.027666 punch from 0x019f9f to 0x01b862 (0x18c4 bytes) 000048[1] 1788651656.057205 mapwrite 0x34fe05 thru 0x3505e9 (0x7e5 bytes) 000049[0] 1788651656.173155 write 0x941bc8 thru 0x9430f7 (0x1530 bytes) 000050[1] 1788651656.231370 trunc from 0x9eec69 to 0x0c4c6f (0x929ff9 bytes) 000051[1] 1788651656.297264 write 0x9449cd thru 0x9460e0 (0x1714 bytes) 000052[1] 1788651656.326582 mapwrite 0x4ea6f3 thru 0x4ea884 (0x192 bytes) 000054[0] 1788651656.438176 read 0x23931b thru 0x23a53b (0x1221 bytes) 000057[1] 1788651656.480783 punch from 0x42886f to 0x428a05 (0x197 bytes) 000058[1] 1788651656.487246 trunc from 0x9460e1 to 0x6f4c49 (0x251497 bytes) 000059[0] 1788651656.564595 trunc from 0x6f4c4a to 0x9af427 (0x2ba7de bytes) 000060[0] 1788651656.644455 mapread 0x456abc thru 0x457ff3 (0x1538 bytes) 000065[0] 1788651656.722279 read 0x44721b thru 0x448357 (0x113d bytes) 000066[0] 1788651656.749481 write 0x028406 thru 0x0299a0 (0x159b bytes) 000068[0] 1788651656.770597 mapwrite 0x8e10f0 thru 0x8e2969 (0x187a bytes) 000069[0] 1788651656.934253 punch from 0x414ccb to 0x416214 (0x154a bytes) 000073[1] 1788651656.966635 trunc from 0x9af428 to 0x921ab4 (0x8d973 bytes) 000075[1] 1788651657.072012 punch from 0x1e67de to 0x1e7d06 (0x1529 bytes) 000076[1] 1788651657.078571 mapread 0x5f7265 thru 0x5f80f7 (0xe93 bytes) 000077[0] 1788651657.117597 read 0x6d451c thru 0x6d49cc (0x4b1 bytes) 000079[1] 1788651657.173592 trunc from 0x921ab5 to 0x59f960 (0x382154 bytes) 000080[0] 1788651657.236612 mapwrite 0x354097 thru 0x35524e (0x11b8 bytes) 000081[1] 1788651657.408879 mapread 0x3f6077 thru 0x3f66a1 (0x62b bytes) 000082[0] 1788651657.466635 write 0x0f3ade thru 0x0f4197 (0x6ba bytes) 000084[0] 1788651657.573488 punch from 0x46d10c to 0x46dff0 (0xee5 bytes) 000087[1] 1788651657.611781 mapwrite 0x07a5d9 thru 0x07acff (0x727 bytes) 000089[0] 1788651657.752474 mapread 0x385cdb thru 0x3873ee (0x1714 bytes) 000092[0] 1788651657.814647 read 0x3743da thru 0x376219 (0x1e40 bytes) 000093[1] 1788651657.834290 mapread 0x4f5e33 thru 0x4f7846 (0x1a14 bytes) 000097[1] 1788651657.883732 mapread 0x55f33e thru 0x561066 (0x1d29 bytes) 000101[1] 1788651657.896147 trunc from 0x59f961 to 0x0f40ca (0x4ab896 bytes) 000102[1] 1788651657.965405 mapwrite 0x9a0cf2 thru 0x9a1ba8 (0xeb7 bytes) 000103[0] 1788651658.088435 mapwrite 0x149a42 thru 0x14acfd (0x12bc bytes) 000104[1] 1788651658.166395 write 0x55b296 thru 0x55c223 (0xf8e bytes) 000105[1] 1788651658.196152 mapread 0x1f9303 thru 0x1fb0ba (0x1db8 bytes) 000106[0] 1788651658.258348 read 0x5ca2c1 thru 0x5ca790 (0x4d0 bytes) 000107[1] 1788651658.282625 trunc from 0x9a1ba9 to 0x47db75 (0x524033 bytes) 000108[0] 1788651658.334329 trunc from 0x47db76 to 0x786685 (0x308b10 bytes) 000109[1] 1788651658.432533 trunc from 0x786686 to 0x239be7 (0x54ca9e bytes) 000111[1] 1788651658.529641 read 0x11a5b3 thru 0x11baf5 (0x1543 bytes) 000112[0] 1788651658.607234 mapwrite 0x28eaa4 thru 0x28ee7a (0x3d7 bytes) 000113[0] 1788651658.776653 punch from 0x0ba3fa to 0x0baec7 (0xace bytes) 000114[1] 1788651658.817648 read 0x1d9578 thru 0x1da8bf (0x1348 bytes) 000117[1] 1788651658.930180 mapread 0x0f5b5e thru 0x0f6b24 (0xfc7 bytes) 000118[0] 1788651658.953105 trunc from 0x28ee7b to 0x5c42d3 (0x335459 bytes) 000119[0] 1788651659.009653 trunc from 0x5c42d4 to 0x560681 (0x63c52 bytes) 000120[0] 1788651659.033498 read 0x40475d thru 0x405408 (0xcac bytes) 000121[1] 1788651659.042156 write 0x0e29d1 thru 0x0e3fcc (0x15fc bytes) 000122[0] 1788651659.117099 trunc from 0x560682 to 0x352dda (0x20d8a7 bytes) 000124[0] 1788651659.191682 mapread 0x165a95 thru 0x166eb5 (0x1421 bytes) 000127[1] 1788651659.222267 trunc from 0x352ddb to 0x90c0f5 (0x5b931b bytes) 000129[1] 1788651659.369684 trunc from 0x90c0f6 to 0x33b157 (0x5d0f9e bytes) 000131[1] 1788651659.487461 punch from 0x25084c to 0x250e75 (0x62a bytes) 000132[1] 1788651659.497094 trunc from 0x33b158 to 0x8016f6 (0x4c659f bytes) 000133[0] 1788651659.539437 read 0x5d632f thru 0x5d6448 (0x11a bytes) 000134[0] 1788651659.562624 read 0x55e1c7 thru 0x560101 (0x1f3b bytes) 000135[0] 1788651659.571251 punch from 0x4cfe43 to 0x4d1034 (0x11f2 bytes) 000138[1] 1788651659.669676 read 0x35f5c0 thru 0x360e62 (0x18a3 bytes) 000139[1] 1788651659.706645 mapread 0x5457c9 thru 0x546b82 (0x13ba bytes) 000141[0] 1788651659.745043 read 0x4beaaa thru 0x4c094c (0x1ea3 bytes) 000142[0] 1788651659.775615 trunc from 0x8016f7 to 0x7b7468 (0x4a28e bytes) 000144[1] 1788651659.840330 mapwrite 0x13f3f9 thru 0x13fc6a (0x872 bytes) 000145[1] 1788651659.970497 mapwrite 0x560581 thru 0x560b49 (0x5c9 bytes) 000146[0] 1788651660.070054 read 0x73c7ce thru 0x73d79a (0xfcd bytes) 000148[1] 1788651660.140773 trunc from 0x7b7469 to 0x9a3079 (0x1ebc11 bytes) 000149[1] 1788651660.193651 punch from 0x200744 to 0x2026eb (0x1fa8 bytes) 000153[0] 1788651660.261374 mapread 0x6e4fd7 thru 0x6e6a13 (0x1a3d bytes) 000154[0] 1788651660.338394 punch from 0x5b4950 to 0x5b68b5 (0x1f66 bytes) 000155[1] 1788651660.367671 read 0x483734 thru 0x483877 (0x144 bytes) 000156[0] 1788651660.399890 read 0x501c3f thru 0x5024c3 (0x885 bytes) 000157[1] 1788651660.419552 mapread 0x8cf64d thru 0x8d1407 (0x1dbb bytes) 000159[0] 1788651660.460219 write 0x8b8833 thru 0x8b945c (0xc2a bytes) 000161[1] 1788651660.527130 write 0x4a839b thru 0x4a8ef2 (0xb58 bytes) 000165[1] 1788651660.661792 mapread 0x345103 thru 0x345848 (0x746 bytes) 000169[1] 1788651660.726787 trunc from 0x9a307a to 0x0fec11 (0x8a4468 bytes) 000170[1] 1788651660.769050 mapwrite 0x3c62ae thru 0x3c64f6 (0x249 bytes) 000171[1] 1788651660.828198 write 0x34452b thru 0x346368 (0x1e3e bytes) 000172[0] 1788651660.878043 mapread 0x18b47d thru 0x18b872 (0x3f6 bytes) 000173[1] 1788651660.924345 write 0x08ca2c thru 0x08d246 (0x81b bytes) 000174[1] 1788651661.004758 read 0x36fbfc thru 0x371103 (0x1508 bytes) 000175[0] 1788651661.091215 write 0x65adda thru 0x65bd13 (0xf3a bytes) 000176[1] 1788651661.122638 write 0x11e805 thru 0x1204ac (0x1ca8 bytes) 000177[0] 1788651661.136599 mapread 0x063ced thru 0x064ef1 (0x1205 bytes) 000178[0] 1788651661.211779 read 0x5adc53 thru 0x5aec0a (0xfb8 bytes) 000179[1] 1788651661.246174 trunc from 0x65bd14 to 0x043709 (0x61860a bytes) 000180[1] 1788651661.329163 trunc from 0x04370a to 0x36c936 (0x32922d bytes) 000182[0] 1788651661.407842 write 0x0a0810 thru 0x0a1648 (0xe39 bytes) 000183[1] 1788651661.478780 punch from 0x0498e7 to 0x049b86 (0x2a0 bytes) 000185[1] 1788651661.555086 punch from 0x180329 to 0x1808e8 (0x5c0 bytes) 000186[0] 1788651661.609707 trunc from 0x36c937 to 0x843540 (0x4d6c0a bytes) 000187[0] 1788651661.695452 mapwrite 0x9b9954 thru 0x9ba107 (0x7b4 bytes) 000188[1] 1788651661.879369 punch from 0x29b353 to 0x29c65d (0x130b bytes) 000189[0] 1788651661.924667 mapwrite 0x6214ba thru 0x62231f (0xe66 bytes) 000190[0] 1788651662.038686 mapwrite 0x6574ab thru 0x659130 (0x1c86 bytes) 000191[1] 1788651662.126417 trunc from 0x9ba108 to 0x951942 (0x687c5 bytes) 000192[1] 1788651662.209218 read 0x9514ca thru 0x951809 (0x340 bytes) 000193[1] 1788651662.218351 read 0x3a1c0a thru 0x3a2f04 (0x12fb bytes) 000195[1] 1788651662.279679 punch from 0x6ceb63 to 0x6cf5c6 (0xa64 bytes) 000197[0] 1788651662.340770 mapread 0x2cd403 thru 0x2ce818 (0x1416 bytes) 000198[0] 1788651662.411870 punch from 0x81e240 to 0x81f798 (0x1559 bytes) 000201[0] 1788651662.517441 mapwrite 0x51548b thru 0x515537 (0xad bytes) 000202[1] 1788651662.656734 mapread 0x63f22f thru 0x63f569 (0x33b bytes) 000203[0] 1788651662.711178 trunc from 0x951943 to 0x627453 (0x32a4ef bytes) 000204[1] 1788651662.756646 mapwrite 0x931de2 thru 0x932f3d (0x115c bytes) 000205[0] 1788651662.953496 read 0x82550c thru 0x825e4f (0x944 bytes) 000206[1] 1788651662.984124 read 0x7eb536 thru 0x7ed202 (0x1ccd bytes) 000207[0] 1788651662.989609 write 0x66f81f thru 0x670cc4 (0x14a6 bytes) 000210[1] 1788651663.049455 trunc from 0x932f3e to 0x311bdd (0x621360 bytes) 000211[1] 1788651663.135717 trunc from 0x311bde to 0x9546f2 (0x642b15 bytes) 000212[1] 1788651663.231249 read 0x8524a3 thru 0x8526fb (0x259 bytes) 000213[1] 1788651663.305938 trunc from 0x9546f3 to 0x2a4dfe (0x6af8f4 bytes) 000214[0] 1788651663.392592 read 0x17dccf thru 0x17de6f (0x1a1 bytes) 000215[0] 1788651663.542358 write 0x008f43 thru 0x0093f4 (0x4b2 bytes) 000218[1] 1788651663.646703 mapread 0x2826bc thru 0x282edc (0x821 bytes) 000219[0] 1788651663.701095 read 0x01da92 thru 0x01f113 (0x1682 bytes) 000220[1] 1788651663.770618 trunc from 0x2a4dff to 0x586865 (0x2e1a67 bytes) 000221[0] 1788651663.826100 write 0x18dcce thru 0x18e551 (0x884 bytes) 000222[1] 1788651663.904142 trunc from 0x586866 to 0x050986 (0x535edf bytes) 000223[1] 1788651663.971523 mapwrite 0x91c541 thru 0x91d542 (0x1002 bytes) 000224[1] 1788651664.076641 mapwrite 0x644bdd thru 0x645d3c (0x1160 bytes) 000225[0] 1788651664.189230 mapread 0x3cc572 thru 0x3ccaeb (0x57a bytes) 000226[0] 1788651664.251898 punch from 0x79f1f4 to 0x7a0d51 (0x1b5e bytes) 000231[1] 1788651664.343233 punch from 0x6848d1 to 0x6864c9 (0x1bf9 bytes) 000232[1] 1788651664.360350 mapwrite 0x63ffde thru 0x640829 (0x84c bytes) 000233[0] 1788651664.433200 trunc from 0x91d543 to 0x485c23 (0x49791f bytes) 000234[1] 1788651664.539353 trunc from 0x485c24 to 0x03225d (0x4539c6 bytes) 000237[1] 1788651664.669141 write 0x8955d6 thru 0x8964b9 (0xee4 bytes) 000239[0] 1788651664.715053 mapread 0x70efe8 thru 0x710adb (0x1af4 bytes) 000240[0] 1788651664.817217 trunc from 0x8964ba to 0x2d58f8 (0x5c0bc1 bytes) 000242[0] 1788651664.960307 punch from 0x2a0587 to 0x2a23fb (0x1e75 bytes) 000243[0] 1788651664.979255 mapread 0x08e1d9 thru 0x08fc9d (0x1ac5 bytes) 000245[1] 1788651665.003361 write 0x5d325c thru 0x5d4720 (0x14c5 bytes) 000247[1] 1788651665.074561 mapread 0x06f220 thru 0x070754 (0x1535 bytes) 000248[0] 1788651665.103966 mapwrite 0x9d691e thru 0x9d75b8 (0xc9b bytes) 000249[1] 1788651665.397398 trunc from 0x9d75b9 to 0x0e08b1 (0x8f6d07 bytes) 000250[1] 1788651665.466070 mapwrite 0x1942cb thru 0x194bb5 (0x8eb bytes) 000251[0] 1788651665.609701 mapread 0x006ee9 thru 0x008a0d (0x1b25 bytes) 000252[0] 1788651665.691056 mapwrite 0x26c43b thru 0x26c462 (0x28 bytes) 000254[0] 1788651665.826060 punch from 0x1a7c2e to 0x1a83c8 (0x79b bytes) 000256[1] 1788651665.863444 punch from 0x1b0b09 to 0x1b277f (0x1c77 bytes) 000257[1] 1788651665.906237 trunc from 0x26c463 to 0x33eefe (0xd2a9c bytes) 000262[1] 1788651666.013172 punch from 0x198c9b to 0x19969c (0xa02 bytes) 000264[1] 1788651666.060128 mapread 0x24d913 thru 0x24f03a (0x1728 bytes) 000265[0] 1788651666.083587 punch from 0x11b29a to 0x11bdbe (0xb25 bytes) 000266[1] 1788651666.122574 trunc from 0x33eeff to 0x584471 (0x245573 bytes) 000267[1] 1788651666.170692 punch from 0x293ff1 to 0x295460 (0x1470 bytes) 000268[0] 1788651666.184612 trunc from 0x584472 to 0x9bd62f (0x4391be bytes) 000270[0] 1788651666.283594 trunc from 0x9bd630 to 0x13950a (0x884125 bytes) 000271[0] 1788651666.366595 trunc from 0x13950b to 0x35465e (0x21b154 bytes) 000272[1] 1788651666.427771 mapread 0x18ba05 thru 0x18cf14 (0x1510 bytes) 000273[0] 1788651666.473413 mapread 0x049d8c thru 0x04bad0 (0x1d45 bytes) 000275[1] 1788651666.502544 mapread 0x2227b0 thru 0x22362f (0xe80 bytes) 000277[0] 1788651666.549461 mapread 0x0f0891 thru 0x0f1f09 (0x1679 bytes) 000278[0] 1788651666.561865 trunc from 0x35465f to 0x351f13 (0x274b bytes) 000279[1] 1788651666.629533 read 0x195b0f thru 0x19728f (0x1781 bytes) 000280[1] 1788651666.751077 read 0x1e9651 thru 0x1e9e18 (0x7c8 bytes) 000281[0] 1788651666.763569 mapread 0x2a7740 thru 0x2a8311 (0xbd2 bytes) 000282[1] 1788651666.793976 write 0x449413 thru 0x44a1c4 (0xdb2 bytes) 000283[1] 1788651666.808423 trunc from 0x44a1c5 to 0x79734d (0x34d189 bytes) 000284[1] 1788651666.851951 mapwrite 0x014879 thru 0x015111 (0x899 bytes) 000286[0] 1788651666.979477 read 0x5ea774 thru 0x5ebefe (0x178b bytes) 000288[1] 1788651667.038952 mapread 0x558568 thru 0x559ca2 (0x173b bytes) 000289[1] 1788651667.163186 write 0x11a8f4 thru 0x11b275 (0x982 bytes) 000293[0] 1788651667.255420 trunc from 0x79734e to 0x0f2a97 (0x6a48b6 bytes) 000295[1] 1788651667.408616 write 0x72215b thru 0x72411d (0x1fc3 bytes) 000296[1] 1788651667.463518 mapread 0x69fa47 thru 0x6a0899 (0xe53 bytes) 000298[0] 1788651667.533533 mapread 0x2ec8ff thru 0x2ece30 (0x532 bytes) 000301[1] 1788651667.684223 read 0x29663c thru 0x297490 (0xe55 bytes) 000302[0] 1788651667.723527 mapread 0x530bf9 thru 0x531251 (0x659 bytes) 000303[1] 1788651667.782480 trunc from 0x72411e to 0x4107f5 (0x313928 bytes) 000306[1] 1788651667.836691 mapwrite 0x60ab28 thru 0x60b9f7 (0xed0 bytes) 000307[1] 1788651667.914399 read 0x442562 thru 0x443fd9 (0x1a78 bytes) 000308[1] 1788651667.922503 write 0x373fd5 thru 0x3745f4 (0x620 bytes) 000312[0] 1788651667.960745 mapread 0x0e3fdb thru 0x0e46ee (0x714 bytes) 000313[1] 1788651668.121556 punch from 0x164dd5 to 0x16519f (0x3cb bytes) 000314[1] 1788651668.184709 mapread 0x149029 thru 0x14a292 (0x126a bytes) 000315[1] 1788651668.205479 read 0x04f8d4 thru 0x051418 (0x1b45 bytes) 000316[1] 1788651668.239093 trunc from 0x60b9f8 to 0x0909ac (0x57b04b bytes) 000317[1] 1788651668.352722 write 0x4dd8e2 thru 0x4df8c1 (0x1fe0 bytes) 000318[0] 1788651668.378586 write 0x1ddeaa thru 0x1de2df (0x436 bytes) 000319[1] 1788651668.441485 trunc from 0x4df8c2 to 0x41385a (0xcc067 bytes) 000320[0] 1788651668.530357 read 0x0fa175 thru 0x0fade7 (0xc73 bytes) 000321[0] 1788651668.579445 mapread 0x3f51bc thru 0x3f548e (0x2d3 bytes) 000322[1] 1788651668.603620 write 0x0e460b thru 0x0e50fb (0xaf1 bytes) 000323[0] 1788651668.695208 mapread 0x0a748e thru 0x0a86eb (0x125e bytes) 000325[0] 1788651668.840519 punch from 0x0636ed to 0x064315 (0xc29 bytes) 000327[1] 1788651668.888279 trunc from 0x41385b to 0x174022 (0x29f838 bytes) 000328[1] 1788651668.947323 punch from 0x095b54 to 0x097274 (0x1721 bytes) 000330[0] 1788651669.014346 mapread 0x0e0792 thru 0x0e14ff (0xd6e bytes) 000332[0] 1788651669.057682 mapread 0x051729 thru 0x05296a (0x1242 bytes) 000333[1] 1788651669.085751 trunc from 0x174023 to 0x775b5c (0x601b3a bytes) 000335[0] 1788651669.121335 read 0x4c7519 thru 0x4c7cf8 (0x7e0 bytes) 000337[0] 1788651669.160873 read 0x1ff635 thru 0x1ffb56 (0x522 bytes) 000338[0] 1788651669.187078 mapwrite 0x2d66bb thru 0x2d7673 (0xfb9 bytes) 000341[1] 1788651669.340627 trunc from 0x775b5d to 0x7fb4fb (0x8599f bytes) 000346[1] 1788651669.559366 write 0x2c9e4c thru 0x2cacdc (0xe91 bytes) 000347[1] 1788651669.578414 mapwrite 0x110352 thru 0x1115ce (0x127d bytes) 000349[0] 1788651669.750692 trunc from 0x7fb4fc to 0x86b58c (0x70091 bytes) 000353[1] 1788651669.865776 write 0x2be5b2 thru 0x2c0473 (0x1ec2 bytes) 000355[1] 1788651669.921290 punch from 0x7cc34c to 0x7cc50d (0x1c2 bytes) 000357[0] 1788651670.008988 trunc from 0x86b58d to 0x1517b5 (0x719dd7 bytes) 000358[1] 1788651670.126703 trunc from 0x1517b6 to 0x946dfe (0x7f5649 bytes) 000359[0] 1788651670.200341 write 0x575020 thru 0x575c8c (0xc6d bytes) 000360[1] 1788651670.243513 read 0x262345 thru 0x263ff3 (0x1caf bytes) 000361[0] 1788651670.270079 read 0x7dc73f thru 0x7dd380 (0xc42 bytes) 000362[0] 1788651670.293898 mapread 0x7049f4 thru 0x706364 (0x1971 bytes) 000363[0] 1788651670.359732 mapread 0x1e318e thru 0x1e46b9 (0x152c bytes) 000364[0] 1788651670.398751 punch from 0x2cb75e to 0x2cbfb4 (0x857 bytes) 000365[0] 1788651670.425996 punch from 0x5d1a49 to 0x5d27e0 (0xd98 bytes) 000367[0] 1788651670.447577 mapwrite 0x7a3546 thru 0x7a3ed5 (0x990 bytes) 000368[0] 1788651670.528186 mapread 0x2b3a50 thru 0x2b49d9 (0xf8a bytes) 000369[0] 1788651670.546391 mapread 0x759333 thru 0x759765 (0x433 bytes) 000370[1] 1788651670.602358 mapwrite 0x4f8507 thru 0x4f854e (0x48 bytes) 000371[0] 1788651670.719108 read 0x3456a7 thru 0x346bfb (0x1555 bytes) 000372[0] 1788651670.759990 read 0x8c7b60 thru 0x8c894c (0xded bytes) 000373[0] 1788651670.784994 write 0x4c736f thru 0x4c8299 (0xf2b bytes) 000374[0] 1788651670.810178 write 0x529e4b thru 0x52a914 (0xaca bytes) 000375[1] 1788651670.820526 trunc from 0x946dff to 0x2311bd (0x715c41 bytes) 000377[1] 1788651670.927863 read 0x0f770b thru 0x0f7862 (0x158 bytes) 000378[1] 1788651671.018343 write 0x38425a thru 0x385c61 (0x1a08 bytes) 000379[1] 1788651671.046517 punch from 0x1887d0 to 0x18a626 (0x1e57 bytes) 000380[0] 1788651671.057325 write 0x787653 thru 0x7891e1 (0x1b8f bytes) 000381[1] 1788651671.090055 read 0x44ca8e thru 0x44cc31 (0x1a4 bytes) 000382[1] 1788651671.145133 mapread 0x21fb8d thru 0x220af1 (0xf65 bytes) 000384[1] 1788651671.176588 punch from 0x3e63f5 to 0x3e69ff (0x60b bytes) 000385[1] 1788651671.191517 read 0x60a2af thru 0x60a4ff (0x251 bytes) 000387[1] 1788651671.232000 mapwrite 0x66cdc6 thru 0x66e02f (0x126a bytes) 000388[1] 1788651671.374152 punch from 0x257331 to 0x258688 (0x1358 bytes) 000389[1] 1788651671.443475 mapread 0x43c99f thru 0x43dbdb (0x123d bytes) 000392[0] 1788651671.509483 write 0x9f61a6 thru 0x9f68c6 (0x721 bytes) 000393[1] 1788651671.550057 mapread 0x7fc900 thru 0x7fdacd (0x11ce bytes) 000395[0] 1788651671.636259 mapwrite 0x01d0f6 thru 0x01d828 (0x733 bytes) 000396[0] 1788651671.782703 mapwrite 0x9ce68a thru 0x9ceffb (0x972 bytes) 000397[1] 1788651671.879219 punch from 0x1a0407 to 0x1a0f79 (0xb73 bytes) 000398[1] 1788651671.920875 mapwrite 0x97b6f0 thru 0x97d46a (0x1d7b bytes) 000400[1] 1788651672.082741 mapread 0x9d3322 thru 0x9d39e7 (0x6c6 bytes) 000401[1] 1788651672.124084 read 0x374bdd thru 0x37529f (0x6c3 bytes) 000404[1] 1788651672.150278 mapread 0x5a2906 thru 0x5a3c9a (0x1395 bytes) 000408[1] 1788651672.250605 mapread 0x643bb0 thru 0x644c7e (0x10cf bytes) 000409[1] 1788651672.268775 mapwrite 0x79234b thru 0x792754 (0x40a bytes) 000410[0] 1788651672.373143 read 0x053702 thru 0x053ea9 (0x7a8 bytes) 000412[1] 1788651672.456449 read 0x06eb61 thru 0x06f3d4 (0x874 bytes) 000414[1] 1788651672.501525 mapwrite 0x8a31b8 thru 0x8a43ba (0x1203 bytes) 000415[0] 1788651672.608988 mapwrite 0x150bf8 thru 0x151c33 (0x103c bytes) 000417[1] 1788651672.845127 mapread 0x1322ed thru 0x132807 (0x51b bytes) 000419[0] 1788651672.958421 read 0x68a30f thru 0x68c0f1 (0x1de3 bytes) 000420[0] 1788651672.999734 punch from 0x7a3cca to 0x7a3fbd (0x2f4 bytes) 000422[1] 1788651673.061363 write 0x592844 thru 0x594198 (0x1955 bytes) 000423[0] 1788651673.117659 trunc from 0x9f68c7 to 0x9e3de3 (0x12ae3 bytes) 000424[1] 1788651673.167674 mapread 0x2d2df0 thru 0x2d4bc7 (0x1dd8 bytes) 000425[1] 1788651673.215384 write 0x4f0955 thru 0x4f1800 (0xeac bytes) 000426[1] 1788651673.281155 read 0x9adb4f thru 0x9af8a3 (0x1d55 bytes) 000429[0] 1788651673.379284 write 0x2f2642 thru 0x2f2f7f (0x93e bytes) 000430[1] 1788651673.444332 trunc from 0x9e3de4 to 0x2bd9b0 (0x726433 bytes) 000431[0] 1788651673.550720 punch from 0x02064e to 0x020a38 (0x3eb bytes) 000432[0] 1788651673.582859 write 0x0d2f08 thru 0x0d44ea (0x15e3 bytes) 000433[1] 1788651673.640083 mapwrite 0x1219aa thru 0x121e2b (0x482 bytes) 000435[1] 1788651673.801564 trunc from 0x2bd9b1 to 0x5449f6 (0x287046 bytes) 000437[0] 1788651673.830902 write 0x6727be thru 0x672a0d (0x250 bytes) 000439[0] 1788651673.887279 punch from 0x0e2810 to 0x0e2a9d (0x28e bytes) 000440[0] 1788651673.937303 write 0x877137 thru 0x878e6b (0x1d35 bytes) 000441[0] 1788651673.944771 mapwrite 0x55105b thru 0x5527d5 (0x177b bytes) 000442[1] 1788651674.014482 mapread 0x41dfb5 thru 0x41ee43 (0xe8f bytes) 000443[0] 1788651674.071273 read 0x80647c thru 0x806567 (0xec bytes) 000444[1] 1788651674.081034 read 0x070ee2 thru 0x07262d (0x174c bytes) 000448[1] 1788651674.107150 read 0x78c596 thru 0x78c961 (0x3cc bytes) 000450[1] 1788651674.112582 trunc from 0x878e6c to 0x033efe (0x844f6d bytes) 000451[1] 1788651674.177622 write 0x3ef68c thru 0x3f09c3 (0x1338 bytes) 000453[0] 1788651674.237300 punch from 0x37ce2a to 0x37da8c (0xc63 bytes) 000454[1] 1788651674.281733 write 0x3c961b thru 0x3c9aac (0x492 bytes) 000455[0] 1788651674.331531 write 0x8a737f thru 0x8a80d6 (0xd58 bytes) 000456[0] 1788651674.372999 read 0x56f800 thru 0x57021e (0xa1f bytes) 000458[1] 1788651674.435784 punch from 0x6096fc to 0x60a7f8 (0x10fd bytes) 000460[0] 1788651674.506569 mapread 0x5ff30c thru 0x6008cc (0x15c1 bytes) 000462[1] 1788651674.539118 write 0x0b089d thru 0x0b20b9 (0x181d bytes) 000463[1] 1788651674.583509 trunc from 0x8a80d7 to 0x92d84f (0x85779 bytes) 000464[1] 1788651674.621612 punch from 0x464367 to 0x466333 (0x1fcd bytes) 000465[1] 1788651674.626044 mapwrite 0x7a86ca thru 0x7a948b (0xdc2 bytes) 000466[0] 1788651674.713044 read 0x685ad6 thru 0x6867eb (0xd16 bytes) 000469[0] 1788651674.772309 trunc from 0x92d850 to 0x12f91d (0x7fdf32 bytes) 000471[0] 1788651674.858609 mapread 0x0df638 thru 0x0e0894 (0x125d bytes) 000472[0] 1788651674.894631 write 0x9277e6 thru 0x92836c (0xb87 bytes) 000473[1] 1788651674.899537 punch from 0x079e22 to 0x07a84c (0xa2b bytes) 000474[1] 1788651674.946082 punch from 0x77a548 to 0x77a828 (0x2e1 bytes) 000475[1] 1788651674.961170 trunc from 0x92836d to 0x400c77 (0x5276f5 bytes) 000476[0] 1788651675.027712 punch from 0x0006a0 to 0x0020e0 (0x1a41 bytes) 000477[0] 1788651675.077191 punch from 0x264ce5 to 0x265eaa (0x11c6 bytes) 000478[1] 1788651675.128562 mapwrite 0x7e22f7 thru 0x7e2dc7 (0xad1 bytes) 000479[0] 1788651675.235354 punch from 0x1cbfa4 to 0x1cc69f (0x6fc bytes) 000480[1] 1788651675.280507 write 0x0e8850 thru 0x0e9f9f (0x1750 bytes) 000481[1] 1788651675.307010 mapwrite 0x956b21 thru 0x957161 (0x641 bytes) 000482[1] 1788651675.425231 mapwrite 0x956265 thru 0x957673 (0x140f bytes) 000483[0] 1788651675.533640 trunc from 0x957674 to 0x6090bb (0x34e5b8 bytes) 000484[1] 1788651675.599487 punch from 0x5169de to 0x517494 (0xab7 bytes) 000487[0] 1788651675.663273 trunc from 0x6090bc to 0x739c0a (0x130b4f bytes) 000489[0] 1788651675.701992 punch from 0x6c7574 to 0x6c772b (0x1b8 bytes) 000490[0] 1788651675.706318 punch from 0x300124 to 0x3011b7 (0x1094 bytes) 000491[0] 1788651675.736393 write 0x3eca02 thru 0x3ed240 (0x83f bytes) 000492[1] 1788651675.747302 read 0x619dff thru 0x61bc72 (0x1e74 bytes) 000494[0] 1788651675.782255 trunc from 0x739c0b to 0x41d17e (0x31ca8c bytes) 000496[0] 1788651675.815278 mapwrite 0x2329a4 thru 0x233c4b (0x12a8 bytes) 000497[0] 1788651675.864570 write 0x85ff0c thru 0x86145c (0x1551 bytes) 000498[0] 1788651675.891214 trunc from 0x86145d to 0x55fae3 (0x301979 bytes) 000499[1] 1788651675.914436 mapwrite 0x326615 thru 0x327061 (0xa4d bytes) 000500[1] 1788651676.045060 read 0x50830a thru 0x50a303 (0x1ffa bytes) All operations completed A-OK! PASS 16b (33s) == sanityn test 16c: verify data consistency on ldiskfs with cache disabled (b=17397) ========================================================== 19:41:21 (1788651681) SKIP: sanityn test_16c dio on ldiskfs only SKIP 16c (4s) == sanityn test 16d: Verify DIO and buffer IO with two clients ========================================================== 19:41:25 (1788651685) 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.400298 s, 21.0 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.311997 s, 26.9 MB/s ldlm.namespaces.MGC192.168.203.136@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff88adc7b35000.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff88add8dd7800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff88adc7b35000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff88add8dd7800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff88adc7b35000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff88add8dd7800.lru_size=clear 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 13.1598 s, 8.0 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.42796 s, 23.7 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 2.77324 s, 37.8 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.08587 s, 14.8 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.35103 s, 12.6 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 3.48402 s, 30.1 MB/s PASS 16d (54s) == sanityn test 16e: Verify size consistency for O_DIRECT write ========================================================== 19:42:19 (1788651739) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.37888 s, 27.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153742 s, 6.8 MB/s PASS 16e (7s) == sanityn test 16f: rw sequential consistency vs drop_caches ========================================================== 19:42:27 (1788651747) TIMEOUT PASS 16f (29s) == sanityn test 16g: mmap rw sequential consistency vs drop_caches ========================================================== 19:42:55 (1788651775) TIMEOUT PASS 16g (29s) == sanityn test 16h: mmap read after truncate file ======= 19:43:25 (1788651805) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16h (9s) == sanityn test 16i: read after truncate file ============ 19:43:33 (1788651813) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16i (7s) == sanityn test 16j: race dio with buffered i/o ========== 19:43:41 (1788651821) 1+0 records in 1+0 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.51311 s, 17.3 MB/s bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.73344 s, 15.1 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.40735 s, 7.7 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.02016 s, 13.0 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.72176 s, 9.6 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.60266 s, 10.1 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.03035 s, 8.7 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.0377 s, 12.9 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.17931 s, 12.0 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.1082 s, 8.4 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.31501 s, 7.9 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.26974 s, 11.5 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.56529 s, 10.2 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.95974 s, 8.9 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.05231 s, 8.6 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.51601 s, 10.4 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.93 s, 8.9 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.48108 s, 7.5 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.69423 s, 7.1 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK PASS 16j (41s) == sanityn test 16k: Parallel FSX and drop caches should not panic ========================================================== 19:44:23 (1788651863) + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.1 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.2 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.3 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.4 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.5 -l 67108864 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! + fsx 1 done: rc=0 Sat Sep 5 19:45:14 EDT 2026 + fsx 2 done: rc=0 Sat Sep 5 19:45:14 EDT 2026 + fsx 3 done: rc=0 Sat Sep 5 19:45:14 EDT 2026 + fsx 4 done: rc=0 Sat Sep 5 19:45:14 EDT 2026 + fsx 5 done: rc=0 Sat Sep 5 19:45:14 EDT 2026 PASS 16k (60s) == sanityn test 17: resource creation/LVB creation race ========================================================================= 19:45:23 (1788651923) fail_loc=0x8000030a PASS 17 (10s) == sanityn test 18: mmap sanity check =========================================================================================== 19:45:34 (1788651934) mmap test1: basic mmap operation (PASS, 0.17443s) mmap test2: MAP_PRIVATE not write back (PASS, 0.082293s) mmap test3: concurrent mmap ops on two nodes (PASS, 13.198s) mmap test4: c1 write to f1 from mmapped f2, c2 write to f1 from mmapped f1 (PASS, 4.5994s) mmap test5: read/write file to/from the buffer which mmapped to just this file (PASS, 0.11634s) mmap test6: check mmap write/read content on two nodes (PASS, 0.31034s) mmap test7: file i/o with an unmapped buffer (SKIPPED, 0s) mmap test8: SIGBUS for beyond file size (SKIPPED, 0s) mmap test9: SIGBUS for negative file offset (SKIPPED, 0s) mmap test10: multi-thread mmap access (PASS, 1.198s) mmap test11: multi-thread shared mmap (PASS, 1.5813s) mmap test10: mtime not change for readonly mmap access (PASS, 1.0959s) PASS 18 (30s) == sanityn test 19: test concurrent uncached read races ========================================================================= 19:46:04 (1788651964) oleg336-server: error: get_param: param_path 'osd-*/lustre-OST*/read_cache_enable': No such file or directory pdsh@oleg336-client: oleg336-server: ssh exited with exit code 2 SKIP: sanityn test_19 not cache-capable obdfilter SKIP 19 (4s) == sanityn test 20: test extra readahead page left in cache ============================================================== 19:46:08 (1788651968) PASS 20 (7s) == sanityn test 21: Try to remove mountpoint on another dir ============================================================== 19:46:15 (1788651975) rmdir: removing directory, '/mnt/lustre/d21.sanityn' rmdir: failed to remove '/mnt/lustre/d21.sanityn': Device or resource busy rmdir: removing directory, '/mnt/lustre2/d21.sanityn' Removed mounted directory from another mountpoint, needs to be fixed PASS 21 (7s) == sanityn test 23: others should see updated atime while another read============================================================== 19:46:22 (1788651982) now is 1788651985 starting reads multiop /mnt/lustre/f23.sanityn vor20_c TMPPIPE=/tmp/multiop_open_wait_pipe.7631 new atime is 1788652046 PASS 23 (70s) == sanityn test 24a: lfs df [-ih] [path] test =================================================================================== 19:47:33 (1788652053) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 5120 3764224 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 5120 3764224 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre2 UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 115.3K 403 114.9K 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 114.5K 399 114.1K 1% /mnt/lustre[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.3K 403 114.9K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 399 114.1K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre2 UUID bytes Used Available Use% Mounted on lustre-MDT0000_UUID 2.1G 2.1M 2.1G 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3.6G 5.0M 3.6G 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3.6G 32.0M 3.6G 1% /mnt/lustre[OST:1] filesystem_summary: 7.2G 37.0M 7.2G 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118099 403 117696 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117231 399 116832 1% /mnt/lustre2[OST:1] filesystem_summary: 234914 386 234528 1% /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 5120 3764224 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.3K 403 114.9K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 399 114.1K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 118099 403 117696 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 117231 399 116832 1% /mnt/lustre[OST:1] filesystem_summary: 234914 386 234528 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118099 403 117696 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117231 399 116832 1% /mnt/lustre2[OST:1] filesystem_summary: 234914 386 234528 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 5120 3764224 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 5120 3764224 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre2 PASS 24a (8s) == sanityn test 24b: lfs df should show both filesystems ========================================================================= 19:47:42 (1788652062) PASS 24b (8s) == sanityn test 25a: change ACL on one mountpoint be seen on another ============================================================= 19:47:49 (1788652069) running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] PASS 25a (8s) == sanityn test 25b: change ACL under remote dir on one mountpoint be seen on another ========================================================== 19:47:57 (1788652077) SKIP: sanityn test_25b needs >= 2 MDTs SKIP 25b (3s) == sanityn test 26a: allow mtime to get older ============ 19:48:00 (1788652080) utime: good mknod times 1788652080 <= 1788652081 <= 1788652081 for /mnt/lustre/f26a utime: good mknod times 1788652080 <= 1788652081 <= 1788652081 for /mnt/lustre2/f26a utime: good utime mtimes 100000, atime 200000 utime: updated times for /mnt/lustre2/f26a PASS 26a (7s) == sanityn test 26b: sync mtime between ost and mds ====== 19:48:08 (1788652088) PASS 26b (8s) == sanityn test 26c: set-in-past on open file is not lost on close ========================================================== 19:48:16 (1788652096) multiop /mnt/lustre/f26c.sanityn vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7631 File: /mnt/lustre/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502398 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-09-05 19:48:18.000000000 -0400 Birth: 2026-09-05 19:48:18.000000000 -0400 File: /mnt/lustre2/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502398 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-09-05 19:48:18.000000000 -0400 Birth: 2026-09-05 19:48:18.000000000 -0400 PASS 26c (10s) SKIP: sanityn test_27 skipping excluded test 27 == sanityn test 30: recreate file race =================== 19:48:28 (1788652108) 0 PASS 30 (10s) == sanityn test 31a: voluntary cancel / blocking ast race======================================================================== 19:48:38 (1788652118) fail_loc=0x314 PASS 31a (10s) == sanityn test 31b: voluntary OST cancel / blocking ast race======================================================================== 19:48:48 (1788652128) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg336-server mds-ost sync done. sleep 5 for ZFS MDS Waiting for MDT destroys to complete fail_loc=0x314 fail_loc=0x316 fail_loc=0 fail_loc=0 Filesystem 1K-blocks Used Available Use% Mounted on 192.168.203.136@tcp:/lustre 7542784 38912 7497728 1% /mnt/lustre2 PASS 31b (25s) == sanityn test 31r: open-rename(replace) race =========== 19:49:13 (1788652153) -rw-r--r-- 1 root root 0 Sep 5 19:49 /mnt/lustre/f31r.sanityn.target fail_loc=0x1419 fail_val=3 PASS 31r (9s) == sanityn test 31s: open should not revalidate invalid dentry ========================================================== 19:49:22 (1788652162) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 1205: 43173 Terminated tail -f $tfile PASS 31s (8s) == sanityn test 31t: getattr should not revalidate invalid dentry ========================================================== 19:49:30 (1788652170) PASS 31t (8s) == sanityn test 32b: lockless i/o ======================== 19:49:38 (1788652178) SKIP: sanityn test_32b max_nolock_bytes is removed >= 2.17.53 SKIP 32b (4s) == sanityn test 32c: contention detection ================ 19:49:42 (1788652182) ldlm.namespaces.filter-lustre-OST0000_UUID.contended_locks=2 ldlm.namespaces.filter-lustre-OST0001_UUID.contended_locks=2 ldlm.namespaces.filter-lustre-OST0000_UUID.contention_events=0 ldlm.namespaces.filter-lustre-OST0001_UUID.contention_events=0 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.74541 s, 30.0 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.35792 s, 38.6 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.93202 s, 27.1 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.14779 s, 24.4 MB/s contention_seconds 4, contention_events 19 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.49648 s, 21.0 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.57528 s, 20.4 MB/s ldlm.namespaces.filter-lustre-OST0000_UUID.contention_events=0 ldlm.namespaces.filter-lustre-OST0001_UUID.contention_events=0 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.4693 s, 35.7 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.53178 s, 34.2 MB/s PASS 32c (30s) SKIP: sanityn test_33a skipping SLOW test 33a == sanityn test 33b: COS: cross create/delete, 2 clients, benchmark under remote dir ========================================================== 19:50:14 (1788652214) SKIP: sanityn test_33b Need two or more clients, have 1 SKIP 33b (4s) == sanityn test 33c: Cancel cross-MDT lock should trigger Sync-on-Lock-Cancel ========================================================== 19:50:17 (1788652217) SKIP: sanityn test_33c needs >= 2 MDTs SKIP 33c (3s) == sanityn test 33d: dependent transactions should trigger COS ========================================================== 19:50:20 (1788652220) SKIP: sanityn test_33d needs >= 2 MDTs SKIP 33d (3s) == sanityn test 33e: independent transactions shouldn't trigger COS ========================================================== 19:50:24 (1788652224) SKIP: sanityn test_33e needs >= 2 MDTs SKIP 33e (3s) == sanityn test 34: no lock timeout under IO ============= 19:50:27 (1788652227) fail_loc=0x512 fail_loc=0x512 lock should not expire writing on client1 reading on client2 success fail_loc=0x511 fail_loc=0x511 lock should expire writing on client1 reading on client2 success oleg336-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff88adc7b35000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88adc7b35000.ost_server_uuid in FULL state after 0 sec oleg336-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff88adc7b35000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88adc7b35000.ost_server_uuid in IDLE state after 0 sec oleg336-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff88adc7b35000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88adc7b35000.ost_server_uuid in IDLE state after 0 sec oleg336-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff88adc7b35000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88adc7b35000.ost_server_uuid in IDLE state after 0 sec oleg336-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff88adc7b35000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88adc7b35000.ost_server_uuid in IDLE state after 0 sec oleg336-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff88adc7b35000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88adc7b35000.ost_server_uuid in IDLE state after 0 sec PASS 34 (133s) == sanityn test 35: -EINTR cp_ast vs. bl_ast race does not evict client ========================================================== 19:52:40 (1788652360) fail_loc=0x80000317 Race attempt 0 Wait for 50484 50539 for 60 sec... - open/close 677 (time 1788652375.69 total 10.01 last 67.66) - open/close 1341 (time 1788652385.69 total 20.01 last 66.37) - open/close 1973 (time 1788652395.70 total 30.02 last 63.14) - open/close 2663 (time 1788652405.72 total 40.04 last 68.89) - open/close 3325 (time 1788652415.72 total 50.04 last 66.16) total: 4000 open/close in 59.55 seconds: 67.17 ops/second fail_loc=0x0 Filesystem Size Used Avail Use% Mounted on 192.168.203.136@tcp:/lustre 7.2G 39M 7.2G 1% /mnt/lustre 192.168.203.136@tcp:/lustre 7.2G 39M 7.2G 1% /mnt/lustre2 PASS 35 (70s) == sanityn test 36: handle ESTALE/open-unlink correctly == 19:53:50 (1788652430) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg336-server mds-ost sync done. Waiting for MDT destroys to complete 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.71452 s, 30.6 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7631 sleep 5 for ZFS MDS Waiting for MDT destroys to complete *** cycle(0) *** before(7498752) after_dd(7443456) after(7494656) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.37803 s, 38.0 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7631 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(1) *** before(7494656) after_dd(7438336) after(7498752) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.54589 s, 33.9 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7631 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(2) *** before(7498752) after_dd(7440384) after(7491584) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.43371 s, 36.6 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7631 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(3) *** before(7491584) after_dd(7447552) after(7498752) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.6674 s, 31.4 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7631 sleep 5 for ZFS MDS Waiting for MDT destroys to complete *** cycle(4) *** before(7498752) after_dd(7443456) after(7494656) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.45022 s, 36.2 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7631 sleep 5 for ZFS MDS Waiting for MDT destroys to complete *** cycle(5) *** before(7494656) after_dd(7443456) after(7494656) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.64876 s, 31.8 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7631 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(6) *** before(7494656) after_dd(7445504) after(7495680) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.55637 s, 33.7 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7631 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(7) *** before(7495680) after_dd(7440384) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.40876 s, 37.2 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7631 sleep 5 for ZFS MDS Waiting for MDT destroys to complete *** cycle(8) *** before(7497728) after_dd(7442432) after(7493632) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.4093 s, 37.2 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7631 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(9) *** before(7493632) after_dd(7444480) after(7495680) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.53612 s, 34.1 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7631 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(10) *** before(7495680) after_dd(7444480) after(7495680) PASS 36 (302s) == sanityn test 37: check i_size is not updated for directory on close (bug 18695) ======================================================================== 19:58:53 (1788652733) multiop /mnt/lustre/d37.sanityn vD_c TMPPIPE=/tmp/multiop_open_wait_pipe.7631 - create 1158 (time 1788652747.36 total 10.00 last 115.77) - create 2385 (time 1788652757.37 total 20.01 last 122.64) - create 3653 (time 1788652767.38 total 30.01 last 126.71) - create 4931 (time 1788652777.38 total 40.02 last 127.76) - create 6114 (time 1788652787.39 total 50.03 last 118.14) - create 7400 (time 1788652797.39 total 60.03 last 128.58) - create 8629 (time 1788652807.41 total 70.04 last 122.75) - create 9915 (time 1788652817.41 total 80.05 last 128.57) total: 10000 create in 80.90 seconds: 123.60 ops/second PASS 37 (93s) == sanityn test 39a: file mtime does not change after rename ========================================================== 20:00:25 (1788652825) repeat after cancel_lru_locks PASS 39a (7s) == sanityn test 39b: file mtime the same on clients with/out lock ========================================================== 20:00:32 (1788652832) repeat after cancel_lru_locks PASS 39b (8s) == sanityn test 39c: check truncate mtime update ================================================================================ 20:00:41 (1788652841) repeat after cancel_lru_locks PASS 39c (9s) == sanityn test 39d: sync write should update mtime ====== 20:00:49 (1788652849) fail_loc=0x411 fail_loc=0 PASS 39d (7s) SKIP: sanityn test_40a skipping ALWAYS excluded test 40a == sanityn test 40b: pdirops: open|create and others ======================================================================== 20:00:58 (1788652858) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40b (20s) == sanityn test 40c: pdirops: link and others ======================================================================== 20:01:18 (1788652878) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40c (19s) == sanityn test 40d: pdirops: unlink and others ======================================================================== 20:01:37 (1788652897) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40d (21s) == sanityn test 40e: pdirops: rename and others ======================================================================== 20:01:58 (1788652918) No conflict No conflict No conflict No conflict No conflict No conflict PASS 40e (18s) == sanityn test 41a: pdirops: create vs mkdir ======================================================================== 20:02:16 (1788652936) mkdir: cannot create directory '/mnt/lustre2/f41a.sanityn': File exists Conflict PASS 41a (13s) == sanityn test 41b: pdirops: create vs create ======================================================================== 20:02:29 (1788652949) open: File exists Conflict PASS 41b (14s) == sanityn test 41c: pdirops: create vs link ======================================================================== 20:02:43 (1788652963) link: cannot create link '/mnt/lustre2/f41c.sanityn' to '/mnt/lustre2/f41c.sanityn-2': File exists Conflict PASS 41c (14s) == sanityn test 41d: pdirops: create vs unlink ======================================================================== 20:02:57 (1788652977) Conflict PASS 41d (14s) == sanityn test 41e: pdirops: create and rename (tgt) ======================================================================== 20:03:11 (1788652991) Conflict PASS 41e (15s) == sanityn test 41f: pdirops: create and rename (src) ======================================================================== 20:03:26 (1788653006) Conflict PASS 41f (13s) == sanityn test 41g: pdirops: create vs getattr ======================================================================== 20:03:39 (1788653019) Conflict PASS 41g (15s) == sanityn test 41h: pdirops: create vs readdir ======================================================================== 20:03:54 (1788653034) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 2427: lctl set_param -n fail_loc=0 2>/dev/null || true: No such file or directory Conflict PASS 41h (15s) == sanityn test 41i: reint_open: create vs create ======== 20:04:09 (1788653049) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 41i (1090s) == sanityn test 42a: pdirops: mkdir vs mkdir ======================================================================== 20:22:20 (1788654140) mkdir: cannot create directory '/mnt/lustre2/f42a.sanityn': File exists Conflict PASS 42a (17s) == sanityn test 42b: pdirops: mkdir vs create ======================================================================== 20:22:36 (1788654156) open: File exists Conflict PASS 42b (14s) == sanityn test 42c: pdirops: mkdir vs link ======================================================================== 20:22:51 (1788654171) link: cannot create link '/mnt/lustre2/f42c.sanityn' to '/mnt/lustre2/f42c.sanityn-2': File exists Conflict PASS 42c (14s) == sanityn test 42d: pdirops: mkdir vs unlink ======================================================================== 20:23:05 (1788654185) Conflict PASS 42d (13s) == sanityn test 42e: pdirops: mkdir and rename (tgt) ======================================================================== 20:23:18 (1788654198) mv: cannot overwrite directory '/mnt/lustre2/f42e.sanityn' with non-directory Conflict PASS 42e (13s) == sanityn test 42f: pdirops: mkdir and rename (src) ======================================================================== 20:23:32 (1788654212) Conflict PASS 42f (12s) == sanityn test 42g: pdirops: mkdir vs getattr ======================================================================== 20:23:45 (1788654225) Conflict PASS 42g (14s) == sanityn test 42h: pdirops: mkdir vs readdir ======================================================================== 20:23:58 (1788654238) Conflict PASS 42h (15s) == sanityn test 43a: rmdir,mkdir doesn't return -EEXIST ======================================================================== 20:24:13 (1788654253) PASS 43a (70s) == sanityn test 43b: pdirops: unlink vs create ======================================================================== 20:25:23 (1788654323) Conflict PASS 43b (14s) == sanityn test 43c: pdirops: unlink vs link ======================================================================== 20:25:37 (1788654337) Conflict PASS 43c (12s) == sanityn test 43d: pdirops: unlink vs unlink ======================================================================== 20:25:49 (1788654349) rm: cannot remove '/mnt/lustre2/f43d.sanityn': No such file or directory Conflict PASS 43d (14s) == sanityn test 43e: pdirops: unlink and rename (tgt) ======================================================================== 20:26:03 (1788654363) Conflict PASS 43e (13s) == sanityn test 43f: pdirops: unlink and rename (src) ======================================================================== 20:26:17 (1788654377) mv: cannot stat '/mnt/lustre2/f43f.sanityn': No such file or directory Conflict PASS 43f (15s) == sanityn test 43g: pdirops: unlink vs getattr ======================================================================== 20:26:31 (1788654391) stat: cannot statx '/mnt/lustre2/f43g.sanityn': No such file or directory Conflict PASS 43g (13s) == sanityn test 43h: pdirops: unlink vs readdir ======================================================================== 20:26:44 (1788654404) Conflict PASS 43h (13s) == sanityn test 43i: pdirops: unlink vs remote mkdir ===== 20:26:57 (1788654417) SKIP: sanityn test_43i needs >= 2 MDTs SKIP 43i (3s) == sanityn test 43j: racy mkdir return EEXIST ======================================================================== 20:27:00 (1788654420) mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists PASS 43j (125s) == sanityn test 43k: unlink vs create ==================== 20:29:05 (1788654545) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 43k (1135s) == sanityn test 44a: pdirops: rename tgt vs mkdir ======================================================================== 20:48:00 (1788655680) mkdir: cannot create directory '/mnt/lustre2/f44a.sanityn': File exists Conflict Sat Sep 5 20:48:07 EDT 2026 PASS 44a (11s) == sanityn test 44b: pdirops: rename tgt vs create ======================================================================== 20:48:11 (1788655691) open: File exists Conflict PASS 44b (12s) == sanityn test 44c: pdirops: rename tgt vs link ======================================================================== 20:48:23 (1788655703) link: cannot create link '/mnt/lustre2/f44c.sanityn' to '/mnt/lustre2/f44c.sanityn-3': File exists Conflict PASS 44c (14s) == sanityn test 44d: pdirops: rename tgt vs unlink ======================================================================== 20:48:37 (1788655717) Conflict PASS 44d (11s) == sanityn test 44e: pdirops: rename tgt and rename (tgt) ======================================================================== 20:48:49 (1788655729) Conflict PASS 44e (12s) == sanityn test 44f: pdirops: rename tgt and rename (src) ======================================================================== 20:49:01 (1788655741) Conflict PASS 44f (12s) == sanityn test 44g: pdirops: rename tgt vs getattr ======================================================================== 20:49:12 (1788655752) Conflict PASS 44g (12s) == sanityn test 44h: pdirops: rename tgt vs readdir ======================================================================== 20:49:24 (1788655764) Conflict PASS 44h (11s) == sanityn test 44i: pdirops: rename tgt vs remote mkdir ========================================================== 20:49:35 (1788655775) SKIP: sanityn test_44i needs >= 2 MDTs SKIP 44i (3s) == sanityn test 45a: rename,mkdir doesn't return -EEXIST ======================================================================== 20:49:38 (1788655778) PASS 45a (149s) == sanityn test 45b: pdirops: rename src vs create ======================================================================== 20:52:07 (1788655927) Conflict PASS 45b (15s) == sanityn test 45c: pdirops: rename src vs link ======================================================================== 20:52:22 (1788655942) Conflict PASS 45c (14s) == sanityn test 45d: pdirops: rename src vs unlink ======================================================================== 20:52:36 (1788655956) rm: cannot remove '/mnt/lustre2/f45d.sanityn': No such file or directory Conflict PASS 45d (14s) == sanityn test 45e: pdirops: rename src and rename (tgt) ======================================================================== 20:52:50 (1788655970) Conflict PASS 45e (12s) == sanityn test 45f: pdirops: rename src and rename (src) ======================================================================== 20:53:03 (1788655983) mv: cannot stat '/mnt/lustre2/f45f.sanityn': No such file or directory Conflict PASS 45f (14s) == sanityn test 45g: pdirops: rename src vs getattr ======================================================================== 20:53:16 (1788655996) stat: cannot statx '/mnt/lustre2/f45g.sanityn': No such file or directory Conflict PASS 45g (15s) == sanityn test 45h: pdirops: unlink vs readdir ======================================================================== 20:53:31 (1788656011) Conflict PASS 45h (14s) == sanityn test 45i: pdirops: rename src vs remote mkdir ========================================================== 20:53:45 (1788656025) SKIP: sanityn test_45i needs >= 2 MDTs SKIP 45i (4s) == sanityn test 45j: read vs rename ====================== 20:53:49 (1788656029) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 45j (1249s) == sanityn test 46a: pdirops: link vs mkdir ======================================================================== 21:14:38 (1788657278) mkdir: cannot create directory '/mnt/lustre2/f46a.sanityn': File exists Conflict PASS 46a (13s) == sanityn test 46b: pdirops: link vs create ======================================================================== 21:14:51 (1788657291) open: File exists Conflict PASS 46b (13s) == sanityn test 46c: pdirops: link vs link ======================================================================== 21:15:04 (1788657304) link: cannot create link '/mnt/lustre2/f46c.sanityn' to '/mnt/lustre2/f46c.sanityn': File exists Conflict PASS 46c (12s) == sanityn test 46d: pdirops: link vs unlink ======================================================================== 21:15:17 (1788657317) Conflict PASS 46d (14s) == sanityn test 46e: pdirops: link and rename (tgt) ======================================================================== 21:15:30 (1788657330) Conflict PASS 46e (13s) == sanityn test 46f: pdirops: link and rename (src) ======================================================================== 21:15:43 (1788657343) Conflict PASS 46f (15s) == sanityn test 46g: pdirops: link vs getattr ======================================================================== 21:15:58 (1788657358) Conflict PASS 46g (14s) == sanityn test 46h: pdirops: link vs readdir ======================================================================== 21:16:12 (1788657372) Conflict PASS 46h (14s) == sanityn test 46i: pdirops: link vs remote mkdir ======= 21:16:26 (1788657386) SKIP: sanityn test_46i needs >= 2 MDTs SKIP 46i (4s) == sanityn test 47a: pdirops: remote mkdir vs mkdir ====== 21:16:30 (1788657390) SKIP: sanityn test_47a needs >= 2 MDTs SKIP 47a (4s) == sanityn test 47b: pdirops: remote mkdir vs create ===== 21:16:34 (1788657394) SKIP: sanityn test_47b needs >= 2 MDTs SKIP 47b (4s) == sanityn test 47c: pdirops: remote mkdir vs link ======= 21:16:38 (1788657398) SKIP: sanityn test_47c needs >= 2 MDTs SKIP 47c (3s) == sanityn test 47d: pdirops: remote mkdir vs unlink ===== 21:16:42 (1788657402) SKIP: sanityn test_47d needs >= 2 MDTs SKIP 47d (3s) == sanityn test 47e: pdirops: remote mkdir and rename (tgt) ========================================================== 21:16:46 (1788657406) SKIP: sanityn test_47e needs >= 2 MDTs SKIP 47e (4s) == sanityn test 47f: pdirops: remote mkdir and rename (src) ========================================================== 21:16:49 (1788657409) SKIP: sanityn test_47f needs >= 2 MDTs SKIP 47f (4s) == sanityn test 47g: pdirops: remote mkdir vs getattr ==== 21:16:53 (1788657413) SKIP: sanityn test_47g needs >= 2 MDTs SKIP 47g (3s) == sanityn test 50: osc lvb attrs: enqueue vs. CP AST ======================================================================== 21:16:57 (1788657417) 10+0 records in 10+0 records out 10240 bytes (10 kB, 10 KiB) copied, 0.122628 s, 83.5 kB/s fail_loc=0x410 fail_loc=0x0 PASS 50 (13s) == sanityn test 51a: layout lock: refresh layout should work ========================================================== 21:17:10 (1788657430) 0+1 records in 0+1 records out 158 bytes copied, 0.0419139 s, 3.8 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 3690: kill: (218454) - No such process PASS 51a (11s) == sanityn test 51b: layout lock: glimpse should be able to restart if layout changed ========================================================== 21:17:22 (1788657442) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0184964 s, 55.4 kB/s fail_loc=0x1404 fail_val=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132752 s, 7.9 MB/s 3145728 PASS 51b (22s) == sanityn test 51c: layout lock: IT_LAYOUT blocked and correct layout can be returned ========================================================== 21:17:43 (1788657463) fail_loc=0x172 Setting layout to have 2 stripes ... 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0212056 s, 48.3 kB/s PASS 51c (14s) == sanityn test 51d: layout lock: losing layout lock should clean up memory map region ========================================================== 21:17:57 (1788657477) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0988371 s, 10.6 MB/s Before revoking layout lock: 1024 KB mapped PASS 51d (9s) == sanityn test 51e: lfs getstripe does not break leases, part 2 ========================================================== 21:18:06 (1788657486) /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 2450 0x992 0x280000400 write lease(2) has applied. /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 2450 0x992 0x280000400 read lease(1) has applied. PASS 51e (11s) == sanityn test 54: rename locking ======================= 21:18:17 (1788657497) llite.lustre-ffff88adc7b35000.xattr_cache=1 llite.lustre-ffff88add8dd7800.xattr_cache=1 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505015 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-05 21:18:20.000000000 -0400 Modify: 2026-09-05 21:18:20.000000000 -0400 Change: 2026-09-05 21:18:20.000000000 -0400 Birth: 2026-09-05 21:18:20.000000000 -0400 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000154 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505018 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-05 21:18:26.000000000 -0400 Modify: 2026-09-05 21:18:27.000000000 -0400 Change: 2026-09-05 21:18:27.000000000 -0400 Birth: 2026-09-05 21:18:26.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505021 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-05 21:18:32.000000000 -0400 Modify: 2026-09-05 21:18:32.000000000 -0400 Change: 2026-09-05 21:18:32.000000000 -0400 Birth: 2026-09-05 21:18:32.000000000 -0400 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505020 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-05 21:18:32.000000000 -0400 Modify: 2026-09-05 21:18:33.000000000 -0400 Change: 2026-09-05 21:18:33.000000000 -0400 Birth: 2026-09-05 21:18:32.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000154 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505023 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-05 21:18:39.000000000 -0400 Modify: 2026-09-05 21:18:40.000000000 -0400 Change: 2026-09-05 21:18:40.000000000 -0400 Birth: 2026-09-05 21:18:39.000000000 -0400 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505024 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-05 21:18:39.000000000 -0400 Modify: 2026-09-05 21:18:40.000000000 -0400 Change: 2026-09-05 21:18:40.000000000 -0400 Birth: 2026-09-05 21:18:39.000000000 -0400 PASS 54 (34s) == sanityn test 55a: rename vs unlink target dir ========= 21:18:51 (1788657531) fail_loc=0x80000156 mv: cannot move '/mnt/lustre/d55a.sanityn/d1/d2' to '/mnt/lustre/d55a.sanityn/d3/d2': No such file or directory PASS 55a (14s) == sanityn test 55b: rename vs unlink source dir ========= 21:19:05 (1788657545) fail_loc=0x80000156 mv: cannot stat '/mnt/lustre/d55b.sanityn/d1/d2': No such file or directory PASS 55b (13s) == sanityn test 55c: rename vs unlink orphan target dir == 21:19:19 (1788657559) fail_loc=0x156 mv: cannot move '/mnt/lustre/d55c.sanityn/d1/d2' to '/mnt/lustre/d55c.sanityn/d3/d2': No such file or directory PASS 55c (19s) == sanityn test 55d: rename file vs link ================= 21:19:38 (1788657578) fail_loc=0x155 mv: '/mnt/lustre/d55d.sanityn/f1' and '/mnt/lustre/d55d.sanityn/d55d.sanityn/f1' are the same file PASS 55d (16s) == sanityn test 55e: rename race AB/BA under the same parent dir ========================================================== 21:19:54 (1788657594) SKIP: sanityn test_55e needs >= 2 MDTs SKIP 55e (5s) == sanityn test 55f: rename: (P1/A -> P2/B) race with (P2/B -> P1/A) ========================================================== 21:19:59 (1788657599) fail_loc=0x155 We survived after P1A_P2B/P2B_P1A race test fail_loc=0x0 PASS 55f (21s) == sanityn test 55g: rename: race with trylock =========== 21:20:20 (1788657620) mdt.lustre-MDT0000.enable_rename_trylock=1 fail_loc=0x155 mv: cannot move '/mnt/lustre/d55g.sanityn/a1/a2' to a subdirectory of itself, '/mnt/lustre/d55g.sanityn/b1/b2/b3/b4/a2' We survived race with trylock test fail_loc=0x0 PASS 55g (24s) == sanityn test 56a: test llverdev with single large stripe ========================================================== 21:20:45 (1788657645) run_llverdev /mnt/lustre/f56a.sanityn -p -s 16000000 -c 4k llverdev: /mnt/lustre/f56a.sanityn is 16777216000000 bytes (15625 GB) in size timestamp: 1788657647 chunksize: 4096 size: 16777216000000 write offset: 16384000000kB 0.455515 MB/s read offset: 16384000000kB inf MB/s llverdev: data verified successfully PASS 56a (146s) == sanityn test 56b: test llverdev and partial verify of wide stripe file ========================================================== 21:23:11 (1788657791) run_llverdev /mnt/lustre/f56b.sanityn -p -s 32000000 -c 4k llverdev: /mnt/lustre/f56b.sanityn is 33554432000000 bytes (31250 GB) in size timestamp: 1788657797 chunksize: 4096 size: 33554432000000 write offset: 32768000000kB 0.968843 MB/s read offset: 32768000000kB inf MB/s llverdev: data verified successfully PASS 56b (140s) == sanityn test 60: Verify data_version behaviour ======== 21:25:31 (1788657931) PASS 60 (11s) == sanityn test 70a: cd directory && rm directory ======== 21:25:43 (1788657943) PASS 70a (8s) == sanityn test 70b: remove files after calling rm_entry ========================================================== 21:25:52 (1788657952) PASS 70b (11s) == sanityn test 71a: correct file map just after write operation is finished ========================================================== 21:26:03 (1788657963) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0106381 s, 6.2 MB/s SKIP: sanityn test_71a ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71a (5s) == sanityn test 71b: check fiemap support for stripecount > 1 ========================================================== 21:26:08 (1788657968) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.023505 s, 2.8 MB/s SKIP: sanityn test_71b ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71b (4s) == sanityn test 71c: check FIEMAP_EXTENT_LAST flag with different extents number ========================================================== 21:26:13 (1788657973) SKIP: sanityn test_71c support only ldiskfs ost SKIP 71c (4s) == sanityn test 71d: fiemap corruption test with fm_extent_count=0 ========================================================== 21:26:16 (1788657976) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0334464 s, 2.0 MB/s SKIP: sanityn test_71d ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71d (4s) == sanityn test 72: getxattr/setxattr cache should be consistent between nodes ========================================================== 21:26:20 (1788657980) llite.lustre-ffff88adc7b35000.xattr_cache=1 llite.lustre-ffff88add8dd7800.xattr_cache=1 getfattr: Removing leading '/' from absolute path names user.attr1="value1" getfattr: Removing leading '/' from absolute path names user.attr1="value2" getfattr: Removing leading '/' from absolute path names getfattr: Removing leading '/' from absolute path names b362740ad371eae580e7f85452e8830b - 870e072da1ebfc20ea8c0cd3d6bdf3b6 - PASS 72 (8s) == sanityn test 73: getxattr should not cause xattr lock cancellation ========================================================== 21:26:29 (1788657989) llite.lustre-ffff88adc7b35000.xattr_cache=1 llite.lustre-ffff88add8dd7800.xattr_cache=1 getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" PASS 73 (8s) == sanityn test 74: flock deadlock: different mounts ======================================================================== 21:26:37 (1788657997) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0139348 s, 73.5 kB/s 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.021313 s, 48.0 kB/s 231267: get lock2 231271: get lock1 231267: done 231267: get lock1 231271: done 231271: get lock2 fcntl cmd 7 failed: Resource deadlock avoided 231271: cannot get lock2: Resource deadlock avoided 231271: exit rc=0 231267: done 231267: exit rc=0 PASS 74 (13s) == sanityn test 75: osc: upcall after unuse lock============================================================================= 21:26:50 (1788658010) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.082361 s, 25.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.051453 s, 20.4 MB/s fail_loc=0x31d /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 4373: 231852 Killed stat -c %s $DIR1/$tfile fail_loc=0 PASS 75 (13s) == sanityn test 76: Verify MDT open_files listing ======== 21:27:03 (1788658023) opening files: ........ closing files: ........ PASS 76 (81s) == sanityn test 77a: check FIFO NRS policy =============== 21:28:24 (1788658104) ost.OSS.ost.nrs_policies=fifo ost.OSS.ost_create.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_out.nrs_policies=fifo ost.OSS.ost_seq.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.499206 s, 33.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0370493 s, 28.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100934 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102694 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170893 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143202 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.236581 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147769 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242806 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0568683 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103321 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.273648 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.343313 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.360886 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.51111 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.474835 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.53553 s, 2.0 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff88adc7b35000.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137395 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0743338 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139063 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199748 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196158 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0579587 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135107 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0661651 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141773 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.219786 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125127 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.239606 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10462 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118691 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134181 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0189401 s, 55.4 MB/s removed '/mnt/lustre/d77a.sanityn/nrs_w_oleg336-client.virtnet' removed '/mnt/lustre/d77a.sanityn/nrs_r_oleg336-client.virtnet' removed directory '/mnt/lustre/d77a.sanityn' PASS 77a (10s) == sanityn test 77b: check CRR-N NRS policy ============== 21:28:35 (1788658115) ost.OSS.ost.nrs_policies=crrn ost.OSS.ost_create.nrs_policies=crrn ost.OSS.ost_io.nrs_policies=crrn ost.OSS.ost_out.nrs_policies=crrn ost.OSS.ost_seq.nrs_policies=crrn ost.OSS.ost.nrs_crrn_quantum=1 ost.OSS.ost_create.nrs_crrn_quantum=1 ost.OSS.ost_io.nrs_crrn_quantum=1 ost.OSS.ost_out.nrs_crrn_quantum=1 ost.OSS.ost_seq.nrs_crrn_quantum=1 policy: crr-n, crrn_quantum 1 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.457192 s, 36.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0627953 s, 16.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.079291 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188298 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23494 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.277065 s, 3.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.257405 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.299243 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.261959 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.310756 s, 3.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.29715 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.289671 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.272189 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.268143 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.277708 s, 3.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff88adc7b35000.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff88adc7b35000.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.263992 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.307564 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0301053 s, 34.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0578131 s, 18.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0755098 s, 13.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0313755 s, 33.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0418364 s, 25.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.048289 s, 21.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0462184 s, 22.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0366485 s, 28.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.045275 s, 23.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0482274 s, 21.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0644744 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0672773 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0649343 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0330959 s, 31.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0895347 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0939412 s, 11.2 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg336-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg336-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost.nrs_crrn_quantum=64 ost.OSS.ost_create.nrs_crrn_quantum=64 ost.OSS.ost_io.nrs_crrn_quantum=64 ost.OSS.ost_out.nrs_crrn_quantum=64 ost.OSS.ost_seq.nrs_crrn_quantum=64 policy: crr-n, crrn_quantum 64 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.531698 s, 31.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0764225 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107959 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130606 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15284 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149688 s, 7.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146673 s, 7.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193948 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19115 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159402 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.311933 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.372083 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.395569 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.410896 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.427746 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.396985 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.386452 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00727622 s, 144 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0557595 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0794155 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0606313 s, 17.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.041937 s, 25.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10527 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0926168 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0396873 s, 26.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0574248 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0582256 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0537933 s, 19.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0576941 s, 18.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106119 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0573434 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.084598 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0760293 s, 13.8 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg336-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg336-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77b (16s) == sanityn test 77c: check ORR NRS policy ================ 21:28:50 (1788658130) ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_orr_quantum=1 ost.OSS.ost_io.nrs_orr_offset_type=physical ost.OSS.ost_io.nrs_orr_supported=reads policy: orr, orr_quantum 1, orr_offset_type physical, orr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.518922 s, 32.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0565152 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0838395 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148849 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104631 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1748 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203449 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192408 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.287956 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198697 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.261664 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.280981 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.253069 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.300564 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.249647 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.263345 s, 4.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff88adc7b35000.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff88adc7b35000.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.273174 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0471008 s, 22.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00447799 s, 234 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157111 s, 66.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0108685 s, 96.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00915993 s, 114 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147144 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0835499 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146836 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0922648 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122887 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164132 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154915 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149606 s, 70.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170702 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0793298 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.066507 s, 15.8 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg336-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg336-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=writes ost.OSS.ost_io.nrs_orr_quantum=64 policy: orr, orr_quantum 64, orr_offset_type physical, orr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.596632 s, 28.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0626093 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140223 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173885 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223171 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227177 s, 4.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.255142 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.272045 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.251291 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197986 s, 5.3 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff88adc7b35000.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.412157 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.393606 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.515099 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.43939 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.452715 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.46817 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0119085 s, 88.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.090087 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0477165 s, 22.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0902576 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0943843 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0954907 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0494201 s, 21.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101344 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100956 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173599 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107495 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0659858 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0178202 s, 58.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131884 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.525809 s, 2.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff88adc7b35000.lock_unused_count=8 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0107698 s, 97.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13333 s, 7.9 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg336-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg336-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=reads_and_writes ost.OSS.ost_io.nrs_orr_offset_type=logical policy: orr, orr_quantum 64, orr_offset_type logical, orr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.417588 s, 40.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0973762 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100196 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.07801 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.098123 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0877133 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177797 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195313 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.321532 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.318513 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.264633 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.276893 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.358368 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.314394 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0112479 s, 93.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129339 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00358184 s, 293 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.39433 s, 2.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.358993 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0319842 s, 32.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0916704 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0685683 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0062684 s, 167 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.432529 s, 2.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181483 s, 5.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00884179 s, 119 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206845 s, 5.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182235 s, 5.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162247 s, 64.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00535973 s, 196 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0727582 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0516267 s, 20.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00621026 s, 169 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg336-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg336-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77c (18s) == sanityn test 77d: check TRR nrs policy ================ 21:29:08 (1788658148) ost.OSS.ost_io.nrs_policies=trr ost.OSS.ost_io.nrs_trr_quantum=1 ost.OSS.ost_io.nrs_trr_offset_type=physical ost.OSS.ost_io.nrs_trr_supported=reads policy: trr, trr_quantum 1, trr_offset_type physical, trr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.582286 s, 28.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126053 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128263 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14199 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0776963 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102353 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209877 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0858874 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.230108 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142253 s, 7.4 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff88adc7b35000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.233179 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.337169 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.286615 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.262248 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.363195 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.353871 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.383465 s, 2.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff88adc7b35000.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0472539 s, 22.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0311001 s, 33.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0364907 s, 28.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0901162 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0119846 s, 87.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116897 s, 9.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0927154 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0964684 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0121389 s, 86.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102267 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0175361 s, 59.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.062512 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0198429 s, 52.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0485665 s, 21.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00827825 s, 127 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0468431 s, 22.4 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg336-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg336-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=writes ost.OSS.ost_io.nrs_trr_quantum=64 policy: trr, trr_quantum 64, trr_offset_type physical, trr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.571092 s, 29.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0811512 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0802838 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127703 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169213 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183206 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121718 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107454 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171849 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135506 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0863886 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.217551 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.32207 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.351293 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.248843 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.29437 s, 3.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff88adc7b35000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff88adc7b35000.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.277988 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0600131 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0515765 s, 20.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0973636 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.06034 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0844109 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0838643 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125984 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0814543 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.098833 s, 10.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0909835 s, 11.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.077359 s, 13.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100437 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0925067 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158378 s, 66.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0899483 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101691 s, 10.3 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg336-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg336-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=reads_and_writes ost.OSS.ost_io.nrs_trr_offset_type=logical policy: trr, trr_quantum 64, trr_offset_type logical, trr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.667028 s, 25.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0639082 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0653127 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0917987 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113555 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194701 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196606 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.27363 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.243968 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.289809 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.277057 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.261302 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.249891 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.319103 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.262179 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.314501 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.364848 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0800176 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.054219 s, 19.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0329856 s, 31.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0860648 s, 12.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111219 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0531075 s, 19.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0973593 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102376 s, 10.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0826119 s, 12.7 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0691415 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0819023 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0968833 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0942364 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158594 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154741 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105122 s, 10.0 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg336-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg336-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77d (18s) == sanityn test 77e: check TBF NID nrs policy ============ 21:29:27 (1788658167) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_tbf_rule=start localhost nid={0@lo} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start clients nid={192.168.203.36@tcp} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start others nid={*.*.*.*@tcp} rate=50 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.527932 s, 31.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106156 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117948 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.267034 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.238762 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.281817 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.256117 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.323766 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.310898 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.29957 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.346586 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.35677 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.393155 s, 2.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.411292 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.437244 s, 2.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.397358 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.464316 s, 2.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0413894 s, 25.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0761185 s, 13.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0379174 s, 27.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0615699 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0555221 s, 18.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0549969 s, 19.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0232303 s, 45.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0784917 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0960547 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134449 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0710164 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0869746 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139416 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0792255 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0218154 s, 48.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0814365 s, 12.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff88adc7b35000.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff88adc7b35000.lock_unused_count=1 removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg336-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg336-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change localhost rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change clients rate=101 ost.OSS.ost_io.nrs_tbf_rule=change others rate=51 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.573536 s, 29.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110992 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12207 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.259999 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221721 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.330606 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.321467 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.422974 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212776 s, 4.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.380725 s, 2.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170014 s, 6.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.241478 s, 4.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166694 s, 62.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.470797 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.44336 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.464714 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.320374 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.261634 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.459209 s, 2.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0172419 s, 60.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0257343 s, 40.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18552 s, 5.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.526267 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.570466 s, 1.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0646094 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152186 s, 68.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.394404 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.593916 s, 1.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00846231 s, 124 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0668273 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0654494 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.598293 s, 1.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0936662 s, 11.2 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg336-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg336-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop localhost ost.OSS.ost_io.nrs_tbf_rule=stop clients ost.OSS.ost_io.nrs_tbf_rule=stop others 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.668894 s, 25.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0652707 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100239 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0437446 s, 24.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0869868 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.070306 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0667241 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114065 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110375 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0522687 s, 20.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207332 s, 5.1 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff88adc7b35000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.277703 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197502 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223141 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.297414 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.261665 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.3518 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171893 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0974655 s, 10.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183117 s, 5.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178183 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175652 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178252 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0676541 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0949426 s, 11.0 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.07521 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0751667 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0849591 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0878443 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0976465 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126051 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104624 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112538 s, 9.3 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg336-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg336-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.419195 s, 40.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0448554 s, 23.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0706281 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10356 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11695 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109712 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0696611 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0796722 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0919759 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.069187 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201091 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.250115 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176064 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.248635 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.311825 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.308135 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.281769 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.046455 s, 22.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00530492 s, 198 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff88adc7b35000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14584 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0982731 s, 10.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0826129 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0701561 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0607799 s, 17.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124646 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0602348 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0598038 s, 17.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0434874 s, 24.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0401014 s, 26.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0374501 s, 28.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0316181 s, 33.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0289103 s, 36.3 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.017006 s, 61.7 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg336-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg336-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' PASS 77e (26s) == sanityn test 77f: check TBF JobID nrs policy ========== 21:29:52 (1788658192) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start runas jobid={iozone.500 dd.500 tiotest.500} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start iozone_runas jobid={iozone.500} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg336-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.310392 s, 54.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0455498 s, 23.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0542258 s, 19.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206791 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181769 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194252 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.255162 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220522 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.243638 s, 4.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.298517 s, 3.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.291691 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.332333 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.323092 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.343908 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.343759 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.342728 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.34208 s, 3.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied ldlm.namespaces.lustre-OST0000-osc-ffff88adc7b35000.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff88adc7b35000.lock_unused_count=2 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00892932 s, 117 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136774 s, 76.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00809582 s, 130 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158776 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171679 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136021 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20454 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109721 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0945471 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0538316 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0495286 s, 21.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0484968 s, 21.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0434246 s, 24.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0382719 s, 27.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.030296 s, 34.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0457414 s, 22.9 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg336-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg336-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change runas rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change iozone_runas rate=101 ost.OSS.ost_io.nrs_tbf_rule=change dd_runas rate=51 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg336-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.538245 s, 31.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.07663 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0709733 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0733939 s, 14.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1093 s, 9.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142094 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213257 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124576 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16469 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.286588 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157438 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184734 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196392 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23592 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.281454 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.298193 s, 3.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.33315 s, 3.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff88adc7b35000.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff88adc7b35000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00385871 s, 272 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.024984 s, 42.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0483479 s, 21.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00238541 s, 440 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00245927 s, 426 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00249162 s, 421 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0118725 s, 88.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00901566 s, 116 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00255443 s, 410 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142164 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148606 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166377 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.07329 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0118038 s, 88.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.02989 s, 35.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0307517 s, 34.1 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg336-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg336-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop runas ost.OSS.ost_io.nrs_tbf_rule=stop iozone_runas ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg336-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.734204 s, 22.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0891167 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103642 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165928 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167886 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206587 s, 5.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.416772 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.425211 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.416823 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.492118 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.44092 s, 2.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff88adc7b35000.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.483318 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.668244 s, 1.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.601474 s, 1.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.690157 s, 1.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.693532 s, 1.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00240012 s, 437 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00738399 s, 142 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.709439 s, 1.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00907012 s, 116 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149219 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.31716 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00323501 s, 324 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.286722 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.421647 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120477 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11972 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.368119 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0781977 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0476557 s, 22.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0644653 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0764362 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0369433 s, 28.4 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg336-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg336-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_policies=fifo running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg336-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.59872 s, 28.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112724 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124264 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113206 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106243 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146921 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190074 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203695 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.266413 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.255258 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.225571 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.348721 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147353 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.330649 s, 3.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178903 s, 5.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff88adc7b35000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff88adc7b35000.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.327228 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.40534 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174535 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153929 s, 68.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00236364 s, 444 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00222895 s, 470 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00240264 s, 436 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141225 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212168 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148739 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.256003 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198709 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172153 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0100689 s, 104 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0759465 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0377963 s, 27.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0606509 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0350015 s, 30.0 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg336-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg336-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' PASS 77f (26s) == sanityn test 77g: Change TBF type directly ============ 21:30:18 (1788658218) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 ost.OSS.ost_io.nrs_policies=fifo PASS 77g (13s) == sanityn test 77h: Wrong policy name should report error, not LBUG ========================================================== 21:30:31 (1788658231) oleg336-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=abc: No such file or directory oleg336-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='abc': No such file or directory pdsh@oleg336-client: oleg336-server: ssh exited with exit code 2 oleg336-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc: Invalid argument oleg336-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc': Invalid argument pdsh@oleg336-client: oleg336-server: ssh exited with exit code 22 oleg336-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf reg abc: Invalid argument oleg336-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf reg abc': Invalid argument pdsh@oleg336-client: oleg336-server: ssh exited with exit code 22 oleg336-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc efg: Invalid argument oleg336-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc efg': Invalid argument pdsh@oleg336-client: oleg336-server: ssh exited with exit code 22 PASS 77h (12s) == sanityn test 77i: Change rank of TBF rule ============= 21:30:43 (1788658243) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start before jobid={jobid} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start after jobid={jobid} rate=1000 rank=default ost.OSS.ost_io.nrs_tbf_rule=start target jobid={jobid} rate=1000 rank=after Move before itself ost.OSS.ost_io.nrs_tbf_rule=change target rank=target Move to higher rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=before Move to lower rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=after Move before default ost.OSS.ost_io.nrs_tbf_rule=change target rank=default ost.OSS.ost_io.nrs_policies=fifo PASS 77i (22s) == sanityn test 77j: check TBF-OPCode NRS policy ========= 21:31:06 (1788658266) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf opcode ost.OSS.ost_io.nrs_tbf_rule=start ost_r opcode={ost_read} rate=5 ost.OSS.ost_io.nrs_tbf_rule=start ost_w opcode={ost_write} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.614716 s, 27.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.083132 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0880538 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153864 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172431 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207984 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.260845 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215625 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237344 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.269491 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.252292 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.239877 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.273334 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.263771 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.247582 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.285998 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.358601 s, 2.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff88adc7b35000.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff88adc7b35000.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.300477 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.361356 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.359315 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.317624 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.32396 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.334154 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.383042 s, 2.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.341701 s, 3.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.388945 s, 2.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.355591 s, 2.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.358076 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.371762 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.373277 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.400568 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.387036 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.421905 s, 2.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff88adc7b35000.lock_unused_count=4 removed '/mnt/lustre/d77j.sanityn/nrs_r_oleg336-client.virtnet' removed '/mnt/lustre/d77j.sanityn/nrs_w_oleg336-client.virtnet' removed directory '/mnt/lustre/d77j.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 5 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.11252 s, 12.9 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.5979 s, 5.4 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77j.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_r ost.OSS.ost_io.nrs_tbf_rule=stop ost_w ost.OSS.ost_io.nrs_policies=fifo PASS 77j (65s) == sanityn test 77ja: check TBF-UID/GID NRS policy ======= 21:32:11 (1788658331) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg336-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg336-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg336-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg336-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg336-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.31162 s, 12.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101171 s, 10.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118276 s, 8.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.085838 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0876026 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176678 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137409 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160822 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20492 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176664 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212843 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.27507 s, 3.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184419 s, 5.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196752 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.325949 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.291848 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.304745 s, 3.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0597793 s, 17.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0649856 s, 16.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0452019 s, 23.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0662009 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0625142 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0644028 s, 16.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0627793 s, 16.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0607203 s, 17.3 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0447423 s, 23.4 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0591256 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0458821 s, 22.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0495561 s, 21.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0480614 s, 21.8 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0839362 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0626153 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0670743 s, 15.6 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg336-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg336-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.91 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.0031 s, 5.2 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg336-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg336-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg336-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg336-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg336-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.29091 s, 13.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105183 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14619 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132687 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.224169 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171377 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205625 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.382769 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.378299 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.302785 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.343121 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.380223 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.274892 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.326213 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.410467 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.371533 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.400019 s, 2.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0592684 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0944334 s, 11.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117963 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0690842 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0684358 s, 15.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0732235 s, 14.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0763916 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0799059 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0821695 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102626 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0900539 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0916442 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0996327 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103267 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106442 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143751 s, 7.3 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg336-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg336-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9874 s, 5.2 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9464 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77ja (154s) == sanityn test 77jb: check TBF-UID/GID NRS policy on files that don't belong to us ========================================================== 21:34:45 (1788658485) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg336-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg336-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg336-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg336-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg336-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.33356 s, 12.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132458 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0871518 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191836 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202083 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202655 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108604 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222726 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139015 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.27452 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.228051 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.353332 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.388647 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.251678 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.377887 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.25223 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.316411 s, 3.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0625893 s, 16.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0680709 s, 15.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0634019 s, 16.5 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0627887 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0281059 s, 37.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0786479 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101572 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0622984 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00743307 s, 141 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00588299 s, 178 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0978669 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0778792 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00568051 s, 185 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0857498 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101919 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118872 s, 8.8 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg336-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg336-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.982 s, 5.2 MB/s Write runtime is 22 s, speed is 4.545454 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.95 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg336-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg336-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg336-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg336-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg336-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.30205 s, 12.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0806075 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149224 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141428 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213434 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.244537 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160955 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.218421 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.266361 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.238804 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222181 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.267415 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183207 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.304723 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163749 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.253626 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.24489 s, 4.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.080573 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0711601 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0693285 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0497424 s, 21.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0949495 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0608186 s, 17.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0676447 s, 15.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0720257 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0708312 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0668855 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0896575 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.099846 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134268 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130529 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0920643 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113083 s, 9.3 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg336-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg336-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9123 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.97 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77jb (153s) == sanityn test 77k: check TBF policy with UID/GID/JobID/OPCode expression ========================================================== 21:37:19 (1788658639) ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg336-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.530296 s, 31.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0485535 s, 21.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0710474 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177775 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157681 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207197 s, 5.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.270298 s, 3.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.271917 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.230776 s, 4.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.225805 s, 4.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.270917 s, 3.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.282435 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.288146 s, 3.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.281488 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.281981 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.255289 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.312717 s, 3.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0501607 s, 20.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0558491 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00678773 s, 154 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154743 s, 67.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00944006 s, 111 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153228 s, 68.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0687702 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101548 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0826679 s, 12.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103962 s, 10.1 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116915 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14597 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107739 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112358 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.044858 s, 23.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115035 s, 9.1 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg336-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg336-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.65378 s, 13.7 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.9802 s, 10.5 MB/s Read runtime is 12 s, speed is 8.333333 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.203.36@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.203.36@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.741901 s, 22.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.073638 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0897631 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0861777 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.067646 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0927389 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18112 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21399 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199325 s, 5.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200401 s, 5.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.230665 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.292871 s, 3.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.217909 s, 4.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.229008 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.26365 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.238206 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.324084 s, 3.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0703583 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0991664 s, 10.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0429421 s, 24.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0798114 s, 13.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0619845 s, 16.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0680191 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0939285 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.074201 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103583 s, 10.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128197 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106739 s, 9.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117523 s, 8.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0928048 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0956639 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154033 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100058 s, 10.5 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg336-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg336-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.66697 s, 12.1 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.91552 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.203.36@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg336-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.784689 s, 21.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0415592 s, 25.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0750489 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111222 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0882608 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14241 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145134 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118757 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0372997 s, 28.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143485 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159896 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.230298 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141176 s, 7.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff88adc7b35000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff88adc7b35000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.246302 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.25079 s, 4.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.254542 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.31147 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0933466 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.053261 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.075234 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137175 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0686137 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.057484 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0952561 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0916361 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105284 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103285 s, 10.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167682 s, 6.3 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120532 s, 8.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177557 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110932 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125168 s, 8.4 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131079 s, 8.0 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg336-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg336-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.0725 s, 9.5 MB/s Write runtime is 13 s, speed is 7.692307 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.46347 s, 19.2 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg336-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.929231 s, 18.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0423893 s, 24.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0282077 s, 37.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0668335 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0314463 s, 33.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0276556 s, 37.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0442461 s, 23.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100175 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126513 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183828 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.2034 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188567 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212818 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222702 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192702 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186664 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.287512 s, 3.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.090474 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0304708 s, 34.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0545661 s, 19.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109986 s, 9.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106001 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120401 s, 8.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0989252 s, 10.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106511 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0871221 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109758 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0413997 s, 25.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0741601 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0487654 s, 21.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0531662 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0600492 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0642502 s, 16.3 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg336-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg336-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.5009 s, 10.0 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.93891 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.93224 s, 13.2 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.90183 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg336-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.01518 s, 16.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0809848 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0769008 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0963028 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188225 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169892 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168444 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.324403 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.2977 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.252033 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201462 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.246166 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.356813 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.256195 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.463367 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.324873 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.37306 s, 2.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0459864 s, 22.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0713205 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0722976 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10641 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102839 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0416628 s, 25.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00287545 s, 365 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101045 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12349 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124782 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146349 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113306 s, 9.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152664 s, 68.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102006 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111512 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0338673 s, 31.0 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg336-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg336-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8198 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.6025 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_ug ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg336-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.48006 s, 34.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0525149 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0776883 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0846689 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208739 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212227 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.266921 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.258868 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.284391 s, 3.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.297592 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.282442 s, 3.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.293851 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.28623 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.307756 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.344831 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.348464 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.362126 s, 2.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff88adc7b35000.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff88adc7b35000.lock_unused_count=4 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0571673 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00716007 s, 146 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0428423 s, 24.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152269 s, 68.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0261218 s, 40.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0334135 s, 31.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0305461 s, 34.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0653621 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0379386 s, 27.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0388875 s, 27.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0651592 s, 16.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0647949 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0086946 s, 121 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00356807 s, 294 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0218809 s, 47.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0208712 s, 50.2 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg336-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg336-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.51385 s, 12.3 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.9099 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg336-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.746345 s, 22.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12773 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179497 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119533 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.262339 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203072 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.272793 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.278402 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212893 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.265522 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.239529 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.274533 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.319247 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17376 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.32843 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.318484 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171094 s, 6.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0614268 s, 17.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0730884 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0631608 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0976282 s, 10.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103944 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106616 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0871889 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0744952 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11062 s, 9.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118764 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106483 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0854254 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0971226 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121675 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112724 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132745 s, 7.9 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg336-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg336-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.7098 s, 9.8 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.94384 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.9133 s, 9.6 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.94086 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_policies=fifo PASS 77k (431s) == sanityn test 77kb: Check different granular TBF type combination: jobid+opcode ========================================================== 21:44:29 (1788659069) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/nrs_r_oleg336-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.503383 s, 33.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0544739 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0943688 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203722 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176355 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203823 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.233891 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.27471 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.253723 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242229 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.306808 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.254612 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.301838 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.253562 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.238261 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.275266 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.313539 s, 3.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff88adc7b35000.lock_unused_count=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0371522 s, 28.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0384362 s, 27.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0577063 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0459125 s, 22.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0517777 s, 20.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0900041 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0626019 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.093018 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0674509 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.071369 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0717237 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0726498 s, 14.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0812891 s, 12.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119885 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0843197 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0984143 s, 10.7 MB/s removed '/mnt/lustre/d77kb.sanityn/nrs_r_oleg336-client.virtnet' removed '/mnt/lustre/d77kb.sanityn/nrs_w_oleg336-client.virtnet' removed directory '/mnt/lustre/d77kb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.63586 s, 12.1 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.99822 s, 10.5 MB/s Read runtime is 12 s, speed is 8.333333 IOPS cleanup_tbf /mnt/lustre/d77kb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kb (54s) == sanityn test 77kc: Check different granular TBF type combination: nid+opcode ========================================================== 21:45:23 (1788659123) ost.OSS.ost_io.nrs_policies=tbf nid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.203.36@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.203.36@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.665886 s, 25.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0883225 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0964262 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0907359 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172024 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124593 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193813 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152649 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216135 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172853 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.233738 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.286433 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.236825 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.268411 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178937 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.276886 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.307722 s, 3.4 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff88adc7b35000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.024438 s, 42.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0649622 s, 16.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0493918 s, 21.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0436097 s, 24.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0443657 s, 23.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.029896 s, 35.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0503911 s, 20.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0501612 s, 20.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0508136 s, 20.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0597746 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0674971 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0724585 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0724109 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0777145 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0729995 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.081814 s, 12.8 MB/s removed '/mnt/lustre/d77kc.sanityn/nrs_w_oleg336-client.virtnet' removed '/mnt/lustre/d77kc.sanityn/nrs_r_oleg336-client.virtnet' removed directory '/mnt/lustre/d77kc.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.96087 s, 11.7 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.94079 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kc.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kc (58s) == sanityn test 77kd: Check different granular TBF type combination: nid+jobid ========================================================== 21:46:21 (1788659181) ost.OSS.ost_io.nrs_policies=tbf nid+jobid ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.203.36@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/nrs_r_oleg336-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.61602 s, 27.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.061425 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0597807 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0666398 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0530282 s, 19.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.090344 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137911 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126781 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136583 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0636113 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.271273 s, 3.9 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff88adc7b35000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.315565 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208122 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.265888 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.274964 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.335392 s, 3.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.29589 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0607812 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0631664 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0547439 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106725 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0258713 s, 40.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112124 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0598645 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0725423 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0530234 s, 19.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0632019 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0942846 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0873981 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0744542 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00301547 s, 348 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.022121 s, 47.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00876994 s, 120 MB/s removed '/mnt/lustre/d77kd.sanityn/nrs_w_oleg336-client.virtnet' removed '/mnt/lustre/d77kd.sanityn/nrs_r_oleg336-client.virtnet' removed directory '/mnt/lustre/d77kd.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.82122 s, 13.4 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kd.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.35027 s, 19.6 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kd.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kd (48s) == sanityn test 77ke: Check different granular TBF type combination: jobid+opcode ========================================================== 21:47:09 (1788659229) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/nrs_r_oleg336-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.677864 s, 24.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0899762 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0855722 s, 12.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209428 s, 5.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1637 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221218 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.25204 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.263851 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.301066 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.330619 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.350408 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.347177 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.288766 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.379133 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.385875 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.362177 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.393545 s, 2.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0326846 s, 32.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0367022 s, 28.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0246644 s, 42.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0500098 s, 21.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0422755 s, 24.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0305178 s, 34.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0312657 s, 33.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0484028 s, 21.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0408082 s, 25.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0390403 s, 26.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0426987 s, 24.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0442253 s, 23.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0460884 s, 22.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.054668 s, 19.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0486311 s, 21.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0527981 s, 19.9 MB/s removed '/mnt/lustre/d77ke.sanityn/nrs_r_oleg336-client.virtnet' removed '/mnt/lustre/d77ke.sanityn/nrs_w_oleg336-client.virtnet' removed directory '/mnt/lustre/d77ke.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.1349 s, 9.4 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ke.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.90389 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.28409 s, 11.3 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0038 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77ke (97s) == sanityn test 77kf: Check different granular TBF type combination: uid+gid ========================================================== 21:48:46 (1788659326) ost.OSS.ost_io.nrs_policies=tbf uid+gid ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/nrs_r_oleg336-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.08155 s, 15.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0406062 s, 25.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0532042 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0759481 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0831178 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0771312 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216689 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136403 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.225348 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.226348 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193537 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186304 s, 5.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.270522 s, 3.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18722 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.272738 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216391 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223913 s, 4.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0461746 s, 22.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.075269 s, 13.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0857149 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0614396 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0777012 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0830605 s, 12.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0814101 s, 12.9 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113948 s, 9.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1228 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132122 s, 7.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100687 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0963961 s, 10.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119491 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12143 s, 8.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120757 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137583 s, 7.6 MB/s removed '/mnt/lustre/d77kf.sanityn/nrs_r_oleg336-client.virtnet' removed '/mnt/lustre/d77kf.sanityn/nrs_w_oleg336-client.virtnet' removed directory '/mnt/lustre/d77kf.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8366 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kf.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9576 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77kf.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kf (75s) == sanityn test 77kg: check different granular TBF type combination: uid+opcode ========================================================== 21:50:01 (1788659401) ost.OSS.ost_io.nrs_policies=tbf uid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg336-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.496243 s, 33.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0604018 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0218288 s, 48.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0687806 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143916 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174823 s, 6.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20805 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212969 s, 4.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.248458 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.218739 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216851 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237191 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.245958 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.231781 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.218992 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.252316 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.230471 s, 4.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0406008 s, 25.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0418067 s, 25.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.04518 s, 23.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0620655 s, 16.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0404711 s, 25.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0375878 s, 27.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0662698 s, 15.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0498959 s, 21.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0481264 s, 21.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0474296 s, 22.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0561704 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0572747 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0565077 s, 18.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0463573 s, 22.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0581099 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.072131 s, 14.5 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg336-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg336-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.6954 s, 13.6 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.97209 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg336-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.653594 s, 25.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0577091 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0684589 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117719 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101105 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135459 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145994 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.27007 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.248328 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.243396 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.272437 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.288453 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.260159 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.430473 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.449341 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.473782 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.491947 s, 2.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0912072 s, 11.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0391469 s, 26.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0792034 s, 13.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0192232 s, 54.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0774914 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0581589 s, 18.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0957917 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109526 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0761474 s, 13.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104116 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0932807 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0901612 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0799346 s, 13.1 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0989143 s, 10.6 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0665175 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0977499 s, 10.7 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg336-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg336-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.7293 s, 9.8 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.89562 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.6672 s, 9.8 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.95542 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kg (145s) == sanityn test 77kh: Verify that Project ID support for NRS TBF rule ========================================================== 21:52:26 (1788659546) using SAVE_PROJECT_SUPPORTED=0 Stopping clients: oleg336-client.virtnet /mnt/lustre (opts:) Stopping client oleg336-client.virtnet /mnt/lustre opts: Stopping clients: oleg336-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg336-client.virtnet /mnt/lustre2 opts: Stopping /mnt/lustre-mds1 (opts:-f) on oleg336-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg336-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg336-server sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime) proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) devtmpfs on /dev type devtmpfs (rw,nosuid,mode=755) securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime) tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev) devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000) tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755) tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755) cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd) pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime) bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700) cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory) cgroup on /sys/fs/cgroup/misc type cgroup (rw,nosuid,nodev,noexec,relatime,misc) cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset) cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer) cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids) cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event) cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma) cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio) cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct) cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb) cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices) cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio) none on /sys/kernel/tracing type tracefs (rw,relatime) configfs on /sys/kernel/config type configfs (rw,relatime) /dev/nbd0 on / type ext4 (ro,relatime,stripe=32) rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) mqueue on /dev/mqueue type mqueue (rw,relatime) debugfs on /sys/kernel/debug type debugfs (rw,relatime) hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M) none on /mnt type ramfs (rw,relatime) /dev/vda on /home/green/git/lustre-release type squashfs (ro,relatime) none on /var/lib/stateless/writable type tmpfs (rw,relatime) none on /var/cache/man type tmpfs (rw,relatime) none on /var/log type tmpfs (rw,relatime) none on /var/lib/dbus type tmpfs (rw,relatime) none on /tmp type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) none on /var/tmp type tmpfs (rw,relatime) none on /var/lib/NetworkManager type tmpfs (rw,relatime) none on /var/lib/systemd/random-seed type tmpfs (rw,relatime) none on /var/spool type tmpfs (rw,relatime) none on /var/lib/nfs type tmpfs (rw,relatime) none on /var/lib/gssproxy type tmpfs (rw,relatime) none on /var/lib/logrotate type tmpfs (rw,relatime) none on /etc type tmpfs (rw,relatime) none on /var/lib/rsyslog type tmpfs (rw,relatime) none on /root/.ssh type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) 192.168.200.253:/exports/state/oleg336-client.virtnet on /var/lib/stateless/state type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.203.36,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg336-client.virtnet/boot on /boot type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.203.36,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg336-client.virtnet/etc/kdump.conf on /etc/kdump.conf type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.203.36,local_lock=none,addr=192.168.200.253) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=378180k,mode=700) 192.168.200.253:/exports/testreports/69622/testresults/sanityn-zfs-rocky8.10_x86_64-rocky8.10_x86_64 on /tmp/testlogs type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.203.36,local_lock=none,addr=192.168.200.253) /dev/vda on /usr/sbin/mount.lustre type squashfs (ro,relatime) Checking servers environments Checking clients oleg336-client.virtnet environments Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg336-server' oleg336-server: oleg336-server.virtnet: executing load_modules_local oleg336-server: kmemleak disabled oleg336-server: Loading modules from /home/green/git/lustre-release/lustre oleg336-server: detected 4 online CPUs by sysfs oleg336-server: MODOPTS_LIBCFS= oleg336-server: Force libcfs to create 2 CPU partitions oleg336-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Setup mgs, mdt, osts Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg336-server: oleg336-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg336-client: oleg336-server: ssh exited with exit code 1 Started lustre-MDT0000 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg336-server: oleg336-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg336-client: oleg336-server: ssh exited with exit code 1 Started lustre-OST0000 Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg336-server: oleg336-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg336-client: oleg336-server: ssh exited with exit code 1 Started lustre-OST0001 Starting client: oleg336-client.virtnet: -o user_xattr,flock 192.168.203.136@tcp:/lustre /mnt/lustre Starting client oleg336-client.virtnet: -o user_xattr,flock 192.168.203.136@tcp:/lustre /mnt/lustre Started clients oleg336-client.virtnet: 192.168.203.136@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Starting client: oleg336-client.virtnet: -o user_xattr,flock 192.168.203.136@tcp:/lustre /mnt/lustre2 Starting client oleg336-client.virtnet: -o user_xattr,flock 192.168.203.136@tcp:/lustre /mnt/lustre2 Started clients oleg336-client.virtnet: 192.168.203.136@tcp:/lustre on /mnt/lustre2 type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff88adc7508800.idle_timeout=debug osc.lustre-OST0000-osc-ffff88add1a6e800.idle_timeout=debug osc.lustre-OST0001-osc-ffff88adc7508800.idle_timeout=debug osc.lustre-OST0001-osc-ffff88add1a6e800.idle_timeout=debug disable quota as required ost.OSS.ost_io.nrs_policies=tbf projid ost.OSS.ost_io.nrs_tbf_rule=start ext_proj_rw projid={100} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.642243 s, 26.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0822676 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114673 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142592 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0875247 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207018 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0500545 s, 20.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157498 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23032 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189797 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.267074 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.279077 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.267666 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.300689 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.234961 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.264664 s, 4.0 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff88adc7508800.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.375386 s, 2.8 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff88adc7508800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.257745 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.265636 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.25668 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.285209 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.295881 s, 3.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.294107 s, 3.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.258059 s, 4.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.277647 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.313213 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.325923 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.300303 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.362689 s, 2.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.305055 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.306929 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.292395 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.321398 s, 3.3 MB/s removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg336-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg336-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.74955 s, 10.8 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.32333 s, 19.7 MB/s Read runtime is 7 s, speed is 14.285714 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_proj_rw ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf projid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w projid={100}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r projid={100}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.869955 s, 19.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0865793 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127765 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177913 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175947 s, 6.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242296 s, 4.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.264794 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.273497 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.305053 s, 3.4 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.290806 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.288861 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.355339 s, 3.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.378442 s, 2.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.352639 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.409361 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.403674 s, 2.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff88adc7508800.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff88adc7508800.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.4471 s, 2.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0669344 s, 15.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166242 s, 6.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163585 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0899663 s, 11.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0983429 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186359 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139419 s, 7.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211401 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14891 s, 7.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106861 s, 9.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113144 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.230975 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123192 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121556 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129677 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143271 s, 7.3 MB/s removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg336-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg336-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.43304 s, 12.4 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.9508 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77kh (211s) == sanityn test 77ki: Add rule with unsupported TBF type should fail for generic TBF ========================================================== 21:55:58 (1788659758) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 oleg336-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_fail projid={100}&opcode={ost_write} rate=20: Invalid argument oleg336-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_fail projid={100}&opcode={ost_write} rate=20': Invalid argument pdsh@oleg336-client: oleg336-server: ssh exited with exit code 22 ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77ki (18s) == sanityn test 77kj: Verify nodemap support for NRS TBF rule ========================================================== 21:56:17 (1788659777) On MGS 192.168.203.136, active = nodemap.active=1 On MGS 192.168.203.136, TBF = ost.OSS.ost_io.nrs_policies=tbf nodemap ost.OSS.ost_io.nrs_tbf_rule=start ext_nm_rw nodemap={TBF} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.656846 s, 25.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0531 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0688766 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205685 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242695 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.250139 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.25319 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.265005 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.268327 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.248334 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.258246 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.304599 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.236764 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23669 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.240035 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.311051 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.294508 s, 3.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff88adc7508800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff88adc7508800.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.089349 s, 11.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0779343 s, 13.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108086 s, 9.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102848 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106458 s, 9.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12728 s, 8.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128812 s, 8.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0953426 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189962 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106893 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143928 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117286 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159249 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153699 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136588 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124088 s, 8.5 MB/s removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg336-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg336-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.4208 s, 10.1 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.21208 s, 20.1 MB/s Read runtime is 7 s, speed is 14.285714 IOPS cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete On MGS 192.168.203.136, TBF.id = On MGS 192.168.203.136, active = nodemap.active=0 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.55632 s, 30.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114502 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118183 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119277 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187152 s, 5.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.244705 s, 4.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.253373 s, 4.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.262985 s, 4.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.253641 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203913 s, 5.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.277693 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.236782 s, 4.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221701 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.252568 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223805 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.311232 s, 3.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff88adc7508800.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff88adc7508800.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.264924 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0511833 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0432184 s, 24.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0248115 s, 42.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0447849 s, 23.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0273572 s, 38.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0445689 s, 23.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0388372 s, 27.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0578899 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0373758 s, 28.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0694693 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0564564 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0486514 s, 21.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0686988 s, 15.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0790504 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0771536 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0684512 s, 15.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff88adc7508800.lock_unused_count=2 removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg336-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg336-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 1 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.46496 s, 14.0 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 2.92965 s, 35.8 MB/s Read runtime is 4 s, speed is 25.000000 IOPS read rate (25.000000) should exceed limit (20 * 1) cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_nm_rw ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf nodemap+opcode oleg336-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_w nodemap={TBF}&opcode={ost_write} rate=20: Invalid argument oleg336-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_w nodemap={TBF}&opcode={ost_write} rate=20': Invalid argument oleg336-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_r nodemap={TBF}&opcode={ost_read} rate=10: Invalid argument oleg336-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_r nodemap={TBF}&opcode={ost_read} rate=10': Invalid argument pdsh@oleg336-client: oleg336-server: ssh exited with exit code 22 On MGS 192.168.203.136, active = nodemap.active=1 On MGS 192.168.203.136, TBF = ost.OSS.ost_io.nrs_tbf_rule=start ext_w nodemap={TBF}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nodemap={TBF}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.584135 s, 28.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0511141 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.059205 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0264179 s, 39.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.253966 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.281663 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.28589 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.290558 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.337999 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.334619 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.357872 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.388675 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.392221 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.385569 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.361396 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.399221 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.419961 s, 2.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0905636 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0733471 s, 14.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111731 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0894479 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0802447 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137977 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0585494 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0511101 s, 20.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125911 s, 8.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0515432 s, 20.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0683505 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0646001 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108677 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159889 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115216 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0786019 s, 13.3 MB/s removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg336-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg336-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.5985 s, 9.9 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.89623 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo On MGS 192.168.203.136, TBF.id = On MGS 192.168.203.136, active = nodemap.active=0 PASS 77kj (187s) == sanityn test 77l: check the output of NRS policies for generic TBF ========================================================== 21:59:24 (1788659964) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_policies=fifo PASS 77l (10s) == sanityn test 77m: check NRS Delay slows write RPC processing ========================================================== 21:59:34 (1788659974) ost.OSS.ost_io.nrs_policies=delay ost.OSS.ost_io.nrs_delay_min=4 ost.OSS.ost_io.nrs_delay_max=4 ost.OSS.ost_io.nrs_delay_pct=100 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg336-client.virtnet] [bs=1M] [count=1] [oflag=direct] [conv=fdatasync] 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 3.94467 s, 266 kB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg336-client.virtnet] [bs=1M] [count=10] [oflag=direct] [conv=fdatasync] 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 40.0598 s, 262 kB/s ost.OSS.ost_io.nrs_policies=fifo PASS 77m (58s) == sanityn test 77n: check wildcard support for TBF JobID NRS policy ========================================================== 22:00:32 (1788660032) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={*.500} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.385237 s, 43.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.05503 s, 19.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0553745 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0704023 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133908 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208748 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128265 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198632 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165098 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.248827 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197999 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.254726 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.230362 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.229419 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227247 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198866 s, 5.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff88adc7508800.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff88adc7508800.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.283467 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.063609 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0352155 s, 29.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.036197 s, 29.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0400699 s, 26.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0585461 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0443923 s, 23.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100622 s, 10.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0470032 s, 22.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0506118 s, 20.7 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0508713 s, 20.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117476 s, 8.9 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0571992 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0700348 s, 15.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0586758 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0647586 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0656918 s, 16.0 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg336-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg336-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77n.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.05264 s, 13.0 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77n.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.15837 s, 20.3 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.*} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.612958 s, 27.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0281589 s, 37.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0898781 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0658507 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101418 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.077008 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.230433 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.262025 s, 4.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.294269 s, 3.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.302538 s, 3.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.359022 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.362837 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.360953 s, 2.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.359398 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.380579 s, 2.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff88adc7508800.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffff88adc7508800.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.412104 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.470917 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.296084 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.273528 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.344429 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.280464 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.32722 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.283364 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.354841 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.29405 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.317615 s, 3.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.323362 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.333496 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.347347 s, 3.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.335633 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.358463 s, 2.9 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff88adc7508800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.374681 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.373981 s, 2.8 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg336-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg336-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.39156 s, 12.5 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.13892 s, 20.4 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_policies=fifo PASS 77n (91s) == sanityn test 77o: Changing rank should not panic ====== 22:02:04 (1788660124) mds.MDS.mdt.nrs_policies=tbf nid mds.MDS.mdt.nrs_tbf_rule=start name nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=start name1 nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=change name1 rank=name mds.MDS.mdt.nrs_tbf_rule=stop name mds.MDS.mdt.nrs_policies=fifo PASS 77o (12s) == sanityn test 77q: Parallel TBF rule definitions should not panic ========================================================== 22:02:16 (1788660136) mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_policies=fifo PASS 77q (110s) == sanityn test 77p: Check validity of rule names for TBF policies ========================================================== 22:04:06 (1788660246) mds.MDS.mdt.nrs_policies=tbf oleg336-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_overflo uid={500} rate=500: File name too long oleg336-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_overflo uid={500} rate=500': File name too long pdsh@oleg336-client: oleg336-server: ssh exited with exit code 36 oleg336-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start uid={500} rate=500: Invalid argument oleg336-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start uid={500} rate=500': Invalid argument pdsh@oleg336-client: oleg336-server: ssh exited with exit code 22 oleg336-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_empty: Invalid argument oleg336-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_empty': Invalid argument pdsh@oleg336-client: oleg336-server: ssh exited with exit code 22 oleg336-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p@spec uid={500} rate=500: Invalid argument oleg336-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p@spec uid={500} rate=500': Invalid argument pdsh@oleg336-client: oleg336-server: ssh exited with exit code 22 oleg336-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p.spec uid={500} rate=500: Invalid argument oleg336-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p.spec uid={500} rate=500': Invalid argument pdsh@oleg336-client: oleg336-server: ssh exited with exit code 22 oleg336-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p~spec uid={500} rate=500: Invalid argument oleg336-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p~spec uid={500} rate=500': Invalid argument pdsh@oleg336-client: oleg336-server: ssh exited with exit code 22 oleg336-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p#spec uid={500} rate=500: Invalid argument oleg336-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p#spec uid={500} rate=500': Invalid argument pdsh@oleg336-client: oleg336-server: ssh exited with exit code 22 oleg336-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p/spec uid={500} rate=500: Invalid argument oleg336-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p/spec uid={500} rate=500': Invalid argument pdsh@oleg336-client: oleg336-server: ssh exited with exit code 22 oleg336-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p^spec uid={500} rate=500: Invalid argument oleg336-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p^spec uid={500} rate=500': Invalid argument pdsh@oleg336-client: oleg336-server: ssh exited with exit code 22 oleg336-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p%spec uid={500} rate=500: Invalid argument oleg336-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p%spec uid={500} rate=500': Invalid argument pdsh@oleg336-client: oleg336-server: ssh exited with exit code 22 oleg336-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p*spec uid={500} rate=500: Invalid argument oleg336-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p*spec uid={500} rate=500': Invalid argument pdsh@oleg336-client: oleg336-server: ssh exited with exit code 22 oleg336-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p;spec uid={500} rate=500: Invalid argument oleg336-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p;spec uid={500} rate=500': Invalid argument pdsh@oleg336-client: oleg336-server: ssh exited with exit code 22 oleg336-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p,spec uid={500} rate=500: Invalid argument oleg336-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p,spec uid={500} rate=500': Invalid argument pdsh@oleg336-client: oleg336-server: ssh exited with exit code 22 oleg336-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p?spec uid={500} rate=500: Invalid argument oleg336-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p?spec uid={500} rate=500': Invalid argument pdsh@oleg336-client: oleg336-server: ssh exited with exit code 22 oleg336-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77pspec uid={500} rate=500: Invalid argument oleg336-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p>spec uid={500} rate=500': Invalid argument pdsh@oleg336-client: oleg336-server: ssh exited with exit code 22 oleg336-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p:spec uid={500} rate=500: Invalid argument oleg336-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p:spec uid={500} rate=500': Invalid argument pdsh@oleg336-client: oleg336-server: ssh exited with exit code 22 oleg336-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p+spec uid={500} rate=500: Invalid argument oleg336-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p+spec uid={500} rate=500': Invalid argument pdsh@oleg336-client: oleg336-server: ssh exited with exit code 22 oleg336-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p=spec uid={500} rate=500: Invalid argument oleg336-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p=spec uid={500} rate=500': Invalid argument pdsh@oleg336-client: oleg336-server: ssh exited with exit code 22 oleg336-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p)spec uid={500} rate=500: Invalid argument oleg336-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p)spec uid={500} rate=500': Invalid argument pdsh@oleg336-client: oleg336-server: ssh exited with exit code 22 oleg336-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p(spec uid={500} rate=500: Invalid argument oleg336-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p(spec uid={500} rate=500': Invalid argument pdsh@oleg336-client: oleg336-server: ssh exited with exit code 22 oleg336-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p{spec uid={500} rate=500: Invalid argument oleg336-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p{spec uid={500} rate=500': Invalid argument pdsh@oleg336-client: oleg336-server: ssh exited with exit code 22 oleg336-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p}spec uid={500} rate=500: Invalid argument oleg336-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p}spec uid={500} rate=500': Invalid argument pdsh@oleg336-client: oleg336-server: ssh exited with exit code 22 oleg336-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p|spec uid={500} rate=500: Invalid argument oleg336-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p|spec uid={500} rate=500': Invalid argument pdsh@oleg336-client: oleg336-server: ssh exited with exit code 22 oleg336-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p[spec uid={500} rate=500: Invalid argument oleg336-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p[spec uid={500} rate=500': Invalid argument pdsh@oleg336-client: oleg336-server: ssh exited with exit code 22 oleg336-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p]spec uid={500} rate=500: Invalid argument oleg336-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p]spec uid={500} rate=500': Invalid argument pdsh@oleg336-client: oleg336-server: ssh exited with exit code 22 oleg336-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p!spec uid={500} rate=500: Invalid argument oleg336-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p!spec uid={500} rate=500': Invalid argument pdsh@oleg336-client: oleg336-server: ssh exited with exit code 22 oleg336-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p&spec uid={500} rate=500: Invalid argument oleg336-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p&spec uid={500} rate=500': Invalid argument pdsh@oleg336-client: oleg336-server: ssh exited with exit code 22 oleg336-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p$spec uid={500} rate=500: Invalid argument oleg336-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p$spec uid={500} rate=500': Invalid argument pdsh@oleg336-client: oleg336-server: ssh exited with exit code 22 oleg336-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p`spec uid={500} rate=500: Invalid argument oleg336-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p`spec uid={500} rate=500': Invalid argument pdsh@oleg336-client: oleg336-server: ssh exited with exit code 22 oleg336-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p\spec uid={500} rate=500: Invalid argument oleg336-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p\spec uid={500} rate=500': Invalid argument pdsh@oleg336-client: oleg336-server: ssh exited with exit code 22 mds.MDS.mdt.nrs_policies=fifo PASS 77p (33s) == sanityn test 77r: Change type of tbf policy at run time ========================================================== 22:04:39 (1788660279) mds.MDS.mdt.nrs_policies=tbf uid mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf gid mds.MDS.mdt.nrs_tbf_rule=start md_rule gid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start md_rule jobid={*.500} rate=1 mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=fifo check the number of requests in queue: fifo: 0 crrn: 0 tbf: 0 delay: 0 fifo: 0 crrn: 0 tbf: 0 delay: 0 cleanup 77r 273814 kill md_thread xargs PASS 77r (54s) == sanityn test 77s: Check TBF LRU shrinker ============== 22:05:34 (1788660334) jobid_name=%e.%p jobid_var=TEST_77s_JOBVAR mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start touch jobid={touch.*} rate=1000 mds.MDS.mdt.nrs_tbf_rule=start unlink jobid={unlink.*} rate=1000 touch jobid={touch.*} 1000, ref 9 touch jobid={touch.*} 1000, ref 0 unlink jobid={unlink.*} 1000, ref 9 unlink jobid={unlink.*} 1000, ref 0 mds.MDS.mdt.nrs_policies=fifo jobid_var=procname_uid jobid_name=%e.%u PASS 77s (66s) == sanityn test 78: Enable policy and specify tunings right away ========================================================== 22:06:39 (1788660399) oleg336-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_orr_quantum=1: No such device oleg336-server: error: set_param: setting 'ost/OSS/*/nrs_orr_quantum'='1': No such device pdsh@oleg336-client: oleg336-server: ssh exited with exit code 19 ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_policies=fifo PASS 78 (9s) == sanityn test 79: xattr: intent error ================== 22:06:48 (1788660408) File: /mnt/lustre/d79.sanityn Size: 16384 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115238826935667 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-05 22:06:50.000000000 -0400 Modify: 2026-09-05 22:06:50.000000000 -0400 Change: 2026-09-05 22:06:50.000000000 -0400 Birth: 2026-09-05 22:06:50.000000000 -0400 fail_val=0 fail_loc=0x80000160 fail_val=0 fail_loc=0x80000131 # file: mnt/lustre/d79.sanityn trusted.name1="value1" PASS 79 (17s) == sanityn test 80a: migrate directory when some children is being opened ========================================================== 22:07:06 (1788660426) SKIP: sanityn test_80a needs >= 2 MDTs SKIP 80a (3s) == sanityn test 80b: Accessing directory during migration ========================================================== 22:07:09 (1788660429) SKIP: sanityn test_80b needs >= 2 MDTs SKIP 80b (4s) == sanityn test 81a: rename and stat under striped directory ========================================================== 22:07:12 (1788660432) SKIP: sanityn test_81a needs >= 2 MDTs SKIP 81a (3s) == sanityn test 81b: rename under striped directory doesn't deadlock ========================================================== 22:07:15 (1788660435) SKIP: sanityn test_81b We need at least 2 MDTs for this test SKIP 81b (3s) == sanityn test 81c: rename revoke LOOKUP lock for remote object ========================================================== 22:07:18 (1788660438) SKIP: sanityn test_81c needs >= 4 MDTs SKIP 81c (3s) == sanityn test 81d: parallel rename file cross-dir on same MDT ========================================================== 22:07:21 (1788660441) - open/close 609 (time 1788660457.35 total 10.00 last 60.87) - open/close 1247 (time 1788660467.37 total 20.02 last 63.68) - open/close 1901 (time 1788660477.37 total 30.03 last 65.35) total: 2500 open/close in 38.94 seconds: 64.20 ops/second snapshot_time 1788660579.826473849 secs.nsecs start_time 1788660444.402662411 secs.nsecs elapsed_time 135.423811438 secs.nsecs open 2505 samples [usecs] 128 17050 4234485 12990569053 close 2504 samples [usecs] 183 18132 1027588 1338850046 mknod 2501 samples [usecs] 593 16865 3728288 10740223898 unlink 2500 samples [usecs] 770 104239 6529097 102219139943 mkdir 2 samples [usecs] 3625 4123 7748 30139754 rmdir 2 samples [usecs] 4373 14422 18795 227117213 rename 2500 samples [usecs] 485 86508 19896660 350402679494 getattr 3761 samples [usecs] 7 21302 344109 1882544929 parallel_rename_file 2500 samples [usecs] 485 86508 19896660 350402679494 crossdir_rename 2500 samples [usecs] 485 86508 19896660 350402679494 readpage 33 samples [usecs] 11748 85704 1031310 42601655704 PASS 81d (143s) == sanityn test 82: fsetxattr and fgetxattr on orphan files ========================================================== 22:09:44 (1788660584) multiop /mnt/lustre/f82.sanityn vO_ac TMPPIPE=/tmp/multiop_open_wait_pipe.7631 multiop /mnt/lustre2/f82.sanityn vo_Ac TMPPIPE=/tmp/multiop_open_wait_pipe.7631 PASS 82 (7s) == sanityn test 83: access striped directory while it is being created/unlinked ========================================================== 22:09:51 (1788660591) SKIP: sanityn test_83 needs >= 2 MDTs SKIP 83 (4s) == sanityn test 84: 0-nlink race in lu_object_find() ===== 22:09:55 (1788660595) fail_loc=0x8000060b PASS 84 (12s) == sanityn test 85: Lustre API root cache race =========== 22:10:07 (1788660607) Starting: llapi_root_test /mnt/lustre/f85.sanityn /mnt/lustre2/f85.sanityn Starting test test1 at 1788660609 Finishing test test1 at 1788660613 PASS 85 (10s) == sanityn test 90: open/create and unlink striped directory ========================================================== 22:10:18 (1788660618) SKIP: sanityn test_90 needs >= 2 MDTs SKIP 90 (4s) == sanityn test 91: chmod and unlink striped directory === 22:10:21 (1788660621) SKIP: sanityn test_91 needs >= 2 MDTs SKIP 91 (3s) == sanityn test 92: create remote directory under orphan directory ========================================================== 22:10:24 (1788660624) SKIP: sanityn test_92 needs >= 2 MDTs SKIP 92 (3s) == sanityn test 93: alloc_rr should not allocate on same ost ========================================================== 22:10:28 (1788660628) fail_loc=0x00000163 fail_loc=0x0 /mnt/lustre/f93.sanityn-1/file1 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5800 0x16a8 0x240000400 1 5760 0x1680 0x280000400 /mnt/lustre2/f93.sanityn-2/file2 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5801 0x16a9 0x240000400 1 5761 0x1681 0x280000400 rc1=0 and rc2=0 PASS 93 (17s) == sanityn test 94: signal vs CP callback race =========== 22:10:45 (1788660645) osc.lustre-OST0000-osc-ffff88adc7508800.idle_timeout=0 osc.lustre-OST0000-osc-ffff88add1a6e800.idle_timeout=0 osc.lustre-OST0001-osc-ffff88adc7508800.idle_timeout=0 osc.lustre-OST0001-osc-ffff88add1a6e800.idle_timeout=0 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.262234 s, 31.2 kB/s fail_val=5 fail_loc=0x80000312 fail_val=6 fail_loc=0x80000329 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 2.9984 s, 1.4 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 6389: 290688 Alarm clock dd if=/dev/zero of=$DIR/$tfile conv=notrunc oflag=append bs=4k count=1 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.104517 s, 39.2 kB/s osc.lustre-OST0000-osc-ffff88adc7508800.idle_timeout=debug osc.lustre-OST0000-osc-ffff88add1a6e800.idle_timeout=debug osc.lustre-OST0001-osc-ffff88adc7508800.idle_timeout=debug osc.lustre-OST0001-osc-ffff88add1a6e800.idle_timeout=debug PASS 94 (15s) == sanityn test 95a: Check readpage() on a page that was removed from page cache ========================================================== 22:11:00 (1788660660) llite.lustre-ffff88adc7508800.fast_read=0 llite.lustre-ffff88add1a6e800.fast_read=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0902139 s, 23.2 MB/s fail_loc=0x80001421 fail_val=7 fail_loc=0x80001422 fail_val=10 llite.lustre-ffff88adc7508800.fast_read=1 llite.lustre-ffff88add1a6e800.fast_read=1 PASS 95a (21s) == sanityn test 95b: Check readpage() on a page that is no longer uptodate ========================================================== 22:11:21 (1788660681) llite.lustre-ffff88adc7508800.fast_read=0 llite.lustre-ffff88add1a6e800.fast_read=0 1+0 records in 1+0 records out 12288 bytes (12 kB, 12 KiB) copied, 0.00601063 s, 2.0 MB/s fail_loc=0x80001424 fail_val=5 fail_loc=0 llite.lustre-ffff88adc7508800.fast_read=1 llite.lustre-ffff88add1a6e800.fast_read=1 PASS 95b (12s) == sanityn test 100a: DoM: glimpse RPCs for stat without IO lock (DoM only file) ========================================================== 22:11:33 (1788660693) SKIP: sanityn test_100a Reserved for glimpse-ahead SKIP 100a (4s) == sanityn test 100b: DoM: no glimpse RPC for stat with IO lock (DoM only file) ========================================================== 22:11:37 (1788660697) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00975428 s, 420 kB/s /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK PASS 100b (7s) == sanityn test 100c: DoM: write vs stat without IO lock (combined file) ========================================================== 22:11:44 (1788660704) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.20876 s, 10.0 MB/s /mnt/lustre/d100c.sanityn/dom has type file OK /mnt/lustre/d100c.sanityn/dom has size 2097152 OK PASS 100c (8s) == sanityn test 100d: DoM: write+truncate vs stat without IO lock (combined file) ========================================================== 22:11:52 (1788660712) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.250369 s, 8.4 MB/s /mnt/lustre/d100d.sanityn/dom has type file OK /mnt/lustre/d100d.sanityn/dom has size 4096 OK PASS 100d (8s) == sanityn test 100e: DoM: read on open and file size ==== 22:12:00 (1788660720) 1+0 records in 1+0 records out 12000 bytes (12 kB, 12 KiB) copied, 0.0164865 s, 728 kB/s PASS 100e (7s) == sanityn test 101a: Discard DoM data on unlink ========= 22:12:08 (1788660728) /mnt/lustre/f101a.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0181709 s, 225 kB/s PASS 101a (6s) == sanityn test 101b: Discard DoM data on rename ========= 22:12:15 (1788660735) /mnt/lustre/f101b.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.024139 s, 170 kB/s PASS 101b (8s) == sanityn test 101c: Discard DoM data on close-unlink === 22:12:23 (1788660743) /mnt/lustre/f101c.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0175551 s, 233 kB/s PASS 101c (8s) == sanityn test 102: Test open by handle of unlinked file ========================================================== 22:12:31 (1788660751) file: /mnt/lustre/f102.sanityn fh_bytes: 32 fh_type: 151 fh_data: d1 0b 00 00 02 00 00 00 54 0f 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 file's parent FID is [0:0x0:0x0] file FID is [0x200000bd1:0xf54:0x0] access via mount point '/mnt/lustre2' - OK access after unlink - OK check_fhandle_syscalls test Passed! PASS 102 (7s) == sanityn test 103: Test size correctness with lockahead ========================================================== 22:12:38 (1788660758) fail_loc=0x415 fail_loc=0x415 fail_val=2 Incorrect size expected (no glimpse fix): Starting test test23 at 1788660761 lockahead_test: lockahead_test.c:1171: test23: assertion 'sb.st_size == sb2.st_size' failed: size on /mnt/lustre/d103.sanityn/f103.sanityn and /mnt/lustre2/d103.sanityn/f103.sanityn differs: 0 vs 1048576 fail_loc=0 fail_loc=0x214 fail_val=2 Starting test test23 at 1788660764 Finishing test test23 at 1788660764 PASS 103 (15s) == sanityn test 104: Verify that MDS stores atime/mtime/ctime during close ========================================================== 22:12:54 (1788660774) SKIP: sanityn test_104 ldiskfs only test SKIP 104 (4s) == sanityn test 105: Glimpse and lock cancel race ======== 22:12:57 (1788660777) fail_loc=0x416 PASS 105 (24s) == sanityn test 106a: Verify the btime via statx() ======= 22:13:21 (1788660801) SKIP: sanityn test_106a Test only for ldiskfs and statx() supported SKIP 106a (3s) == sanityn test 106b: Glimpse RPCs test for statx ======== 22:13:24 (1788660804) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0195077 s, 52.5 kB/s File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238826938203 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-05 22:13:26.000000000 -0400 Modify: 2026-09-05 22:13:26.000000000 -0400 Change: 2026-09-05 22:13:26.000000000 -0400 Birth: 2026-09-05 22:13:26.000000000 -0400 /mnt/lustre/f106b.sanityn 144115238826938203 -rw-r--r-- 1 0 0 1788660806 1788660806 1788660806 File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238826938203 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-05 22:13:26.000000000 -0400 Modify: 2026-09-05 22:13:26.000000000 -0400 Change: 2026-09-05 22:13:26.000000000 -0400 Birth: 2026-09-05 22:13:26.000000000 -0400 1788660806 1024 1 PASS 106b (8s) == sanityn test 106c: Verify statx attributes mask ======= 22:13:32 (1788660812) PASS 106c (8s) == sanityn test 107a: Basic grouplock conflict =========== 22:13:40 (1788660820) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.308882 s, 33.9 MB/s /mnt/lustre/f107a.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5810 0x16b2 0x240000400 multiop /mnt/lustre/f107a.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7631 multiop /mnt/lustre2/f107a.sanityn vO_G16022000r10g16022000c TMPPIPE=/tmp/multiop_open_wait_pipe.7631 First grouplock blocks second one PASS 107a (9s) == sanityn test 107b: Grouplock is added to the head of waiting list ========================================================== 22:13:49 (1788660829) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.241109 s, 43.5 MB/s /mnt/lustre/f107b.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 5768 0x1688 0x280000400 multiop /mnt/lustre/f107b.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7631 Grouplock blocks IO First grouplock blocks second one Second grouplock blocks IO PASS 107b (13s) == sanityn test 108a: lseek: parallel updates ============ 22:14:02 (1788660842) 1+0 records in 1+0 records out 10000 bytes (10 kB, 9.8 KiB) copied, 0.00937895 s, 1.1 MB/s llite.lustre-ffff88adc7508800.hybrid_io=0 llite.lustre-ffff88add1a6e800.hybrid_io=0 fail_val=4 fail_loc=0x80000414 1+0 records in 1+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 4.68246 s, 1.8 MB/s PASS 108a (12s) == sanityn test 109: Race with several mount instances on 1 node ========================================================== 22:14:15 (1788660855) 192.168.203.136@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg336-client.virtnet /mnt/lustre (opts:) 192.168.203.136@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg336-client.virtnet /mnt/lustre2 (opts:) Starting race between client mounts (duration 30) Iteration 0 Starting client: oleg336-client.virtnet: -o user_xattr,flock 192.168.203.136@tcp:/lustre /mnt/lustre Starting client: oleg336-client.virtnet: -o user_xattr,flock 192.168.203.136@tcp:/lustre /mnt/lustre2 192.168.203.136@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.136@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg336-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg336-client.virtnet /mnt/lustre (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg336-server' oleg336-server: oleg336-server.virtnet: executing load_modules_local oleg336-server: kmemleak disabled oleg336-server: Loading modules from /home/green/git/lustre-release/lustre oleg336-server: detected 4 online CPUs by sysfs oleg336-server: MODOPTS_LIBCFS= oleg336-server: Force libcfs to create 2 CPU partitions oleg336-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Iteration 1 Starting client: oleg336-client.virtnet: -o user_xattr,flock 192.168.203.136@tcp:/lustre /mnt/lustre Starting client: oleg336-client.virtnet: -o user_xattr,flock 192.168.203.136@tcp:/lustre /mnt/lustre2 192.168.203.136@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.136@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg336-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg336-client.virtnet /mnt/lustre (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg336-server' oleg336-server: oleg336-server.virtnet: executing load_modules_local oleg336-server: kmemleak disabled oleg336-server: Loading modules from /home/green/git/lustre-release/lustre oleg336-server: detected 4 online CPUs by sysfs oleg336-server: MODOPTS_LIBCFS= oleg336-server: Force libcfs to create 2 CPU partitions oleg336-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Starting client: oleg336-client.virtnet: -o user_xattr,flock 192.168.203.136@tcp:/lustre /mnt/lustre Starting client: oleg336-client.virtnet: -o user_xattr,flock 192.168.203.136@tcp:/lustre /mnt/lustre2 PASS 109 (65s) == sanityn test 110: do not grant another lock on resend ========================================================== 22:15:20 (1788660920) fail_loc=0x525 fail_val=3 fail_loc=0x534 File: /mnt/lustre/f110.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238860488707 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-05 22:15:22.000000000 -0400 Modify: 2026-09-05 22:15:22.000000000 -0400 Change: 2026-09-05 22:15:22.000000000 -0400 Birth: 2026-09-05 22:15:22.000000000 -0400 fail_loc=0 fail_val=0 Connected clients: oleg336-client.virtnet oleg336-client.virtnet PASS 110 (168s) == sanityn test 111: A racy rename/link an open file should not cause fs corruption ========================================================== 22:18:08 (1788661088) SKIP: sanityn test_111 needs >= 2 MDTs SKIP 111 (3s) == sanityn test 112: update max-inherit in default LMV === 22:18:11 (1788661091) SKIP: sanityn test_112 We need at least 2 MDTs for this test SKIP 112 (3s) == sanityn test 113: check servers of specified fs ======= 22:18:14 (1788661094) lustre-OST0000-osc-ffff88adc7509000 active. lustre-OST0001-osc-ffff88adc7509000 active. lustre-OST0000-osc-ffff88adc7509000 active. lustre-OST0001-osc-ffff88adc7509000 active. MGC192.168.203.136@tcp active. PASS 113 (6s) == sanityn test 114: implicit default LMV inherit ======== 22:18:20 (1788661100) SKIP: sanityn test_114 We need at least 2 MDTs for this test SKIP 114 (3s) == sanityn test 115: ldiskfs doesn't check direntry for uniqueness ========================================================== 22:18:23 (1788661103) SKIP: sanityn test_115 ldiskfs only test SKIP 115 (3s) == sanityn test 116: DNE: Set default LMV layout from a remote client ========================================================== 22:18:26 (1788661106) SKIP: sanityn test_116 needs >= 2 MDTs SKIP 116 (3s) == sanityn test 121: trunc append race =================== 22:18:29 (1788661109) 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0169952 s, 1.2 MB/s fail_loc=0x80001436 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0808953 s, 253 kB/s 0000000 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0005000 0000 0000 0000 0000 0000 0000 0000 0000 * 000c800 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0011800 PASS 121 (9s) == sanityn test 122: directory size is consistent across mounts ========================================================== 22:18:38 (1788661118) PASS 122 (8s) == sanityn test 200: service remains healthy while able to process request ========================================================== 22:18:46 (1788661126) at_max=10 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=9 fail_loc=0x1e1 fail_val=20 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1476 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1957 s, 0.2 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.2034 s, 0.2 kB/s 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1517 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.2518 s, 0.2 kB/s1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1786 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.2201 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.0754 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.0709 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1456 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1589 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.2386 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.2465 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.2061 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2658 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.3153 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2652 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2295 s, 0.1 kB/s at_max=600 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=116 oleg336-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff88adc7509000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88adc7509000.ost_server_uuid in FULL state after 0 sec PASS 200 (81s) cleanup: ====================================================== == sanityn test complete, duration 10040 sec ============= 22:20:09 (1788661209) === sanityn: start cleanup 22:20:10 (1788661210) === Stopping clients: oleg336-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg336-client.virtnet /mnt/lustre2 opts: === sanityn: finish cleanup 22:24:58 (1788661498) ===