-----============= acceptance-small: sanityn ============----- Mon Sep 7 19:23:31 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg351-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanityn.*ex': No such file or directory excepting tests: 27 40a skipping tests SLOW=no: 33a === sanityn: start setup 19:23:50 (1788823430) === Starting client oleg351-client.virtnet: -o user_xattr,flock 192.168.203.151@tcp:/lustre /mnt/lustre2 Mount client oleg351-client.virtnet: mount -t lustre -o user_xattr,flock 192.168.203.151@tcp:/lustre /mnt/lustre2 Started clients oleg351-client.virtnet: 192.168.203.151@tcp:/lustre on /mnt/lustre2 type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) oleg351-client.virtnet: executing check_config_client /mnt/lustre oleg351-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg351-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9f8d8666f000.idle_timeout=debug osc.lustre-OST0000-osc-ffff9f8d92a09000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9f8d8666f000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9f8d92a09000.idle_timeout=debug disable quota as required oleg351-server: oleg351-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanityn: finish setup 19:24:20 (1788823460) === 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0180196 s, 58.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7629] == sanityn test 0: do_node_vp() and do_facet_vp() do the right thing ========================================================== 19:24:22 (1788823462) PASS 0 (6s) == sanityn test 1: Check attribute updates on 2 mount points ========================================================== 19:24:29 (1788823469) /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0777 OK /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0666 OK /mnt/lustre/f1.sanityn: absent OK PASS 1 (5s) == sanityn test 2a: check cached attribute updates on 2 mtpt's ================================================================== 19:24:34 (1788823474) -rw-r--r-- 1 root root 0 Sep 7 19:24 /mnt/lustre2/f2a /mnt/lustre/f2a has type file OK /mnt/lustre/f2a has perms 0777 OK PASS 2a (5s) == sanityn test 2b: check cached attribute updates on 2 mtpt's ================================================================== 19:24:39 (1788823479) -rw-r--r-- 1 root root 0 Sep 7 19:24 /mnt/lustre2/f2b /mnt/lustre2/f2b has type file OK /mnt/lustre2/f2b has perms 0777 OK PASS 2b (5s) == sanityn test 2c: check cached attribute updates on 2 mtpt's root ============================================================= 19:24:44 (1788823484) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0777 OK PASS 2c (5s) == sanityn test 2d: check cached attribute updates on 2 mtpt's root ============================================================= 19:24:49 (1788823489) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0755 OK PASS 2d (5s) == sanityn test 2e: check chmod on root is propagated to others ========================================================== 19:24:54 (1788823494) total 1 -rwxrwxrwx 1 root root 0 Sep 7 19:24 f2a -rwxrwxrwx 1 root root 0 Sep 7 19:24 f2b total 1 -rwxrwxrwx 1 root root 0 Sep 7 19:24 f2a -rwxrwxrwx 1 root root 0 Sep 7 19:24 f2b running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre2/f2e.sanityn] [count=1] 1+0 records in 1+0 records out 512 bytes copied, 0.00867149 s, 59.0 kB/s PASS 2e (6s) == sanityn test 2f: check attr/owner updates on DNE with 2 mtpt's ========================================================== 19:25:00 (1788823500) SKIP: sanityn test_2f needs >= 2 MDTs SKIP 2f (2s) == sanityn test 2g: check blocks update on sync write ==== 19:25:03 (1788823503) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.314335 s, 6.7 MB/s /mnt/lustre/f2g.sanityn has 4103 blocks /mnt/lustre2/f2g.sanityn has 4103 blocks PASS 2g (5s) == sanityn test 3: symlink on one mtpt, readlink on another ===================================================================== 19:25:08 (1788823508) PASS 3 (5s) == sanityn test 4: fstat validation on multiple mount points ==================================================================== 19:25:13 (1788823513) PASS 4 (7s) == sanityn test 5: create a file on one mount, truncate it on the other ========================================================== 19:25:20 (1788823520) /mnt/lustre/f5 has type file OK /mnt/lustre/f5 has size 100 OK PASS 5 (6s) == sanityn test 6: remove of open file on other node ============================================================================ 19:25:26 (1788823526) opening writing unlinking /mnt/lustre2/f6.sanityn accessing (1) seeking (1) accessing (2) fstat... reading comparing data truncating seeking (2) writing again seeking (3) reading again comparing data again closing SUCCESS - goto beer PASS 6 (5s) == sanityn test 7: remove of open directory on other node ======================================================================= 19:25:31 (1788823531) creating directory /mnt/lustre/d7 opening directory unlinking /mnt/lustre/d7 Ok, everything goes well. PASS 7 (9s) == sanityn test 8: remove of open special file on other node ==================================================================== 19:25:40 (1788823540) creating special file /mnt/lustre/f8.sanityn opening file unlinking /mnt/lustre/f8.sanityn Ok, everything goes well. PASS 8 (7s) == sanityn test 9a: append of file with sub-page size on multiple mounts ========================================================== 19:25:48 (1788823548) PASS 9a (8s) == sanityn test 9b: append to striped sparse file ======== 19:25:55 (1788823555) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.052785 s, 19.9 MB/s 3+0 records in 3+0 records out 3 bytes copied, 0.0292679 s, 0.1 kB/s Data read (expecting 'foo'): foo PASS 9b (7s) == sanityn test 10a: write of file with sub-page size on multiple mounts ========================================================== 19:26:03 (1788823563) 1+0 records in 1+0 records out 1 byte copied, 0.0189941 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0238156 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0276963 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0108852 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0345001 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0178621 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0206658 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0104507 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0151901 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0154291 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0110005 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0141865 s, 0.1 kB/s PASS 10a (11s) == sanityn test 10b: write of file with sub-page size on multiple mounts ========================================================== 19:26:14 (1788823574) 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.0196395 s, 156 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0184367 s, 222 kB/s 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.000336132 s, 9.1 MB/s PASS 10b (7s) == sanityn test 11: execution of file opened for write should return error ============================================================== 19:26:21 (1788823581) multiop /mnt/lustre/d11/f vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7629 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 298: /mnt/lustre2/d11/f: Text file busy PASS 11 (6s) == sanityn test 12: test lock ordering (link, stat, unlink) ========================================================== 19:26:27 (1788823587) warning: '-runtime' deprecated, use '-t runtime' instead - link 404 (time 1788823599.79 total 10.01 last 40.36) - link 764 (time 1788823609.82 total 20.04 last 35.90) - link 1202 (time 1788823619.83 total 30.05 last 43.74) - link 1599 (time 1788823629.86 total 40.08 last 39.57) - link 1972 (time 1788823639.88 total 50.10 last 37.22) - link 2327 (time 1788823649.89 total 60.11 last 35.47) - link 2745 (time 1788823659.90 total 70.12 last 41.78) - link 3074 (time 1788823669.91 total 80.14 last 32.85) - link 3444 (time 1788823679.92 total 90.14 last 36.97) total: 3782 link in 100.01 seconds: 37.81 ops/second using seed 3869465810 running for 100 seconds - stat 5688 (time 1788823601 ; total 11 ; last 11) - stat 10000 (time 1788823608 ; total 18 ; last 7) - stat 16846 (time 1788823619 ; total 29 ; last 11) - stat 20000 (time 1788823623 ; total 33 ; last 4) - stat 26122 (time 1788823634 ; total 44 ; last 11) - stat 30000 (time 1788823642 ; total 52 ; last 8) - stat 33063 (time 1788823653 ; total 63 ; last 11) - stat 36572 (time 1788823664 ; total 74 ; last 11) - stat 39622 (time 1788823675 ; total 85 ; last 11) - stat 40000 (time 1788823675 ; total 85 ; last 0) - stat 42964 (time 1788823686 ; total 96 ; last 11) total: 44034 stats in 100 seconds: 440.339996 stats/second - unlinked 0 (time 1788823640 ; total 0 ; last 0) unlink(/mnt/lustre2/lockdir/lockfile3782) error: No such file or directory total: 3781 unlinks in 90 seconds: 42.011112 unlinks/second /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 77: kill: (21425) - No such process /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 78: kill: (21427) - No such process PASS 12 (148s) == sanityn test 13: test directory page revocation ======= 19:28:55 (1788823735) f13 f13 PASS 13 (8s) == sanityn test 14aa: execution of file open for write returns -ETXTBSY ========================================================== 19:29:03 (1788823743) multiop /mnt/lustre/d14aa.sanityn/f14aa.sanityn vOw_c TMPPIPE=/tmp/multiop_open_wait_pipe.7629 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 333: /mnt/lustre2/d14aa.sanityn/f14aa.sanityn: Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 22614 Terminated $MULTIOP_PROG $FILE v$ARGS > $TMPPIPE (wd: ~) PASS 14aa (5s) == sanityn test 14ab: open(RDWR) of executing file returns -ETXTBSY ========================================================== 19:29:08 (1788823748) open(O_RDWR|O_CREAT): Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 23193 Terminated $DIR1/$tdir/sleep 60 (wd: ~) PASS 14ab (7s) == sanityn test 14b: truncate of executing file returns -ETXTBSY ================================================================ 19:29:15 (1788823755) truncate: cannot truncate '/mnt/lustre2/d14b.sanityn/sleep' to length 60: Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 348: 23771 Terminated $DIR1/$tdir/sleep 60 PASS 14b (6s) == sanityn test 14c: open(O_TRUNC) of executing file return -ETXTBSY ============================================================ 19:29:21 (1788823761) cp: cannot create regular file '/mnt/lustre2/d14c.sanityn/sleep': Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 360: 24353 Terminated $DIR1/$tdir/sleep 60 PASS 14c (6s) == sanityn test 14d: chmod of executing file is still possible ================================================================== 19:29:27 (1788823767) chmod /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 371: 24936 Terminated $DIR1/$tdir/sleep 60 PASS 14d (8s) == sanityn test 15: test out-of-space with multiple writers ===================================================================== 19:29:35 (1788823775) PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin: MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky sleep 5 for ZFS MDS Waiting for MDT destroys to complete fallocate on zfs doesn't consume space STRIPECOUNT=2 ORIGFREE=7531520 MAXFREE=800000 SKIP: oos2 test_15 oos2.sh: 7531520KB free > 800000KB, increase MAXFREE (or reduce fs size) sleep 5 for ZFS MDS Waiting for MDT destroys to complete SKIP 15 (49s) == sanityn test 16a: 500 iterations of dual-mount fsx ==== 19:30:24 (1788823824) keep default fallocate mode: 2 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.609362 s, 13.8 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.425309 s, 19.7 MB/s Chance of close/open is 1 in 50 Seed set to 3829 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero length punch hole 000004[0] 1788823829.798094 write 0x6c5948 thru 0x6cb317 (0x59d0 bytes) 000005[0] 1788823829.828297 mapwrite 0x431fca thru 0x43ed04 (0xcd3b bytes) 000006[0] 1788823829.931662 read 0x0a1a3e thru 0x0ac71e (0xace1 bytes) 000007[0] 1788823829.948090 mapread 0x1d222c thru 0x1df168 (0xcf3d bytes) 000008[1] 1788823830.001927 punch from 0x470e6e to 0x4776b7 (0x684a bytes) 000009[1] 1788823830.028547 mapwrite 0x426a79 thru 0x42cf2c (0x64b4 bytes) truncating to largest ever: 0x7bf103 000010[0] 1788823830.164912 trunc from 0x6cb318 to 0x7bf102 (0xf3deb bytes) 000011[1] 1788823830.246438 write 0x86b6a6 thru 0x874df2 (0x974d bytes) 000013[0] 1788823830.298733 write 0x977814 thru 0x97cd46 (0x5533 bytes) 000015[0] 1788823830.389743 read 0x394ef7 thru 0x39ab61 (0x5c6b bytes) 000016[0] 1788823830.410985 punch from 0x19dc55 to 0x1a46e6 (0x6a92 bytes) 000022[1] 1788823830.548409 trunc from 0x97cd47 to 0x10be70 (0x870ed6 bytes) 000023[0] 1788823830.706527 mapwrite 0x39c21f thru 0x3a23b3 (0x6195 bytes) 000024[1] 1788823830.844509 write 0x8baca2 thru 0x8c7412 (0xc771 bytes) 000025[0] 1788823830.888901 write 0x4e41c4 thru 0x4f270b (0xe548 bytes) 000027[1] 1788823830.939780 punch from 0x17c30b to 0x189d8c (0xda82 bytes) truncating to largest ever: 0x879e94 000028[0] 1788823830.956857 trunc from 0x8c7413 to 0x879e93 (0x4d57f bytes) 000029[0] 1788823831.036152 write 0x074184 thru 0x0820c1 (0xdf3e bytes) 000030[1] 1788823831.054119 read 0x314ff6 thru 0x3169c1 (0x19cc bytes) 000031[0] 1788823831.098248 write 0x023fdd thru 0x02cebb (0x8edf bytes) 000032[0] 1788823831.151587 write 0x6279c2 thru 0x629c74 (0x22b3 bytes) 000034[1] 1788823831.191617 read 0x55bffe thru 0x56a120 (0xe123 bytes) 000035[1] 1788823831.231998 mapread 0x6e66d4 thru 0x6e67e8 (0x115 bytes) 000036[1] 1788823831.245160 write 0x4a25f3 thru 0x4a3de8 (0x17f6 bytes) 000037[1] 1788823831.287937 punch from 0x6e3a8f to 0x6e740f (0x3981 bytes) 000038[1] 1788823831.313756 mapwrite 0x506de0 thru 0x515e2a (0xf04b bytes) 000039[1] 1788823831.478978 write 0x459532 thru 0x45afd5 (0x1aa4 bytes) 000041[1] 1788823831.569875 trunc from 0x879e94 to 0x18d297 (0x6ecbfc bytes) 000042[1] 1788823831.633731 write 0x6c11ab thru 0x6cfac4 (0xe91a bytes) 000044[0] 1788823831.687711 punch from 0x26c071 to 0x2725d6 (0x6566 bytes) 000047[0] 1788823831.756769 mapwrite 0x8cef1d thru 0x8d30a1 (0x4185 bytes) 000048[0] 1788823831.914156 punch from 0x6dd8ab to 0x6e7db9 (0xa50f bytes) 000050[0] 1788823831.978673 mapread 0x1b7f01 thru 0x1b9700 (0x1800 bytes) 000053[1] 1788823832.037069 mapread 0x2e9d45 thru 0x2ef7f7 (0x5ab3 bytes) 000054[1] 1788823832.075156 read 0x4ea5b2 thru 0x4ee50f (0x3f5e bytes) 000056[1] 1788823832.100368 mapwrite 0x1ad5a2 thru 0x1b3f42 (0x69a1 bytes) 000061[1] 1788823832.185886 write 0x5bd066 thru 0x5c5bc0 (0x8b5b bytes) 000063[1] 1788823832.247553 read 0x45f615 thru 0x46f242 (0xfc2e bytes) 000064[1] 1788823832.267102 punch from 0x0c7051 to 0x0c85d5 (0x1585 bytes) 000065[1] 1788823832.288266 write 0x734255 thru 0x737dfb (0x3ba7 bytes) 000066[0] 1788823832.316920 read 0x80a36d thru 0x80a517 (0x1ab bytes) 000068[1] 1788823832.361950 mapread 0x6e1822 thru 0x6e98cb (0x80aa bytes) truncating to largest ever: 0x8e9277 000071[1] 1788823832.418319 trunc from 0x8d30a2 to 0x8e9276 (0x161d5 bytes) 000072[0] 1788823832.456827 punch from 0x08b455 to 0x08d13d (0x1ce9 bytes) 000073[0] 1788823832.513259 mapread 0x5c63d3 thru 0x5c8f3b (0x2b69 bytes) 000074[0] 1788823832.600786 punch from 0x77c72d to 0x7850fd (0x89d1 bytes) 000076[0] 1788823832.661265 write 0x769173 thru 0x7739b3 (0xa841 bytes) 000078[1] 1788823832.732157 mapread 0x8c1d94 thru 0x8cfd2b (0xdf98 bytes) 000080[0] 1788823832.798106 punch from 0x124d0c to 0x12f211 (0xa506 bytes) 000081[1] 1788823832.830687 mapwrite 0x4551eb thru 0x45be52 (0x6c68 bytes) 000083[1] 1788823833.013650 trunc from 0x8e9277 to 0x699373 (0x24ff03 bytes) 000084[1] 1788823833.062903 write 0x3cb8ba thru 0x3d323c (0x7983 bytes) 000085[0] 1788823833.090930 read 0x4b9c8a thru 0x4be6d7 (0x4a4e bytes) 000087[1] 1788823833.126346 mapwrite 0x9e8faf thru 0x9ea8ee (0x1940 bytes) 000088[1] 1788823833.226698 mapread 0x091fd2 thru 0x09ca72 (0xaaa1 bytes) 000089[0] 1788823833.251678 punch from 0x61dd15 to 0x62db8f (0xfe7b bytes) 000090[1] 1788823833.275869 mapwrite 0x077030 thru 0x082370 (0xb341 bytes) 000091[0] 1788823833.367174 write 0x7d0747 thru 0x7d32e8 (0x2ba2 bytes) 000092[0] 1788823833.405201 mapwrite 0x6b065d thru 0x6bac0b (0xa5af bytes) 000093[0] 1788823833.524391 read 0x9779f8 thru 0x980fd1 (0x95da bytes) 000096[0] 1788823833.549925 trunc from 0x9ea8ef to 0x0141ce (0x9d6720 bytes) 000097[0] 1788823833.609883 mapwrite 0x49a63e thru 0x4a9775 (0xf138 bytes) 000098[1] 1788823833.783107 mapread 0x382fe2 thru 0x38491b (0x193a bytes) 000099[1] 1788823833.848099 punch from 0x1b949f to 0x1c3638 (0xa19a bytes) 000101[1] 1788823833.905972 mapwrite 0x1584f2 thru 0x1613ee (0x8efd bytes) 000103[0] 1788823834.050284 read 0x1260c3 thru 0x12cb2d (0x6a6b bytes) 000104[0] 1788823834.077799 punch from 0x17cc2f to 0x17dea4 (0x1276 bytes) 000105[1] 1788823834.109365 punch from 0x373ee3 to 0x37bff2 (0x8110 bytes) 000108[0] 1788823834.207254 punch from 0x2bc9d6 to 0x2c2898 (0x5ec3 bytes) 000109[0] 1788823834.233220 read 0x1fe43a thru 0x20b507 (0xd0ce bytes) 000110[1] 1788823834.288815 mapwrite 0x9f7ccf thru 0x9fd595 (0x58c7 bytes) 000112[1] 1788823834.464250 read 0x65510b thru 0x663ffc (0xeef2 bytes) 000113[1] 1788823834.517449 mapread 0x03ccac thru 0x03fedf (0x3234 bytes) 000114[1] 1788823834.601708 mapwrite 0x385206 thru 0x38f1e2 (0x9fdd bytes) 000115[1] 1788823834.756055 trunc from 0x9fd596 to 0x897d6d (0x165828 bytes) 000118[1] 1788823834.845900 read 0x269472 thru 0x273107 (0x9c96 bytes) 000120[0] 1788823834.889120 mapread 0x2d58b0 thru 0x2d99c5 (0x4116 bytes) 000121[1] 1788823834.944062 read 0x7be16e thru 0x7c7753 (0x95e6 bytes) 000125[0] 1788823835.040417 punch from 0x2e3f35 to 0x2f0f68 (0xd034 bytes) 000128[1] 1788823835.110021 write 0x8cd94c thru 0x8dca67 (0xf11c bytes) 000133[1] 1788823835.205605 read 0x8d13d3 thru 0x8db8b4 (0xa4e2 bytes) 000137[1] 1788823835.268295 write 0x29c7d5 thru 0x2ab05c (0xe888 bytes) 000138[0] 1788823835.280877 mapread 0x454dfa thru 0x45e752 (0x9959 bytes) 000141[0] 1788823835.336023 trunc from 0x8dca68 to 0x7fc479 (0xe05ee bytes) 000143[0] 1788823835.409829 trunc from 0x7fc47a to 0x6de137 (0x11e342 bytes) 000146[1] 1788823835.464821 mapwrite 0x393c23 thru 0x3a38be (0xfc9c bytes) 000152[1] 1788823835.696224 read 0x0cf229 thru 0x0d6a9b (0x7873 bytes) 000153[1] 1788823835.714786 trunc from 0x6de138 to 0x243cd3 (0x49a464 bytes) 000154[1] 1788823835.764072 write 0x7e82f5 thru 0x7f058b (0x8297 bytes) 000155[1] 1788823835.784410 read 0x4d01ba thru 0x4d9180 (0x8fc7 bytes) 000160[1] 1788823835.822358 punch from 0x7af173 to 0x7bc93b (0xd7c9 bytes) 000161[0] 1788823835.850136 write 0x15536c thru 0x157a03 (0x2698 bytes) 000162[1] 1788823835.881671 punch from 0x1c1bdd to 0x1c8fa4 (0x73c8 bytes) 000163[0] 1788823835.914166 write 0x98dde7 thru 0x98df01 (0x11b bytes) 000164[0] 1788823835.938941 mapwrite 0x9684d7 thru 0x96c332 (0x3e5c bytes) truncating to largest ever: 0x9f3937 000165[1] 1788823836.015167 trunc from 0x98df02 to 0x9f3936 (0x65a35 bytes) 000170[1] 1788823836.166316 trunc from 0x9f3937 to 0x6ed917 (0x30601f bytes) 000171[0] 1788823836.219297 read 0x0fd978 thru 0x10bd48 (0xe3d1 bytes) 000172[1] 1788823836.257134 mapread 0x0b8f61 thru 0x0c42fb (0xb39b bytes) 000173[1] 1788823836.295172 mapwrite 0x0b36eb thru 0x0b3cd2 (0x5e8 bytes) 000174[0] 1788823836.381268 mapwrite 0x0416b3 thru 0x0421f9 (0xb47 bytes) 000175[0] 1788823836.460195 mapread 0x2660da thru 0x26c536 (0x645d bytes) 000176[1] 1788823836.492769 punch from 0x2acf28 to 0x2ba782 (0xd85b bytes) 000177[1] 1788823836.502799 mapread 0x1bf46b thru 0x1c050d (0x10a3 bytes) 000179[1] 1788823836.545164 write 0x258122 thru 0x26089f (0x877e bytes) 000180[0] 1788823836.562681 read 0x55812d thru 0x5676bd (0xf591 bytes) 000181[0] 1788823836.602733 mapwrite 0x3c6fc1 thru 0x3cb80e (0x484e bytes) 000182[0] 1788823836.784697 write 0x7cbc4e thru 0x7cc010 (0x3c3 bytes) 000183[1] 1788823836.856164 trunc from 0x7cc011 to 0x162dbe (0x669252 bytes) 000185[0] 1788823836.942191 trunc from 0x162dbf to 0x2ac9e0 (0x149c22 bytes) 000187[1] 1788823837.070661 mapwrite 0x1c5ef7 thru 0x1d0bf2 (0xacfc bytes) 000188[0] 1788823837.223356 punch from 0x1be3e3 to 0x1cd069 (0xec87 bytes) 000189[1] 1788823837.247064 read 0x01205f thru 0x01730a (0x52ac bytes) 000192[0] 1788823837.406428 punch from 0x16d242 to 0x17acbd (0xda7c bytes) 000193[0] 1788823837.440298 punch from 0x011b93 to 0x0211bb (0xf629 bytes) 000195[1] 1788823837.477094 punch from 0x06a591 to 0x0767c2 (0xc232 bytes) 000196[1] 1788823837.538409 trunc from 0x2ac9e1 to 0x8103ef (0x563a0f bytes) 000198[1] 1788823837.643409 read 0x70f4e8 thru 0x71f058 (0xfb71 bytes) 000199[1] 1788823837.673052 punch from 0x68ad38 to 0x68f0b2 (0x437b bytes) 000202[1] 1788823837.734375 read 0x50b5f8 thru 0x50f241 (0x3c4a bytes) 000203[0] 1788823837.747856 trunc from 0x8103f0 to 0x1adfa9 (0x662446 bytes) 000204[0] 1788823837.801826 mapwrite 0x38577c thru 0x387547 (0x1dcc bytes) 000208[1] 1788823837.936584 write 0x48ddd6 thru 0x4963df (0x860a bytes) 000209[0] 1788823837.959884 write 0x449737 thru 0x453039 (0x9903 bytes) 000210[0] 1788823838.012178 mapwrite 0x98bc96 thru 0x98bff9 (0x364 bytes) 000211[1] 1788823838.191669 punch from 0x1c9e37 to 0x1cabfd (0xdc7 bytes) 000214[0] 1788823838.325593 trunc from 0x98bffa to 0x235301 (0x756cf8 bytes) 000216[1] 1788823838.475231 read 0x21f0a3 thru 0x225ea3 (0x6e01 bytes) 000217[1] 1788823838.576847 trunc from 0x235302 to 0x7ef318 (0x5ba017 bytes) 000218[0] 1788823838.675518 trunc from 0x7ef319 to 0x5b738b (0x237f8d bytes) 000220[0] 1788823838.766622 trunc from 0x5b738c to 0x48d4f3 (0x129e98 bytes) 000221[0] 1788823838.801400 mapwrite 0x040056 thru 0x045a83 (0x5a2e bytes) 000223[0] 1788823838.992222 write 0x9305c2 thru 0x937ca5 (0x76e4 bytes) 000224[0] 1788823839.022171 trunc from 0x937ca6 to 0x997f40 (0x6029b bytes) 000225[1] 1788823839.059115 trunc from 0x997f41 to 0x740c38 (0x257308 bytes) 000227[0] 1788823839.206605 mapread 0x1705a2 thru 0x17fabd (0xf51c bytes) 000228[1] 1788823839.315260 write 0x68411d thru 0x693ece (0xfdb2 bytes) 000230[0] 1788823839.550971 mapwrite 0x4d6dc8 thru 0x4e12ac (0xa4e5 bytes) 000231[0] 1788823839.710688 read 0x0604bf thru 0x06f3a9 (0xeeeb bytes) 000232[1] 1788823839.720795 punch from 0x641a3f to 0x643e3f (0x2401 bytes) 000233[1] 1788823839.751559 read 0x02e77a thru 0x0361e4 (0x7a6b bytes) 000234[1] 1788823839.789968 read 0x3368f2 thru 0x33cc21 (0x6330 bytes) 000236[1] 1788823839.873134 mapwrite 0x0a66c2 thru 0x0b46fb (0xe03a bytes) 000238[1] 1788823840.110292 read 0x0c2273 thru 0x0c4d0d (0x2a9b bytes) 000239[0] 1788823840.135154 read 0x569515 thru 0x5774dc (0xdfc8 bytes) 000240[0] 1788823840.164694 mapwrite 0x1e6604 thru 0x1e6de1 (0x7de bytes) 000241[0] 1788823840.243901 trunc from 0x740c39 to 0x229b1a (0x51711e bytes) 000242[0] 1788823840.323807 read 0x13dc2b thru 0x13e74f (0xb25 bytes) 000243[0] 1788823840.445299 write 0x1e1361 thru 0x1ea4a7 (0x9147 bytes) 000244[1] 1788823840.453530 punch from 0x1146af to 0x11f781 (0xb0d3 bytes) 000245[1] 1788823840.491203 mapwrite 0x1ebbf8 thru 0x1fa390 (0xe799 bytes) 000249[0] 1788823840.660979 mapwrite 0x096530 thru 0x0a004e (0x9b1f bytes) 000254[0] 1788823840.980538 write 0x4b0707 thru 0x4b5f9c (0x5896 bytes) 000255[1] 1788823840.986675 mapwrite 0x152da1 thru 0x15e846 (0xbaa6 bytes) 000261[1] 1788823841.171081 punch from 0x2099a3 to 0x2154af (0xbb0d bytes) 000262[1] 1788823841.195981 write 0x5403ca thru 0x54eb77 (0xe7ae bytes) 000263[1] 1788823841.242319 punch from 0x4447a6 to 0x44b437 (0x6c92 bytes) 000264[1] 1788823841.250607 punch from 0x50f2b2 to 0x51d6cd (0xe41c bytes) 000265[1] 1788823841.289187 write 0x51acb3 thru 0x524c7c (0x9fca bytes) 000266[1] 1788823841.317972 mapwrite 0x2ef0ca thru 0x2f0498 (0x13cf bytes) 000267[1] 1788823841.370745 mapwrite 0x1e9351 thru 0x1f433c (0xafec bytes) 000268[1] 1788823841.484088 punch from 0x4b88a6 to 0x4c6244 (0xd99f bytes) 000269[0] 1788823841.516744 write 0x6aaa4d thru 0x6ab91a (0xece bytes) 000270[1] 1788823841.567141 read 0x0ca6cf thru 0x0d8a14 (0xe346 bytes) 000271[1] 1788823841.641989 read 0x04668d thru 0x0481b5 (0x1b29 bytes) 000273[1] 1788823841.776188 mapwrite 0x0546d0 thru 0x062dd7 (0xe708 bytes) 000274[0] 1788823841.859573 write 0x565e59 thru 0x573d9a (0xdf42 bytes) 000275[1] 1788823841.894705 punch from 0x0b6264 to 0x0c5b4f (0xf8ec bytes) 000277[1] 1788823841.930082 punch from 0x00133a to 0x004d43 (0x3a0a bytes) 000280[1] 1788823842.005805 punch from 0x04c9ae to 0x04debe (0x1511 bytes) 000281[1] 1788823842.033301 trunc from 0x6ab91b to 0x95483c (0x2a8f22 bytes) 000282[0] 1788823842.097056 read 0x6e0752 thru 0x6e0cfc (0x5ab bytes) 000283[1] 1788823842.148584 read 0x1c9bcb thru 0x1ca7b4 (0xbea bytes) 000284[1] 1788823842.179446 punch from 0x3b3c48 to 0x3be67e (0xaa37 bytes) 000285[1] 1788823842.229547 read 0x073789 thru 0x08153b (0xddb3 bytes) 000287[0] 1788823842.261141 read 0x5af4f1 thru 0x5b01fa (0xd0a bytes) 000289[0] 1788823842.312570 write 0x1f4d44 thru 0x1fed22 (0x9fdf bytes) 000290[0] 1788823842.375780 mapwrite 0x72c6e3 thru 0x7389e2 (0xc300 bytes) 000291[1] 1788823842.503182 mapwrite 0x22a100 thru 0x2305b9 (0x64ba bytes) 000293[0] 1788823842.631022 mapwrite 0x25565e thru 0x25d088 (0x7a2b bytes) 000295[1] 1788823842.716175 mapread 0x429468 thru 0x42a2df (0xe78 bytes) 000296[0] 1788823842.756178 trunc from 0x95483d to 0x3c93ff (0x58b43d bytes) 000297[0] 1788823842.791986 trunc from 0x3c9400 to 0x7d1e77 (0x408a78 bytes) 000298[0] 1788823842.809195 write 0x11ff31 thru 0x12a282 (0xa352 bytes) 000299[0] 1788823842.844184 read 0x258bb5 thru 0x261a0c (0x8e58 bytes) 000301[0] 1788823842.873906 write 0x9a34d3 thru 0x9ad741 (0xa26f bytes) 000307[0] 1788823842.894530 write 0x31471b thru 0x31d541 (0x8e27 bytes) 000308[0] 1788823842.939546 mapread 0x37dda1 thru 0x388ecf (0xb12f bytes) 000309[0] 1788823842.994607 read 0x170abc thru 0x180054 (0xf599 bytes) 000311[0] 1788823843.026578 write 0x238825 thru 0x23f542 (0x6d1e bytes) 000312[1] 1788823843.034309 mapwrite 0x9abe33 thru 0x9b465c (0x882a bytes) 000313[0] 1788823843.191329 punch from 0x4f5599 to 0x4f9762 (0x41ca bytes) 000314[0] 1788823843.253913 punch from 0x34269a to 0x34de3e (0xb7a5 bytes) 000315[0] 1788823843.332241 trunc from 0x9b465d to 0x341209 (0x673453 bytes) 000316[0] 1788823843.416797 trunc from 0x34120a to 0x92646c (0x5e5263 bytes) 000319[0] 1788823843.535683 trunc from 0x92646d to 0x986f90 (0x60b24 bytes) 000320[1] 1788823843.557142 mapwrite 0x17b919 thru 0x17fe49 (0x4531 bytes) 000321[1] 1788823843.661721 trunc from 0x986f91 to 0x070796 (0x9167fa bytes) 000322[1] 1788823843.731365 trunc from 0x070797 to 0x6893c2 (0x618c2c bytes) 000324[0] 1788823843.786265 write 0x83b855 thru 0x84135e (0x5b0a bytes) 000325[0] 1788823843.815580 read 0x0f4fcf thru 0x100ecd (0xbeff bytes) 000330[1] 1788823843.895144 punch from 0x4e5eb4 to 0x4eba56 (0x5ba3 bytes) 000331[0] 1788823843.910395 read 0x46eb54 thru 0x47a39d (0xb84a bytes) 000332[1] 1788823843.928828 punch from 0x0fb50a to 0x10976f (0xe266 bytes) 000334[0] 1788823843.955175 mapread 0x3553d4 thru 0x362e7c (0xdaa9 bytes) 000335[0] 1788823843.999672 trunc from 0x84135f to 0x887945 (0x465e7 bytes) 000338[0] 1788823844.057716 trunc from 0x887946 to 0x2d75ed (0x5b0358 bytes) 000339[1] 1788823844.077616 read 0x099df9 thru 0x09cef6 (0x30fe bytes) 000342[0] 1788823844.160580 mapwrite 0x3376e6 thru 0x338909 (0x1224 bytes) 000351[1] 1788823844.375896 trunc from 0x33890a to 0x823af8 (0x4eb1ef bytes) 000353[1] 1788823844.473139 punch from 0x05f592 to 0x0688bd (0x932c bytes) 000355[1] 1788823844.541414 mapread 0x57d45b thru 0x585cd2 (0x8878 bytes) 000358[1] 1788823844.590147 trunc from 0x823af9 to 0x30742a (0x51c6ce bytes) 000360[0] 1788823844.685406 mapread 0x086840 thru 0x095d8b (0xf54c bytes) 000361[0] 1788823844.757188 read 0x0ad7c7 thru 0x0b6f7b (0x97b5 bytes) 000363[0] 1788823844.841753 mapread 0x226aad thru 0x2342c0 (0xd814 bytes) 000365[0] 1788823844.913186 write 0x071a82 thru 0x07a3d8 (0x8957 bytes) 000366[1] 1788823844.971145 punch from 0x2e4d1f to 0x2f150e (0xc7f0 bytes) 000370[0] 1788823845.067095 punch from 0x2a761b to 0x2affd4 (0x89ba bytes) 000371[1] 1788823845.094797 read 0x105da6 thru 0x110d42 (0xaf9d bytes) 000373[0] 1788823845.200600 write 0x4f2419 thru 0x4faf49 (0x8b31 bytes) 000375[0] 1788823845.229993 punch from 0x3c1463 to 0x3cd92e (0xc4cc bytes) 000376[1] 1788823845.262378 trunc from 0x4faf4a to 0x80daf6 (0x312bad bytes) 000377[0] 1788823845.321674 trunc from 0x80daf7 to 0x7f5aa3 (0x18053 bytes) 000380[0] 1788823845.415854 mapread 0x52ddc3 thru 0x537ef8 (0xa136 bytes) 000381[0] 1788823845.482046 trunc from 0x7f5aa4 to 0x2b6356 (0x53f74d bytes) 000382[1] 1788823845.513512 punch from 0x015d74 to 0x01bbff (0x5e8c bytes) 000383[0] 1788823845.557027 mapread 0x0eb57e thru 0x0f9ca7 (0xe72a bytes) 000384[1] 1788823845.655001 mapread 0x1d4ee6 thru 0x1de8d1 (0x99ec bytes) 000385[1] 1788823845.766167 read 0x09c19b thru 0x0a808c (0xbef2 bytes) 000386[1] 1788823845.807751 trunc from 0x2b6357 to 0x43ee12 (0x188abc bytes) 000388[1] 1788823845.846819 punch from 0x23baa6 to 0x245332 (0x988d bytes) 000391[1] 1788823845.872041 mapread 0x3691f7 thru 0x36cc94 (0x3a9e bytes) 000392[1] 1788823845.907145 mapread 0x133a24 thru 0x13d759 (0x9d36 bytes) 000393[0] 1788823845.946326 mapwrite 0x92f892 thru 0x932dc5 (0x3534 bytes) 000395[0] 1788823846.040216 punch from 0x19d068 to 0x1a424e (0x71e7 bytes) 000396[1] 1788823846.064656 mapread 0x5d374f thru 0x5df61e (0xbed0 bytes) 000399[0] 1788823846.166144 punch from 0x2d37b3 to 0x2e3009 (0xf857 bytes) 000402[1] 1788823846.197791 mapread 0x7d36f2 thru 0x7e2635 (0xef44 bytes) 000408[1] 1788823846.290726 write 0x779307 thru 0x785fa9 (0xcca3 bytes) 000412[1] 1788823846.408988 trunc from 0x932dc6 to 0x85ab42 (0xd8283 bytes) 000414[1] 1788823846.470316 write 0x43708f thru 0x44656a (0xf4dc bytes) 000417[1] 1788823846.484110 write 0x40c026 thru 0x40daff (0x1ada bytes) 000418[0] 1788823846.492231 punch from 0x4b067f to 0x4b8571 (0x7ef3 bytes) 000420[0] 1788823846.535433 trunc from 0x85ab43 to 0x04d7b7 (0x80d38b bytes) 000422[0] 1788823846.584957 trunc from 0x04d7b8 to 0x9c6427 (0x978c70 bytes) 000427[1] 1788823846.719741 write 0x297a9b thru 0x2a5125 (0xd68b bytes) 000428[0] 1788823846.794392 write 0x362193 thru 0x367def (0x5c5d bytes) 000431[0] 1788823846.922925 read 0x1b0b7a thru 0x1b575c (0x4be3 bytes) 000432[0] 1788823846.950016 mapread 0x0753fe thru 0x07afa5 (0x5ba8 bytes) 000433[0] 1788823847.075499 write 0x7ec2a7 thru 0x7ef0cc (0x2e26 bytes) 000434[1] 1788823847.154797 punch from 0x28e343 to 0x297ad4 (0x9792 bytes) 000436[0] 1788823847.220594 mapread 0x4d4d81 thru 0x4e2cc6 (0xdf46 bytes) 000441[0] 1788823847.358411 mapwrite 0x9d1dad thru 0x9df3e7 (0xd63b bytes) 000442[0] 1788823847.591752 read 0x9379ca thru 0x93927c (0x18b3 bytes) 000443[0] 1788823847.633347 trunc from 0x9df3e8 to 0x378c58 (0x66678f bytes) 000445[0] 1788823847.706145 mapwrite 0x2e5567 thru 0x2eece6 (0x9780 bytes) 000447[1] 1788823847.782911 write 0x4b2d8b thru 0x4b3300 (0x576 bytes) 000448[0] 1788823847.817105 mapread 0x33c505 thru 0x342d32 (0x682e bytes) 000449[1] 1788823847.854201 punch from 0x3066d0 to 0x30cb1c (0x644d bytes) 000450[1] 1788823847.871393 mapwrite 0x7c7871 thru 0x7ce4c4 (0x6c54 bytes) 000451[0] 1788823848.008883 trunc from 0x7ce4c5 to 0x3d5958 (0x3f8b6c bytes) 000453[1] 1788823848.079060 read 0x172fa1 thru 0x1828a3 (0xf903 bytes) 000455[1] 1788823848.168156 punch from 0x129d1c to 0x12ca06 (0x2ceb bytes) 000457[1] 1788823848.213665 mapread 0x269f80 thru 0x2730d8 (0x9159 bytes) 000458[1] 1788823848.283507 mapread 0x12ed09 thru 0x137874 (0x8b6c bytes) 000460[0] 1788823848.418116 write 0x8e3a03 thru 0x8f23b8 (0xe9b6 bytes) 000461[0] 1788823848.483156 mapread 0x519fff thru 0x52300f (0x9011 bytes) 000462[1] 1788823848.508906 trunc from 0x8f23b9 to 0x3b1f0f (0x5404a9 bytes) 000464[1] 1788823848.637662 write 0x7a04f9 thru 0x7acc07 (0xc70f bytes) 000465[0] 1788823848.673630 trunc from 0x7acc08 to 0x18f9c3 (0x61d244 bytes) 000466[0] 1788823848.774808 trunc from 0x18f9c4 to 0x9a3585 (0x813bc2 bytes) 000467[0] 1788823848.793827 read 0x52ae3d thru 0x5395d8 (0xe79c bytes) 000468[1] 1788823848.814439 write 0x082c75 thru 0x087081 (0x440d bytes) 000469[0] 1788823848.844679 mapread 0x2c8112 thru 0x2d2b81 (0xaa70 bytes) 000471[1] 1788823848.918921 mapread 0x8cb757 thru 0x8d68a8 (0xb152 bytes) 000473[1] 1788823849.050387 punch from 0x5fc790 to 0x601dd5 (0x5646 bytes) 000474[0] 1788823849.113330 mapwrite 0x9b004c thru 0x9bded3 (0xde88 bytes) 000475[0] 1788823849.428175 mapread 0x158e6c thru 0x1590b7 (0x24c bytes) 000476[0] 1788823849.483440 mapread 0x250997 thru 0x255bb3 (0x521d bytes) 000479[1] 1788823849.544289 trunc from 0x9bded4 to 0x74812c (0x275da7 bytes) 000480[1] 1788823849.646198 read 0x11c3aa thru 0x121686 (0x52dd bytes) 000482[0] 1788823849.707328 write 0x8e9265 thru 0x8f202d (0x8dc9 bytes) 000483[0] 1788823849.766088 trunc from 0x8f202e to 0x178cdf (0x77934e bytes) 000484[1] 1788823849.839762 write 0x9adbbc thru 0x9b092e (0x2d73 bytes) 000485[0] 1788823849.887761 mapwrite 0x9d3ac8 thru 0x9d8920 (0x4e59 bytes) 000486[1] 1788823850.109217 read 0x9a90d9 thru 0x9b9083 (0xffab bytes) 000487[1] 1788823850.187732 mapwrite 0x2491dd thru 0x2553f5 (0xc219 bytes) 000488[0] 1788823850.402400 mapwrite 0x279672 thru 0x27d30a (0x3c99 bytes) 000493[0] 1788823850.564031 punch from 0x0ae503 to 0x0b3622 (0x5120 bytes) 000494[1] 1788823850.571922 trunc from 0x9d8921 to 0x253e66 (0x784aba bytes) 000500[1] 1788823850.754444 punch from 0x1bad65 to 0x1c14c3 (0x675f bytes) All operations completed A-OK! Chance of close/open is 1 in 50 Seed set to 3850 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read truncating to largest ever: 0x7701cb 000003[0] 1788823851.509938 trunc from 00000000 to 0x7701ca (0x7701cb bytes) 000005[1] 1788823851.559844 mapwrite 0x7bd000 thru 0x7c5f04 (0x8f05 bytes) 000007[1] 1788823851.670613 write_OD 0x922000 thru 0x928fff (0x7000 bytes) 000008[0] 1788823851.757873 trunc from 0x929000 to 0x217383 (0x711c7c bytes) 000009[0] 1788823851.795738 read_OD 0x044000 thru 0x052fff (0xf000 bytes) 000011[1] 1788823851.834313 mapwrite 0x113000 thru 0x1200e3 (0xd0e4 bytes) 000012[1] 1788823852.107424 mapread 0x00a000 thru 0x00a718 (0x719 bytes) 000014[0] 1788823852.127428 mapwrite 0x86c000 thru 0x86fdfd (0x3dfe bytes) 000015[0] 1788823852.276735 mapread 0x23b000 thru 0x245165 (0xa166 bytes) 000016[0] 1788823852.335365 punch from 0x7d3901 to 0x7e2291 (0xe991 bytes) 000017[0] 1788823852.348669 mapread 0x6ef000 thru 0x6f5f31 (0x6f32 bytes) 000018[1] 1788823852.392943 punch from 0x4e863d to 0x4ef3d3 (0x6d97 bytes) 000022[1] 1788823852.415025 mapwrite 0x52a000 thru 0x531523 (0x7524 bytes) 000023[1] 1788823852.535944 trunc from 0x86fdfe to 0x5d42a2 (0x29bb5b bytes) 000024[1] 1788823852.589703 write_OD 0x827000 thru 0x830fff (0xa000 bytes) 000025[1] 1788823852.734997 trunc from 0x831000 to 0x37badb (0x4b5524 bytes) 000027[0] 1788823852.825866 mapwrite 0x688000 thru 0x69633c (0xe33d bytes) 000028[1] 1788823852.913655 punch from 0x2b9c54 to 0x2be87d (0x4c2a bytes) 000029[1] 1788823852.936840 punch from 0x169b3c to 0x1756a4 (0xbb69 bytes) 000031[1] 1788823852.963850 trunc from 0x69633d to 0x4eaa4a (0x1ab8f2 bytes) 000032[0] 1788823852.984834 write_OD 0x808000 thru 0x811fff (0xa000 bytes) 000033[1] 1788823853.096008 read_OD 0x21e000 thru 0x224fff (0x7000 bytes) 000038[1] 1788823853.118049 read_OD 0x2d3000 thru 0x2ddfff (0xb000 bytes) 000039[0] 1788823853.142197 read_OD 0x1ed000 thru 0x1eefff (0x2000 bytes) truncating to largest ever: 0x9c9b1f 000040[0] 1788823853.156723 trunc from 0x812000 to 0x9c9b1e (0x1b7b1f bytes) 000042[1] 1788823853.269863 read_OD 0x653000 thru 0x656fff (0x4000 bytes) 000043[0] 1788823853.306693 punch from 0x0a1a5a to 0x0ade62 (0xc409 bytes) 000044[1] 1788823853.349036 read_OD 0x7c0000 thru 0x7cdfff (0xe000 bytes) 000045[1] 1788823853.391972 write_OD 0x676000 thru 0x67bfff (0x6000 bytes) 000047[1] 1788823853.516875 punch from 0x3dfdcd to 0x3ecab1 (0xcce5 bytes) 000051[0] 1788823853.599745 trunc from 0x9c9b1f to 0x68e6a7 (0x33b477 bytes) 000053[0] 1788823853.635887 mapwrite 0x05c000 thru 0x06864a (0xc64b bytes) 000054[1] 1788823853.742943 read_OD 0x55f000 thru 0x561fff (0x3000 bytes) 000055[1] 1788823853.775388 read_OD 0x38f000 thru 0x394fff (0x6000 bytes) 000056[0] 1788823853.781946 trunc from 0x68e6a8 to 0x481812 (0x20ce95 bytes) 000057[1] 1788823853.820819 punch from 0x184be0 to 0x190e0d (0xc22e bytes) 000059[1] 1788823853.861213 trunc from 0x481813 to 0x04340c (0x43e406 bytes) 000062[1] 1788823853.910857 punch from 0x01b165 to 0x01df58 (0x2df4 bytes) 000063[0] 1788823853.927305 trunc from 0x04340d to 0x92fc91 (0x8ec885 bytes) 000064[0] 1788823853.960020 trunc from 0x92fc92 to 0x805d06 (0x129f8b bytes) 000072[0] 1788823854.091945 mapwrite 0x1c5000 thru 0x1c6c33 (0x1c34 bytes) 000073[0] 1788823854.177057 punch from 0x12f8e1 to 0x136ac4 (0x71e4 bytes) 000074[1] 1788823854.223356 mapread 0x7ac000 thru 0x7b9b87 (0xdb88 bytes) 000075[1] 1788823854.293016 punch from 0x3a489f to 0x3ae01d (0x977f bytes) 000078[0] 1788823854.371147 write_OD 0x77c000 thru 0x781fff (0x6000 bytes) 000079[1] 1788823854.440990 mapread 0x5e6000 thru 0x5ed36e (0x736f bytes) 000081[1] 1788823854.501976 mapwrite 0x7eb000 thru 0x7fa705 (0xf706 bytes) 000082[1] 1788823854.674904 mapwrite 0x723000 thru 0x727d60 (0x4d61 bytes) 000083[0] 1788823854.803763 mapwrite 0x2d2000 thru 0x2d5b91 (0x3b92 bytes) 000084[1] 1788823854.871931 punch from 0x1ccf5a to 0x1d007f (0x3126 bytes) 000085[1] 1788823854.902606 mapwrite 0x267000 thru 0x26d6aa (0x66ab bytes) 000088[0] 1788823855.054028 read_OD 0x314000 thru 0x317fff (0x4000 bytes) 000091[1] 1788823855.082387 write_OD 0x817000 thru 0x81bfff (0x5000 bytes) 000092[0] 1788823855.123870 read_OD 0x403000 thru 0x407fff (0x5000 bytes) 000095[0] 1788823855.135685 trunc from 0x81c000 to 0x43fe5d (0x3dc1a2 bytes) 000096[0] 1788823855.169888 write_OD 0x6db000 thru 0x6e4fff (0xa000 bytes) 000099[0] 1788823855.222982 read_OD 0x5a3000 thru 0x5adfff (0xb000 bytes) 000101[1] 1788823855.234900 trunc from 0x6e5000 to 0x9b7910 (0x2d2911 bytes) 000102[1] 1788823855.267019 mapread 0x2d1000 thru 0x2d1511 (0x512 bytes) 000103[1] 1788823855.331952 mapwrite 0x07e000 thru 0x08863c (0xa63d bytes) 000104[1] 1788823855.411343 mapwrite 0x861000 thru 0x8657e6 (0x47e7 bytes) 000107[0] 1788823855.572446 write_OD 0x0f1000 thru 0x0fafff (0xa000 bytes) 000109[1] 1788823855.641940 mapread 0x88f000 thru 0x893c75 (0x4c76 bytes) 000110[1] 1788823855.664426 mapwrite 0x887000 thru 0x891144 (0xa145 bytes) 000111[1] 1788823855.847589 trunc from 0x9b7911 to 0x08a744 (0x92d1cc bytes) 000112[0] 1788823855.875884 mapwrite 0x58d000 thru 0x597dbc (0xadbd bytes) 000113[0] 1788823855.994119 read_OD 0x01e000 thru 0x029fff (0xc000 bytes) 000114[0] 1788823856.060082 mapread 0x1ae000 thru 0x1b6af2 (0x8af3 bytes) 000115[1] 1788823856.111143 read_OD 0x0ca000 thru 0x0d5fff (0xc000 bytes) 000116[0] 1788823856.135112 mapread 0x061000 thru 0x06bcf1 (0xacf2 bytes) 000117[1] 1788823856.167273 read_OD 0x30f000 thru 0x316fff (0x8000 bytes) 000119[1] 1788823856.189684 punch from 0x29a7e2 to 0x2a9cc8 (0xf4e7 bytes) 000120[1] 1788823856.235093 mapread 0x0b7000 thru 0x0b8c87 (0x1c88 bytes) 000121[0] 1788823856.251330 write_OD 0x4a6000 thru 0x4a7fff (0x2000 bytes) 000122[0] 1788823856.306869 write_OD 0x2d7000 thru 0x2d8fff (0x2000 bytes) 000124[0] 1788823856.401512 write_OD 0x96f000 thru 0x975fff (0x7000 bytes) 000125[0] 1788823856.485237 punch from 0x7f1a3b to 0x7f9da1 (0x8367 bytes) 000126[0] 1788823856.539034 read_OD 0x8cc000 thru 0x8ccfff (0x1000 bytes) 000128[0] 1788823856.562719 punch from 0x0b3f84 to 0x0b8087 (0x4104 bytes) 000130[0] 1788823856.612100 trunc from 0x976000 to 0x4e8a86 (0x48d579 bytes) 000132[0] 1788823856.695181 mapwrite 0x3fd000 thru 0x40c91c (0xf91d bytes) 000133[1] 1788823856.823229 mapwrite 0x863000 thru 0x8664e7 (0x34e8 bytes) 000136[0] 1788823856.965817 trunc from 0x8664e8 to 0x9b58b1 (0x14f3ca bytes) 000139[0] 1788823857.055587 mapread 0x8c5000 thru 0x8cc545 (0x7546 bytes) 000142[1] 1788823857.150923 punch from 0x23d86d to 0x249fe2 (0xc776 bytes) 000143[1] 1788823857.203771 mapread 0x3de000 thru 0x3e48a6 (0x68a7 bytes) 000144[0] 1788823857.223107 mapwrite 0x58c000 thru 0x599431 (0xd432 bytes) 000145[1] 1788823857.457760 read_OD 0x2b9000 thru 0x2c6fff (0xe000 bytes) 000146[0] 1788823857.480525 punch from 0x411abc to 0x418d9d (0x72e2 bytes) 000149[0] 1788823857.561353 mapread 0x8c3000 thru 0x8d19fe (0xe9ff bytes) 000153[1] 1788823857.597555 mapread 0x445000 thru 0x44c1f4 (0x71f5 bytes) 000154[0] 1788823857.633731 read_OD 0x825000 thru 0x825fff (0x1000 bytes) 000155[0] 1788823857.641983 punch from 0x2b3649 to 0x2bf87c (0xc234 bytes) 000160[1] 1788823857.734188 punch from 0x5db321 to 0x5dccae (0x198e bytes) 000161[0] 1788823857.775036 punch from 0x01df73 to 0x029af9 (0xbb87 bytes) 000162[0] 1788823857.800414 trunc from 0x9b58b2 to 0x7194d6 (0x29c3db bytes) 000164[0] 1788823857.854408 mapread 0x4b9000 thru 0x4b912f (0x130 bytes) 000166[0] 1788823857.872107 punch from 0x1387ea to 0x144f61 (0xc778 bytes) 000167[1] 1788823857.894940 mapread 0x210000 thru 0x214b86 (0x4b87 bytes) 000168[1] 1788823857.936875 trunc from 0x7194d7 to 0x5854e1 (0x193ff5 bytes) 000169[0] 1788823857.980079 trunc from 0x5854e2 to 0x7771d1 (0x1f1cf0 bytes) 000170[0] 1788823858.024697 punch from 0x096b9d to 0x09ab98 (0x3ffc bytes) 000171[0] 1788823858.031874 punch from 0x111dee to 0x117f5c (0x616f bytes) 000172[1] 1788823858.036228 mapwrite 0x5fe000 thru 0x60661d (0x861e bytes) 000175[1] 1788823858.201256 mapwrite 0x2b8000 thru 0x2c7182 (0xf183 bytes) 000176[0] 1788823858.352588 mapwrite 0x66d000 thru 0x671ceb (0x4cec bytes) 000177[1] 1788823858.561126 punch from 0x00d3c1 to 0x00fe14 (0x2a54 bytes) 000178[0] 1788823858.628122 write_OD 0x6bb000 thru 0x6c3fff (0x9000 bytes) 000182[1] 1788823858.733291 trunc from 0x7771d2 to 0x2531ea (0x523fe7 bytes) 000183[0] 1788823858.798721 mapwrite 0x275000 thru 0x27efd9 (0x9fda bytes) 000188[0] 1788823859.044306 mapread 0x201000 thru 0x2054ed (0x44ee bytes) 000189[1] 1788823859.057902 mapread 0x023000 thru 0x028cb0 (0x5cb1 bytes) 000190[1] 1788823859.114661 mapread 0x11e000 thru 0x124c57 (0x6c58 bytes) 000191[1] 1788823859.124098 mapread 0x221000 thru 0x229f24 (0x8f25 bytes) 000195[1] 1788823859.171544 trunc from 0x27efda to 0x5f39e6 (0x374a0d bytes) 000196[1] 1788823859.277584 trunc from 0x5f39e7 to 0x449c93 (0x1a9d53 bytes) 000197[1] 1788823859.363047 punch from 0x1d4230 to 0x1dc138 (0x7f09 bytes) 000199[1] 1788823859.375088 mapread 0x081000 thru 0x088133 (0x7134 bytes) 000200[0] 1788823859.426451 trunc from 0x449c94 to 0x4a4204 (0x5a571 bytes) 000202[0] 1788823859.549969 write_OD 0x91f000 thru 0x923fff (0x5000 bytes) 000204[0] 1788823859.763597 trunc from 0x924000 to 0x95da5b (0x39a5c bytes) 000205[0] 1788823859.805360 punch from 0x2fc97a to 0x30affe (0xe685 bytes) 000206[0] 1788823859.825048 trunc from 0x95da5c to 0x81d284 (0x1407d7 bytes) 000209[0] 1788823859.894931 mapread 0x116000 thru 0x11981a (0x381b bytes) 000210[1] 1788823859.929047 trunc from 0x81d285 to 0x2dd63a (0x53fc4a bytes) 000215[0] 1788823860.028697 write_OD 0x437000 thru 0x439fff (0x3000 bytes) 000218[0] 1788823860.077497 punch from 0x12a95c to 0x12e18d (0x3832 bytes) 000219[1] 1788823860.102794 mapwrite 0x416000 thru 0x41a082 (0x4083 bytes) 000220[0] 1788823860.239199 mapread 0x17f000 thru 0x17fc49 (0xc4a bytes) 000222[0] 1788823860.278880 mapwrite 0x2d5000 thru 0x2dbd9d (0x6d9e bytes) 000223[0] 1788823860.366126 punch from 0x26fb32 to 0x275482 (0x5951 bytes) 000226[1] 1788823860.429882 punch from 0x0ac855 to 0x0b9f65 (0xd711 bytes) 000228[1] 1788823860.448189 punch from 0x320a4c to 0x32a562 (0x9b17 bytes) 000230[0] 1788823860.462257 trunc from 0x43a000 to 0x5f3796 (0x1b9797 bytes) 000231[1] 1788823860.503761 mapread 0x085000 thru 0x092eaa (0xdeab bytes) 000232[0] 1788823860.569733 mapwrite 0x3eb000 thru 0x3ebb97 (0xb98 bytes) 000235[1] 1788823860.691362 read_OD 0x246000 thru 0x247fff (0x2000 bytes) 000238[1] 1788823860.724282 read_OD 0x1d1000 thru 0x1ddfff (0xd000 bytes) 000239[0] 1788823860.742843 read_OD 0x4ce000 thru 0x4d8fff (0xb000 bytes) 000240[1] 1788823860.756893 write_OD 0x9e7000 thru 0x9e8fff (0x2000 bytes) 000241[0] 1788823860.827574 punch from 0x90e2cc to 0x90ef37 (0xc6c bytes) 000244[0] 1788823860.859404 punch from 0x5c81a5 to 0x5d714d (0xefa9 bytes) 000246[1] 1788823860.877130 read_OD 0x94e000 thru 0x956fff (0x9000 bytes) 000247[0] 1788823860.910909 read_OD 0x541000 thru 0x544fff (0x4000 bytes) 000248[1] 1788823860.916422 mapwrite 0x95a000 thru 0x9627fd (0x87fe bytes) 000250[0] 1788823861.015960 trunc from 0x9e9000 to 0x3bdfed (0x62b012 bytes) 000251[1] 1788823861.058032 punch from 0x097172 to 0x0a0493 (0x9322 bytes) 000252[1] 1788823861.092563 write_OD 0x3b2000 thru 0x3b7fff (0x6000 bytes) 000254[1] 1788823861.164639 punch from 0x0c6904 to 0x0cae6f (0x456c bytes) 000256[0] 1788823861.184742 punch from 0x074371 to 0x0837fb (0xf48b bytes) 000257[1] 1788823861.223243 read_OD 0x1c2000 thru 0x1cdfff (0xc000 bytes) 000259[0] 1788823861.266743 mapwrite 0x5be000 thru 0x5c7578 (0x9579 bytes) 000260[0] 1788823861.404811 punch from 0x46b223 to 0x473f9b (0x8d79 bytes) 000261[0] 1788823861.452881 write_OD 0x25e000 thru 0x25ffff (0x2000 bytes) 000263[1] 1788823861.558063 mapwrite 0x243000 thru 0x24b186 (0x8187 bytes) 000267[0] 1788823861.756642 read_OD 0x1e3000 thru 0x1e4fff (0x2000 bytes) 000269[0] 1788823861.813517 mapwrite 0x0dd000 thru 0x0e9088 (0xc089 bytes) 000273[1] 1788823862.033170 trunc from 0x5c7579 to 0x80d096 (0x245b1e bytes) 000274[1] 1788823862.076718 write_OD 0x3cd000 thru 0x3cefff (0x2000 bytes) 000283[1] 1788823862.162833 read_OD 0x6c6000 thru 0x6c8fff (0x3000 bytes) 000284[0] 1788823862.167524 trunc from 0x80d097 to 0x147273 (0x6c5e23 bytes) 000285[1] 1788823862.211252 read_OD 0x0a7000 thru 0x0aefff (0x8000 bytes) 000286[0] 1788823862.241769 punch from 0x10e796 to 0x1124db (0x3d46 bytes) 000288[0] 1788823862.280945 mapread 0x0cb000 thru 0x0d69a0 (0xb9a1 bytes) 000289[1] 1788823862.326937 read_OD 0x039000 thru 0x045fff (0xd000 bytes) 000290[0] 1788823862.372640 write_OD 0x216000 thru 0x218fff (0x3000 bytes) 000292[0] 1788823862.446751 mapread 0x086000 thru 0x095671 (0xf672 bytes) 000296[1] 1788823862.514499 trunc from 0x219000 to 0x2709c6 (0x579c7 bytes) 000298[1] 1788823862.607772 read_OD 0x093000 thru 0x098fff (0x6000 bytes) 000299[0] 1788823862.640563 mapwrite 0x276000 thru 0x27ba4c (0x5a4d bytes) 000301[1] 1788823862.838285 write_OD 0x449000 thru 0x457fff (0xf000 bytes) 000303[1] 1788823862.986206 trunc from 0x458000 to 0x6c7180 (0x26f181 bytes) 000304[1] 1788823863.065256 mapread 0x3d9000 thru 0x3dbc49 (0x2c4a bytes) 000308[0] 1788823863.260098 write_OD 0x42e000 thru 0x43cfff (0xf000 bytes) 000310[1] 1788823863.351642 mapwrite 0x22e000 thru 0x23442d (0x642e bytes) 000312[0] 1788823863.541307 mapwrite 0x6f5000 thru 0x6fdb06 (0x8b07 bytes) 000313[0] 1788823863.667113 read_OD 0x533000 thru 0x539fff (0x7000 bytes) 000314[1] 1788823863.700004 punch from 0x4ed923 to 0x4eed0c (0x13ea bytes) 000315[0] 1788823863.735094 mapwrite 0x7fa000 thru 0x80707d (0xd07e bytes) 000316[0] 1788823863.952046 trunc from 0x80707e to 0x956c40 (0x14fbc3 bytes) 000318[1] 1788823863.970925 read_OD 0x785000 thru 0x785fff (0x1000 bytes) 000319[0] 1788823863.988381 trunc from 0x956c41 to 0x660df4 (0x2f5e4c bytes) 000320[0] 1788823864.023933 punch from 0x272d05 to 0x275907 (0x2c03 bytes) 000321[0] 1788823864.033200 read_OD 0x496000 thru 0x49cfff (0x7000 bytes) 000324[1] 1788823864.059698 mapread 0x299000 thru 0x2a853f (0xf540 bytes) 000326[0] 1788823864.086157 punch from 0x4c9fef to 0x4d4475 (0xa487 bytes) 000327[0] 1788823864.111607 write_OD 0x02a000 thru 0x02bfff (0x2000 bytes) 000331[0] 1788823864.158846 mapread 0x154000 thru 0x159dd5 (0x5dd6 bytes) 000335[1] 1788823864.212413 mapwrite 0x190000 thru 0x193223 (0x3224 bytes) 000337[1] 1788823864.384308 write_OD 0x963000 thru 0x96afff (0x8000 bytes) 000338[0] 1788823864.526903 read_OD 0x498000 thru 0x49bfff (0x4000 bytes) 000339[0] 1788823864.542058 read_OD 0x62f000 thru 0x632fff (0x4000 bytes) 000340[1] 1788823864.553228 trunc from 0x96b000 to 0x7e3687 (0x187978 bytes) 000345[1] 1788823864.638096 trunc from 0x7e3688 to 0x7b0036 (0x33651 bytes) 000346[0] 1788823864.664896 punch from 0x18e011 to 0x19ace2 (0xccd2 bytes) 000347[1] 1788823864.695588 read_OD 0x5a9000 thru 0x5affff (0x7000 bytes) 000348[1] 1788823864.728578 mapread 0x6b8000 thru 0x6bfa68 (0x7a69 bytes) 000349[0] 1788823864.764561 write_OD 0x211000 thru 0x218fff (0x8000 bytes) 000350[0] 1788823864.808269 mapread 0x632000 thru 0x63d9df (0xb9e0 bytes) 000352[1] 1788823864.834674 mapread 0x16c000 thru 0x16fb2d (0x3b2e bytes) 000356[1] 1788823864.853532 mapwrite 0x5eb000 thru 0x5f2d83 (0x7d84 bytes) 000357[0] 1788823864.956887 mapread 0x2fd000 thru 0x305f6e (0x8f6f bytes) 000360[0] 1788823865.016906 trunc from 0x7b0037 to 0x699bbf (0x116477 bytes) 000361[0] 1788823865.066767 punch from 0x14afaf to 0x152c1c (0x7c6e bytes) 000362[0] 1788823865.090797 read_OD 0x192000 thru 0x19ffff (0xe000 bytes) 000363[0] 1788823865.127615 write_OD 0x0ff000 thru 0x10dfff (0xf000 bytes) 000365[0] 1788823865.179693 mapwrite 0x117000 thru 0x12470d (0xd70e bytes) 000367[0] 1788823865.301954 write_OD 0x207000 thru 0x20cfff (0x6000 bytes) 000371[1] 1788823865.389410 mapwrite 0x857000 thru 0x85f035 (0x8036 bytes) 000373[1] 1788823865.523667 trunc from 0x85f036 to 0x1fa71d (0x664918 bytes) 000375[0] 1788823865.543760 mapread 0x183000 thru 0x18b3c9 (0x83ca bytes) 000378[0] 1788823865.591632 write_OD 0x606000 thru 0x607fff (0x2000 bytes) 000379[0] 1788823865.644432 punch from 0x3fc9d5 to 0x405318 (0x8944 bytes) 000380[1] 1788823865.672062 trunc from 0x608000 to 0x3b55db (0x252a24 bytes) 000382[0] 1788823865.761178 mapread 0x36f000 thru 0x37eb25 (0xfb26 bytes) 000383[0] 1788823865.804785 write_OD 0x6dd000 thru 0x6dffff (0x3000 bytes) 000384[1] 1788823865.837588 write_OD 0x76c000 thru 0x779fff (0xe000 bytes) 000385[0] 1788823865.878179 read_OD 0x758000 thru 0x762fff (0xb000 bytes) 000386[0] 1788823865.888113 read_OD 0x2d7000 thru 0x2e1fff (0xb000 bytes) 000388[1] 1788823865.904549 read_OD 0x0bb000 thru 0x0c0fff (0x6000 bytes) 000390[0] 1788823865.910987 read_OD 0x24e000 thru 0x25afff (0xd000 bytes) 000391[0] 1788823865.923037 mapwrite 0x7ad000 thru 0x7bb497 (0xe498 bytes) 000392[0] 1788823866.024742 mapread 0x0ba000 thru 0x0c855c (0xe55d bytes) 000393[1] 1788823866.054977 mapread 0x4bd000 thru 0x4c720c (0xa20d bytes) 000394[1] 1788823866.088363 punch from 0x02a7f1 to 0x02f606 (0x4e16 bytes) 000397[0] 1788823866.112670 mapread 0x4eb000 thru 0x4f4f1d (0x9f1e bytes) 000398[0] 1788823866.151733 punch from 0x14e9b8 to 0x156555 (0x7b9e bytes) 000399[1] 1788823866.167221 punch from 0x45c5f7 to 0x4675b2 (0xafbc bytes) 000400[0] 1788823866.184905 mapwrite 0x4ab000 thru 0x4b5d31 (0xad32 bytes) 000402[1] 1788823866.301773 mapwrite 0x2bd000 thru 0x2cae7d (0xde7e bytes) 000408[0] 1788823866.484213 mapread 0x579000 thru 0x5859a5 (0xc9a6 bytes) 000410[1] 1788823866.526448 read_OD 0x18d000 thru 0x18ffff (0x3000 bytes) 000412[0] 1788823866.538107 mapwrite 0x7cb000 thru 0x7d7174 (0xc175 bytes) 000413[0] 1788823866.711009 write_OD 0x895000 thru 0x897fff (0x3000 bytes) 000414[1] 1788823866.795004 trunc from 0x898000 to 0x7b91c4 (0xdee3b bytes) 000416[1] 1788823866.862174 write_OD 0x60a000 thru 0x60efff (0x5000 bytes) 000417[1] 1788823866.929146 trunc from 0x7b91c5 to 0x423596 (0x395c2e bytes) 000419[1] 1788823866.968814 mapread 0x153000 thru 0x15e0bc (0xb0bd bytes) 000420[0] 1788823867.005633 write_OD 0x7db000 thru 0x7e1fff (0x7000 bytes) 000421[0] 1788823867.095927 mapwrite 0x035000 thru 0x037acb (0x2acc bytes) 000422[1] 1788823867.196529 write_OD 0x07e000 thru 0x080fff (0x3000 bytes) 000425[1] 1788823867.305856 mapread 0x268000 thru 0x277d2e (0xfd2f bytes) 000429[1] 1788823867.348186 trunc from 0x7e2000 to 0x3794bc (0x468b43 bytes) 000430[1] 1788823867.387457 trunc from 0x3794bd to 0x802436 (0x488f7a bytes) 000431[1] 1788823867.423892 punch from 0x69dbdf to 0x6a9aa0 (0xbec2 bytes) 000433[1] 1788823867.446018 mapwrite 0x972000 thru 0x9781b8 (0x61b9 bytes) 000434[1] 1788823867.555722 punch from 0x75bcb7 to 0x768b79 (0xcec3 bytes) 000435[0] 1788823867.562438 mapread 0x09a000 thru 0x09b700 (0x1701 bytes) 000436[1] 1788823867.582592 punch from 0x0f1997 to 0x0f83fc (0x6a66 bytes) 000438[0] 1788823867.624060 write_OD 0x9a5000 thru 0x9a9fff (0x5000 bytes) 000439[1] 1788823867.677273 trunc from 0x9aa000 to 0x0fe624 (0x8ab9db bytes) 000442[0] 1788823867.736334 write_OD 0x0b4000 thru 0x0befff (0xb000 bytes) 000443[0] 1788823867.772684 punch from 0x0cf278 to 0x0d4c60 (0x59e9 bytes) 000444[1] 1788823867.789748 mapread 0x0db000 thru 0x0dfe56 (0x4e57 bytes) 000445[0] 1788823867.820505 punch from 0x0c727f to 0x0ca418 (0x319a bytes) 000446[1] 1788823867.833820 trunc from 0x0fe625 to 0x6e251e (0x5e3efa bytes) 000447[0] 1788823867.895557 trunc from 0x6e251f to 0x41acc5 (0x2c7859 bytes) 000449[0] 1788823867.928529 punch from 0x17d1da to 0x18aa3f (0xd866 bytes) 000450[0] 1788823867.963951 write_OD 0x8c3000 thru 0x8c4fff (0x2000 bytes) 000452[0] 1788823868.022835 mapwrite 0x399000 thru 0x39bae7 (0x2ae8 bytes) 000456[1] 1788823868.135038 trunc from 0x8c5000 to 0x71f1ab (0x1a5e54 bytes) 000459[1] 1788823868.233169 punch from 0x1f7d77 to 0x205b9d (0xde27 bytes) 000462[0] 1788823868.253533 mapread 0x436000 thru 0x43ad84 (0x4d85 bytes) 000464[0] 1788823868.294398 trunc from 0x71f1ac to 0x3f22d7 (0x32ced4 bytes) 000465[0] 1788823868.325626 mapread 0x3a1000 thru 0x3b057f (0xf580 bytes) 000466[0] 1788823868.364335 write_OD 0x71b000 thru 0x729fff (0xf000 bytes) 000467[0] 1788823868.458450 mapread 0x026000 thru 0x02987b (0x387c bytes) 000468[1] 1788823868.547310 trunc from 0x72a000 to 0x7af1b4 (0x851b5 bytes) 000469[1] 1788823868.597294 trunc from 0x7af1b5 to 0x83d927 (0x8e773 bytes) 000471[0] 1788823868.679711 write_OD 0x9b8000 thru 0x9bbfff (0x4000 bytes) 000473[0] 1788823868.764992 read_OD 0x015000 thru 0x021fff (0xd000 bytes) 000474[0] 1788823868.776185 punch from 0x3a28d3 to 0x3ae3e6 (0xbb14 bytes) 000476[0] 1788823868.805200 trunc from 0x9bc000 to 0x561071 (0x45af8e bytes) 000477[1] 1788823868.821614 punch from 0x1b2067 to 0x1b4c8f (0x2c29 bytes) 000479[0] 1788823868.853032 mapread 0x0d7000 thru 0x0ddf3f (0x6f40 bytes) 000481[1] 1788823868.898835 mapread 0x0a6000 thru 0x0b1789 (0xb78a bytes) 000482[1] 1788823868.920320 mapread 0x0aa000 thru 0x0b5343 (0xb344 bytes) 000485[0] 1788823868.922014 read_OD 0x36a000 thru 0x373fff (0xa000 bytes) 000486[0] 1788823868.930167 write_OD 0x910000 thru 0x913fff (0x4000 bytes) 000487[1] 1788823868.970964 trunc from 0x914000 to 0x296b8c (0x67d473 bytes) 000488[0] 1788823869.024630 trunc from 0x296b8d to 0x76ef1a (0x4d838e bytes) 000489[1] 1788823869.049104 read_OD 0x525000 thru 0x52ffff (0xb000 bytes) 000491[0] 1788823869.094242 write_OD 0x4d6000 thru 0x4dcfff (0x7000 bytes) 000492[0] 1788823869.181985 mapwrite 0x0d9000 thru 0x0dacdd (0x1cde bytes) 000493[1] 1788823869.263523 read_OD 0x086000 thru 0x086fff (0x1000 bytes) 000494[0] 1788823869.289177 punch from 0x56a735 to 0x578c49 (0xe515 bytes) 000495[0] 1788823869.314129 mapread 0x1a6000 thru 0x1b12e1 (0xb2e2 bytes) 000496[1] 1788823869.373884 mapread 0x631000 thru 0x63a386 (0x9387 bytes) 000498[0] 1788823869.442588 mapwrite 0x4fc000 thru 0x5080c2 (0xc0c3 bytes) 000500[1] 1788823869.575453 read_OD 0x6e3000 thru 0x6f0fff (0xe000 bytes) All operations completed A-OK! PASS 16a (49s) == sanityn test 16b: 500 iterations of dual-mount fsx at small size ========================================================== 19:31:13 (1788823873) keep default fallocate mode: 2 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.297033 s, 28.2 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.249812 s, 33.6 MB/s Chance of close/open is 1 in 50 Seed set to 3876 fd 0: /mnt/lustre/f16b.sanityn fd 1: /mnt/lustre2/f16b.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read truncating to largest ever: 0x1b645 000004[1] 1788823877.411703 trunc from 00000000 to 0x01b644 (0x1b645 bytes) 000008[0] 1788823877.484676 punch from 0x001b47 to 0x003867 (0x1d21 bytes) 000012[1] 1788823877.584099 mapread 0x010e96 thru 0x012d3b (0x1ea6 bytes) 000014[0] 1788823877.718025 mapwrite 0x705193 thru 0x7068bc (0x172a bytes) 000016[1] 1788823877.916417 read 0x5b1394 thru 0x5b3087 (0x1cf4 bytes) truncating to largest ever: 0x40b6ca 000017[1] 1788823877.963444 trunc from 0x7068bd to 0x40b6c9 (0x2fb1f3 bytes) 000019[1] 1788823878.061893 read 0x30daae thru 0x30eda4 (0x12f7 bytes) 000022[1] 1788823878.089780 punch from 0x30ec6c to 0x30ee6f (0x204 bytes) 000024[0] 1788823878.094559 write 0x9830c5 thru 0x984e65 (0x1da1 bytes) 000025[0] 1788823878.116067 mapwrite 0x3cc361 thru 0x3ccfb5 (0xc55 bytes) 000026[1] 1788823878.187696 write 0x0ad89a thru 0x0ae478 (0xbdf bytes) truncating to largest ever: 0x80b2cf 000029[1] 1788823878.304779 trunc from 0x984e66 to 0x80b2ce (0x179b97 bytes) 000030[1] 1788823878.351149 mapwrite 0x2b9d2d thru 0x2baee5 (0x11b9 bytes) 000031[1] 1788823878.401768 read 0x40ffb9 thru 0x411a01 (0x1a49 bytes) 000034[0] 1788823878.409076 write 0x66fba1 thru 0x66ff3c (0x39c bytes) 000037[1] 1788823878.456795 mapwrite 0x54ebe6 thru 0x54f00e (0x429 bytes) 000038[1] 1788823878.557020 write 0x220105 thru 0x221c75 (0x1b71 bytes) 000041[1] 1788823878.624771 mapread 0x074f2c thru 0x0758c8 (0x99d bytes) 000042[0] 1788823878.645505 mapread 0x574a00 thru 0x574df2 (0x3f3 bytes) 000043[0] 1788823878.677766 write 0x52d3c4 thru 0x52e957 (0x1594 bytes) truncating to largest ever: 0x9fe48d 000045[0] 1788823878.739941 trunc from 0x80b2cf to 0x9fe48c (0x1f31be bytes) 000051[0] 1788823878.837929 write 0x58f7ff thru 0x58f873 (0x75 bytes) 000053[0] 1788823878.896576 trunc from 0x9fe48d to 0x64e1c5 (0x3b02c7 bytes) 000054[1] 1788823878.958626 mapread 0x16c07f thru 0x16c82d (0x7af bytes) 000055[0] 1788823879.009331 punch from 0x618223 to 0x619ba4 (0x1982 bytes) 000060[1] 1788823879.042927 write 0x653b3a thru 0x653c8f (0x156 bytes) 000061[0] 1788823879.083497 trunc from 0x653c90 to 0x67ff35 (0x2c2a6 bytes) 000062[1] 1788823879.154837 read 0x1e1e90 thru 0x1e301b (0x118c bytes) 000064[0] 1788823879.187559 trunc from 0x67ff36 to 0x76c60d (0xec6d8 bytes) 000065[1] 1788823879.215371 mapwrite 0x7553ab thru 0x75647d (0x10d3 bytes) 000066[1] 1788823879.322024 punch from 0x4b5f96 to 0x4b6512 (0x57d bytes) 000067[1] 1788823879.371189 trunc from 0x76c60e to 0x210e7a (0x55b793 bytes) 000068[1] 1788823879.424018 write 0x838880 thru 0x83944b (0xbcc bytes) 000069[0] 1788823879.430926 mapwrite 0x06d54f thru 0x06efed (0x1a9f bytes) 000070[0] 1788823879.536026 read 0x0b9d6b thru 0x0ba348 (0x5de bytes) 000071[1] 1788823879.564273 mapread 0x06bed2 thru 0x06c06e (0x19d bytes) 000074[1] 1788823879.609715 write 0x4c162f thru 0x4c2363 (0xd35 bytes) 000075[1] 1788823879.644350 mapread 0x219abf thru 0x21a867 (0xda9 bytes) 000077[1] 1788823879.683102 punch from 0x0a4133 to 0x0a4d83 (0xc51 bytes) 000080[0] 1788823879.792574 mapwrite 0x0d1707 thru 0x0d228b (0xb85 bytes) 000081[0] 1788823879.886898 mapread 0x401f0b thru 0x402d82 (0xe78 bytes) 000086[0] 1788823879.969896 mapread 0x755c3d thru 0x7562c1 (0x685 bytes) 000087[0] 1788823879.979515 mapwrite 0x6c1b7a thru 0x6c22b9 (0x740 bytes) 000088[1] 1788823880.068571 punch from 0x57c115 to 0x57c3a1 (0x28d bytes) 000091[0] 1788823880.181683 mapwrite 0x47737d thru 0x477cae (0x932 bytes) 000093[0] 1788823880.391988 read 0x4104b7 thru 0x4114aa (0xff4 bytes) 000094[1] 1788823880.398409 mapread 0x29e311 thru 0x29f552 (0x1242 bytes) 000095[0] 1788823880.472191 write 0x89d642 thru 0x89e0a6 (0xa65 bytes) 000097[0] 1788823880.574434 write 0x347984 thru 0x347efe (0x57b bytes) 000098[1] 1788823880.581097 mapwrite 0x5edb38 thru 0x5ef536 (0x19ff bytes) 000100[0] 1788823880.729735 trunc from 0x89e0a7 to 0x2233d4 (0x67acd2 bytes) 000101[0] 1788823880.770387 mapread 0x173182 thru 0x174af2 (0x1971 bytes) 000103[1] 1788823880.805597 write 0x4d7d78 thru 0x4d9030 (0x12b9 bytes) 000104[1] 1788823880.831902 read 0x168c60 thru 0x16a035 (0x13d6 bytes) 000106[0] 1788823880.850690 write 0x85106b thru 0x852804 (0x179a bytes) 000107[0] 1788823880.884562 write 0x30a2cb thru 0x30b5e9 (0x131f bytes) 000109[1] 1788823880.970435 mapwrite 0x34f73d thru 0x3509f0 (0x12b4 bytes) 000115[1] 1788823881.204030 write 0x89886b thru 0x899780 (0xf16 bytes) 000116[0] 1788823881.224754 read 0x5a9549 thru 0x5a998b (0x443 bytes) 000117[0] 1788823881.255208 trunc from 0x899781 to 0x604e41 (0x29493f bytes) 000118[1] 1788823881.305436 trunc from 0x604e42 to 0x2eadef (0x31a052 bytes) 000120[0] 1788823881.378406 mapread 0x1bfeb9 thru 0x1c1ba6 (0x1cee bytes) 000122[0] 1788823881.452299 write 0x10c328 thru 0x10c92a (0x603 bytes) 000125[1] 1788823881.509582 punch from 0x09924f to 0x09995f (0x711 bytes) 000127[0] 1788823881.557696 punch from 0x1b733e to 0x1b830d (0xfd0 bytes) 000128[1] 1788823881.585436 write 0x11f407 thru 0x120ba0 (0x179a bytes) 000129[0] 1788823881.624589 punch from 0x058734 to 0x059580 (0xe4d bytes) 000130[0] 1788823881.654107 mapwrite 0x398395 thru 0x398855 (0x4c1 bytes) 000132[1] 1788823881.723819 mapwrite 0x188d27 thru 0x188fe4 (0x2be bytes) 000134[0] 1788823881.814363 mapread 0x2c4fe6 thru 0x2c5b52 (0xb6d bytes) 000136[1] 1788823881.863264 read 0x30769b thru 0x308f6b (0x18d1 bytes) 000137[1] 1788823881.971676 trunc from 0x398856 to 0x4ec6b6 (0x153e61 bytes) 000140[1] 1788823882.079838 punch from 0x00f67f to 0x0101ee (0xb70 bytes) 000141[1] 1788823882.147629 mapread 0x4a96a7 thru 0x4ab55a (0x1eb4 bytes) 000142[0] 1788823882.194439 punch from 0x2da5ef to 0x2dc039 (0x1a4b bytes) 000143[1] 1788823882.217838 trunc from 0x4ec6b7 to 0x472e56 (0x79860 bytes) 000144[0] 1788823882.253040 trunc from 0x472e57 to 0x897857 (0x424a01 bytes) 000145[0] 1788823882.362045 mapread 0x0e630e thru 0x0e78a5 (0x1598 bytes) 000146[1] 1788823882.390728 trunc from 0x897858 to 0x2f4708 (0x5a314f bytes) 000148[1] 1788823882.429204 write 0x4a6b1f thru 0x4a7ac7 (0xfa9 bytes) 000150[1] 1788823882.431428 mapread 0x0c4d82 thru 0x0c5f19 (0x1198 bytes) 000151[0] 1788823882.470672 trunc from 0x4a7ac8 to 0x20277e (0x2a5349 bytes) 000153[1] 1788823882.590041 read 0x0f9146 thru 0x0f9fef (0xeaa bytes) 000154[1] 1788823882.651103 punch from 0x1e0cc1 to 0x1e2860 (0x1ba0 bytes) 000158[1] 1788823882.751620 write 0x944c1e thru 0x944ee6 (0x2c9 bytes) 000160[0] 1788823882.779501 mapwrite 0x20e7d4 thru 0x2103a8 (0x1bd5 bytes) 000164[0] 1788823882.921831 mapwrite 0x83be33 thru 0x83d5d5 (0x17a3 bytes) 000165[0] 1788823882.975768 punch from 0x819bfa to 0x81a47a (0x881 bytes) 000166[1] 1788823882.980032 read 0x6b23cf thru 0x6b2afb (0x72d bytes) 000169[0] 1788823883.015762 write 0x081308 thru 0x081f44 (0xc3d bytes) 000170[0] 1788823883.023597 mapwrite 0x8beee6 thru 0x8c0a21 (0x1b3c bytes) 000171[1] 1788823883.082222 mapread 0x08bcfa thru 0x08cd1e (0x1025 bytes) 000172[1] 1788823883.130807 mapwrite 0x624bba thru 0x62640a (0x1851 bytes) 000173[1] 1788823883.218747 trunc from 0x944ee7 to 0x3d677a (0x56e76c bytes) 000175[1] 1788823883.298921 mapread 0x0cdcc6 thru 0x0ce03f (0x37a bytes) 000177[0] 1788823883.339281 punch from 0x00f27d to 0x00fd02 (0xa86 bytes) 000178[0] 1788823883.399417 mapwrite 0x9efbc5 thru 0x9f032a (0x766 bytes) 000182[0] 1788823883.580662 read 0x243edb thru 0x24405c (0x182 bytes) 000184[0] 1788823883.594005 mapwrite 0x03f3b4 thru 0x040b4c (0x1799 bytes) 000186[0] 1788823883.654888 mapread 0x41426a thru 0x414e0e (0xba5 bytes) 000187[0] 1788823883.662874 punch from 0x4a1198 to 0x4a26e2 (0x154b bytes) 000189[0] 1788823883.672295 read 0x16e7ed thru 0x16f29c (0xab0 bytes) 000190[0] 1788823883.703577 read 0x651ecb thru 0x653e80 (0x1fb6 bytes) 000191[0] 1788823883.709302 trunc from 0x9f032b to 0x613638 (0x3dccf2 bytes) 000192[0] 1788823883.751091 mapread 0x45d087 thru 0x45d42b (0x3a5 bytes) 000193[1] 1788823883.779210 trunc from 0x613639 to 0x3fbc94 (0x2179a4 bytes) 000195[0] 1788823883.912887 write 0x83363f thru 0x833bfd (0x5bf bytes) 000196[1] 1788823883.950051 mapread 0x371604 thru 0x372c90 (0x168d bytes) 000197[0] 1788823883.997846 mapread 0x727ece thru 0x7298ea (0x1a1d bytes) 000201[0] 1788823884.047427 write 0x800bd7 thru 0x8023ba (0x17e4 bytes) 000202[0] 1788823884.076550 trunc from 0x833bfe to 0x5aa620 (0x2895dd bytes) 000203[1] 1788823884.143714 punch from 0x06e9fa to 0x06ef3a (0x541 bytes) 000205[0] 1788823884.225005 mapwrite 0x9d3fcf thru 0x9d479d (0x7cf bytes) 000210[1] 1788823884.357017 mapwrite 0x87f845 thru 0x87ffd6 (0x792 bytes) 000211[0] 1788823884.449870 trunc from 0x9d479e to 0x6bb2e1 (0x3194bc bytes) 000213[1] 1788823884.506625 mapread 0x0449ad thru 0x0455ff (0xc53 bytes) 000215[0] 1788823884.537726 read 0x16d1fe thru 0x16d423 (0x226 bytes) 000217[1] 1788823884.557351 trunc from 0x6bb2e2 to 0x0c2e56 (0x5f848b bytes) 000219[1] 1788823884.647310 write 0x258291 thru 0x259247 (0xfb7 bytes) 000220[1] 1788823884.688430 punch from 0x10cf3c to 0x10cfa8 (0x6d bytes) 000223[0] 1788823884.826545 mapwrite 0x5202d4 thru 0x52131a (0x1047 bytes) 000224[1] 1788823884.966902 mapwrite 0x025cbf thru 0x026b9c (0xede bytes) 000225[1] 1788823885.067328 mapwrite 0x851d20 thru 0x85263c (0x91d bytes) 000226[0] 1788823885.189411 mapwrite 0x12b83d thru 0x12c697 (0xe5b bytes) 000227[1] 1788823885.290348 punch from 0x160baf to 0x161472 (0x8c4 bytes) 000229[1] 1788823885.344971 write 0x4cd12d thru 0x4ce8c9 (0x179d bytes) 000230[1] 1788823885.354329 punch from 0x1e3677 to 0x1e5612 (0x1f9c bytes) 000231[1] 1788823885.357929 read 0x8417e3 thru 0x843285 (0x1aa3 bytes) 000232[0] 1788823885.383006 trunc from 0x85263d to 0x2cb824 (0x586e18 bytes) 000233[0] 1788823885.464363 write 0x116b7f thru 0x1187e0 (0x1c62 bytes) 000235[0] 1788823885.500381 mapread 0x0bba85 thru 0x0bd69b (0x1c17 bytes) 000236[1] 1788823885.540879 write 0x1801a2 thru 0x1820c5 (0x1f24 bytes) 000237[1] 1788823885.591749 punch from 0x23482f to 0x235a69 (0x123b bytes) 000238[0] 1788823885.612673 read 0x1ce737 thru 0x1cec48 (0x512 bytes) 000239[0] 1788823885.696739 read 0x261b1b thru 0x262c0a (0x10f0 bytes) 000241[1] 1788823885.717369 punch from 0x027ec0 to 0x0291c4 (0x1305 bytes) 000243[0] 1788823885.799058 mapwrite 0x1fbe98 thru 0x1fc5c9 (0x732 bytes) 000244[0] 1788823885.879177 punch from 0x0c2375 to 0x0c3557 (0x11e3 bytes) 000245[0] 1788823885.885895 trunc from 0x2cb825 to 0x763c40 (0x49841c bytes) 000246[0] 1788823885.928104 punch from 0x37fe27 to 0x380615 (0x7ef bytes) 000247[0] 1788823885.935071 mapread 0x3deda6 thru 0x3df7aa (0xa05 bytes) 000251[1] 1788823886.009024 write 0x44e69f thru 0x450119 (0x1a7b bytes) 000252[0] 1788823886.041850 trunc from 0x763c41 to 0x664767 (0xff4d9 bytes) 000255[0] 1788823886.145298 read 0x42b7ca thru 0x42bcfc (0x533 bytes) 000257[1] 1788823886.165261 read 0x3b6122 thru 0x3b6953 (0x832 bytes) 000258[0] 1788823886.221923 punch from 0x04f764 to 0x0512dd (0x1b7a bytes) 000262[0] 1788823886.290325 write 0x71ab58 thru 0x71c2ac (0x1755 bytes) 000265[1] 1788823886.383080 mapwrite 0x215da5 thru 0x2178cf (0x1b2b bytes) 000268[1] 1788823886.591818 write 0x99f7f3 thru 0x9a014c (0x95a bytes) 000272[1] 1788823886.672902 trunc from 0x9a014d to 0x7471ee (0x258f5e bytes) 000273[0] 1788823886.726276 read 0x699d6c thru 0x69afdf (0x1274 bytes) 000274[1] 1788823886.766412 mapread 0x2461c1 thru 0x247ab6 (0x18f6 bytes) 000276[0] 1788823886.812271 mapread 0x6ba12b thru 0x6bb3b0 (0x1286 bytes) 000278[0] 1788823886.831375 read 0x19b1e1 thru 0x19b9f3 (0x813 bytes) 000279[0] 1788823886.859441 mapread 0x197bf3 thru 0x199b46 (0x1f54 bytes) 000280[0] 1788823886.875253 punch from 0x5bb364 to 0x5bcc71 (0x190e bytes) 000281[1] 1788823886.891210 punch from 0x1cdb56 to 0x1cea8b (0xf36 bytes) 000282[0] 1788823886.922938 punch from 0x25a85e to 0x25ac52 (0x3f5 bytes) 000283[0] 1788823887.004754 write 0x253618 thru 0x253ecb (0x8b4 bytes) 000285[1] 1788823887.016046 trunc from 0x7471ef to 0x82c082 (0xe4e94 bytes) 000286[0] 1788823887.101890 mapwrite 0x3bbba1 thru 0x3bbfec (0x44c bytes) 000287[0] 1788823887.215057 read 0x68f0ca thru 0x68f5f2 (0x529 bytes) 000290[0] 1788823887.246414 punch from 0x6528e5 to 0x652c2c (0x348 bytes) 000291[1] 1788823887.278542 read 0x8107df thru 0x81182e (0x1050 bytes) 000292[1] 1788823887.295435 write 0x9dd391 thru 0x9de28f (0xeff bytes) 000296[1] 1788823887.381711 read 0x311560 thru 0x312033 (0xad4 bytes) 000299[1] 1788823887.449389 punch from 0x00a5c7 to 0x00bc5e (0x1698 bytes) 000300[0] 1788823887.473088 write 0x6a1bf0 thru 0x6a22d8 (0x6e9 bytes) 000301[0] 1788823887.501763 mapread 0x9217c5 thru 0x923130 (0x196c bytes) 000303[1] 1788823887.554039 mapwrite 0x046665 thru 0x04742f (0xdcb bytes) 000305[1] 1788823887.679980 read 0x81b167 thru 0x81c594 (0x142e bytes) 000306[0] 1788823887.696689 punch from 0x9ab402 to 0x9acc15 (0x1814 bytes) 000307[1] 1788823887.766005 trunc from 0x9de290 to 0x643cd5 (0x39a5ba bytes) 000309[0] 1788823887.924762 punch from 0x337b86 to 0x3397d5 (0x1c50 bytes) 000310[0] 1788823887.958094 mapwrite 0x92a024 thru 0x92a77c (0x759 bytes) 000312[0] 1788823888.113892 mapread 0x50259a thru 0x50415b (0x1bc2 bytes) 000313[1] 1788823888.148596 trunc from 0x92a77d to 0x910875 (0x19f07 bytes) 000316[1] 1788823888.179558 read 0x4c6691 thru 0x4c7d29 (0x1699 bytes) 000322[0] 1788823888.291840 mapwrite 0x0bce8a thru 0x0bd73b (0x8b2 bytes) 000323[1] 1788823888.362759 punch from 0x68ccb6 to 0x68df25 (0x1270 bytes) 000324[1] 1788823888.381256 punch from 0x5c5684 to 0x5c593e (0x2bb bytes) 000325[0] 1788823888.395131 trunc from 0x910876 to 0x4777af (0x4990c6 bytes) 000327[1] 1788823888.455371 write 0x969ce9 thru 0x96b229 (0x1541 bytes) 000328[0] 1788823888.502101 trunc from 0x96b22a to 0x9f28e0 (0x876b7 bytes) 000330[1] 1788823888.665595 trunc from 0x9f28e1 to 0x01bf38 (0x9d69a8 bytes) 000332[1] 1788823888.764880 mapread 0x0155fc thru 0x0161a9 (0xbae bytes) 000333[1] 1788823888.810810 punch from 0x003c50 to 0x00418a (0x53b bytes) 000334[1] 1788823888.818047 write 0x136b60 thru 0x13870e (0x1baf bytes) 000335[0] 1788823888.819867 read 0x135032 thru 0x1354cb (0x49a bytes) 000337[1] 1788823888.880202 mapread 0x003804 thru 0x00449a (0xc97 bytes) 000343[0] 1788823888.960841 mapread 0x0d052e thru 0x0d119d (0xc70 bytes) 000346[1] 1788823889.009047 mapread 0x0cabcf thru 0x0cbfd4 (0x1406 bytes) 000347[0] 1788823889.020220 mapwrite 0x9ea300 thru 0x9ebf1a (0x1c1b bytes) 000349[0] 1788823889.272407 read 0x566d22 thru 0x567f34 (0x1213 bytes) 000351[0] 1788823889.288992 mapread 0x8f1be1 thru 0x8f2f9d (0x13bd bytes) 000352[0] 1788823889.290034 punch from 0x2264c9 to 0x22679c (0x2d4 bytes) 000353[1] 1788823889.313789 write 0x5e5acd thru 0x5e6860 (0xd94 bytes) 000355[0] 1788823889.368568 mapread 0x7139ac thru 0x714d1f (0x1374 bytes) 000356[0] 1788823889.424828 write 0x13793f thru 0x138361 (0xa23 bytes) 000357[1] 1788823889.435385 mapwrite 0x409729 thru 0x409a98 (0x370 bytes) 000358[1] 1788823889.532491 mapwrite 0x44a557 thru 0x44bae0 (0x158a bytes) 000359[0] 1788823889.589777 trunc from 0x9ebf1b to 0x401e36 (0x5ea0e4 bytes) 000360[1] 1788823889.632227 mapread 0x226925 thru 0x228055 (0x1731 bytes) 000362[0] 1788823889.749181 trunc from 0x401e37 to 0x358570 (0xa98c6 bytes) 000363[1] 1788823889.824146 write 0x6e6e96 thru 0x6e7418 (0x583 bytes) 000364[0] 1788823889.867115 trunc from 0x6e7419 to 0x27e9da (0x468a3e bytes) 000367[0] 1788823889.937791 mapwrite 0x3bff19 thru 0x3c0f09 (0xff1 bytes) 000368[0] 1788823890.104755 mapread 0x2df16c thru 0x2df5fc (0x491 bytes) 000369[0] 1788823890.119242 read 0x1a8166 thru 0x1a9cfd (0x1b98 bytes) 000372[1] 1788823890.237330 punch from 0x24b552 to 0x24c63d (0x10ec bytes) 000373[1] 1788823890.302568 read 0x052a3d thru 0x05369b (0xc5f bytes) 000375[0] 1788823890.365684 mapwrite 0x9828f7 thru 0x9833aa (0xab4 bytes) 000376[0] 1788823890.612328 mapread 0x8370ec thru 0x837423 (0x338 bytes) 000377[1] 1788823890.644177 punch from 0x8d0379 to 0x8d0859 (0x4e1 bytes) 000379[1] 1788823890.716026 mapwrite 0x0511df thru 0x051fb6 (0xdd8 bytes) 000380[0] 1788823890.815027 trunc from 0x9833ab to 0x253cf2 (0x72f6b8 bytes) 000381[0] 1788823890.889538 mapwrite 0x6d5e96 thru 0x6d768d (0x17f8 bytes) 000383[1] 1788823891.001548 read 0x407f1b thru 0x4096dc (0x17c2 bytes) 000384[0] 1788823891.025808 read 0x54850a thru 0x54a285 (0x1d7c bytes) 000386[0] 1788823891.048568 write 0x3b170a thru 0x3b1881 (0x178 bytes) 000387[0] 1788823891.072977 trunc from 0x6d768e to 0x50389e (0x1d3def bytes) 000388[0] 1788823891.176720 read 0x291abd thru 0x293a5b (0x1f9f bytes) 000389[0] 1788823891.188161 trunc from 0x50389f to 0x780f85 (0x27d6e7 bytes) 000393[1] 1788823891.325444 mapwrite 0x9b94c6 thru 0x9ba55e (0x1099 bytes) 000394[0] 1788823891.425196 trunc from 0x9ba55f to 0x2baab7 (0x6ffaa7 bytes) 000395[0] 1788823891.461805 write 0x780bd7 thru 0x780f06 (0x330 bytes) 000396[1] 1788823891.465838 write 0x4e3b62 thru 0x4e54d3 (0x1972 bytes) 000399[1] 1788823891.584437 mapread 0x42694e thru 0x42866b (0x1d1e bytes) 000400[1] 1788823891.631582 mapwrite 0x40ba4a thru 0x40c925 (0xedc bytes) 000401[1] 1788823891.764040 trunc from 0x780f07 to 0x3522d9 (0x42ec2d bytes) 000403[1] 1788823891.808972 write 0x976310 thru 0x9768cc (0x5bd bytes) 000405[0] 1788823891.814745 punch from 0x2ac639 to 0x2ae1d3 (0x1b9b bytes) 000407[1] 1788823891.887298 read 0x0f80ce thru 0x0f92e6 (0x1219 bytes) 000411[1] 1788823891.935936 read 0x593c67 thru 0x594c6e (0x1008 bytes) 000412[0] 1788823891.945414 mapwrite 0x510aa0 thru 0x51101e (0x57f bytes) 000414[0] 1788823892.100994 mapwrite 0x17cb31 thru 0x17d096 (0x566 bytes) 000415[1] 1788823892.188566 mapread 0x21be43 thru 0x21d1bb (0x1379 bytes) 000416[1] 1788823892.257640 trunc from 0x9768cd to 0x34c959 (0x629f73 bytes) 000417[1] 1788823892.322555 mapwrite 0x5bdc18 thru 0x5be9bc (0xda5 bytes) 000419[1] 1788823892.448384 write 0x8d2aa9 thru 0x8d3ca7 (0x11ff bytes) 000420[0] 1788823892.450112 trunc from 0x8d3ca8 to 0x71f4a9 (0x1b47fe bytes) 000421[0] 1788823892.503456 mapwrite 0x713bc9 thru 0x714438 (0x870 bytes) 000422[0] 1788823892.573415 write 0x4d247b thru 0x4d3fd3 (0x1b59 bytes) 000423[1] 1788823892.584453 mapread 0x47591c thru 0x477869 (0x1f4e bytes) 000424[0] 1788823892.659300 trunc from 0x71f4aa to 0x2dd92b (0x441b7e bytes) 000425[0] 1788823892.719141 write 0x55b048 thru 0x55c973 (0x192c bytes) 000426[1] 1788823892.727298 write 0x6d7617 thru 0x6d864f (0x1039 bytes) 000428[0] 1788823892.819716 write 0x25a0c2 thru 0x25a6b9 (0x5f8 bytes) 000429[0] 1788823892.845086 write 0x2976d2 thru 0x298b47 (0x1476 bytes) 000431[1] 1788823892.884648 write 0x36adf9 thru 0x36af31 (0x139 bytes) 000432[1] 1788823892.957697 read 0x3169c0 thru 0x316e85 (0x4c6 bytes) 000435[1] 1788823893.001050 mapread 0x6755d9 thru 0x67567b (0xa3 bytes) 000436[1] 1788823893.035930 mapread 0x461948 thru 0x461d2e (0x3e7 bytes) 000437[1] 1788823893.041614 trunc from 0x6d8650 to 0x310219 (0x3c8436 bytes) 000438[1] 1788823893.060257 read 0x15e210 thru 0x15f22f (0x1020 bytes) 000439[1] 1788823893.170641 trunc from 0x31021a to 0x9aefc2 (0x69eda9 bytes) 000441[1] 1788823893.223683 write 0x5f67ab thru 0x5f8161 (0x19b7 bytes) 000442[0] 1788823893.305720 write 0x9eaf43 thru 0x9eb504 (0x5c2 bytes) 000444[1] 1788823893.405893 write 0x62519e thru 0x625e62 (0xcc5 bytes) 000445[1] 1788823893.419276 trunc from 0x9eb505 to 0x2289f9 (0x7c2b0b bytes) 000446[1] 1788823893.546355 read 0x03a59a thru 0x03b35a (0xdc1 bytes) 000449[0] 1788823893.699966 write 0x712651 thru 0x712f4a (0x8fa bytes) 000450[1] 1788823893.723954 read 0x3e7ef3 thru 0x3e820f (0x31d bytes) 000452[1] 1788823893.821261 trunc from 0x712f4b to 0x69a3b5 (0x78b95 bytes) 000454[0] 1788823893.930407 punch from 0x0e1bcb to 0x0e30e5 (0x151b bytes) 000456[0] 1788823894.021428 read 0x466f37 thru 0x466fd3 (0x9d bytes) 000458[1] 1788823894.108275 trunc from 0x69a3b6 to 0x70d2f9 (0x72f44 bytes) 000459[1] 1788823894.144978 write 0x07760b thru 0x077df0 (0x7e6 bytes) 000462[0] 1788823894.215027 mapwrite 0x1f771d thru 0x1f9538 (0x1e1c bytes) 000465[0] 1788823894.400153 punch from 0x42c3a5 to 0x42e287 (0x1ee3 bytes) 000468[0] 1788823894.508836 mapwrite 0x22a9c5 thru 0x22c392 (0x19ce bytes) 000471[1] 1788823894.616041 write 0x31b9d8 thru 0x31d5f2 (0x1c1b bytes) 000473[1] 1788823894.725315 trunc from 0x70d2fa to 0x9a63f5 (0x2990fc bytes) 000475[0] 1788823894.758124 read 0x1ac1df thru 0x1adec6 (0x1ce8 bytes) 000477[1] 1788823894.798858 mapread 0x6a200d thru 0x6a2c98 (0xc8c bytes) 000483[1] 1788823895.025416 trunc from 0x9a63f6 to 0x0054fe (0x9a0ef7 bytes) 000487[1] 1788823895.161296 write 0x9044eb thru 0x905fb2 (0x1ac8 bytes) 000488[0] 1788823895.171532 trunc from 0x905fb3 to 0x8f7374 (0xec3e bytes) 000491[1] 1788823895.292447 read 0x4ce28d thru 0x4ce8cd (0x641 bytes) 000492[1] 1788823895.334285 mapread 0x3897d1 thru 0x38a5bb (0xdeb bytes) 000493[0] 1788823895.367318 mapwrite 0x035347 thru 0x0363be (0x1078 bytes) 000494[1] 1788823895.423755 write 0x379ee7 thru 0x37aea2 (0xfbc bytes) 000495[0] 1788823895.440205 write 0x915acb thru 0x915d9d (0x2d3 bytes) 000496[0] 1788823895.454142 trunc from 0x915d9e to 0x49f588 (0x476815 bytes) 000497[1] 1788823895.499205 read 0x136528 thru 0x1383ac (0x1e85 bytes) 000498[1] 1788823895.520637 write 0x0dd723 thru 0x0dd996 (0x274 bytes) 000499[0] 1788823895.547720 punch from 0x0aa199 to 0x0ab374 (0x11dc bytes) 000500[0] 1788823895.589849 punch from 0x46e9ef to 0x46f399 (0x9ab bytes) All operations completed A-OK! PASS 16b (27s) == sanityn test 16c: verify data consistency on ldiskfs with cache disabled (b=17397) ========================================================== 19:31:40 (1788823900) SKIP: sanityn test_16c dio on ldiskfs only SKIP 16c (4s) == sanityn test 16d: Verify DIO and buffer IO with two clients ========================================================== 19:31:44 (1788823904) 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.372518 s, 22.5 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.375969 s, 22.3 MB/s ldlm.namespaces.MGC192.168.203.151@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff9f8d8666f000.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff9f8d92a09000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff9f8d8666f000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff9f8d92a09000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff9f8d8666f000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff9f8d92a09000.lru_size=clear 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.9383 s, 9.6 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 3.25766 s, 32.2 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 2.19169 s, 47.8 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.59698 s, 18.7 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.75227 s, 12.0 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 3.92368 s, 26.7 MB/s PASS 16d (47s) == sanityn test 16e: Verify size consistency for O_DIRECT write ========================================================== 19:32:31 (1788823951) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.27408 s, 38.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181088 s, 5.8 MB/s PASS 16e (8s) == sanityn test 16f: rw sequential consistency vs drop_caches ========================================================== 19:32:39 (1788823959) TIMEOUT PASS 16f (28s) == sanityn test 16g: mmap rw sequential consistency vs drop_caches ========================================================== 19:33:07 (1788823987) TIMEOUT PASS 16g (28s) == sanityn test 16h: mmap read after truncate file ======= 19:33:35 (1788824015) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16h (7s) == sanityn test 16i: read after truncate file ============ 19:33:42 (1788824022) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16i (6s) == sanityn test 16j: race dio with buffered i/o ========== 19:33:48 (1788824028) 1+0 records in 1+0 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.74216 s, 15.0 MB/s bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.39793 s, 18.8 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.98849 s, 8.8 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.84101 s, 14.2 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.60775 s, 10.1 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.20969 s, 11.9 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.1608 s, 8.3 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.750839 s, 34.9 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.09576 s, 23.9 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.71671 s, 9.6 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.01088 s, 8.7 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.50121 s, 17.5 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.5864 s, 16.5 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.07279 s, 12.6 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.19909 s, 11.9 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.37159 s, 11.1 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.67265 s, 9.8 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.31501 s, 7.9 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.37649 s, 7.8 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK PASS 16j (38s) == sanityn test 16k: Parallel FSX and drop caches should not panic ========================================================== 19:34:26 (1788824066) + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.1 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.2 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.3 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.4 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.5 -l 67108864 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported truncating to largest ever: 0x1a7062c skipping zero size read truncating to largest ever: 0x232e1d3 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: skipping zero size read test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported truncating to largest ever: 0x1a7062c skipping zero size read truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x3c569db truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x3c569db truncating to largest ever: 0x32cce75 truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab All operations completed A-OK! All operations completed A-OK! + fsx 1 done: rc=0 Mon Sep 7 19:35:16 EDT 2026 All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! + fsx 2 done: rc=0 Mon Sep 7 19:35:16 EDT 2026 + fsx 3 done: rc=0 Mon Sep 7 19:35:17 EDT 2026 + fsx 4 done: rc=0 Mon Sep 7 19:35:17 EDT 2026 + fsx 5 done: rc=0 Mon Sep 7 19:35:17 EDT 2026 PASS 16k (57s) == sanityn test 17: resource creation/LVB creation race ========================================================================= 19:35:23 (1788824123) fail_loc=0x8000030a PASS 17 (10s) == sanityn test 18: mmap sanity check =========================================================================================== 19:35:33 (1788824133) mmap test1: basic mmap operation (PASS, 0.12739s) mmap test2: MAP_PRIVATE not write back (PASS, 0.054355s) mmap test3: concurrent mmap ops on two nodes (PASS, 6.3091s) mmap test4: c1 write to f1 from mmapped f2, c2 write to f1 from mmapped f1 (PASS, 4.4447s) mmap test5: read/write file to/from the buffer which mmapped to just this file (PASS, 0.11622s) mmap test6: check mmap write/read content on two nodes (PASS, 0.14926s) mmap test7: file i/o with an unmapped buffer (SKIPPED, 0s) mmap test8: SIGBUS for beyond file size (SKIPPED, 0s) mmap test9: SIGBUS for negative file offset (SKIPPED, 0s) mmap test10: multi-thread mmap access (PASS, 1.4447s) mmap test11: multi-thread shared mmap (PASS, 1.2697s) mmap test10: mtime not change for readonly mmap access (PASS, 1.127s) PASS 18 (23s) == sanityn test 19: test concurrent uncached read races ========================================================================= 19:35:56 (1788824156) oleg351-server: error: get_param: param_path 'osd-*/lustre-OST*/read_cache_enable': No such file or directory pdsh@oleg351-client: oleg351-server: ssh exited with exit code 2 SKIP: sanityn test_19 not cache-capable obdfilter SKIP 19 (4s) == sanityn test 20: test extra readahead page left in cache ============================================================== 19:36:00 (1788824160) PASS 20 (6s) == sanityn test 21: Try to remove mountpoint on another dir ============================================================== 19:36:06 (1788824166) rmdir: removing directory, '/mnt/lustre/d21.sanityn' rmdir: failed to remove '/mnt/lustre/d21.sanityn': Device or resource busy rmdir: removing directory, '/mnt/lustre2/d21.sanityn' Removed mounted directory from another mountpoint, needs to be fixed PASS 21 (6s) == sanityn test 23: others should see updated atime while another read============================================================== 19:36:13 (1788824173) now is 1788824175 starting reads multiop /mnt/lustre/f23.sanityn vor20_c TMPPIPE=/tmp/multiop_open_wait_pipe.7629 new atime is 1788824236 PASS 23 (69s) == sanityn test 24a: lfs df [-ih] [path] test =================================================================================== 19:37:22 (1788824242) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 6144 3763200 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 6144 3763200 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre2 UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 115.3K 403 114.9K 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 114.5K 399 114.1K 1% /mnt/lustre[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.3K 403 114.9K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 399 114.1K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre2 UUID bytes Used Available Use% Mounted on lustre-MDT0000_UUID 2.1G 2.1M 2.1G 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3.6G 6.0M 3.6G 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3.6G 32.0M 3.6G 1% /mnt/lustre[OST:1] filesystem_summary: 7.2G 38.0M 7.2G 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118067 403 117664 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117231 399 116832 1% /mnt/lustre2[OST:1] filesystem_summary: 234882 386 234496 1% /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 6144 3763200 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.3K 403 114.9K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 399 114.1K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 118067 403 117664 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 117231 399 116832 1% /mnt/lustre[OST:1] filesystem_summary: 234882 386 234496 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118067 403 117664 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117231 399 116832 1% /mnt/lustre2[OST:1] filesystem_summary: 234882 386 234496 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 6144 3763200 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 6144 3763200 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre2 PASS 24a (7s) == sanityn test 24b: lfs df should show both filesystems ========================================================================= 19:37:29 (1788824249) PASS 24b (8s) == sanityn test 25a: change ACL on one mountpoint be seen on another ============================================================= 19:37:38 (1788824258) running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] PASS 25a (12s) == sanityn test 25b: change ACL under remote dir on one mountpoint be seen on another ========================================================== 19:37:50 (1788824270) SKIP: sanityn test_25b needs >= 2 MDTs SKIP 25b (5s) == sanityn test 26a: allow mtime to get older ============ 19:37:55 (1788824275) utime: good mknod times 1788824275 <= 1788824276 <= 1788824276 for /mnt/lustre/f26a utime: good mknod times 1788824275 <= 1788824276 <= 1788824276 for /mnt/lustre2/f26a utime: good utime mtimes 100000, atime 200000 utime: updated times for /mnt/lustre2/f26a PASS 26a (7s) == sanityn test 26b: sync mtime between ost and mds ====== 19:38:02 (1788824282) PASS 26b (11s) == sanityn test 26c: set-in-past on open file is not lost on close ========================================================== 19:38:13 (1788824293) multiop /mnt/lustre/f26c.sanityn vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7629 File: /mnt/lustre/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502398 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-09-07 19:38:15.000000000 -0400 Birth: 2026-09-07 19:38:16.000000000 -0400 File: /mnt/lustre2/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502398 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-09-07 19:38:15.000000000 -0400 Birth: 2026-09-07 19:38:16.000000000 -0400 PASS 26c (9s) SKIP: sanityn test_27 skipping excluded test 27 == sanityn test 30: recreate file race =================== 19:38:24 (1788824304) 0 PASS 30 (9s) == sanityn test 31a: voluntary cancel / blocking ast race======================================================================== 19:38:33 (1788824313) fail_loc=0x314 PASS 31a (8s) == sanityn test 31b: voluntary OST cancel / blocking ast race======================================================================== 19:38:41 (1788824321) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg351-server mds-ost sync done. sleep 5 for ZFS MDS Waiting for MDT destroys to complete fail_loc=0x314 fail_loc=0x316 fail_loc=0 fail_loc=0 Filesystem 1K-blocks Used Available Use% Mounted on 192.168.203.151@tcp:/lustre 7542784 39936 7497728 1% /mnt/lustre2 PASS 31b (21s) == sanityn test 31r: open-rename(replace) race =========== 19:39:03 (1788824343) -rw-r--r-- 1 root root 0 Sep 7 19:39 /mnt/lustre/f31r.sanityn.target fail_loc=0x1419 fail_val=3 PASS 31r (10s) == sanityn test 31s: open should not revalidate invalid dentry ========================================================== 19:39:12 (1788824352) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 1205: 43152 Terminated tail -f $tfile PASS 31s (7s) == sanityn test 31t: getattr should not revalidate invalid dentry ========================================================== 19:39:19 (1788824359) PASS 31t (7s) == sanityn test 32b: lockless i/o ======================== 19:39:26 (1788824366) SKIP: sanityn test_32b max_nolock_bytes is removed >= 2.17.53 SKIP 32b (4s) == sanityn test 32c: contention detection ================ 19:39:30 (1788824370) ldlm.namespaces.filter-lustre-OST0000_UUID.contended_locks=2 ldlm.namespaces.filter-lustre-OST0001_UUID.contended_locks=2 ldlm.namespaces.filter-lustre-OST0000_UUID.contention_events=0 ldlm.namespaces.filter-lustre-OST0001_UUID.contention_events=0 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.5311 s, 34.2 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.60593 s, 32.6 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.15617 s, 24.3 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.39377 s, 21.9 MB/s contention_seconds 4, contention_events 18 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.91826 s, 27.3 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.25197 s, 23.3 MB/s ldlm.namespaces.filter-lustre-OST0000_UUID.contention_events=0 ldlm.namespaces.filter-lustre-OST0001_UUID.contention_events=0 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.38339 s, 37.9 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.37118 s, 38.2 MB/s PASS 32c (29s) SKIP: sanityn test_33a skipping SLOW test 33a == sanityn test 33b: COS: cross create/delete, 2 clients, benchmark under remote dir ========================================================== 19:40:00 (1788824400) SKIP: sanityn test_33b Need two or more clients, have 1 SKIP 33b (3s) == sanityn test 33c: Cancel cross-MDT lock should trigger Sync-on-Lock-Cancel ========================================================== 19:40:03 (1788824403) SKIP: sanityn test_33c needs >= 2 MDTs SKIP 33c (4s) == sanityn test 33d: dependent transactions should trigger COS ========================================================== 19:40:07 (1788824407) SKIP: sanityn test_33d needs >= 2 MDTs SKIP 33d (3s) == sanityn test 33e: independent transactions shouldn't trigger COS ========================================================== 19:40:10 (1788824410) SKIP: sanityn test_33e needs >= 2 MDTs SKIP 33e (3s) == sanityn test 34: no lock timeout under IO ============= 19:40:13 (1788824413) fail_loc=0x512 fail_loc=0x512 lock should not expire writing on client1 reading on client2 success fail_loc=0x511 fail_loc=0x511 lock should expire writing on client1 reading on client2 success oleg351-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9f8d8666f000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9f8d8666f000.ost_server_uuid in FULL state after 0 sec oleg351-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9f8d8666f000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9f8d8666f000.ost_server_uuid in IDLE state after 0 sec oleg351-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9f8d8666f000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9f8d8666f000.ost_server_uuid in FULL state after 0 sec oleg351-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9f8d8666f000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9f8d8666f000.ost_server_uuid in IDLE state after 0 sec oleg351-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9f8d8666f000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9f8d8666f000.ost_server_uuid in FULL state after 0 sec oleg351-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9f8d8666f000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9f8d8666f000.ost_server_uuid in IDLE state after 0 sec PASS 34 (126s) == sanityn test 35: -EINTR cp_ast vs. bl_ast race does not evict client ========================================================== 19:42:19 (1788824539) fail_loc=0x80000317 Race attempt 0 Wait for 50458 50537 for 60 sec... - open/close 769 (time 1788824554.88 total 10.01 last 76.83) - open/close 1550 (time 1788824564.88 total 20.01 last 78.08) - open/close 2410 (time 1788824574.88 total 30.02 last 85.95) - open/close 3218 (time 1788824584.89 total 40.02 last 80.77) total: 4000 open/close in 49.75 seconds: 80.40 ops/second fail_loc=0x0 Filesystem Size Used Avail Use% Mounted on 192.168.203.151@tcp:/lustre 7.2G 39M 7.2G 1% /mnt/lustre 192.168.203.151@tcp:/lustre 7.2G 39M 7.2G 1% /mnt/lustre2 PASS 35 (70s) == sanityn test 36: handle ESTALE/open-unlink correctly == 19:43:29 (1788824609) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg351-server mds-ost sync done. Waiting for MDT destroys to complete 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.21835 s, 43.0 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7629 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(0) *** before(7498752) after_dd(7443456) after(7494656) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.11427 s, 47.1 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7629 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(1) *** before(7494656) after_dd(7443456) after(7494656) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.87298 s, 60.1 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7629 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(2) *** before(7494656) after_dd(7443456) after(7494656) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.34222 s, 39.1 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7629 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(3) *** before(7494656) after_dd(7445504) after(7496704) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.07475 s, 48.8 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7629 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(4) *** before(7496704) after_dd(7445504) after(7496704) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.09801 s, 47.7 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7629 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(5) *** before(7496704) after_dd(7445504) after(7496704) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.902474 s, 58.1 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7629 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(6) *** before(7496704) after_dd(7443456) after(7494656) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.873249 s, 60.0 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7629 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(7) *** before(7494656) after_dd(7443456) after(7494656) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.729312 s, 71.9 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7629 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(8) *** before(7494656) after_dd(7443456) after(7494656) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.841201 s, 62.3 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7629 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(9) *** before(7494656) after_dd(7443456) after(7494656) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.916032 s, 57.2 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7629 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(10) *** before(7494656) after_dd(7445504) after(7496704) PASS 36 (294s) == sanityn test 37: check i_size is not updated for directory on close (bug 18695) ======================================================================== 19:48:23 (1788824903) multiop /mnt/lustre/d37.sanityn vD_c TMPPIPE=/tmp/multiop_open_wait_pipe.7629 - create 1685 (time 1788824916.01 total 10.00 last 168.46) - create 3399 (time 1788824926.01 total 20.00 last 171.38) - create 5424 (time 1788824936.01 total 30.01 last 202.45) - create 7316 (time 1788824946.01 total 40.01 last 189.18) - create 9421 (time 1788824956.01 total 50.01 last 210.48) total: 10000 create in 53.36 seconds: 187.41 ops/second PASS 37 (61s) == sanityn test 39a: file mtime does not change after rename ========================================================== 19:49:24 (1788824964) repeat after cancel_lru_locks PASS 39a (4s) == sanityn test 39b: file mtime the same on clients with/out lock ========================================================== 19:49:28 (1788824968) repeat after cancel_lru_locks PASS 39b (6s) == sanityn test 39c: check truncate mtime update ================================================================================ 19:49:34 (1788824974) repeat after cancel_lru_locks PASS 39c (5s) == sanityn test 39d: sync write should update mtime ====== 19:49:39 (1788824979) fail_loc=0x411 fail_loc=0 PASS 39d (4s) SKIP: sanityn test_40a skipping ALWAYS excluded test 40a == sanityn test 40b: pdirops: open|create and others ======================================================================== 19:49:44 (1788824984) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40b (12s) == sanityn test 40c: pdirops: link and others ======================================================================== 19:49:56 (1788824996) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40c (12s) == sanityn test 40d: pdirops: unlink and others ======================================================================== 19:50:08 (1788825008) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40d (12s) == sanityn test 40e: pdirops: rename and others ======================================================================== 19:50:20 (1788825020) No conflict No conflict No conflict No conflict No conflict No conflict PASS 40e (12s) == sanityn test 41a: pdirops: create vs mkdir ======================================================================== 19:50:32 (1788825032) mkdir: cannot create directory '/mnt/lustre2/f41a.sanityn': File exists Conflict PASS 41a (9s) == sanityn test 41b: pdirops: create vs create ======================================================================== 19:50:41 (1788825041) open: File exists Conflict PASS 41b (8s) == sanityn test 41c: pdirops: create vs link ======================================================================== 19:50:49 (1788825049) link: cannot create link '/mnt/lustre2/f41c.sanityn' to '/mnt/lustre2/f41c.sanityn-2': File exists Conflict PASS 41c (9s) == sanityn test 41d: pdirops: create vs unlink ======================================================================== 19:50:58 (1788825058) Conflict PASS 41d (8s) == sanityn test 41e: pdirops: create and rename (tgt) ======================================================================== 19:51:06 (1788825066) Conflict PASS 41e (9s) == sanityn test 41f: pdirops: create and rename (src) ======================================================================== 19:51:15 (1788825075) Conflict PASS 41f (8s) == sanityn test 41g: pdirops: create vs getattr ======================================================================== 19:51:23 (1788825083) Conflict PASS 41g (9s) == sanityn test 41h: pdirops: create vs readdir ======================================================================== 19:51:32 (1788825092) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 2427: lctl set_param -n fail_loc=0 2>/dev/null || true: No such file or directory Conflict PASS 41h (9s) == sanityn test 41i: reint_open: create vs create ======== 19:51:41 (1788825101) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 41i (822s) == sanityn test 42a: pdirops: mkdir vs mkdir ======================================================================== 20:05:23 (1788825923) mkdir: cannot create directory '/mnt/lustre2/f42a.sanityn': File exists Conflict PASS 42a (7s) == sanityn test 42b: pdirops: mkdir vs create ======================================================================== 20:05:30 (1788825930) open: File exists Conflict PASS 42b (6s) == sanityn test 42c: pdirops: mkdir vs link ======================================================================== 20:05:37 (1788825937) link: cannot create link '/mnt/lustre2/f42c.sanityn' to '/mnt/lustre2/f42c.sanityn-2': File exists Conflict PASS 42c (7s) == sanityn test 42d: pdirops: mkdir vs unlink ======================================================================== 20:05:43 (1788825943) Conflict PASS 42d (7s) == sanityn test 42e: pdirops: mkdir and rename (tgt) ======================================================================== 20:05:51 (1788825951) mv: cannot overwrite directory '/mnt/lustre2/f42e.sanityn' with non-directory Conflict PASS 42e (6s) == sanityn test 42f: pdirops: mkdir and rename (src) ======================================================================== 20:05:57 (1788825957) Conflict PASS 42f (8s) == sanityn test 42g: pdirops: mkdir vs getattr ======================================================================== 20:06:05 (1788825965) Conflict PASS 42g (6s) == sanityn test 42h: pdirops: mkdir vs readdir ======================================================================== 20:06:11 (1788825971) Conflict PASS 42h (7s) == sanityn test 43a: rmdir,mkdir doesn't return -EEXIST ======================================================================== 20:06:18 (1788825978) PASS 43a (27s) == sanityn test 43b: pdirops: unlink vs create ======================================================================== 20:06:45 (1788826005) Conflict PASS 43b (6s) == sanityn test 43c: pdirops: unlink vs link ======================================================================== 20:06:51 (1788826011) Conflict PASS 43c (7s) == sanityn test 43d: pdirops: unlink vs unlink ======================================================================== 20:06:58 (1788826018) rm: cannot remove '/mnt/lustre2/f43d.sanityn': No such file or directory Conflict PASS 43d (7s) == sanityn test 43e: pdirops: unlink and rename (tgt) ======================================================================== 20:07:05 (1788826025) Conflict PASS 43e (7s) == sanityn test 43f: pdirops: unlink and rename (src) ======================================================================== 20:07:13 (1788826033) mv: cannot stat '/mnt/lustre2/f43f.sanityn': No such file or directory Conflict PASS 43f (7s) == sanityn test 43g: pdirops: unlink vs getattr ======================================================================== 20:07:20 (1788826040) stat: cannot statx '/mnt/lustre2/f43g.sanityn': No such file or directory Conflict PASS 43g (7s) == sanityn test 43h: pdirops: unlink vs readdir ======================================================================== 20:07:27 (1788826047) Conflict PASS 43h (6s) == sanityn test 43i: pdirops: unlink vs remote mkdir ===== 20:07:33 (1788826053) SKIP: sanityn test_43i needs >= 2 MDTs SKIP 43i (2s) == sanityn test 43j: racy mkdir return EEXIST ======================================================================== 20:07:35 (1788826055) mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists PASS 43j (56s) == sanityn test 43k: unlink vs create ==================== 20:08:31 (1788826111) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 43k (884s) == sanityn test 44a: pdirops: rename tgt vs mkdir ======================================================================== 20:23:15 (1788826995) mkdir: cannot create directory '/mnt/lustre2/f44a.sanityn': File exists Conflict Mon Sep 7 20:23:20 EDT 2026 PASS 44a (8s) == sanityn test 44b: pdirops: rename tgt vs create ======================================================================== 20:23:23 (1788827003) open: File exists Conflict PASS 44b (8s) == sanityn test 44c: pdirops: rename tgt vs link ======================================================================== 20:23:31 (1788827011) link: cannot create link '/mnt/lustre2/f44c.sanityn' to '/mnt/lustre2/f44c.sanityn-3': File exists Conflict PASS 44c (8s) == sanityn test 44d: pdirops: rename tgt vs unlink ======================================================================== 20:23:39 (1788827019) Conflict PASS 44d (7s) == sanityn test 44e: pdirops: rename tgt and rename (tgt) ======================================================================== 20:23:46 (1788827026) Conflict PASS 44e (9s) == sanityn test 44f: pdirops: rename tgt and rename (src) ======================================================================== 20:23:55 (1788827035) Conflict PASS 44f (7s) == sanityn test 44g: pdirops: rename tgt vs getattr ======================================================================== 20:24:02 (1788827042) Conflict PASS 44g (7s) == sanityn test 44h: pdirops: rename tgt vs readdir ======================================================================== 20:24:09 (1788827049) Conflict PASS 44h (8s) == sanityn test 44i: pdirops: rename tgt vs remote mkdir ========================================================== 20:24:17 (1788827057) SKIP: sanityn test_44i needs >= 2 MDTs SKIP 44i (1s) == sanityn test 45a: rename,mkdir doesn't return -EEXIST ======================================================================== 20:24:18 (1788827058) PASS 45a (70s) == sanityn test 45b: pdirops: rename src vs create ======================================================================== 20:25:28 (1788827128) Conflict PASS 45b (8s) == sanityn test 45c: pdirops: rename src vs link ======================================================================== 20:25:36 (1788827136) Conflict PASS 45c (8s) == sanityn test 45d: pdirops: rename src vs unlink ======================================================================== 20:25:44 (1788827144) rm: cannot remove '/mnt/lustre2/f45d.sanityn': No such file or directory Conflict PASS 45d (7s) == sanityn test 45e: pdirops: rename src and rename (tgt) ======================================================================== 20:25:51 (1788827151) Conflict PASS 45e (9s) == sanityn test 45f: pdirops: rename src and rename (src) ======================================================================== 20:26:00 (1788827160) mv: cannot stat '/mnt/lustre2/f45f.sanityn': No such file or directory Conflict PASS 45f (8s) == sanityn test 45g: pdirops: rename src vs getattr ======================================================================== 20:26:09 (1788827169) stat: cannot statx '/mnt/lustre2/f45g.sanityn': No such file or directory Conflict PASS 45g (7s) == sanityn test 45h: pdirops: unlink vs readdir ======================================================================== 20:26:17 (1788827177) Conflict PASS 45h (7s) == sanityn test 45i: pdirops: rename src vs remote mkdir ========================================================== 20:26:24 (1788827184) SKIP: sanityn test_45i needs >= 2 MDTs SKIP 45i (1s) == sanityn test 45j: read vs rename ====================== 20:26:26 (1788827186) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 45j (756s) == sanityn test 46a: pdirops: link vs mkdir ======================================================================== 20:39:02 (1788827942) mkdir: cannot create directory '/mnt/lustre2/f46a.sanityn': File exists Conflict PASS 46a (12s) == sanityn test 46b: pdirops: link vs create ======================================================================== 20:39:14 (1788827954) open: File exists Conflict PASS 46b (12s) == sanityn test 46c: pdirops: link vs link ======================================================================== 20:39:26 (1788827966) link: cannot create link '/mnt/lustre2/f46c.sanityn' to '/mnt/lustre2/f46c.sanityn': File exists Conflict PASS 46c (12s) == sanityn test 46d: pdirops: link vs unlink ======================================================================== 20:39:38 (1788827978) Conflict PASS 46d (13s) == sanityn test 46e: pdirops: link and rename (tgt) ======================================================================== 20:39:51 (1788827991) Conflict PASS 46e (11s) == sanityn test 46f: pdirops: link and rename (src) ======================================================================== 20:40:02 (1788828002) Conflict PASS 46f (12s) == sanityn test 46g: pdirops: link vs getattr ======================================================================== 20:40:14 (1788828014) Conflict PASS 46g (11s) == sanityn test 46h: pdirops: link vs readdir ======================================================================== 20:40:25 (1788828025) Conflict PASS 46h (12s) == sanityn test 46i: pdirops: link vs remote mkdir ======= 20:40:37 (1788828037) SKIP: sanityn test_46i needs >= 2 MDTs SKIP 46i (3s) == sanityn test 47a: pdirops: remote mkdir vs mkdir ====== 20:40:40 (1788828040) SKIP: sanityn test_47a needs >= 2 MDTs SKIP 47a (3s) == sanityn test 47b: pdirops: remote mkdir vs create ===== 20:40:44 (1788828044) SKIP: sanityn test_47b needs >= 2 MDTs SKIP 47b (3s) == sanityn test 47c: pdirops: remote mkdir vs link ======= 20:40:47 (1788828047) SKIP: sanityn test_47c needs >= 2 MDTs SKIP 47c (2s) == sanityn test 47d: pdirops: remote mkdir vs unlink ===== 20:40:50 (1788828050) SKIP: sanityn test_47d needs >= 2 MDTs SKIP 47d (2s) == sanityn test 47e: pdirops: remote mkdir and rename (tgt) ========================================================== 20:40:53 (1788828053) SKIP: sanityn test_47e needs >= 2 MDTs SKIP 47e (3s) == sanityn test 47f: pdirops: remote mkdir and rename (src) ========================================================== 20:40:55 (1788828055) SKIP: sanityn test_47f needs >= 2 MDTs SKIP 47f (3s) == sanityn test 47g: pdirops: remote mkdir vs getattr ==== 20:40:59 (1788828059) SKIP: sanityn test_47g needs >= 2 MDTs SKIP 47g (3s) == sanityn test 50: osc lvb attrs: enqueue vs. CP AST ======================================================================== 20:41:02 (1788828062) 10+0 records in 10+0 records out 10240 bytes (10 kB, 10 KiB) copied, 0.175016 s, 58.5 kB/s fail_loc=0x410 fail_loc=0x0 PASS 50 (11s) == sanityn test 51a: layout lock: refresh layout should work ========================================================== 20:41:13 (1788828073) 0+1 records in 0+1 records out 158 bytes copied, 0.0341114 s, 4.6 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 3690: kill: (218933) - No such process PASS 51a (7s) == sanityn test 51b: layout lock: glimpse should be able to restart if layout changed ========================================================== 20:41:20 (1788828080) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00734313 s, 139 kB/s fail_loc=0x1404 fail_val=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.073118 s, 14.3 MB/s 3145728 PASS 51b (18s) == sanityn test 51c: layout lock: IT_LAYOUT blocked and correct layout can be returned ========================================================== 20:41:38 (1788828098) fail_loc=0x172 Setting layout to have 2 stripes ... 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.029764 s, 34.4 kB/s PASS 51c (10s) == sanityn test 51d: layout lock: losing layout lock should clean up memory map region ========================================================== 20:41:48 (1788828108) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0508917 s, 20.6 MB/s Before revoking layout lock: 1024 KB mapped PASS 51d (7s) == sanityn test 51e: lfs getstripe does not break leases, part 2 ========================================================== 20:41:55 (1788828115) /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 2451 0x993 0x280000400 write lease(2) has applied. /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 2451 0x993 0x280000400 read lease(1) has applied. PASS 51e (7s) == sanityn test 54: rename locking ======================= 20:42:02 (1788828122) llite.lustre-ffff9f8d8666f000.xattr_cache=1 llite.lustre-ffff9f8d92a09000.xattr_cache=1 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505015 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-07 20:42:04.000000000 -0400 Modify: 2026-09-07 20:42:04.000000000 -0400 Change: 2026-09-07 20:42:04.000000000 -0400 Birth: 2026-09-07 20:42:05.000000000 -0400 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000154 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505018 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-07 20:42:10.000000000 -0400 Modify: 2026-09-07 20:42:10.000000000 -0400 Change: 2026-09-07 20:42:10.000000000 -0400 Birth: 2026-09-07 20:42:11.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505021 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-07 20:42:16.000000000 -0400 Modify: 2026-09-07 20:42:16.000000000 -0400 Change: 2026-09-07 20:42:16.000000000 -0400 Birth: 2026-09-07 20:42:17.000000000 -0400 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505020 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-07 20:42:16.000000000 -0400 Modify: 2026-09-07 20:42:16.000000000 -0400 Change: 2026-09-07 20:42:16.000000000 -0400 Birth: 2026-09-07 20:42:17.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000154 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505024 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-07 20:42:22.000000000 -0400 Modify: 2026-09-07 20:42:23.000000000 -0400 Change: 2026-09-07 20:42:23.000000000 -0400 Birth: 2026-09-07 20:42:23.000000000 -0400 File: /mnt/lustre/d1 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505023 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-07 20:42:22.000000000 -0400 Modify: 2026-09-07 20:42:23.000000000 -0400 Change: 2026-09-07 20:42:23.000000000 -0400 Birth: 2026-09-07 20:42:23.000000000 -0400 PASS 54 (30s) == sanityn test 55a: rename vs unlink target dir ========= 20:42:32 (1788828152) fail_loc=0x80000156 mv: cannot move '/mnt/lustre/d55a.sanityn/d1/d2' to '/mnt/lustre/d55a.sanityn/d3/d2': No such file or directory PASS 55a (10s) == sanityn test 55b: rename vs unlink source dir ========= 20:42:43 (1788828163) fail_loc=0x80000156 mv: cannot stat '/mnt/lustre/d55b.sanityn/d1/d2': No such file or directory PASS 55b (10s) == sanityn test 55c: rename vs unlink orphan target dir == 20:42:53 (1788828173) fail_loc=0x156 mv: cannot move '/mnt/lustre/d55c.sanityn/d1/d2' to '/mnt/lustre/d55c.sanityn/d3/d2': No such file or directory PASS 55c (16s) == sanityn test 55d: rename file vs link ================= 20:43:09 (1788828189) fail_loc=0x155 mv: '/mnt/lustre/d55d.sanityn/f1' and '/mnt/lustre/d55d.sanityn/d55d.sanityn/f1' are the same file PASS 55d (14s) == sanityn test 55e: rename race AB/BA under the same parent dir ========================================================== 20:43:23 (1788828203) SKIP: sanityn test_55e needs >= 2 MDTs SKIP 55e (2s) == sanityn test 55f: rename: (P1/A -> P2/B) race with (P2/B -> P1/A) ========================================================== 20:43:25 (1788828205) fail_loc=0x155 We survived after P1A_P2B/P2B_P1A race test fail_loc=0x0 PASS 55f (18s) == sanityn test 55g: rename: race with trylock =========== 20:43:43 (1788828223) mdt.lustre-MDT0000.enable_rename_trylock=1 fail_loc=0x155 mv: cannot move '/mnt/lustre2/d55g.sanityn/b1/b2' to a subdirectory of itself, '/mnt/lustre2/d55g.sanityn/a1/a2/a3/a4/b2' We survived race with trylock test fail_loc=0x0 PASS 55g (20s) == sanityn test 56a: test llverdev with single large stripe ========================================================== 20:44:03 (1788828243) run_llverdev /mnt/lustre/f56a.sanityn -p -s 16000000 -c 4k llverdev: /mnt/lustre/f56a.sanityn is 16777216000000 bytes (15625 GB) in size timestamp: 1788828244 chunksize: 4096 size: 16777216000000 write offset: 16384000000kB 0.744379 MB/s read offset: 16384000000kB inf MB/s llverdev: data verified successfully PASS 56a (87s) == sanityn test 56b: test llverdev and partial verify of wide stripe file ========================================================== 20:45:30 (1788828330) run_llverdev /mnt/lustre/f56b.sanityn -p -s 32000000 -c 4k llverdev: /mnt/lustre/f56b.sanityn is 33554432000000 bytes (31250 GB) in size timestamp: 1788828333 chunksize: 4096 size: 33554432000000 write offset: 32768000000kB 1.74392 MB/s read offset: 32768000000kB inf MB/s llverdev: data verified successfully PASS 56b (79s) == sanityn test 60: Verify data_version behaviour ======== 20:46:49 (1788828409) PASS 60 (5s) == sanityn test 70a: cd directory && rm directory ======== 20:46:54 (1788828414) PASS 70a (5s) == sanityn test 70b: remove files after calling rm_entry ========================================================== 20:46:59 (1788828419) PASS 70b (6s) == sanityn test 71a: correct file map just after write operation is finished ========================================================== 20:47:05 (1788828425) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0289097 s, 2.3 MB/s SKIP: sanityn test_71a ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71a (3s) == sanityn test 71b: check fiemap support for stripecount > 1 ========================================================== 20:47:08 (1788828428) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0111226 s, 5.9 MB/s SKIP: sanityn test_71b ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71b (2s) == sanityn test 71c: check FIEMAP_EXTENT_LAST flag with different extents number ========================================================== 20:47:10 (1788828430) SKIP: sanityn test_71c support only ldiskfs ost SKIP 71c (3s) == sanityn test 71d: fiemap corruption test with fm_extent_count=0 ========================================================== 20:47:13 (1788828433) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0161538 s, 4.1 MB/s SKIP: sanityn test_71d ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71d (2s) == sanityn test 72: getxattr/setxattr cache should be consistent between nodes ========================================================== 20:47:15 (1788828435) llite.lustre-ffff9f8d8666f000.xattr_cache=1 llite.lustre-ffff9f8d92a09000.xattr_cache=1 getfattr: Removing leading '/' from absolute path names user.attr1="value1" getfattr: Removing leading '/' from absolute path names user.attr1="value2" getfattr: Removing leading '/' from absolute path names getfattr: Removing leading '/' from absolute path names b362740ad371eae580e7f85452e8830b - 870e072da1ebfc20ea8c0cd3d6bdf3b6 - PASS 72 (5s) == sanityn test 73: getxattr should not cause xattr lock cancellation ========================================================== 20:47:20 (1788828440) llite.lustre-ffff9f8d8666f000.xattr_cache=1 llite.lustre-ffff9f8d92a09000.xattr_cache=1 getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" PASS 73 (5s) == sanityn test 74: flock deadlock: different mounts ======================================================================== 20:47:25 (1788828445) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00770588 s, 133 kB/s 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0116438 s, 87.9 kB/s 231722: get lock2 231723: get lock1 231722: done 231722: get lock1 231723: done 231723: get lock2 fcntl cmd 7 failed: Resource deadlock avoided 231723: cannot get lock2: Resource deadlock avoided 231723: exit rc=0 231722: done 231722: exit rc=0 PASS 74 (9s) == sanityn test 75: osc: upcall after unuse lock============================================================================= 20:47:34 (1788828454) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0504513 s, 41.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0239222 s, 43.8 MB/s fail_loc=0x31d /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 4373: 232304 Killed stat -c %s $DIR1/$tfile fail_loc=0 PASS 75 (11s) == sanityn test 76: Verify MDT open_files listing ======== 20:47:45 (1788828465) opening files: ........ closing files: ........ PASS 76 (51s) == sanityn test 77a: check FIFO NRS policy =============== 20:48:37 (1788828517) ost.OSS.ost.nrs_policies=fifo ost.OSS.ost_create.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_out.nrs_policies=fifo ost.OSS.ost_seq.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.293943 s, 57.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0551853 s, 19.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.061693 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0987989 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105199 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0795469 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0886515 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145923 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147588 s, 7.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182283 s, 5.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159186 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144611 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165281 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152524 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158955 s, 6.6 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9f8d92a09000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213201 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182927 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162026 s, 64.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.030474 s, 34.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.023064 s, 45.5 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0216053 s, 48.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0258972 s, 40.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0391814 s, 26.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0258236 s, 40.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0549258 s, 19.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0354251 s, 29.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0377213 s, 27.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0383947 s, 27.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.056826 s, 18.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0476116 s, 22.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.039259 s, 26.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0436109 s, 24.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0480816 s, 21.8 MB/s removed '/mnt/lustre/d77a.sanityn/nrs_r_oleg351-client.virtnet' removed '/mnt/lustre/d77a.sanityn/nrs_w_oleg351-client.virtnet' removed directory '/mnt/lustre/d77a.sanityn' PASS 77a (7s) == sanityn test 77b: check CRR-N NRS policy ============== 20:48:43 (1788828523) ost.OSS.ost.nrs_policies=crrn ost.OSS.ost_create.nrs_policies=crrn ost.OSS.ost_io.nrs_policies=crrn ost.OSS.ost_out.nrs_policies=crrn ost.OSS.ost_seq.nrs_policies=crrn ost.OSS.ost.nrs_crrn_quantum=1 ost.OSS.ost_create.nrs_crrn_quantum=1 ost.OSS.ost_io.nrs_crrn_quantum=1 ost.OSS.ost_out.nrs_crrn_quantum=1 ost.OSS.ost_seq.nrs_crrn_quantum=1 policy: crr-n, crrn_quantum 1 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.313864 s, 53.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0477774 s, 21.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0630721 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0649898 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0622578 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0639562 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0865277 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0643299 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0730764 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0582603 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132527 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131787 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16879 s, 6.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183839 s, 5.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182342 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185567 s, 5.7 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9f8d92a09000.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21625 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00536809 s, 195 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0104061 s, 101 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0207422 s, 50.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00418816 s, 250 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155372 s, 67.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0668356 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0715766 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.046315 s, 22.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00388988 s, 270 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0363472 s, 28.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0500477 s, 21.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0644726 s, 16.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0106283 s, 98.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00429726 s, 244 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0228491 s, 45.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0262954 s, 39.9 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg351-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg351-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost.nrs_crrn_quantum=64 ost.OSS.ost_create.nrs_crrn_quantum=64 ost.OSS.ost_io.nrs_crrn_quantum=64 ost.OSS.ost_out.nrs_crrn_quantum=64 ost.OSS.ost_seq.nrs_crrn_quantum=64 policy: crr-n, crrn_quantum 64 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.348377 s, 48.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0395668 s, 26.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0445288 s, 23.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127167 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123997 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12538 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132354 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114812 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122961 s, 8.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128187 s, 8.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13181 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100776 s, 10.4 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9f8d92a09000.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116315 s, 9.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151915 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132566 s, 7.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135326 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170379 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165152 s, 63.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0237302 s, 44.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.020005 s, 52.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0395233 s, 26.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.058491 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.034297 s, 30.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0363678 s, 28.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.040407 s, 26.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0427239 s, 24.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.042774 s, 24.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0547586 s, 19.1 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0464366 s, 22.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0437906 s, 23.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0509011 s, 20.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0468219 s, 22.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.074681 s, 14.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9f8d92a09000.lock_unused_count=2 removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg351-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg351-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77b (8s) == sanityn test 77c: check ORR NRS policy ================ 20:48:51 (1788828531) ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_orr_quantum=1 ost.OSS.ost_io.nrs_orr_offset_type=physical ost.OSS.ost_io.nrs_orr_supported=reads policy: orr, orr_quantum 1, orr_offset_type physical, orr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.408166 s, 41.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0467462 s, 22.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0561344 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0624271 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0473502 s, 22.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0963919 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140526 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164324 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175688 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191923 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175343 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0783103 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222376 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220102 s, 4.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20413 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0845237 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206142 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0818839 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0452869 s, 23.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0490367 s, 21.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.219878 s, 4.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0546326 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10011 s, 10.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0559863 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00438523 s, 239 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0461411 s, 22.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237824 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0595901 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0701652 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0704869 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0728646 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.07434 s, 14.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9f8d92a09000.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100848 s, 10.4 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg351-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg351-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=writes ost.OSS.ost_io.nrs_orr_quantum=64 policy: orr, orr_quantum 64, orr_offset_type physical, orr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.311731 s, 53.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0709632 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0776705 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0713491 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105513 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184892 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188881 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186731 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199107 s, 5.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9f8d92a09000.lock_unused_count=3 ldlm.namespaces.lustre-OST0001-osc-ffff9f8d92a09000.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.24106 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196633 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.231233 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.249541 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22403 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.235369 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.249528 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.262361 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.055096 s, 19.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0358281 s, 29.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00383135 s, 274 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0500177 s, 21.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0985515 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0305986 s, 34.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0588303 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.061361 s, 17.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0838034 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0645305 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0632305 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0439601 s, 23.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0645928 s, 16.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0668656 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0679033 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.069874 s, 15.0 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg351-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg351-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=reads_and_writes ost.OSS.ost_io.nrs_orr_offset_type=logical policy: orr, orr_quantum 64, orr_offset_type logical, orr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.495976 s, 33.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0649109 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0621338 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116194 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136953 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123672 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101088 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123826 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123814 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13549 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142054 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125441 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12618 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166988 s, 6.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166245 s, 6.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160512 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166046 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0263756 s, 39.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0252951 s, 41.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0268555 s, 39.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0353107 s, 29.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0666623 s, 15.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0321998 s, 32.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0431701 s, 24.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0623114 s, 16.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0510841 s, 20.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0582939 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0531499 s, 19.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0549887 s, 19.1 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0542939 s, 19.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0542987 s, 19.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0591834 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0676108 s, 15.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9f8d92a09000.lock_unused_count=2 removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg351-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg351-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77c (11s) == sanityn test 77d: check TRR nrs policy ================ 20:49:02 (1788828542) ost.OSS.ost_io.nrs_policies=trr ost.OSS.ost_io.nrs_trr_quantum=1 ost.OSS.ost_io.nrs_trr_offset_type=physical ost.OSS.ost_io.nrs_trr_supported=reads policy: trr, trr_quantum 1, trr_offset_type physical, trr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.320931 s, 52.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0535457 s, 19.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.054109 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0670627 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110566 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143723 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147422 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12477 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153389 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127438 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133375 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126067 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.09748 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122785 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129914 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147733 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126939 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0317052 s, 33.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0174347 s, 60.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0397868 s, 26.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0248699 s, 42.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0187387 s, 56.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.021338 s, 49.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0241597 s, 43.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0627432 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0293058 s, 35.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.026664 s, 39.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0309883 s, 33.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out ldlm.namespaces.lustre-OST0001-osc-ffff9f8d92a09000.lock_unused_count=2 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0322472 s, 32.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0328842 s, 31.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0326699 s, 32.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.03318 s, 31.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0385598 s, 27.2 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg351-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg351-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=writes ost.OSS.ost_io.nrs_trr_quantum=64 policy: trr, trr_quantum 64, trr_offset_type physical, trr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.247171 s, 67.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0549501 s, 19.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.084965 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0933627 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0970812 s, 10.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114113 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131555 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115573 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0951485 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129949 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125251 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1489 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12318 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169467 s, 6.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148427 s, 7.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1571 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142995 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00459587 s, 228 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0650849 s, 16.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0539161 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0779213 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0413237 s, 25.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0378543 s, 27.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00814222 s, 129 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108107 s, 9.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0362819 s, 28.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0333038 s, 31.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0287756 s, 36.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129668 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0212251 s, 49.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01006 s, 104 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0196303 s, 53.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0201103 s, 52.1 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg351-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg351-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=reads_and_writes ost.OSS.ost_io.nrs_trr_offset_type=logical policy: trr, trr_quantum 64, trr_offset_type logical, trr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.417716 s, 40.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0406946 s, 25.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0684503 s, 15.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170672 s, 6.1 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162761 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187867 s, 5.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172434 s, 6.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203108 s, 5.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168806 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162606 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196839 s, 5.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.224014 s, 4.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180522 s, 5.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199039 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175737 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186568 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.233641 s, 4.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9f8d92a09000.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff9f8d92a09000.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0341392 s, 30.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00834372 s, 126 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00423964 s, 247 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158135 s, 66.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0230608 s, 45.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0628121 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.055195 s, 19.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0642136 s, 16.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0390221 s, 26.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0539222 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0684035 s, 15.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0667476 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0486922 s, 21.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.057431 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0223324 s, 47.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0225031 s, 46.6 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg351-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg351-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77d (10s) == sanityn test 77e: check TBF NID nrs policy ============ 20:49:12 (1788828552) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_tbf_rule=start localhost nid={0@lo} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start clients nid={192.168.203.51@tcp} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start others nid={*.*.*.*@tcp} rate=50 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.306867 s, 54.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0372133 s, 28.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0561933 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0912896 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186215 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173416 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170904 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199005 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196326 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206435 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173829 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16837 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199674 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186533 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210378 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194639 s, 5.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9f8d92a09000.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff9f8d92a09000.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.232187 s, 4.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128531 s, 8.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137662 s, 7.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16084 s, 6.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140173 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140007 s, 7.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154897 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180506 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146151 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153516 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161002 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153347 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153556 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169291 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173927 s, 6.0 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173435 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178881 s, 5.9 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg351-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg351-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change localhost rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change clients rate=101 ost.OSS.ost_io.nrs_tbf_rule=change others rate=51 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.315791 s, 53.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0524413 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0588463 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119514 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157954 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134158 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16042 s, 6.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1819 s, 5.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181263 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212786 s, 4.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.2002 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166881 s, 6.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191739 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215788 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189805 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216162 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202635 s, 5.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9f8d92a09000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9f8d92a09000.lock_unused_count=2 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173478 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16693 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172482 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212731 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181203 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20673 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183168 s, 5.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190012 s, 5.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188527 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191373 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190839 s, 5.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200122 s, 5.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191499 s, 5.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19399 s, 5.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202219 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20857 s, 5.0 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg351-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg351-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop localhost ost.OSS.ost_io.nrs_tbf_rule=stop clients ost.OSS.ost_io.nrs_tbf_rule=stop others 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.395489 s, 42.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0740126 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0518548 s, 20.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0559505 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113667 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101008 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137451 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0869033 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152865 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0934339 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0822024 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166651 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187451 s, 5.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.266353 s, 3.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.24976 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.234008 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.319279 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.02122 s, 49.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9f8d92a09000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9f8d92a09000.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129334 s, 81.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121329 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00332486 s, 315 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126523 s, 82.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145209 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0889666 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107267 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0729113 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0803261 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0565097 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0513518 s, 20.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17653 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.044979 s, 23.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0421685 s, 24.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0399696 s, 26.2 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg351-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg351-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.332054 s, 50.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0616091 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0453319 s, 23.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0640478 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0890054 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0518451 s, 20.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0870713 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0845202 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130066 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134026 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182108 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185653 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176595 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207418 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107897 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190673 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.238008 s, 4.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9f8d92a09000.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0026298 s, 399 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0649544 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00403528 s, 260 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0100232 s, 105 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00284706 s, 368 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.040333 s, 26.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00251874 s, 416 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.069692 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0496734 s, 21.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0819717 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0468259 s, 22.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0544255 s, 19.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0562803 s, 18.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0825521 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0215974 s, 48.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.023121 s, 45.4 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg351-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg351-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' PASS 77e (17s) == sanityn test 77f: check TBF JobID nrs policy ========== 20:49:29 (1788828569) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start runas jobid={iozone.500 dd.500 tiotest.500} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start iozone_runas jobid={iozone.500} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg351-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.334928 s, 50.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0523945 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0621066 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0280488 s, 37.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0501678 s, 20.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121667 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132952 s, 7.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155908 s, 6.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125426 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164394 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139083 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188957 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195332 s, 5.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185579 s, 5.7 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163989 s, 6.4 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9f8d92a09000.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211068 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227786 s, 4.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00442344 s, 237 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0178678 s, 58.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0379949 s, 27.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0322054 s, 32.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00921098 s, 114 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00496378 s, 211 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0249954 s, 42.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00807178 s, 130 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00871949 s, 120 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0736715 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0753812 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0298192 s, 35.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00593523 s, 177 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00686063 s, 153 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.019777 s, 53.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0122126 s, 85.9 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg351-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg351-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change runas rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change iozone_runas rate=101 ost.OSS.ost_io.nrs_tbf_rule=change dd_runas rate=51 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg351-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.345043 s, 48.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0512216 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0922154 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175069 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190422 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185703 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213956 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.232318 s, 4.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169848 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237203 s, 4.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.204156 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.257538 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.272652 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173131 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223264 s, 4.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9f8d92a09000.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffff9f8d92a09000.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222454 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.228228 s, 4.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.063403 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.037919 s, 27.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0651155 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.043558 s, 24.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0451695 s, 23.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0434389 s, 24.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0421988 s, 24.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0492425 s, 21.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0595401 s, 17.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0499882 s, 21.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0532244 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0561609 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0600801 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.071817 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0665118 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0807825 s, 13.0 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg351-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg351-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop runas ost.OSS.ost_io.nrs_tbf_rule=stop iozone_runas ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg351-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.366591 s, 45.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0811824 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0803821 s, 13.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.078208 s, 13.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0914842 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123505 s, 8.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133251 s, 7.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172217 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112806 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193662 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18631 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172373 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156235 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22761 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192884 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198138 s, 5.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9f8d92a09000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9f8d92a09000.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220533 s, 4.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0297138 s, 35.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0399371 s, 26.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0315717 s, 33.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0421966 s, 24.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0625189 s, 16.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0370998 s, 28.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.040458 s, 25.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0548565 s, 19.1 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0575525 s, 18.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0508158 s, 20.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0647362 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0489566 s, 21.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0444834 s, 23.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.05505 s, 19.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0651191 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0516626 s, 20.3 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg351-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg351-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_policies=fifo running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg351-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.30558 s, 54.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0719256 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0684439 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0768841 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0878899 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0770649 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196671 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151725 s, 6.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups:1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200662 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167399 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22844 s, 4.6 MB/s 5001+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192231 s, 5.5 MB/s [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161391 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187585 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.247433 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18612 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.381958 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00619743 s, 169 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102224 s, 10.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159068 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.076291 s, 13.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0628543 s, 16.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0745405 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0573934 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0552262 s, 19.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0841968 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175682 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0480462 s, 21.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0433987 s, 24.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.043211 s, 24.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0297478 s, 35.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0309952 s, 33.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0223712 s, 46.9 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg351-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg351-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' PASS 77f (17s) == sanityn test 77g: Change TBF type directly ============ 20:49:46 (1788828586) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 ost.OSS.ost_io.nrs_policies=fifo PASS 77g (8s) == sanityn test 77h: Wrong policy name should report error, not LBUG ========================================================== 20:49:54 (1788828594) oleg351-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=abc: No such file or directory oleg351-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='abc': No such file or directory pdsh@oleg351-client: oleg351-server: ssh exited with exit code 2 oleg351-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc: Invalid argument oleg351-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc': Invalid argument pdsh@oleg351-client: oleg351-server: ssh exited with exit code 22 oleg351-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf reg abc: Invalid argument oleg351-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf reg abc': Invalid argument pdsh@oleg351-client: oleg351-server: ssh exited with exit code 22 oleg351-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc efg: Invalid argument oleg351-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc efg': Invalid argument pdsh@oleg351-client: oleg351-server: ssh exited with exit code 22 PASS 77h (8s) == sanityn test 77i: Change rank of TBF rule ============= 20:50:02 (1788828602) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start before jobid={jobid} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start after jobid={jobid} rate=1000 rank=default ost.OSS.ost_io.nrs_tbf_rule=start target jobid={jobid} rate=1000 rank=after Move before itself ost.OSS.ost_io.nrs_tbf_rule=change target rank=target Move to higher rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=before Move to lower rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=after Move before default ost.OSS.ost_io.nrs_tbf_rule=change target rank=default ost.OSS.ost_io.nrs_policies=fifo PASS 77i (15s) == sanityn test 77j: check TBF-OPCode NRS policy ========= 20:50:17 (1788828617) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf opcode ost.OSS.ost_io.nrs_tbf_rule=start ost_r opcode={ost_read} rate=5 ost.OSS.ost_io.nrs_tbf_rule=start ost_w opcode={ost_write} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.407574 s, 41.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0393071 s, 26.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0576769 s, 18.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133519 s, 7.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148986 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169684 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165407 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16844 s, 6.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188915 s, 5.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180012 s, 5.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175468 s, 6.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154815 s, 6.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174497 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176224 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174173 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20552 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216737 s, 4.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00862893 s, 122 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0179713 s, 58.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0454614 s, 23.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0205894 s, 50.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0322514 s, 32.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0195948 s, 53.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156712 s, 66.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0256219 s, 40.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0252005 s, 41.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0411487 s, 25.5 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138257 s, 75.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00338973 s, 309 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00456837 s, 230 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150184 s, 69.8 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129488 s, 81.0 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154287 s, 68.0 MB/s removed '/mnt/lustre/d77j.sanityn/nrs_r_oleg351-client.virtnet' removed '/mnt/lustre/d77j.sanityn/nrs_w_oleg351-client.virtnet' removed directory '/mnt/lustre/d77j.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 5, bs: 1048576 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.77784 s, 18.1 MB/s Write runtime is 7 s, speed is 14.285714 vs. 20 Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.565 s, 5.4 MB/s Read runtime is 20 s, speed is 5.000000 vs. 5 cleanup_tbf /mnt/lustre/d77j.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_r ost.OSS.ost_io.nrs_tbf_rule=stop ost_w ost.OSS.ost_io.nrs_policies=fifo PASS 77j (58s) == sanityn test 77ja: check TBF-UID/GID NRS policy ======= 20:51:16 (1788828676) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg351-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg351-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg351-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg351-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg351-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.26003 s, 13.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0839721 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0656356 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123991 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13333 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127704 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126029 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124695 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190295 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118822 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172053 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102713 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1948 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15268 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182591 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155166 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145584 s, 7.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0419791 s, 25.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0338326 s, 31.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0355516 s, 29.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0434473 s, 24.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0567427 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0330575 s, 31.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0438283 s, 23.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0410665 s, 25.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.06202 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0631322 s, 16.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0431197 s, 24.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0535112 s, 19.6 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0560363 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0639248 s, 16.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0698445 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0605054 s, 17.3 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg351-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg351-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5, bs: 1048576 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1048576] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8674 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 vs. 5 Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1048576] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8922 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 vs. 5 cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg351-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg351-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg351-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg351-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg351-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.26235 s, 13.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0813344 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119487 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108027 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145048 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12784 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161444 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180298 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185638 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183844 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18626 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196339 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176923 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190003 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168202 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162894 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186226 s, 5.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0547187 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0563933 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.057491 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0562181 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0370999 s, 28.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0599746 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0693977 s, 15.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0606513 s, 17.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0424107 s, 24.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0548749 s, 19.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0734107 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0631685 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0735334 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0624194 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0788865 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.086463 s, 12.1 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg351-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg351-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5, bs: 1048576 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1048576] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8668 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 vs. 5 Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1048576] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9167 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 vs. 5 cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77ja (142s) == sanityn test 77jb: check TBF-UID/GID NRS policy on files that don't belong to us ========================================================== 20:53:38 (1788828818) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg351-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg351-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg351-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg351-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg351-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.27372 s, 13.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0518857 s, 20.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0468448 s, 22.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0576756 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0659524 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105492 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107162 s, 9.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104795 s, 10.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0922546 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106661 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0692109 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140885 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190925 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183267 s, 5.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189003 s, 5.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185588 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186794 s, 5.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.039184 s, 26.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0314315 s, 33.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0380969 s, 27.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0370262 s, 28.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0323804 s, 32.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.043464 s, 24.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0332655 s, 31.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0453893 s, 23.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0525318 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0489991 s, 21.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0443522 s, 23.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0493303 s, 21.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0389151 s, 26.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0529192 s, 19.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0485162 s, 21.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0470124 s, 22.3 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg351-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg351-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5, bs: 1048576 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1048576] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.924 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 vs. 5 Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1048576] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9064 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 vs. 5 cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg351-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg351-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg351-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg351-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg351-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.27068 s, 13.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0519346 s, 20.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0688932 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0833588 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111999 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102612 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117477 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106885 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116955 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108699 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155437 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157517 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139799 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155961 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191669 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183668 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.232705 s, 4.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0280126 s, 37.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0401161 s, 26.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0454319 s, 23.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0512367 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0528229 s, 19.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0396639 s, 26.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0375566 s, 27.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0385798 s, 27.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0465555 s, 22.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0385625 s, 27.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0471942 s, 22.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0478131 s, 21.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0494109 s, 21.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0620081 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0663795 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0825683 s, 12.7 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg351-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg351-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5, bs: 1048576 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1048576] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8296 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 vs. 5 Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1048576] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9601 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 vs. 5 cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77jb (143s) == sanityn test 77k: check TBF policy with UID/GID/JobID/OPCode expression ========================================================== 20:56:01 (1788828961) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode+nid+uid+gid ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg351-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.432088 s, 38.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0469378 s, 22.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0548476 s, 19.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0533539 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0812979 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132575 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106071 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169099 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15279 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152067 s, 6.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168952 s, 6.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182018 s, 5.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169406 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150841 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137138 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192734 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173027 s, 6.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.042437 s, 24.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.049364 s, 21.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.028838 s, 36.4 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0342165 s, 30.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0325304 s, 32.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0263129 s, 39.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0276666 s, 37.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.041523 s, 25.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0422774 s, 24.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0438206 s, 23.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0451444 s, 23.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0429502 s, 24.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0431215 s, 24.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0486752 s, 21.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.037699 s, 27.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0499667 s, 21.0 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg351-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg351-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10, bs: 1048576 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1048576] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.26745 s, 16.7 MB/s Write runtime is 7 s, speed is 14.285714 vs. 20 Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1048576] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.90371 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 vs. 10 cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.203.51@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.203.51@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.408985 s, 41.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0723623 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.045208 s, 23.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0756105 s, 13.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0718296 s, 14.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144208 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132619 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136783 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15964 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120507 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121368 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17666 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180257 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178126 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127678 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165165 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135452 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0358739 s, 29.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00612931 s, 171 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00803819 s, 130 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0268037 s, 39.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00499145 s, 210 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00881263 s, 119 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00996678 s, 105 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0124407 s, 84.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0051236 s, 205 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00336804 s, 311 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0402671 s, 26.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0624627 s, 16.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0805826 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0032851 s, 319 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0584161 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00629112 s, 167 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg351-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg351-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10, bs: 1048576 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.11324 s, 17.2 MB/s Write runtime is 8 s, speed is 12.500000 vs. 20 Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.83896 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 vs. 10 cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.203.51@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg351-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.453515 s, 37.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0307897 s, 34.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0371379 s, 28.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0548964 s, 19.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0846656 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0654536 s, 16.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152585 s, 6.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115967 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155013 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155825 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127376 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171788 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121209 s, 8.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184081 s, 5.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180996 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150853 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199874 s, 5.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0483506 s, 21.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0309407 s, 33.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0393123 s, 26.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0366311 s, 28.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0543451 s, 19.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0440167 s, 23.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0385811 s, 27.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0576208 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0465381 s, 22.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0466408 s, 22.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0442358 s, 23.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0327283 s, 32.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0541116 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.040553 s, 25.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0433069 s, 24.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0544653 s, 19.3 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg351-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg351-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20, bs: 1048576 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1048576] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.84781 s, 17.9 MB/s Write runtime is 7 s, speed is 14.285714 vs. 20 Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1048576] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.01839 s, 20.9 MB/s Read runtime is 6 s, speed is 16.666666 vs. 20 cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg351-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.523838 s, 32.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0440836 s, 23.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0530253 s, 19.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154006 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160306 s, 6.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192756 s, 5.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146553 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171829 s, 6.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18631 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163023 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166804 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194557 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150917 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196056 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160825 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213928 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197498 s, 5.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0393868 s, 26.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00708454 s, 148 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0405153 s, 25.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0292982 s, 35.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0709332 s, 14.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0368277 s, 28.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00687826 s, 152 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.028075 s, 37.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0602706 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0453481 s, 23.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0409587 s, 25.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0424563 s, 24.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00717174 s, 146 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00439978 s, 238 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00601842 s, 174 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00570188 s, 184 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg351-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg351-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10, bs: 1048576 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1048576] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.95653 s, 10.5 MB/s Write runtime is 11 s, speed is 9.090909 vs. 10 Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1048576] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.83108 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 vs. 10 cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10, bs: 1048576 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.62945 s, 18.6 MB/s Write runtime is 7 s, speed is 14.285714 vs. 20 Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.84114 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 vs. 10 cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg351-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.893215 s, 18.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0347965 s, 30.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112357 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0845005 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0956943 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106564 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108183 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105343 s, 10.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154741 s, 6.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116249 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172355 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17813 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164502 s, 6.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.217085 s, 4.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211834 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22554 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207167 s, 5.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00861137 s, 122 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0412945 s, 25.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137038 s, 76.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158406 s, 66.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0272705 s, 38.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0257793 s, 40.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0262946 s, 39.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0314394 s, 33.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134958 s, 77.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0336954 s, 31.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0369894 s, 28.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0408047 s, 25.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0411431 s, 25.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0439941 s, 23.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0498053 s, 21.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0517661 s, 20.3 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg351-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg351-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5, bs: 1048576 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1048576] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.7086 s, 5.3 MB/s Write runtime is 20 s, speed is 5.000000 vs. 5 Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1048576] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.5605 s, 5.4 MB/s Read runtime is 21 s, speed is 4.761904 vs. 5 cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_ug ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg351-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.383042 s, 43.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.041488 s, 25.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0530195 s, 19.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0929407 s, 11.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0696817 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.060329 s, 17.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116235 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120784 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0674087 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116799 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111997 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145185 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207333 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.218513 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206896 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185461 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.25512 s, 4.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0435839 s, 24.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.036173 s, 29.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0652023 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0391318 s, 26.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.052639 s, 19.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0529092 s, 19.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0517494 s, 20.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0502206 s, 20.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0524575 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0553555 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0556063 s, 18.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0565643 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0856262 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0820016 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0611145 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0650344 s, 16.1 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg351-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg351-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10, bs: 1048576 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1048576] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.74633 s, 18.2 MB/s Write runtime is 7 s, speed is 14.285714 vs. 20 Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1048576] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.82381 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 vs. 10 cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg351-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.497516 s, 33.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0515457 s, 20.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.059778 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137616 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14864 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167369 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158024 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174682 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206827 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182343 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182224 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163804 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208961 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216742 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175102 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216304 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187861 s, 5.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0452834 s, 23.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0384951 s, 27.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0335501 s, 31.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0389915 s, 26.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0431738 s, 24.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0443051 s, 23.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0284321 s, 36.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0355926 s, 29.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0463976 s, 22.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0467204 s, 22.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0510261 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0514313 s, 20.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.050677 s, 20.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0480016 s, 21.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0503335 s, 20.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.060485 s, 17.3 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg351-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg351-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10, bs: 1048576 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1048576] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.9554 s, 10.5 MB/s Write runtime is 11 s, speed is 9.090909 vs. 10 Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1048576] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.84389 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 vs. 10 cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10, bs: 1048576 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1048576] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.8924 s, 10.6 MB/s Write runtime is 11 s, speed is 9.090909 vs. 20 Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1048576] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.80451 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 vs. 10 cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Cleanup TBF ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_policies=fifo PASS 77k (383s) == sanityn test 77kb: Check different granular TBF type combination: jobid+opcode ========================================================== 21:02:25 (1788829345) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/nrs_r_oleg351-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.42577 s, 39.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0499526 s, 21.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0516278 s, 20.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0999481 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170158 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168862 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161853 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182237 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154823 s, 6.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160699 s, 6.5 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185582 s, 5.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213472 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188615 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211736 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187028 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211704 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189983 s, 5.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0456702 s, 23.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0468039 s, 22.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.034463 s, 30.4 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0492876 s, 21.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0288551 s, 36.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0418591 s, 25.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0558205 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.037792 s, 27.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0390307 s, 26.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0451732 s, 23.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0441037 s, 23.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0456822 s, 23.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0469535 s, 22.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.047903 s, 21.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0434057 s, 24.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.056748 s, 18.5 MB/s removed '/mnt/lustre/d77kb.sanityn/nrs_r_oleg351-client.virtnet' removed '/mnt/lustre/d77kb.sanityn/nrs_w_oleg351-client.virtnet' removed directory '/mnt/lustre/d77kb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10, bs: 1048576 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/tbf] [bs=1048576] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.3482 s, 19.6 MB/s Write runtime is 6 s, speed is 16.666666 vs. 20 Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kb.sanityn/tbf] [of=/dev/null] [bs=1048576] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.81294 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 vs. 10 cleanup_tbf /mnt/lustre/d77kb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Cleanup TBF ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77kb (46s) == sanityn test 77kc: Check different granular TBF type combination: nid+opcode ========================================================== 21:03:11 (1788829391) ost.OSS.ost_io.nrs_policies=tbf nid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.203.51@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.203.51@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.351259 s, 47.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0509312 s, 20.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0523557 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0584942 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176652 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187019 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175668 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172481 s, 6.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197268 s, 5.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203281 s, 5.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176184 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183499 s, 5.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213861 s, 4.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186056 s, 5.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209053 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176205 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187422 s, 5.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0279556 s, 37.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0351086 s, 29.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0321497 s, 32.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0259847 s, 40.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0360534 s, 29.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0248643 s, 42.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.028887 s, 36.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0335073 s, 31.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0311306 s, 33.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0278952 s, 37.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0464091 s, 22.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0343495 s, 30.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0371086 s, 28.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0365841 s, 28.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0510579 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0632995 s, 16.6 MB/s removed '/mnt/lustre/d77kc.sanityn/nrs_r_oleg351-client.virtnet' removed '/mnt/lustre/d77kc.sanityn/nrs_w_oleg351-client.virtnet' removed directory '/mnt/lustre/d77kc.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10, bs: 1048576 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.2644 s, 19.9 MB/s Write runtime is 7 s, speed is 14.285714 vs. 20 Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.8634 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 vs. 10 cleanup_tbf /mnt/lustre/d77kc.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Cleanup TBF ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77kc (47s) == sanityn test 77kd: Check different granular TBF type combination: nid+jobid ========================================================== 21:03:58 (1788829438) ost.OSS.ost_io.nrs_policies=tbf nid+jobid ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.203.51@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/nrs_r_oleg351-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.366175 s, 45.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0397033 s, 26.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0801448 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0897061 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111184 s, 9.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102636 s, 10.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.085741 s, 12.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12017 s, 8.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0728597 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0990749 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0868973 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107115 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0943647 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116166 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114126 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160948 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158483 s, 6.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0296999 s, 35.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0329001 s, 31.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0215722 s, 48.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0355952 s, 29.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0416755 s, 25.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0292539 s, 35.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0242021 s, 43.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0226029 s, 46.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0339376 s, 30.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0312064 s, 33.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00394808 s, 266 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0295323 s, 35.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0255776 s, 41.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0277871 s, 37.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0398744 s, 26.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.037212 s, 28.2 MB/s removed '/mnt/lustre/d77kd.sanityn/nrs_r_oleg351-client.virtnet' removed '/mnt/lustre/d77kd.sanityn/nrs_w_oleg351-client.virtnet' removed directory '/mnt/lustre/d77kd.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20, bs: 1048576 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/tbf] [bs=1048576] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.13303 s, 20.4 MB/s Write runtime is 6 s, speed is 16.666666 vs. 20 Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kd.sanityn/tbf] [of=/dev/null] [bs=1048576] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.00399 s, 21.0 MB/s Read runtime is 6 s, speed is 16.666666 vs. 20 cleanup_tbf /mnt/lustre/d77kd.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Cleanup TBF ost.OSS.ost_io.nrs_tbf_rule=stop ext ost.OSS.ost_io.nrs_policies=fifo PASS 77kd (41s) == sanityn test 77ke: Check different granular TBF type combination: jobid+opcode ========================================================== 21:04:39 (1788829479) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/nrs_r_oleg351-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.541798 s, 31.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0390363 s, 26.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0481047 s, 21.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0753677 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0900238 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0866712 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0879754 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14357 s, 7.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100459 s, 10.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149451 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111379 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165608 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151284 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16538 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146917 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.479397 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.444075 s, 2.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00755466 s, 139 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0124052 s, 84.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0101151 s, 104 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0467438 s, 22.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0272017 s, 38.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0381329 s, 27.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00253899 s, 413 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00440795 s, 238 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0399442 s, 26.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00649304 s, 161 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00451379 s, 232 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0357397 s, 29.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00382179 s, 274 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00411933 s, 255 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012636 s, 83.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00717965 s, 146 MB/s removed '/mnt/lustre/d77ke.sanityn/nrs_r_oleg351-client.virtnet' removed '/mnt/lustre/d77ke.sanityn/nrs_w_oleg351-client.virtnet' removed directory '/mnt/lustre/d77ke.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10, bs: 1048576 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/tbf] [bs=1048576] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0058 s, 10.5 MB/s Write runtime is 11 s, speed is 9.090909 vs. 10 Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ke.sanityn/tbf] [of=/dev/null] [bs=1048576] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.8218 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 vs. 10 cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10, bs: 1048576 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.13388 s, 17.1 MB/s Write runtime is 8 s, speed is 12.500000 vs. 20 Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.8724 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 vs. 10 cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Cleanup TBF ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_policies=fifo PASS 77ke (89s) == sanityn test 77kf: Check different granular TBF type combination: uid+gid ========================================================== 21:06:08 (1788829568) ost.OSS.ost_io.nrs_policies=tbf uid+gid ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/nrs_r_oleg351-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.923366 s, 18.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0458304 s, 22.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0539726 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0933347 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112311 s, 9.3 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0952917 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103313 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125609 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126715 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134555 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179547 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19491 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182984 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.218587 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200478 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.224121 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23147 s, 4.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0250392 s, 41.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0494493 s, 21.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0436013 s, 24.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0451371 s, 23.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0312697 s, 33.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0326878 s, 32.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0375257 s, 27.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0411898 s, 25.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.043999 s, 23.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0451705 s, 23.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0481414 s, 21.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0506703 s, 20.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.080451 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0515217 s, 20.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.054017 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0629437 s, 16.7 MB/s removed '/mnt/lustre/d77kf.sanityn/nrs_w_oleg351-client.virtnet' removed '/mnt/lustre/d77kf.sanityn/nrs_r_oleg351-client.virtnet' removed directory '/mnt/lustre/d77kf.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5, bs: 1048576 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/tbf] [bs=1048576] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.6891 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 vs. 5 Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kf.sanityn/tbf] [of=/dev/null] [bs=1048576] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9091 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 vs. 5 cleanup_tbf /mnt/lustre/d77kf.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Cleanup TBF ost.OSS.ost_io.nrs_tbf_rule=stop ext_ug ost.OSS.ost_io.nrs_policies=fifo PASS 77kf (73s) == sanityn test 77kg: check different granular TBF type combination: uid+opcode ========================================================== 21:07:21 (1788829641) ost.OSS.ost_io.nrs_policies=tbf uid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg351-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.335349 s, 50.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0717764 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0806998 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0760397 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0842042 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0728134 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0936999 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0860581 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102828 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101325 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11734 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0827219 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185507 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165872 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188331 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183308 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206518 s, 5.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0229009 s, 45.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.024933 s, 42.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0314435 s, 33.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0231288 s, 45.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0331148 s, 31.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0265578 s, 39.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.024665 s, 42.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0267983 s, 39.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0315598 s, 33.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0248349 s, 42.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0292091 s, 35.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0283672 s, 37.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0306318 s, 34.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0304013 s, 34.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0338164 s, 31.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0410507 s, 25.5 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg351-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg351-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10, bs: 1048576 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1048576] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.32113 s, 19.7 MB/s Write runtime is 7 s, speed is 14.285714 vs. 20 Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1048576] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.82392 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 vs. 10 cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg351-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.493812 s, 34.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0286831 s, 36.6 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0390697 s, 26.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127794 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144933 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139686 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163911 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127878 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167644 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15911 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155469 s, 6.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175497 s, 6.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181056 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22416 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.25773 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.459775 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.444579 s, 2.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0232481 s, 45.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0281753 s, 37.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0391012 s, 26.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0300251 s, 34.9 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.034755 s, 30.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0344972 s, 30.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0344551 s, 30.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0346864 s, 30.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0362519 s, 28.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0409069 s, 25.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0381669 s, 27.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0382423 s, 27.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0415934 s, 25.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.037781 s, 27.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0409694 s, 25.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.049746 s, 21.1 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg351-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg351-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10, bs: 1048576 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1048576] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.8987 s, 10.6 MB/s Write runtime is 11 s, speed is 9.090909 vs. 10 Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1048576] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.79283 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 vs. 10 cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10, bs: 1048576 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1048576] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.93457 s, 10.6 MB/s Write runtime is 11 s, speed is 9.090909 vs. 20 Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1048576] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.83717 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 vs. 10 cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Cleanup TBF ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_policies=fifo PASS 77kg (131s) == sanityn test 77kh: Verify that Project ID support for NRS TBF rule ========================================================== 21:09:32 (1788829772) using SAVE_PROJECT_SUPPORTED=0 Stopping clients: oleg351-client.virtnet /mnt/lustre (opts:) Stopping client oleg351-client.virtnet /mnt/lustre opts: Stopping clients: oleg351-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg351-client.virtnet /mnt/lustre2 opts: Stopping /mnt/lustre-mds1 (opts:-f) on oleg351-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg351-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg351-server sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime) proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) devtmpfs on /dev type devtmpfs (rw,nosuid,mode=755) securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime) tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev) devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000) tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755) tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755) cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd) pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime) bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700) cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer) cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma) cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory) cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset) cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio) cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct) cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio) cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices) cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids) cgroup on /sys/fs/cgroup/misc type cgroup (rw,nosuid,nodev,noexec,relatime,misc) cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event) cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb) none on /sys/kernel/tracing type tracefs (rw,relatime) configfs on /sys/kernel/config type configfs (rw,relatime) /dev/nbd0 on / type ext4 (ro,relatime,stripe=32) rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M) mqueue on /dev/mqueue type mqueue (rw,relatime) debugfs on /sys/kernel/debug type debugfs (rw,relatime) none on /mnt type ramfs (rw,relatime) /dev/vda on /home/green/git/lustre-release type squashfs (ro,relatime) none on /var/lib/stateless/writable type tmpfs (rw,relatime) none on /var/cache/man type tmpfs (rw,relatime) none on /var/log type tmpfs (rw,relatime) none on /var/lib/dbus type tmpfs (rw,relatime) none on /tmp type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) none on /var/tmp type tmpfs (rw,relatime) none on /var/lib/NetworkManager type tmpfs (rw,relatime) none on /var/lib/systemd/random-seed type tmpfs (rw,relatime) none on /var/spool type tmpfs (rw,relatime) none on /var/lib/nfs type tmpfs (rw,relatime) none on /var/lib/gssproxy type tmpfs (rw,relatime) none on /var/lib/logrotate type tmpfs (rw,relatime) none on /etc type tmpfs (rw,relatime) none on /var/lib/rsyslog type tmpfs (rw,relatime) none on /root/.ssh type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) 192.168.200.253:/exports/state/oleg351-client.virtnet on /var/lib/stateless/state type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.203.51,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg351-client.virtnet/boot on /boot type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.203.51,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg351-client.virtnet/etc/kdump.conf on /etc/kdump.conf type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.203.51,local_lock=none,addr=192.168.200.253) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=378180k,mode=700) 192.168.200.253:/exports/testreports/69784/testresults/sanityn-zfs-rocky8.10_x86_64-rocky8.10_x86_64 on /tmp/testlogs type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.203.51,local_lock=none,addr=192.168.200.253) /dev/vda on /usr/sbin/mount.lustre type squashfs (ro,relatime) Checking servers environments Checking clients oleg351-client.virtnet environments Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg351-server' oleg351-server: oleg351-server.virtnet: executing load_modules_local oleg351-server: kmemleak disabled oleg351-server: Loading modules from /home/green/git/lustre-release/lustre oleg351-server: detected 4 online CPUs by sysfs oleg351-server: MODOPTS_LIBCFS= oleg351-server: Force libcfs to create 2 CPU partitions oleg351-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Setup mgs, mdt, osts Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg351-server: oleg351-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg351-client: oleg351-server: ssh exited with exit code 1 Started lustre-MDT0000 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg351-server: oleg351-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg351-client: oleg351-server: ssh exited with exit code 1 Started lustre-OST0000 Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg351-server: oleg351-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg351-client: oleg351-server: ssh exited with exit code 1 Started lustre-OST0001 Starting client: oleg351-client.virtnet: -o user_xattr,flock 192.168.203.151@tcp:/lustre /mnt/lustre Starting client oleg351-client.virtnet: -o user_xattr,flock 192.168.203.151@tcp:/lustre /mnt/lustre Started clients oleg351-client.virtnet: 192.168.203.151@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Starting client: oleg351-client.virtnet: -o user_xattr,flock 192.168.203.151@tcp:/lustre /mnt/lustre2 Starting client oleg351-client.virtnet: -o user_xattr,flock 192.168.203.151@tcp:/lustre /mnt/lustre2 Started clients oleg351-client.virtnet: 192.168.203.151@tcp:/lustre on /mnt/lustre2 type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9f8d85c37800.idle_timeout=debug osc.lustre-OST0000-osc-ffff9f8d90919800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9f8d85c37800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9f8d90919800.idle_timeout=debug disable quota as required ost.OSS.ost_io.nrs_policies=tbf projid ost.OSS.ost_io.nrs_tbf_rule=start ext_proj_rw projid={100} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.33318 s, 50.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0522326 s, 20.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0526478 s, 19.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0763164 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13756 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11772 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154499 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137618 s, 7.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16537 s, 6.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14549 s, 7.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151424 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160141 s, 6.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142713 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.2062 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211699 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.24193 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.228087 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0284259 s, 36.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0520036 s, 20.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0233758 s, 44.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0508514 s, 20.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0329728 s, 31.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0345066 s, 30.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0358709 s, 29.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0294182 s, 35.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0577038 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0376085 s, 27.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0431312 s, 24.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0435258 s, 24.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0386372 s, 27.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0443147 s, 23.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0401311 s, 26.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0432181 s, 24.3 MB/s removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg351-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg351-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20, bs: 1048576 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.51433 s, 19.0 MB/s Write runtime is 7 s, speed is 14.285714 vs. 20 Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.9852 s, 21.0 MB/s Read runtime is 6 s, speed is 16.666666 vs. 20 cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Cleanup TBF ost.OSS.ost_io.nrs_tbf_rule=stop ext_proj_rw ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf projid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w projid={100}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r projid={100}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.37725 s, 44.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0567153 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0494798 s, 21.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114564 s, 9.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163762 s, 6.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165427 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175039 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14796 s, 7.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162648 s, 6.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170819 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172584 s, 6.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168935 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150818 s, 7.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175602 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183351 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191233 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180926 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00330677 s, 317 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0026248 s, 399 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0038315 s, 274 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0855244 s, 12.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0745192 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00419512 s, 250 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0685484 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117378 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108219 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0773936 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.048322 s, 21.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0384673 s, 27.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107306 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0451563 s, 23.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0237415 s, 44.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0355562 s, 29.5 MB/s removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg351-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg351-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10, bs: 1048576 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.96144 s, 17.6 MB/s Write runtime is 7 s, speed is 14.285714 vs. 20 Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.83632 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 vs. 10 cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Cleanup TBF ost.OSS.ost_io.nrs_tbf_rule=stop ext_proj_rw ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77kh (151s) == sanityn test 77ki: Add rule with unsupported TBF type should fail for generic TBF ========================================================== 21:12:03 (1788829923) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 oleg351-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_fail projid={100}&opcode={ost_write} rate=20: Invalid argument oleg351-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_fail projid={100}&opcode={ost_write} rate=20': Invalid argument pdsh@oleg351-client: oleg351-server: ssh exited with exit code 22 Cleanup TBF ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77ki (13s) == sanityn test 77kj: Verify nodemap support for NRS TBF rule ========================================================== 21:12:16 (1788829936) On MGS 192.168.203.151, active = nodemap.active=1 On MGS 192.168.203.151, TBF.id = nodemap.TBF.id=1 ost.OSS.ost_io.nrs_policies=tbf nodemap ost.OSS.ost_io.nrs_tbf_rule=start ext_nm_rw nodemap={TBF} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.437789 s, 38.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0414151 s, 25.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0537318 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0404217 s, 25.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0609441 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117364 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146306 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121226 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144442 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108435 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165218 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127752 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158844 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155474 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122291 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131322 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154534 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0348851 s, 30.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154888 s, 67.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162213 s, 64.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0239438 s, 43.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.025044 s, 41.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0386847 s, 27.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0481196 s, 21.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0676076 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0426125 s, 24.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0590231 s, 17.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0417713 s, 25.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0438443 s, 23.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0530699 s, 19.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0485794 s, 21.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.051822 s, 20.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0517861 s, 20.2 MB/s removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg351-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg351-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20, bs: 1048576 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.60186 s, 18.7 MB/s Write runtime is 7 s, speed is 14.285714 vs. 20 Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.98837 s, 21.0 MB/s Read runtime is 6 s, speed is 16.666666 vs. 20 cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete On MGS 192.168.203.151, TBF.id = On MGS 192.168.203.151, active = nodemap.active=0 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.256213 s, 65.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.036384 s, 28.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0394402 s, 26.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0458183 s, 22.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0926564 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0968972 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150074 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14296 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164237 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145065 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142605 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13466 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139976 s, 7.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9f8d85c37800.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffff9f8d85c37800.lock_unused_count=3 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161261 s, 6.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146547 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156567 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189312 s, 5.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0393737 s, 26.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0426771 s, 24.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0397427 s, 26.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0408469 s, 25.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0386834 s, 27.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0477601 s, 22.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0298556 s, 35.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0373066 s, 28.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0354283 s, 29.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0386939 s, 27.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0389935 s, 26.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0400355 s, 26.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0487479 s, 21.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0364538 s, 28.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0427869 s, 24.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0475466 s, 22.1 MB/s removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg351-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg351-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 1 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20, bs: 1048576 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.2097 s, 24.9 MB/s Write runtime is 5 s, speed is 20.000000 vs. 20 Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 1.32011 s, 79.4 MB/s Read runtime is 3 s, speed is 33.333333 vs. 20 read rate (33.333333) should exceed limit (20 * 1) cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Cleanup TBF ost.OSS.ost_io.nrs_tbf_rule=stop ext_nm_rw ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf nodemap+opcode oleg351-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_w nodemap={TBF}&opcode={ost_write} rate=20: Invalid argument oleg351-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_w nodemap={TBF}&opcode={ost_write} rate=20': Invalid argument oleg351-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_r nodemap={TBF}&opcode={ost_read} rate=10: Invalid argument oleg351-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_r nodemap={TBF}&opcode={ost_read} rate=10': Invalid argument pdsh@oleg351-client: oleg351-server: ssh exited with exit code 22 On MGS 192.168.203.151, active = nodemap.active=1 On MGS 192.168.203.151, TBF.id = nodemap.TBF.id=2 ost.OSS.ost_io.nrs_tbf_rule=start ext_w nodemap={TBF}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nodemap={TBF}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.400814 s, 41.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0565155 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0401359 s, 26.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0479748 s, 21.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0634747 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0889692 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141104 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121134 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176457 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164929 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172818 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155759 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198038 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159097 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158016 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195757 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167144 s, 6.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0329023 s, 31.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0420641 s, 24.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0406914 s, 25.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0333841 s, 31.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0338615 s, 31.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0277463 s, 37.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0336823 s, 31.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0320663 s, 32.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0395072 s, 26.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0396109 s, 26.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.043177 s, 24.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0470993 s, 22.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0508569 s, 20.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0430654 s, 24.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0455754 s, 23.0 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0461972 s, 22.7 MB/s removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg351-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg351-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10, bs: 1048576 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.43328 s, 19.3 MB/s Write runtime is 7 s, speed is 14.285714 vs. 20 Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.84346 s, 10.7 MB/s Read runtime is 10 s, speed is 10.000000 vs. 10 cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Cleanup TBF ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf nid classify=dynamic ost.OSS.ost_io.nrs_tbf_rule=start nmdyn nodemap={TBF} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.359338 s, 46.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0299506 s, 35.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0387316 s, 27.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111789 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108823 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107227 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.096342 s, 10.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126429 s, 8.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121366 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117046 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117272 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128911 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138166 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19738 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.204976 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.236158 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.2302 s, 4.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01215 s, 86.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0357909 s, 29.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0214145 s, 49.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0301019 s, 34.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.031282 s, 33.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0268792 s, 39.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0208735 s, 50.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0345422 s, 30.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0359927 s, 29.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0370771 s, 28.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0376846 s, 27.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0639096 s, 16.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0317286 s, 33.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0446246 s, 23.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.046063 s, 22.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0464276 s, 22.6 MB/s removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg351-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg351-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20, bs: 1048576 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.28894 s, 19.8 MB/s Write runtime is 6 s, speed is 16.666666 vs. 20 Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.97877 s, 21.1 MB/s Read runtime is 6 s, speed is 16.666666 vs. 20 cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Cleanup TBF ost.OSS.ost_io.nrs_tbf_rule=stop nmdyn ost.OSS.ost_io.nrs_policies=fifo On MGS 192.168.203.151, TBF.id = On MGS 192.168.203.151, active = nodemap.active=0 Cleanup TBF ost.OSS.ost_io.nrs_policies=fifo Cleanup TBF ost.OSS.ost_io.nrs_policies=fifo Cleanup TBF ost.OSS.ost_io.nrs_policies=fifo On MGS 192.168.203.151, active = nodemap.active=0 PASS 77kj (196s) == sanityn test 77l: check the output of NRS policies for generic TBF ========================================================== 21:15:32 (1788830132) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_policies=fifo PASS 77l (4s) == sanityn test 77m: check NRS Delay slows write RPC processing ========================================================== 21:15:37 (1788830137) ost.OSS.ost_io.nrs_policies=delay ost.OSS.ost_io.nrs_delay_min=4 ost.OSS.ost_io.nrs_delay_max=4 ost.OSS.ost_io.nrs_delay_pct=100 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg351-client.virtnet] [bs=1M] [count=1] [oflag=direct] [conv=fdatasync] 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 3.47491 s, 302 kB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg351-client.virtnet] [bs=1M] [count=10] [oflag=direct] [conv=fdatasync] 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 40.0306 s, 262 kB/s ost.OSS.ost_io.nrs_policies=fifo PASS 77m (52s) == sanityn test 77n: check wildcard support for TBF JobID NRS policy ========================================================== 21:16:29 (1788830189) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={*.500} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.241316 s, 69.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0498075 s, 21.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0425728 s, 24.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111865 s, 9.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0949486 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137801 s, 7.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136109 s, 7.7 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120938 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141443 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126252 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124699 s, 8.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167453 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12802 s, 8.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154108 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135024 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16567 s, 6.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9f8d85c37800.lock_unused_count=3 ldlm.namespaces.lustre-OST0001-osc-ffff9f8d85c37800.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196198 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0451671 s, 23.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0270994 s, 38.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0477289 s, 22.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0282033 s, 37.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0574745 s, 18.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0684099 s, 15.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.064512 s, 16.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0318613 s, 32.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0689785 s, 15.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0332132 s, 31.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0413789 s, 25.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0367183 s, 28.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0446612 s, 23.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0469507 s, 22.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0391252 s, 26.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0765652 s, 13.7 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg351-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg351-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20, bs: 1048576 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77n.sanityn/tbf] [bs=1048576] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.01604 s, 20.9 MB/s Write runtime is 6 s, speed is 16.666666 vs. 20 Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77n.sanityn/tbf] [of=/dev/null] [bs=1048576] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.98853 s, 21.0 MB/s Read runtime is 6 s, speed is 16.666666 vs. 20 cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.*} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.255169 s, 65.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0661902 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0705942 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0767609 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0846622 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0750408 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0818755 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0870843 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0832535 s, 12.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0798551 s, 13.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0854039 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0915109 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102993 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1428 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122279 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18046 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160149 s, 6.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0291174 s, 36.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0198956 s, 52.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0206793 s, 50.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0303285 s, 34.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0206831 s, 50.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0262132 s, 40.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0401373 s, 26.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0256258 s, 40.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0308503 s, 34.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.027486 s, 38.1 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0359389 s, 29.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0388566 s, 27.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0401002 s, 26.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.032434 s, 32.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0462125 s, 22.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0336575 s, 31.2 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg351-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg351-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20, bs: 1048576 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.2519 s, 20.0 MB/s Write runtime is 6 s, speed is 16.666666 vs. 20 Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.00286 s, 21.0 MB/s Read runtime is 6 s, speed is 16.666666 vs. 20 cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_policies=fifo PASS 77n (74s) == sanityn test 77o: Changing rank should not panic ====== 21:17:43 (1788830263) mds.MDS.mdt.nrs_policies=tbf nid mds.MDS.mdt.nrs_tbf_rule=start name nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=start name1 nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=change name1 rank=name mds.MDS.mdt.nrs_tbf_rule=stop name mds.MDS.mdt.nrs_policies=fifo PASS 77o (15s) == sanityn test 77q: Parallel TBF rule definitions should not panic ========================================================== 21:17:59 (1788830279) mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_policies=fifo PASS 77q (113s) == sanityn test 77p: Check validity of rule names for TBF policies ========================================================== 21:19:51 (1788830391) mds.MDS.mdt.nrs_policies=tbf oleg351-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_overflo uid={500} rate=500: File name too long oleg351-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_overflo uid={500} rate=500': File name too long pdsh@oleg351-client: oleg351-server: ssh exited with exit code 36 oleg351-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start uid={500} rate=500: Invalid argument oleg351-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start uid={500} rate=500': Invalid argument pdsh@oleg351-client: oleg351-server: ssh exited with exit code 22 oleg351-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_empty: Invalid argument oleg351-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_empty': Invalid argument pdsh@oleg351-client: oleg351-server: ssh exited with exit code 22 oleg351-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p@spec uid={500} rate=500: Invalid argument oleg351-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p@spec uid={500} rate=500': Invalid argument pdsh@oleg351-client: oleg351-server: ssh exited with exit code 22 oleg351-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p.spec uid={500} rate=500: Invalid argument oleg351-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p.spec uid={500} rate=500': Invalid argument pdsh@oleg351-client: oleg351-server: ssh exited with exit code 22 oleg351-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p~spec uid={500} rate=500: Invalid argument oleg351-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p~spec uid={500} rate=500': Invalid argument pdsh@oleg351-client: oleg351-server: ssh exited with exit code 22 oleg351-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p#spec uid={500} rate=500: Invalid argument oleg351-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p#spec uid={500} rate=500': Invalid argument pdsh@oleg351-client: oleg351-server: ssh exited with exit code 22 oleg351-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p/spec uid={500} rate=500: Invalid argument oleg351-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p/spec uid={500} rate=500': Invalid argument pdsh@oleg351-client: oleg351-server: ssh exited with exit code 22 oleg351-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p^spec uid={500} rate=500: Invalid argument oleg351-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p^spec uid={500} rate=500': Invalid argument pdsh@oleg351-client: oleg351-server: ssh exited with exit code 22 oleg351-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p%spec uid={500} rate=500: Invalid argument oleg351-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p%spec uid={500} rate=500': Invalid argument pdsh@oleg351-client: oleg351-server: ssh exited with exit code 22 oleg351-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p*spec uid={500} rate=500: Invalid argument oleg351-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p*spec uid={500} rate=500': Invalid argument pdsh@oleg351-client: oleg351-server: ssh exited with exit code 22 oleg351-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p;spec uid={500} rate=500: Invalid argument oleg351-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p;spec uid={500} rate=500': Invalid argument pdsh@oleg351-client: oleg351-server: ssh exited with exit code 22 oleg351-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p,spec uid={500} rate=500: Invalid argument oleg351-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p,spec uid={500} rate=500': Invalid argument pdsh@oleg351-client: oleg351-server: ssh exited with exit code 22 oleg351-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p?spec uid={500} rate=500: Invalid argument oleg351-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p?spec uid={500} rate=500': Invalid argument pdsh@oleg351-client: oleg351-server: ssh exited with exit code 22 oleg351-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77pspec uid={500} rate=500: Invalid argument oleg351-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p>spec uid={500} rate=500': Invalid argument pdsh@oleg351-client: oleg351-server: ssh exited with exit code 22 oleg351-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p:spec uid={500} rate=500: Invalid argument oleg351-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p:spec uid={500} rate=500': Invalid argument pdsh@oleg351-client: oleg351-server: ssh exited with exit code 22 oleg351-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p+spec uid={500} rate=500: Invalid argument oleg351-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p+spec uid={500} rate=500': Invalid argument pdsh@oleg351-client: oleg351-server: ssh exited with exit code 22 oleg351-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p=spec uid={500} rate=500: Invalid argument oleg351-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p=spec uid={500} rate=500': Invalid argument pdsh@oleg351-client: oleg351-server: ssh exited with exit code 22 oleg351-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p)spec uid={500} rate=500: Invalid argument oleg351-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p)spec uid={500} rate=500': Invalid argument pdsh@oleg351-client: oleg351-server: ssh exited with exit code 22 oleg351-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p(spec uid={500} rate=500: Invalid argument oleg351-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p(spec uid={500} rate=500': Invalid argument pdsh@oleg351-client: oleg351-server: ssh exited with exit code 22 oleg351-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p{spec uid={500} rate=500: Invalid argument oleg351-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p{spec uid={500} rate=500': Invalid argument pdsh@oleg351-client: oleg351-server: ssh exited with exit code 22 oleg351-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p}spec uid={500} rate=500: Invalid argument oleg351-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p}spec uid={500} rate=500': Invalid argument pdsh@oleg351-client: oleg351-server: ssh exited with exit code 22 oleg351-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p|spec uid={500} rate=500: Invalid argument oleg351-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p|spec uid={500} rate=500': Invalid argument pdsh@oleg351-client: oleg351-server: ssh exited with exit code 22 oleg351-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p[spec uid={500} rate=500: Invalid argument oleg351-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p[spec uid={500} rate=500': Invalid argument pdsh@oleg351-client: oleg351-server: ssh exited with exit code 22 oleg351-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p]spec uid={500} rate=500: Invalid argument oleg351-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p]spec uid={500} rate=500': Invalid argument pdsh@oleg351-client: oleg351-server: ssh exited with exit code 22 oleg351-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p!spec uid={500} rate=500: Invalid argument oleg351-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p!spec uid={500} rate=500': Invalid argument pdsh@oleg351-client: oleg351-server: ssh exited with exit code 22 oleg351-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p&spec uid={500} rate=500: Invalid argument oleg351-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p&spec uid={500} rate=500': Invalid argument pdsh@oleg351-client: oleg351-server: ssh exited with exit code 22 oleg351-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p$spec uid={500} rate=500: Invalid argument oleg351-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p$spec uid={500} rate=500': Invalid argument pdsh@oleg351-client: oleg351-server: ssh exited with exit code 22 oleg351-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p`spec uid={500} rate=500: Invalid argument oleg351-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p`spec uid={500} rate=500': Invalid argument pdsh@oleg351-client: oleg351-server: ssh exited with exit code 22 oleg351-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p\spec uid={500} rate=500: Invalid argument oleg351-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p\spec uid={500} rate=500': Invalid argument pdsh@oleg351-client: oleg351-server: ssh exited with exit code 22 mds.MDS.mdt.nrs_policies=fifo PASS 77p (29s) == sanityn test 77r: Change type of tbf policy at run time ========================================================== 21:20:20 (1788830420) mds.MDS.mdt.nrs_policies=tbf uid mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf gid mds.MDS.mdt.nrs_tbf_rule=start md_rule gid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start md_rule jobid={*.500} rate=1 mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=fifo check the number of requests in queue: fifo: 0 crrn: 0 tbf: 0 mclock: 0 delay: 0 fifo: 0 crrn: 0 tbf: 0 mclock: 0 delay: 0 cleanup 77r 277590 kill md_thread xargs PASS 77r (52s) == sanityn test 77s: Check TBF LRU shrinker ============== 21:21:12 (1788830472) jobid_name=%e.%p jobid_var=TEST_77s_JOBVAR mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start touch jobid={touch.*} rate=1000 mds.MDS.mdt.nrs_tbf_rule=start unlink jobid={unlink.*} rate=1000 touch jobid={touch.*} 1000, seq 2, type=jobid, ref 10 touch jobid={touch.*} 1000, seq 2, type=jobid, ref 0 unlink jobid={unlink.*} 1000, seq 3, type=jobid, ref 10 unlink jobid={unlink.*} 1000, seq 3, type=jobid, ref 0 mds.MDS.mdt.nrs_policies=fifo jobid_var=procname_uid jobid_name=%e.%u PASS 77s (57s) == sanityn test 77ta: Testing for dynamic|static classify parameter setting for TBF ========================================================== 21:22:09 (1788830529) ost.OSS.ost_io.nrs_policies=fifo oleg351-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf nid+uid classify=foo: Invalid argument oleg351-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf nid+uid classify=foo': Invalid argument pdsh@oleg351-client: oleg351-server: ssh exited with exit code 22 oleg351-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf nid+uid foo=static: Invalid argument oleg351-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf nid+uid foo=static': Invalid argument pdsh@oleg351-client: oleg351-server: ssh exited with exit code 22 oleg351-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf nid+uid classify=: Invalid argument oleg351-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf nid+uid classify=': Invalid argument pdsh@oleg351-client: oleg351-server: ssh exited with exit code 22 ost.OSS.ost_io.nrs_policies=tbf nid classify=dynamic ost.OSS.ost_io.nrs_policies=tbf nid classify=static ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf reg nid classify=dynamic ost.OSS.ost_io.nrs_policies=tbf hp nid ost.OSS.ost_io.nrs_tbf_rule=start mixed1 nid={0@lo} rate=100 ost.OSS.ost_io.nrs_tbf_rule=stop mixed1 ost.OSS.ost_io.nrs_policies=tbf reg nid classify=static ost.OSS.ost_io.nrs_policies=tbf hp nid classify=dynamic ost.OSS.ost_io.nrs_tbf_rule=start mixed2 nid={0@lo} rate=100 ost.OSS.ost_io.nrs_tbf_rule=stop mixed2 ost.OSS.ost_io.nrs_policies=tbf reg nid+uid ost.OSS.ost_io.nrs_policies=tbf hp nid oleg351-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start mixed3 nid={0@lo}&uid={500} rate=100: Invalid argument oleg351-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start mixed3 nid={0@lo}&uid={500} rate=100': Invalid argument pdsh@oleg351-client: oleg351-server: ssh exited with exit code 22 ost.OSS.ost_io.nrs_policies=tbf nid classify=static classify=dynamic ost.OSS.ost_io.nrs_policies=tbf reg nid classify=dynamic ost.OSS.ost_io.nrs_policies= regular_requests: - name: fifo state: started fallback: yes queued: 0 active: 0 - name: crrn state: stopped fallback: no queued: 0 active: 0 - name: orr state: stopped fallback: no queued: 0 active: 0 - name: trr state: stopped fallback: no queued: 0 active: 0 - name: tbf nid classify=dynamic state: started fallback: no queued: 0 active: 0 - name: mclock state: stopped fallback: no queued: 0 active: 0 - name: delay state: stopped fallback: no queued: 0 active: 0 high_priority_requests: - name: fifo state: started fallback: yes queued: 0 active: 0 - name: crrn state: stopped fallback: no queued: 0 active: 0 - name: orr state: stopped fallback: no queued: 0 active: 0 - name: trr state: stopped fallback: no queued: 0 active: 0 - name: tbf nid classify=static classify=dynamic state: started fallback: no queued: 0 active: 0 - name: mclock state: stopped fallback: no queued: 0 active: 0 - name: delay state: stopped fallback: no queued: 0 active: 0 ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf hp nid classify=static ost.OSS.ost_io.nrs_policies= regular_requests: - name: fifo state: started fallback: yes queued: 0 active: 0 - name: crrn state: stopped fallback: no queued: 0 active: 0 - name: orr state: stopped fallback: no queued: 0 active: 0 - name: trr state: stopped fallback: no queued: 0 active: 0 - name: tbf state: stopped fallback: no queued: 0 active: 0 - name: mclock state: stopped fallback: no queued: 0 active: 0 - name: delay state: stopped fallback: no queued: 0 active: 0 high_priority_requests: - name: fifo state: started fallback: yes queued: 0 active: 0 - name: crrn state: stopped fallback: no queued: 0 active: 0 - name: orr state: stopped fallback: no queued: 0 active: 0 - name: trr state: stopped fallback: no queued: 0 active: 0 - name: tbf nid classify=static state: started fallback: no queued: 0 active: 0 - name: mclock state: stopped fallback: no queued: 0 active: 0 - name: delay state: stopped fallback: no queued: 0 active: 0 ost.OSS.ost_io.nrs_policies=fifo PASS 77ta (35s) == sanityn test 77tB: Request falling to default rule should not panic for dynamic TBF ========================================================== 21:22:44 (1788830564) ost.OSS.ost_io.nrs_policies=tbf nid classify=dynamic not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5, bs: 1048576 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.7918 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 vs. 5 Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9393 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 vs. 5 cleanup_tbf /mnt/lustre/d77tB.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=start unmatchrule nid={192.0.2.1@tcp} rate=100 not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5, bs: 1048576 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.7534 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 vs. 5 Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.933 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 vs. 5 cleanup_tbf /mnt/lustre/d77tB.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop unmatchrule ost.OSS.ost_io.nrs_policies=fifo PASS 77tB (135s) == sanityn test 77th: TBF: a mixed classify head pair keeps each head type ========================================================== 21:24:59 (1788830699) ost.OSS.ost_io.nrs_policies=tbf reg nid+opcode ost.OSS.ost_io.nrs_policies=tbf hp nid classify=dynamic ost.OSS.ost_io.nrs_tbf_rule=start mixedhp nid={0@lo} rate=100 regular_requests: CPT 0: mixedhp nid={0@lo} 100, seq 2, type=nid+opcode, ref 0 default * 10000, seq 1, type=nid+opcode, ref 0 high_priority_requests: CPT 0: mixedhp nid={0@lo} 100, seq 2, type=nid, ref 0 default * 10000, seq 1, type=nid, ref 0 ost.OSS.ost_io.nrs_tbf_rule=stop mixedhp ost.OSS.ost_io.nrs_policies=fifo PASS 77th (10s) == sanityn test 77tb: Testing for dynamic classify TBF Scheduler ========================================================== 21:25:09 (1788830709) ost.OSS.ost_io.nrs_policies=tbf nid classify=dynamic ost.OSS.ost_io.nrs_policies= regular_requests: - name: fifo state: started fallback: yes queued: 0 active: 0 - name: crrn state: stopped fallback: no queued: 0 active: 0 - name: orr state: stopped fallback: no queued: 0 active: 0 - name: trr state: stopped fallback: no queued: 0 active: 0 - name: tbf nid classify=dynamic state: started fallback: no queued: 0 active: 0 - name: mclock state: stopped fallback: no queued: 0 active: 0 - name: delay state: stopped fallback: no queued: 0 active: 0 high_priority_requests: - name: fifo state: started fallback: yes queued: 0 active: 0 - name: crrn state: stopped fallback: no queued: 0 active: 0 - name: orr state: stopped fallback: no queued: 0 active: 0 - name: trr state: stopped fallback: no queued: 0 active: 0 - name: tbf nid classify=dynamic state: started fallback: no queued: 0 active: 0 - name: mclock state: stopped fallback: no queued: 0 active: 0 - name: delay state: stopped fallback: no queued: 0 active: 0 ost.OSS.ost_io.nrs_tbf_classify= reg_classify:dynamic hp_classify:dynamic ost.OSS.ost_io.nrs_tbf_rule=start nidall nid={0@lo 192.168.203.51@tcp} rate=10 ost.OSS.ost_io.nrs_tbf_rule=start nidrd nid={0@lo 192.168.203.51@tcp}&opcode={ost_read} rate=5 ost.OSS.ost_io.nrs_tbf_rule=start nidwr nid={0@lo 192.168.203.51@tcp}&opcode={ost_write} rate=15 ost.OSS.ost_io.nrs_tbf_rule= regular_requests: CPT 0: nidwr nid={0@lo 192.168.203.51@tcp}&opcode={ost_write} 15, seq 4, type=nid+opcode, ref 0 nidrd nid={0@lo 192.168.203.51@tcp}&opcode={ost_read} 5, seq 3, type=nid+opcode, ref 0 nidall nid={0@lo 192.168.203.51@tcp} 10, seq 2, type=nid, ref 0 default * 10000, seq 1, type=nid, ref 0 high_priority_requests: CPT 0: nidwr nid={0@lo 192.168.203.51@tcp}&opcode={ost_write} 15, seq 4, type=nid+opcode, ref 0 nidrd nid={0@lo 192.168.203.51@tcp}&opcode={ost_read} 5, seq 3, type=nid+opcode, ref 0 nidall nid={0@lo 192.168.203.51@tcp} 10, seq 2, type=nid, ref 0 default * 10000, seq 1, type=nid, ref 0 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.49043 s, 34.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0633224 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0470616 s, 22.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157064 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183473 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15811 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149135 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216304 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20416 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187601 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174694 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21543 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198452 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.25258 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.251884 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.305394 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.32124 s, 3.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0402239 s, 26.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.042571 s, 24.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0451279 s, 23.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0457499 s, 22.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0485006 s, 21.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0575267 s, 18.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0761579 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0573767 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0530839 s, 19.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0623998 s, 16.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0580703 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0615442 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0679488 s, 15.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0688332 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0694908 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0835431 s, 12.6 MB/s removed '/mnt/lustre/d77tb.sanityn/nrs_r_oleg351-client.virtnet' removed '/mnt/lustre/d77tb.sanityn/nrs_w_oleg351-client.virtnet' removed directory '/mnt/lustre/d77tb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 15, read rate: 5, bs: 1048576 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.22676 s, 14.5 MB/s Write runtime is 8 s, speed is 12.500000 vs. 15 Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.5877 s, 5.4 MB/s Read runtime is 21 s, speed is 4.761904 vs. 5 cleanup_tbf /mnt/lustre/d77tb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Cleanup TBF ost.OSS.ost_io.nrs_tbf_rule=stop nidall ost.OSS.ost_io.nrs_tbf_rule=stop nidrd ost.OSS.ost_io.nrs_tbf_rule=stop nidwr ost.OSS.ost_io.nrs_policies=fifo PASS 77tb (70s) == sanityn test 77tc: Testing for the setting of allowed TBF classification type ========================================================== 21:26:19 (1788830779) oleg351-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf nid classify=dynamic allow=nid allow=nid: File exists oleg351-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf nid classify=dynamic allow=nid allow=nid': File exists pdsh@oleg351-client: oleg351-server: ssh exited with exit code 17 oleg351-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf nid allow=nid allow=uid: Invalid argument oleg351-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf nid allow=nid allow=uid': Invalid argument pdsh@oleg351-client: oleg351-server: ssh exited with exit code 22 ost.OSS.ost_io.nrs_policies=tbf nid classify=dynamic allow=nid+opcode allow=nid allow=uid+opcode oleg351-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start another uid={500} rate=10: Operation not permitted oleg351-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start another uid={500} rate=10': Operation not permitted pdsh@oleg351-client: oleg351-server: ssh exited with exit code 1 oleg351-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start another uid={500}&opcode={ost_read}&gid={500} rate=10: Operation not permitted oleg351-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start another uid={500}&opcode={ost_read}&gid={500} rate=10': Operation not permitted pdsh@oleg351-client: oleg351-server: ssh exited with exit code 1 ost.OSS.ost_io.nrs_tbf_rule=start nidall nid={0@lo} rate=10 ost.OSS.ost_io.nrs_tbf_rule=start nidrd nid={0@lo}&opcode={ost_read} rate=5 Cleanup TBF ost.OSS.ost_io.nrs_tbf_rule=stop nidall ost.OSS.ost_io.nrs_tbf_rule=stop nidrd ost.OSS.ost_io.nrs_policies=fifo oleg351-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf nodemap classify=dynamic allow=nid allow=uid allow=gid allow=jobid allow=opcode allow=projid allow=nid+uid allow=nid+gid allow=uid+gid allow=nid+jobid allow=uid+jobid allow=gid+jobid allow=nid+opcode allow=uid+opcode allow=gid+opcode allow=nid+projid: Argument list too long oleg351-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf nodemap classify=dynamic allow=nid allow=uid allow=gid allow=jobid allow=opcode allow=projid allow=nid+uid allow=nid+gid allow=uid+gid allow=nid+jobid allow=uid+jobid allow=gid+jobid allow=nid+opcode allow=uid+opcode allow=gid+opcode allow=nid+projid': Argument list too long pdsh@oleg351-client: oleg351-server: ssh exited with exit code 7 Cleanup TBF ost.OSS.ost_io.nrs_policies=fifo PASS 77tc (25s) == sanityn test 77td: Testing Number of allowed rules larger or smaller than threshold ========================================================== 21:26:44 (1788830804) using SAVE_PROJECT_SUPPORTED=0 Stopping clients: oleg351-client.virtnet /mnt/lustre (opts:) Stopping client oleg351-client.virtnet /mnt/lustre opts: Stopping clients: oleg351-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg351-client.virtnet /mnt/lustre2 opts: Stopping /mnt/lustre-mds1 (opts:-f) on oleg351-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg351-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg351-server sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime) proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) devtmpfs on /dev type devtmpfs (rw,nosuid,mode=755) securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime) tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev) devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000) tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755) tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755) cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd) pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime) bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700) cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer) cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma) cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory) cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset) cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio) cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct) cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio) cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices) cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids) cgroup on /sys/fs/cgroup/misc type cgroup (rw,nosuid,nodev,noexec,relatime,misc) cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event) cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb) none on /sys/kernel/tracing type tracefs (rw,relatime) configfs on /sys/kernel/config type configfs (rw,relatime) /dev/nbd0 on / type ext4 (ro,relatime,stripe=32) rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M) mqueue on /dev/mqueue type mqueue (rw,relatime) debugfs on /sys/kernel/debug type debugfs (rw,relatime) none on /mnt type ramfs (rw,relatime) /dev/vda on /home/green/git/lustre-release type squashfs (ro,relatime) none on /var/lib/stateless/writable type tmpfs (rw,relatime) none on /var/cache/man type tmpfs (rw,relatime) none on /var/log type tmpfs (rw,relatime) none on /var/lib/dbus type tmpfs (rw,relatime) none on /tmp type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) none on /var/tmp type tmpfs (rw,relatime) none on /var/lib/NetworkManager type tmpfs (rw,relatime) none on /var/lib/systemd/random-seed type tmpfs (rw,relatime) none on /var/spool type tmpfs (rw,relatime) none on /var/lib/nfs type tmpfs (rw,relatime) none on /var/lib/gssproxy type tmpfs (rw,relatime) none on /var/lib/logrotate type tmpfs (rw,relatime) none on /etc type tmpfs (rw,relatime) none on /var/lib/rsyslog type tmpfs (rw,relatime) none on /root/.ssh type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) 192.168.200.253:/exports/state/oleg351-client.virtnet on /var/lib/stateless/state type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.203.51,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg351-client.virtnet/boot on /boot type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.203.51,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg351-client.virtnet/etc/kdump.conf on /etc/kdump.conf type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.203.51,local_lock=none,addr=192.168.200.253) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=378180k,mode=700) 192.168.200.253:/exports/testreports/69784/testresults/sanityn-zfs-rocky8.10_x86_64-rocky8.10_x86_64 on /tmp/testlogs type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.203.51,local_lock=none,addr=192.168.200.253) /dev/vda on /usr/sbin/mount.lustre type squashfs (ro,relatime) Checking servers environments Checking clients oleg351-client.virtnet environments Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg351-server' oleg351-server: oleg351-server.virtnet: executing load_modules_local oleg351-server: kmemleak disabled oleg351-server: Loading modules from /home/green/git/lustre-release/lustre oleg351-server: detected 4 online CPUs by sysfs oleg351-server: MODOPTS_LIBCFS= oleg351-server: Force libcfs to create 2 CPU partitions oleg351-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Setup mgs, mdt, osts Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg351-server: oleg351-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg351-client: oleg351-server: ssh exited with exit code 1 Started lustre-MDT0000 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg351-server: oleg351-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg351-client: oleg351-server: ssh exited with exit code 1 Started lustre-OST0000 Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg351-server: oleg351-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg351-client: oleg351-server: ssh exited with exit code 1 Started lustre-OST0001 Starting client: oleg351-client.virtnet: -o user_xattr,flock 192.168.203.151@tcp:/lustre /mnt/lustre Starting client oleg351-client.virtnet: -o user_xattr,flock 192.168.203.151@tcp:/lustre /mnt/lustre Started clients oleg351-client.virtnet: 192.168.203.151@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Starting client: oleg351-client.virtnet: -o user_xattr,flock 192.168.203.151@tcp:/lustre /mnt/lustre2 Starting client oleg351-client.virtnet: -o user_xattr,flock 192.168.203.151@tcp:/lustre /mnt/lustre2 Started clients oleg351-client.virtnet: 192.168.203.151@tcp:/lustre on /mnt/lustre2 type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9f8d82fb1800.idle_timeout=debug osc.lustre-OST0000-osc-ffff9f8d866d9800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9f8d82fb1800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9f8d866d9800.idle_timeout=debug disable quota as required ost.OSS.ost_io.nrs_policies=tbf nid classify=dynamic allow=projid+opcode allow=projid ost.OSS.ost_io.nrs_tbf_allowed_types= regular_allowed_types: CPT 0: 0: 0x00000024 opcode+projid 1: 0x00000020 projid 2: 0x00000002 nid high_priority_allowed_types: CPT 0: 0: 0x00000024 opcode+projid 1: 0x00000020 projid 2: 0x00000002 nid 8 ost.OSS.ost_io.nrs_tbf_rule=start rulerd0 projid={100}&opcode={ost_read} rate=5 ost.OSS.ost_io.nrs_tbf_rule=start rulewr0 projid={100}&opcode={ost_write} rate=10 ost.OSS.ost_io.nrs_tbf_rule=start ruleall0 projid={200} rate=15 ost.OSS.ost_io.nrs_tbf_rule= regular_requests: CPT 0: ruleall0 projid={200} 15, seq 4, type=projid, ref 0 rulewr0 projid={100}&opcode={ost_write} 10, seq 3, type=opcode+projid, ref 0 rulerd0 projid={100}&opcode={ost_read} 5, seq 2, type=opcode+projid, ref 0 default * 10000, seq 1, type=nid+opcode+projid, ref 0 high_priority_requests: CPT 0: ruleall0 projid={200} 15, seq 4, type=projid, ref 0 rulewr0 projid={100}&opcode={ost_write} 10, seq 3, type=opcode+projid, ref 0 rulerd0 projid={100}&opcode={ost_read} 5, seq 2, type=opcode+projid, ref 0 default * 10000, seq 1, type=nid+opcode+projid, ref 0 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.491076 s, 34.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11018 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106221 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215052 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221418 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.250695 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151035 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.29674 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.308277 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196626 s, 5.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.238669 s, 4.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.335795 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.245547 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.271259 s, 3.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9f8d866d9800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9f8d866d9800.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.287154 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.288088 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.340487 s, 3.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9f8d866d9800.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0421905 s, 24.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0352012 s, 29.8 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0445353 s, 23.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0377241 s, 27.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0401444 s, 26.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.040833 s, 25.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0404628 s, 25.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102381 s, 10.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.062354 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115256 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0753612 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0694987 s, 15.1 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0770246 s, 13.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0742679 s, 14.1 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0852532 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0955386 s, 11.0 MB/s removed '/mnt/lustre/d77td.sanityn/nrs_r_oleg351-client.virtnet' removed '/mnt/lustre/d77td.sanityn/nrs_w_oleg351-client.virtnet' removed directory '/mnt/lustre/d77td.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 5, bs: 1048576 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.9059 s, 9.6 MB/s Write runtime is 12 s, speed is 8.333333 vs. 10 Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.5801 s, 5.4 MB/s Read runtime is 21 s, speed is 4.761904 vs. 5 cleanup_tbf /mnt/lustre/d77td.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 15, read rate: 15, bs: 1048576 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.71173 s, 12.0 MB/s Write runtime is 10 s, speed is 10.000000 vs. 15 Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.69838 s, 15.7 MB/s Read runtime is 8 s, speed is 12.500000 vs. 15 cleanup_tbf /mnt/lustre/d77td.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=change default rate=15 ost.OSS.ost_io.nrs_tbf_rule= regular_requests: CPT 0: ruleall0 projid={200} 15, seq 4, type=projid, ref 1 rulewr0 projid={100}&opcode={ost_write} 10, seq 3, type=opcode+projid, ref 1 rulerd0 projid={100}&opcode={ost_read} 5, seq 2, type=opcode+projid, ref 1 default * 15, seq 1, type=nid+opcode+projid, ref 2 high_priority_requests: CPT 0: ruleall0 projid={200} 15, seq 4, type=projid, ref 0 rulewr0 projid={100}&opcode={ost_write} 10, seq 3, type=opcode+projid, ref 0 rulerd0 projid={100}&opcode={ost_read} 5, seq 2, type=opcode+projid, ref 0 default * 15, seq 1, type=nid+opcode+projid, ref 1 not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 15, read rate: 15, bs: 1048576 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.47488 s, 11.1 MB/s Write runtime is 11 s, speed is 9.090909 vs. 15 Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.69834 s, 15.7 MB/s Read runtime is 8 s, speed is 12.500000 vs. 15 cleanup_tbf /mnt/lustre/d77td.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf nid classify=dynamic allow=projid+opcode allow=projid ost.OSS.ost_io.nrs_tbf_allowed_types= regular_allowed_types: CPT 0: 0: 0x00000024 opcode+projid 1: 0x00000020 projid 2: 0x00000002 nid high_priority_allowed_types: CPT 0: 0: 0x00000024 opcode+projid 1: 0x00000020 projid 2: 0x00000002 nid 8 ost.OSS.ost_io.nrs_tbf_rule=start rulerd0 projid={100}&opcode={ost_read} rate=5 ost.OSS.ost_io.nrs_tbf_rule=start rulewr0 projid={100}&opcode={ost_write} rate=10 ost.OSS.ost_io.nrs_tbf_rule=start rulerd1 projid={101}&opcode={ost_read} rate=5 ost.OSS.ost_io.nrs_tbf_rule=start rulewr1 projid={101}&opcode={ost_write} rate=10 ost.OSS.ost_io.nrs_tbf_rule=start rulerd2 projid={102}&opcode={ost_read} rate=5 ost.OSS.ost_io.nrs_tbf_rule=start rulewr2 projid={102}&opcode={ost_write} rate=10 ost.OSS.ost_io.nrs_tbf_rule=start rulerd3 projid={103}&opcode={ost_read} rate=5 ost.OSS.ost_io.nrs_tbf_rule=start rulewr3 projid={103}&opcode={ost_write} rate=10 ost.OSS.ost_io.nrs_tbf_rule=start rulerd4 projid={104}&opcode={ost_read} rate=5 ost.OSS.ost_io.nrs_tbf_rule=start rulewr4 projid={104}&opcode={ost_write} rate=10 ost.OSS.ost_io.nrs_tbf_rule=start ruleall0 projid={200} rate=15 ost.OSS.ost_io.nrs_tbf_rule=start ruleall1 projid={201} rate=15 ost.OSS.ost_io.nrs_tbf_rule=start ruleall2 projid={202} rate=15 ost.OSS.ost_io.nrs_tbf_rule=start ruleall3 projid={203} rate=15 ost.OSS.ost_io.nrs_tbf_rule=start ruleall4 projid={204} rate=15 ost.OSS.ost_io.nrs_tbf_rule= regular_requests: CPT 0: ruleall4 projid={204} 15, seq 16, type=projid, ref 0 ruleall3 projid={203} 15, seq 15, type=projid, ref 0 ruleall2 projid={202} 15, seq 14, type=projid, ref 0 ruleall1 projid={201} 15, seq 13, type=projid, ref 0 ruleall0 projid={200} 15, seq 12, type=projid, ref 0 rulewr4 projid={104}&opcode={ost_write} 10, seq 11, type=opcode+projid, ref 0 rulerd4 projid={104}&opcode={ost_read} 5, seq 10, type=opcode+projid, ref 0 rulewr3 projid={103}&opcode={ost_write} 10, seq 9, type=opcode+projid, ref 0 rulerd3 projid={103}&opcode={ost_read} 5, seq 8, type=opcode+projid, ref 0 rulewr2 projid={102}&opcode={ost_write} 10, seq 7, type=opcode+projid, ref 0 rulerd2 projid={102}&opcode={ost_read} 5, seq 6, type=opcode+projid, ref 0 rulewr1 projid={101}&opcode={ost_write} 10, seq 5, type=opcode+projid, ref 0 rulerd1 projid={101}&opcode={ost_read} 5, seq 4, type=opcode+projid, ref 0 rulewr0 projid={100}&opcode={ost_write} 10, seq 3, type=opcode+projid, ref 0 rulerd0 projid={100}&opcode={ost_read} 5, seq 2, type=opcode+projid, ref 0 default * 10000, seq 1, type=nid+opcode+projid, ref 0 high_priority_requests: CPT 0: ruleall4 projid={204} 15, seq 16, type=projid, ref 0 ruleall3 projid={203} 15, seq 15, type=projid, ref 0 ruleall2 projid={202} 15, seq 14, type=projid, ref 0 ruleall1 projid={201} 15, seq 13, type=projid, ref 0 ruleall0 projid={200} 15, seq 12, type=projid, ref 0 rulewr4 projid={104}&opcode={ost_write} 10, seq 11, type=opcode+projid, ref 0 rulerd4 projid={104}&opcode={ost_read} 5, seq 10, type=opcode+projid, ref 0 rulewr3 projid={103}&opcode={ost_write} 10, seq 9, type=opcode+projid, ref 0 rulerd3 projid={103}&opcode={ost_read} 5, seq 8, type=opcode+projid, ref 0 rulewr2 projid={102}&opcode={ost_write} 10, seq 7, type=opcode+projid, ref 0 rulerd2 projid={102}&opcode={ost_read} 5, seq 6, type=opcode+projid, ref 0 rulewr1 projid={101}&opcode={ost_write} 10, seq 5, type=opcode+projid, ref 0 rulerd1 projid={101}&opcode={ost_read} 5, seq 4, type=opcode+projid, ref 0 rulewr0 projid={100}&opcode={ost_write} 10, seq 3, type=opcode+projid, ref 0 rulerd0 projid={100}&opcode={ost_read} 5, seq 2, type=opcode+projid, ref 0 default * 10000, seq 1, type=nid+opcode+projid, ref 0 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.546922 s, 30.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100144 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0951969 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155591 s, 6.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.258424 s, 4.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.204236 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17741 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.245205 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191437 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.248487 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.276484 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227483 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.33135 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.255852 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.348345 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.243597 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.339158 s, 3.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9f8d866d9800.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff9f8d866d9800.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00307233 s, 341 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00422157 s, 248 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00213448 s, 491 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13663 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133591 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00237989 s, 441 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0761812 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0552789 s, 19.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0644682 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0799879 s, 13.1 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102667 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0495617 s, 21.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0413717 s, 25.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0282689 s, 37.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0248627 s, 42.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0223144 s, 47.0 MB/s removed '/mnt/lustre/d77td.sanityn/nrs_r_oleg351-client.virtnet' removed '/mnt/lustre/d77td.sanityn/nrs_w_oleg351-client.virtnet' removed directory '/mnt/lustre/d77td.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 5, bs: 1048576 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.8743 s, 9.6 MB/s Write runtime is 12 s, speed is 8.333333 vs. 10 Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.6657 s, 5.3 MB/s Read runtime is 20 s, speed is 5.000000 vs. 5 cleanup_tbf /mnt/lustre/d77td.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 15, read rate: 15, bs: 1048576 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.72738 s, 10.8 MB/s Write runtime is 10 s, speed is 10.000000 vs. 15 Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.84809 s, 15.3 MB/s Read runtime is 8 s, speed is 12.500000 vs. 15 cleanup_tbf /mnt/lustre/d77td.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=change default rate=15 ost.OSS.ost_io.nrs_tbf_rule= regular_requests: CPT 0: ruleall4 projid={204} 15, seq 16, type=projid, ref 0 ruleall3 projid={203} 15, seq 15, type=projid, ref 0 ruleall2 projid={202} 15, seq 14, type=projid, ref 0 ruleall1 projid={201} 15, seq 13, type=projid, ref 0 ruleall0 projid={200} 15, seq 12, type=projid, ref 1 rulewr4 projid={104}&opcode={ost_write} 10, seq 11, type=opcode+projid, ref 0 rulerd4 projid={104}&opcode={ost_read} 5, seq 10, type=opcode+projid, ref 0 rulewr3 projid={103}&opcode={ost_write} 10, seq 9, type=opcode+projid, ref 0 rulerd3 projid={103}&opcode={ost_read} 5, seq 8, type=opcode+projid, ref 0 rulewr2 projid={102}&opcode={ost_write} 10, seq 7, type=opcode+projid, ref 0 rulerd2 projid={102}&opcode={ost_read} 5, seq 6, type=opcode+projid, ref 0 rulewr1 projid={101}&opcode={ost_write} 10, seq 5, type=opcode+projid, ref 0 rulerd1 projid={101}&opcode={ost_read} 5, seq 4, type=opcode+projid, ref 0 rulewr0 projid={100}&opcode={ost_write} 10, seq 3, type=opcode+projid, ref 1 rulerd0 projid={100}&opcode={ost_read} 5, seq 2, type=opcode+projid, ref 1 default * 15, seq 1, type=nid+opcode+projid, ref 2 high_priority_requests: CPT 0: ruleall4 projid={204} 15, seq 16, type=projid, ref 0 ruleall3 projid={203} 15, seq 15, type=projid, ref 0 ruleall2 projid={202} 15, seq 14, type=projid, ref 0 ruleall1 projid={201} 15, seq 13, type=projid, ref 0 ruleall0 projid={200} 15, seq 12, type=projid, ref 0 rulewr4 projid={104}&opcode={ost_write} 10, seq 11, type=opcode+projid, ref 0 rulerd4 projid={104}&opcode={ost_read} 5, seq 10, type=opcode+projid, ref 0 rulewr3 projid={103}&opcode={ost_write} 10, seq 9, type=opcode+projid, ref 0 rulerd3 projid={103}&opcode={ost_read} 5, seq 8, type=opcode+projid, ref 0 rulewr2 projid={102}&opcode={ost_write} 10, seq 7, type=opcode+projid, ref 0 rulerd2 projid={102}&opcode={ost_read} 5, seq 6, type=opcode+projid, ref 0 rulewr1 projid={101}&opcode={ost_write} 10, seq 5, type=opcode+projid, ref 0 rulerd1 projid={101}&opcode={ost_read} 5, seq 4, type=opcode+projid, ref 0 rulewr0 projid={100}&opcode={ost_write} 10, seq 3, type=opcode+projid, ref 0 rulerd0 projid={100}&opcode={ost_read} 5, seq 2, type=opcode+projid, ref 0 default * 15, seq 1, type=nid+opcode+projid, ref 1 not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 15, read rate: 15, bs: 1048576 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.05282 s, 11.6 MB/s Write runtime is 10 s, speed is 10.000000 vs. 15 Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.80304 s, 15.4 MB/s Read runtime is 8 s, speed is 12.500000 vs. 15 cleanup_tbf /mnt/lustre/d77td.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=fifo PASS 77td (425s) == sanityn test 77tf: TBF: rule churn under load on a dynamic classify policy ========================================================== 21:33:49 (1788831229) jobid_name=%e.%p jobid_var=TEST_77TF_JOBVAR ost.OSS.ost_io.nrs_policies=tbf jobid+opcode classify=dynamic ost.OSS.ost_io.nrs_tbf_rule=start churn1 jobid={dd.*} rate=2000 ost.OSS.ost_io.nrs_tbf_rule=start churn2 jobid={dd.*}&opcode={ost_write} rate=2000 ost.OSS.ost_io.nrs_tbf_rule=stop churn2 ost.OSS.ost_io.nrs_tbf_rule=stop churn1 ost.OSS.ost_io.nrs_tbf_rule=start churn1 jobid={dd.*} rate=2000 ost.OSS.ost_io.nrs_tbf_rule=start churn2 jobid={dd.*}&opcode={ost_write} rate=2000 ost.OSS.ost_io.nrs_tbf_rule=stop churn2 ost.OSS.ost_io.nrs_tbf_rule=stop churn1 ost.OSS.ost_io.nrs_tbf_rule=start churn1 jobid={dd.*} rate=2000 ost.OSS.ost_io.nrs_tbf_rule=start churn2 jobid={dd.*}&opcode={ost_write} rate=2000 ost.OSS.ost_io.nrs_tbf_rule=stop churn2 ost.OSS.ost_io.nrs_tbf_rule=stop churn1 ost.OSS.ost_io.nrs_tbf_rule=start churn1 jobid={dd.*} rate=2000 ost.OSS.ost_io.nrs_tbf_rule=start churn2 jobid={dd.*}&opcode={ost_write} rate=2000 ost.OSS.ost_io.nrs_tbf_rule=stop churn2 ost.OSS.ost_io.nrs_tbf_rule=stop churn1 ost.OSS.ost_io.nrs_tbf_rule=start churn1 jobid={dd.*} rate=2000 ost.OSS.ost_io.nrs_tbf_rule=start churn2 jobid={dd.*}&opcode={ost_write} rate=2000 ost.OSS.ost_io.nrs_tbf_rule=stop churn2 ost.OSS.ost_io.nrs_tbf_rule=stop churn1 ost.OSS.ost_io.nrs_tbf_rule=start churn1 jobid={dd.*} rate=2000 ost.OSS.ost_io.nrs_tbf_rule=start churn2 jobid={dd.*}&opcode={ost_write} rate=2000 ost.OSS.ost_io.nrs_tbf_rule=stop churn2 ost.OSS.ost_io.nrs_tbf_rule=stop churn1 ost.OSS.ost_io.nrs_tbf_rule=start churn1 jobid={dd.*} rate=2000 ost.OSS.ost_io.nrs_tbf_rule=start churn2 jobid={dd.*}&opcode={ost_write} rate=2000 ost.OSS.ost_io.nrs_tbf_rule=stop churn2 ost.OSS.ost_io.nrs_tbf_rule=stop churn1 ost.OSS.ost_io.nrs_tbf_rule=start churn1 jobid={dd.*} rate=2000 ost.OSS.ost_io.nrs_tbf_rule=start churn2 jobid={dd.*}&opcode={ost_write} rate=2000 ost.OSS.ost_io.nrs_tbf_rule=stop churn2 ost.OSS.ost_io.nrs_tbf_rule=stop churn1 ost.OSS.ost_io.nrs_tbf_rule=start churn1 jobid={dd.*} rate=2000 ost.OSS.ost_io.nrs_tbf_rule=start churn2 jobid={dd.*}&opcode={ost_write} rate=2000 ost.OSS.ost_io.nrs_tbf_rule=stop churn2 ost.OSS.ost_io.nrs_tbf_rule=stop churn1 ost.OSS.ost_io.nrs_tbf_rule=start churn1 jobid={dd.*} rate=2000 ost.OSS.ost_io.nrs_tbf_rule=start churn2 jobid={dd.*}&opcode={ost_write} rate=2000 ost.OSS.ost_io.nrs_tbf_rule=stop churn2 ost.OSS.ost_io.nrs_tbf_rule=stop churn1 Cleanup TBF ost.OSS.ost_io.nrs_tbf_rule=stop churn1 ost.OSS.ost_io.nrs_tbf_rule=stop churn2 ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid jobid_name=%e.%u PASS 77tf (57s) == sanityn test 77te: Dynamic type lookup should obey the rule order ========================================================== 21:34:46 (1788831286) using SAVE_PROJECT_SUPPORTED=0 ost.OSS.ost_io.nrs_policies=tbf projid classify=dynamic allow=projid+opcode allow=uid ost.OSS.ost_io.nrs_tbf_allowed_types= regular_allowed_types: CPT 0: 0: 0x00000024 opcode+projid 1: 0x00000020 projid 2: 0x00000008 uid high_priority_allowed_types: CPT 0: 0: 0x00000024 opcode+projid 1: 0x00000020 projid 2: 0x00000008 uid ost.OSS.ost_io.nrs_tbf_rule=start rule1 projid={100}&opcode={ost_read} rate=10 ost.OSS.ost_io.nrs_tbf_rule=start rule2 projid={100}&opcode={ost_write} rate=10 ost.OSS.ost_io.nrs_tbf_rule=start rule3 uid={500} rate=5 regular_requests: CPT 0: rule3 uid={500} 5, seq 4, type=uid, ref 0 rule2 projid={100}&opcode={ost_write} 10, seq 3, type=opcode+projid, ref 0 rule1 projid={100}&opcode={ost_read} 10, seq 2, type=opcode+projid, ref 0 default * 10000, seq 1, type=opcode+uid+projid, ref 0 high_priority_requests: CPT 0: rule3 uid={500} 5, seq 4, type=uid, ref 0 rule2 projid={100}&opcode={ost_write} 10, seq 3, type=opcode+projid, ref 0 rule1 projid={100}&opcode={ost_read} 10, seq 2, type=opcode+projid, ref 0 default * 10000, seq 1, type=opcode+uid+projid, ref 0 not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10, bs: 1048576 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.9733 s, 9.6 MB/s Write runtime is 12 s, speed is 8.333333 vs. 10 Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.88641 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 vs. 10 cleanup_tbf /mnt/lustre/d77te.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5, bs: 1048576 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77te.sanityn/tbf] [bs=1048576] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8791 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 vs. 5 Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77te.sanityn/tbf] [of=/dev/null] [bs=1048576] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9297 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 vs. 5 cleanup_tbf /mnt/lustre/d77te.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_policies=fifo PASS 77te (124s) == sanityn test 77tg: TBF: the default rule holds its rate with many rules ========================================================== 21:36:50 (1788831410) ost.OSS.ost_io.nrs_policies=tbf uid classify=dynamic allow=uid ost.OSS.ost_io.nrs_tbf_rule=start r5000 uid={5000} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5001 uid={5001} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5002 uid={5002} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5003 uid={5003} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5004 uid={5004} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5005 uid={5005} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5006 uid={5006} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5007 uid={5007} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5008 uid={5008} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5009 uid={5009} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5010 uid={5010} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5011 uid={5011} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5012 uid={5012} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5013 uid={5013} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5014 uid={5014} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5015 uid={5015} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5016 uid={5016} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5017 uid={5017} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5018 uid={5018} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5019 uid={5019} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5020 uid={5020} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5021 uid={5021} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5022 uid={5022} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5023 uid={5023} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5024 uid={5024} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5025 uid={5025} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5026 uid={5026} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5027 uid={5027} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5028 uid={5028} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5029 uid={5029} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5030 uid={5030} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5031 uid={5031} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5032 uid={5032} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5033 uid={5033} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5034 uid={5034} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5035 uid={5035} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5036 uid={5036} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5037 uid={5037} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5038 uid={5038} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5039 uid={5039} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5040 uid={5040} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5041 uid={5041} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5042 uid={5042} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5043 uid={5043} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5044 uid={5044} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5045 uid={5045} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5046 uid={5046} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5047 uid={5047} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5048 uid={5048} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5049 uid={5049} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5050 uid={5050} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5051 uid={5051} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5052 uid={5052} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5053 uid={5053} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5054 uid={5054} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5055 uid={5055} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5056 uid={5056} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5057 uid={5057} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5058 uid={5058} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5059 uid={5059} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5060 uid={5060} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5061 uid={5061} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5062 uid={5062} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5063 uid={5063} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5064 uid={5064} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5065 uid={5065} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5066 uid={5066} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5067 uid={5067} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5068 uid={5068} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5069 uid={5069} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5070 uid={5070} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5071 uid={5071} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5072 uid={5072} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5073 uid={5073} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5074 uid={5074} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5075 uid={5075} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5076 uid={5076} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5077 uid={5077} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5078 uid={5078} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5079 uid={5079} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5080 uid={5080} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5081 uid={5081} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5082 uid={5082} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5083 uid={5083} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5084 uid={5084} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5085 uid={5085} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5086 uid={5086} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5087 uid={5087} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5088 uid={5088} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5089 uid={5089} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5090 uid={5090} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5091 uid={5091} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5092 uid={5092} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5093 uid={5093} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5094 uid={5094} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5095 uid={5095} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5096 uid={5096} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5097 uid={5097} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5098 uid={5098} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5099 uid={5099} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5100 uid={5100} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5101 uid={5101} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5102 uid={5102} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5103 uid={5103} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5104 uid={5104} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5105 uid={5105} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5106 uid={5106} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5107 uid={5107} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5108 uid={5108} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5109 uid={5109} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5110 uid={5110} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5111 uid={5111} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5112 uid={5112} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5113 uid={5113} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5114 uid={5114} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5115 uid={5115} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5116 uid={5116} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5117 uid={5117} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5118 uid={5118} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5119 uid={5119} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5120 uid={5120} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5121 uid={5121} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5122 uid={5122} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5123 uid={5123} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5124 uid={5124} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5125 uid={5125} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5126 uid={5126} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5127 uid={5127} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5128 uid={5128} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5129 uid={5129} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5130 uid={5130} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5131 uid={5131} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5132 uid={5132} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5133 uid={5133} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5134 uid={5134} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5135 uid={5135} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5136 uid={5136} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5137 uid={5137} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5138 uid={5138} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5139 uid={5139} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5140 uid={5140} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5141 uid={5141} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5142 uid={5142} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5143 uid={5143} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5144 uid={5144} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5145 uid={5145} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5146 uid={5146} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5147 uid={5147} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5148 uid={5148} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5149 uid={5149} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5150 uid={5150} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5151 uid={5151} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5152 uid={5152} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5153 uid={5153} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5154 uid={5154} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5155 uid={5155} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5156 uid={5156} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5157 uid={5157} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5158 uid={5158} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5159 uid={5159} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5160 uid={5160} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5161 uid={5161} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5162 uid={5162} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5163 uid={5163} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5164 uid={5164} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5165 uid={5165} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5166 uid={5166} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5167 uid={5167} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5168 uid={5168} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5169 uid={5169} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5170 uid={5170} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5171 uid={5171} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5172 uid={5172} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5173 uid={5173} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5174 uid={5174} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5175 uid={5175} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5176 uid={5176} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5177 uid={5177} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5178 uid={5178} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5179 uid={5179} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5180 uid={5180} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5181 uid={5181} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5182 uid={5182} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5183 uid={5183} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5184 uid={5184} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5185 uid={5185} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5186 uid={5186} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5187 uid={5187} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5188 uid={5188} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5189 uid={5189} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5190 uid={5190} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5191 uid={5191} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5192 uid={5192} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5193 uid={5193} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5194 uid={5194} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5195 uid={5195} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5196 uid={5196} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5197 uid={5197} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5198 uid={5198} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5199 uid={5199} rate=1000 not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10, bs: 1048576 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.4728 s, 10.0 MB/s Write runtime is 12 s, speed is 8.333333 vs. 10 Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.95 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 vs. 10 cleanup_tbf /mnt/lustre/d77tg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Cleanup TBF ost.OSS.ost_io.nrs_policies=fifo PASS 77tg (62s) == sanityn test 77ua: Setting TBF rate performance mode: BPS and IOPS ========================================================== 21:37:53 (1788831473) ost.OSS.ost_io.nrs_policies=tbf nid+opcode ost.OSS.ost_io.nrs_policies= regular_requests: - name: fifo state: started fallback: yes queued: 0 active: 0 - name: crrn state: stopped fallback: no queued: 0 active: 0 - name: orr state: stopped fallback: no queued: 0 active: 0 - name: trr state: stopped fallback: no queued: 0 active: 0 - name: tbf nid+opcode state: started fallback: no queued: 0 active: 0 - name: mclock state: stopped fallback: no queued: 0 active: 0 - name: delay state: stopped fallback: no queued: 0 active: 0 high_priority_requests: - name: fifo state: started fallback: yes queued: 0 active: 0 - name: crrn state: stopped fallback: no queued: 0 active: 0 - name: orr state: stopped fallback: no queued: 0 active: 0 - name: trr state: stopped fallback: no queued: 0 active: 0 - name: tbf nid+opcode state: started fallback: no queued: 0 active: 0 - name: mclock state: stopped fallback: no queued: 0 active: 0 - name: delay state: stopped fallback: no queued: 0 active: 0 ost.OSS.ost_io.nrs_tbf_rule=start nidrd nid={0@lo 192.168.203.51@tcp}&opcode={ost_read} bps=5MiB/s ost.OSS.ost_io.nrs_tbf_rule=start nidwr nid={0@lo 192.168.203.51@tcp}&opcode={ost_write} bps=15MiB/s regular_requests: CPT 0: nidwr nid={0@lo 192.168.203.51@tcp}&opcode={ost_write} 15, seq 3, type=nid+opcode, ref 0 bps=15728640 nidrd nid={0@lo 192.168.203.51@tcp}&opcode={ost_read} 5, seq 2, type=nid+opcode, ref 0 bps=5242880 default * 10000, seq 1, type=nid+opcode, ref 0 high_priority_requests: CPT 0: nidwr nid={0@lo 192.168.203.51@tcp}&opcode={ost_write} 15, seq 3, type=nid+opcode, ref 0 bps=15728640 nidrd nid={0@lo 192.168.203.51@tcp}&opcode={ost_read} 5, seq 2, type=nid+opcode, ref 0 bps=5242880 default * 10000, seq 1, type=nid+opcode, ref 0 ost.OSS.ost_io.nrs_tbf_rule=stop nidrd ost.OSS.ost_io.nrs_tbf_rule=stop nidwr ost.OSS.ost_io.nrs_tbf_rule=start nidrd nid={0@lo 192.168.203.51@tcp}&opcode={ost_read} bps=500K ost.OSS.ost_io.nrs_tbf_rule=start nidwr nid={0@lo 192.168.203.51@tcp}&opcode={ost_write} bps=750K regular_requests: CPT 0: nidwr nid={0@lo 192.168.203.51@tcp}&opcode={ost_write} 1, seq 5, type=nid+opcode, ref 0 bps=768000 nidrd nid={0@lo 192.168.203.51@tcp}&opcode={ost_read} 1, seq 4, type=nid+opcode, ref 0 bps=512000 default * 10000, seq 1, type=nid+opcode, ref 0 high_priority_requests: CPT 0: nidwr nid={0@lo 192.168.203.51@tcp}&opcode={ost_write} 1, seq 5, type=nid+opcode, ref 0 bps=768000 nidrd nid={0@lo 192.168.203.51@tcp}&opcode={ost_read} 1, seq 4, type=nid+opcode, ref 0 bps=512000 default * 10000, seq 1, type=nid+opcode, ref 0 ost.OSS.ost_io.nrs_tbf_rule=stop nidrd ost.OSS.ost_io.nrs_tbf_rule=stop nidwr ost.OSS.ost_io.nrs_tbf_rule=start nidrd nid={0@lo 192.168.203.51@tcp}&opcode={ost_read} rate=5MB/s ost.OSS.ost_io.nrs_tbf_rule=start nidwr nid={0@lo 192.168.203.51@tcp}&opcode={ost_write} rate=15MB/s regular_requests: CPT 0: nidwr nid={0@lo 192.168.203.51@tcp}&opcode={ost_write} 14, seq 7, type=nid+opcode, ref 0 bps=15000000 nidrd nid={0@lo 192.168.203.51@tcp}&opcode={ost_read} 4, seq 6, type=nid+opcode, ref 0 bps=5000000 default * 10000, seq 1, type=nid+opcode, ref 0 high_priority_requests: CPT 0: nidwr nid={0@lo 192.168.203.51@tcp}&opcode={ost_write} 14, seq 7, type=nid+opcode, ref 0 bps=15000000 nidrd nid={0@lo 192.168.203.51@tcp}&opcode={ost_read} 4, seq 6, type=nid+opcode, ref 0 bps=5000000 default * 10000, seq 1, type=nid+opcode, ref 0 ost.OSS.ost_io.nrs_tbf_rule=stop nidrd ost.OSS.ost_io.nrs_tbf_rule=stop nidwr ost.OSS.ost_io.nrs_tbf_rule=start nidrd nid={0@lo 192.168.203.51@tcp}&opcode={ost_read} rate=5MB ost.OSS.ost_io.nrs_tbf_rule=start nidwr nid={0@lo 192.168.203.51@tcp}&opcode={ost_write} rate=15MB regular_requests: CPT 0: nidwr nid={0@lo 192.168.203.51@tcp}&opcode={ost_write} 14, seq 9, type=nid+opcode, ref 0 bps=15000000 nidrd nid={0@lo 192.168.203.51@tcp}&opcode={ost_read} 4, seq 8, type=nid+opcode, ref 0 bps=5000000 default * 10000, seq 1, type=nid+opcode, ref 0 high_priority_requests: CPT 0: nidwr nid={0@lo 192.168.203.51@tcp}&opcode={ost_write} 14, seq 9, type=nid+opcode, ref 0 bps=15000000 nidrd nid={0@lo 192.168.203.51@tcp}&opcode={ost_read} 4, seq 8, type=nid+opcode, ref 0 bps=5000000 default * 10000, seq 1, type=nid+opcode, ref 0 ost.OSS.ost_io.nrs_tbf_rule=stop nidrd ost.OSS.ost_io.nrs_tbf_rule=stop nidwr ost.OSS.ost_io.nrs_tbf_rule=start nidrd nid={0@lo 192.168.203.51@tcp}&opcode={ost_read} bps=5 ost.OSS.ost_io.nrs_tbf_rule=start nidwr nid={0@lo 192.168.203.51@tcp}&opcode={ost_write} bps=15 regular_requests: CPT 0: nidwr nid={0@lo 192.168.203.51@tcp}&opcode={ost_write} 15, seq 11, type=nid+opcode, ref 0 bps=15728640 nidrd nid={0@lo 192.168.203.51@tcp}&opcode={ost_read} 5, seq 10, type=nid+opcode, ref 0 bps=5242880 default * 10000, seq 1, type=nid+opcode, ref 0 high_priority_requests: CPT 0: nidwr nid={0@lo 192.168.203.51@tcp}&opcode={ost_write} 15, seq 11, type=nid+opcode, ref 0 bps=15728640 nidrd nid={0@lo 192.168.203.51@tcp}&opcode={ost_read} 5, seq 10, type=nid+opcode, ref 0 bps=5242880 default * 10000, seq 1, type=nid+opcode, ref 0 ost.OSS.ost_io.nrs_tbf_rule=stop nidrd ost.OSS.ost_io.nrs_tbf_rule=stop nidwr ost.OSS.ost_io.nrs_tbf_rule=start nidrd nid={0@lo 192.168.203.51@tcp}&opcode={ost_read} bps=5 iops=1000 ost.OSS.ost_io.nrs_tbf_rule=start nidwr nid={0@lo 192.168.203.51@tcp}&opcode={ost_write} bps=15 iops=1200 regular_requests: CPT 0: nidwr nid={0@lo 192.168.203.51@tcp}&opcode={ost_write} 1200, seq 13, type=nid+opcode, ref 0 bps=15728640 nidrd nid={0@lo 192.168.203.51@tcp}&opcode={ost_read} 1000, seq 12, type=nid+opcode, ref 0 bps=5242880 default * 10000, seq 1, type=nid+opcode, ref 0 high_priority_requests: CPT 0: nidwr nid={0@lo 192.168.203.51@tcp}&opcode={ost_write} 1200, seq 13, type=nid+opcode, ref 0 bps=15728640 nidrd nid={0@lo 192.168.203.51@tcp}&opcode={ost_read} 1000, seq 12, type=nid+opcode, ref 0 bps=5242880 default * 10000, seq 1, type=nid+opcode, ref 0 ost.OSS.ost_io.nrs_tbf_rule=stop nidrd ost.OSS.ost_io.nrs_tbf_rule=stop nidwr ost.OSS.ost_io.nrs_policies=fifo PASS 77ua (28s) == sanityn test 77ub: Testing for TBF rate control in BPS performance mode ========================================================== 21:38:21 (1788831501) ost.OSS.ost_io.nrs_policies=tbf nid+opcode ost.OSS.ost_io.nrs_tbf_rule=start nidrd nid={0@lo 192.168.203.51@tcp}&opcode={ost_read} bps=5 ost.OSS.ost_io.nrs_tbf_rule=start nidwr nid={0@lo 192.168.203.51@tcp}&opcode={ost_write} bps=15 ost.OSS.ost_io.nrs_tbf_rule= regular_requests: CPT 0: nidwr nid={0@lo 192.168.203.51@tcp}&opcode={ost_write} 15, seq 3, type=nid+opcode, ref 0 bps=15728640 nidrd nid={0@lo 192.168.203.51@tcp}&opcode={ost_read} 5, seq 2, type=nid+opcode, ref 0 bps=5242880 default * 10000, seq 1, type=nid+opcode, ref 0 high_priority_requests: CPT 0: nidwr nid={0@lo 192.168.203.51@tcp}&opcode={ost_write} 15, seq 3, type=nid+opcode, ref 0 bps=15728640 nidrd nid={0@lo 192.168.203.51@tcp}&opcode={ost_read} 5, seq 2, type=nid+opcode, ref 0 bps=5242880 default * 10000, seq 1, type=nid+opcode, ref 0 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.786646 s, 21.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104771 s, 10.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0731634 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115838 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0760715 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.267129 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.374279 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.417714 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.422514 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.350647 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.473765 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.536256 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.45402 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.516308 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.450212 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.528083 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.546641 s, 1.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9f8d866d9800.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.245627 s, 4.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.258541 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.262037 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.280381 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.266412 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.277314 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.286654 s, 3.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.322641 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.300041 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.303482 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.307139 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.307768 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.31462 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.319385 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.330847 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.34198 s, 3.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9f8d866d9800.lock_unused_count=1 removed '/mnt/lustre/d77ub.sanityn/nrs_r_oleg351-client.virtnet' removed '/mnt/lustre/d77ub.sanityn/nrs_w_oleg351-client.virtnet' removed directory '/mnt/lustre/d77ub.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 15, read rate: 5, bs: 4096 Verify the write rate is under TBF control 25600+0 records in 25600+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 1595.66 s, 65.7 kB/s Write runtime is 1596 s, speed is .062656 vs. 15 Verify the read rate is under TBF control 25600+0 records in 25600+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 179.548 s, 584 kB/s Read runtime is 181 s, speed is .552486 vs. 5 cleanup_tbf /mnt/lustre/d77ub.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 15, read rate: 5, bs: 65536 Verify the write rate is under TBF control 1600+0 records in 1600+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 111.353 s, 942 kB/s Write runtime is 112 s, speed is .892857 vs. 15 Verify the read rate is under TBF control 1600+0 records in 1600+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 28.3599 s, 3.7 MB/s Read runtime is 30 s, speed is 3.333333 vs. 5 cleanup_tbf /mnt/lustre/d77ub.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 15, read rate: 5, bs: 131072 Verify the write rate is under TBF control 800+0 records in 800+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 58.6468 s, 1.8 MB/s Write runtime is 59 s, speed is 1.694915 vs. 15 Verify the read rate is under TBF control 800+0 records in 800+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 23.0916 s, 4.5 MB/s Read runtime is 24 s, speed is 4.166666 vs. 5 cleanup_tbf /mnt/lustre/d77ub.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 15, read rate: 5, bs: 262144 Verify the write rate is under TBF control 400+0 records in 400+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 29.3671 s, 3.6 MB/s Write runtime is 30 s, speed is 3.333333 vs. 15 Verify the read rate is under TBF control 400+0 records in 400+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.6414 s, 5.1 MB/s Read runtime is 22 s, speed is 4.545454 vs. 5 cleanup_tbf /mnt/lustre/d77ub.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 15, read rate: 5, bs: 1048576 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.1574 s, 11.5 MB/s Write runtime is 10 s, speed is 10.000000 vs. 15 Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9956 s, 5.2 MB/s Read runtime is 21 s, speed is 4.761904 vs. 5 cleanup_tbf /mnt/lustre/d77ub.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Cleanup TBF ost.OSS.ost_io.nrs_tbf_rule=stop nidrd ost.OSS.ost_io.nrs_tbf_rule=stop nidwr ost.OSS.ost_io.nrs_policies=fifo PASS 77ub (2219s) == sanityn test 77uc: TBF: the realtime key does not work with a byte rate ========================================================== 22:15:20 (1788833720) ost.OSS.ost_io.nrs_policies=tbf nid+opcode oleg351-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start rtwr nid={0@lo 192.168.203.51@tcp}&opcode={ost_write} realtime=1 bps=5MiB/s: Operation not supported oleg351-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start rtwr nid={0@lo 192.168.203.51@tcp}&opcode={ost_write} realtime=1 bps=5MiB/s': Operation not supported pdsh@oleg351-client: oleg351-server: ssh exited with exit code 95 ost.OSS.ost_io.nrs_tbf_rule=start rtwr nid={0@lo 192.168.203.51@tcp}&opcode={ost_write} realtime=1 rate=100 ost.OSS.ost_io.nrs_tbf_rule= regular_requests: CPT 0: rtwr nid={0@lo 192.168.203.51@tcp}&opcode={ost_write} 100, seq 2, type=nid+opcode, ref 0 default * 10000, seq 1, type=nid+opcode, ref 0 high_priority_requests: CPT 0: rtwr nid={0@lo 192.168.203.51@tcp}&opcode={ost_write} 100, seq 2, type=nid+opcode, ref 0 default * 10000, seq 1, type=nid+opcode, ref 0 Cleanup TBF ost.OSS.ost_io.nrs_tbf_rule=stop rtwr ost.OSS.ost_io.nrs_policies=fifo PASS 77uc (18s) == sanityn test 77ud: TBF: a rate change on a BPS rule leaves an IOPS rate ========================================================== 22:15:38 (1788833738) ost.OSS.ost_io.nrs_policies=tbf nid+opcode ost.OSS.ost_io.nrs_tbf_rule=start nidrd nid={0@lo 192.168.203.51@tcp}&opcode={ost_read} bps=5MiB/s ost.OSS.ost_io.nrs_tbf_rule=start nidwr nid={0@lo 192.168.203.51@tcp}&opcode={ost_write} bps=5MiB/s ost.OSS.ost_io.nrs_tbf_rule= regular_requests: CPT 0: nidwr nid={0@lo 192.168.203.51@tcp}&opcode={ost_write} 5, seq 3, type=nid+opcode, ref 0 bps=5242880 nidrd nid={0@lo 192.168.203.51@tcp}&opcode={ost_read} 5, seq 2, type=nid+opcode, ref 0 bps=5242880 default * 10000, seq 1, type=nid+opcode, ref 0 high_priority_requests: CPT 0: nidwr nid={0@lo 192.168.203.51@tcp}&opcode={ost_write} 5, seq 3, type=nid+opcode, ref 0 bps=5242880 nidrd nid={0@lo 192.168.203.51@tcp}&opcode={ost_read} 5, seq 2, type=nid+opcode, ref 0 bps=5242880 default * 10000, seq 1, type=nid+opcode, ref 0 ost.OSS.ost_io.nrs_tbf_rule=change nidrd rate=100 ost.OSS.ost_io.nrs_tbf_rule=change nidwr rate=100 ost.OSS.ost_io.nrs_tbf_rule= regular_requests: CPT 0: nidwr nid={0@lo 192.168.203.51@tcp}&opcode={ost_write} 100, seq 3, type=nid+opcode, ref 0 nidrd nid={0@lo 192.168.203.51@tcp}&opcode={ost_read} 100, seq 2, type=nid+opcode, ref 0 default * 10000, seq 1, type=nid+opcode, ref 0 high_priority_requests: CPT 0: nidwr nid={0@lo 192.168.203.51@tcp}&opcode={ost_write} 100, seq 3, type=nid+opcode, ref 0 nidrd nid={0@lo 192.168.203.51@tcp}&opcode={ost_read} 100, seq 2, type=nid+opcode, ref 0 default * 10000, seq 1, type=nid+opcode, ref 0 ost.OSS.ost_io.nrs_tbf_rule=change nidrd rate=20 ost.OSS.ost_io.nrs_tbf_rule=change nidwr rate=20 not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20, bs: 1048576 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.96377 s, 11.7 MB/s Write runtime is 10 s, speed is 10.000000 vs. 20 Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.26647 s, 19.9 MB/s Read runtime is 6 s, speed is 16.666666 vs. 20 cleanup_tbf /mnt/lustre/d77ud.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Cleanup TBF ost.OSS.ost_io.nrs_tbf_rule=stop nidrd ost.OSS.ost_io.nrs_tbf_rule=stop nidwr ost.OSS.ost_io.nrs_policies=fifo PASS 77ud (58s) == sanityn test 77va: Reserved rate mClock: check OPCode NRS policy ========================================================== 22:16:37 (1788833797) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=mclock opcode ost.OSS.ost_io.nrs_mclock_rule=start ost_r opcode={ost_read} resvd_rate=5 rate=5 ost.OSS.ost_io.nrs_mclock_rule=start ost_w opcode={ost_write} resvd_rate=20 rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.629159 s, 26.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0823641 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0923672 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196533 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.256393 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22338 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.277937 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.332451 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.319082 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.240954 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.308482 s, 3.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.304166 s, 3.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.385242 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.345345 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.374844 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.363094 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.44291 s, 2.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9f8d866d9800.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff9f8d866d9800.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0904806 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0369743 s, 28.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112425 s, 9.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0500228 s, 21.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0797013 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0505267 s, 20.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113216 s, 9.3 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0900267 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0941615 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100112 s, 10.5 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0961556 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.096202 s, 10.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163906 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0951983 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113182 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108387 s, 9.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9f8d866d9800.lock_unused_count=2 removed '/mnt/lustre/d77va.sanityn/nrs_r_oleg351-client.virtnet' removed '/mnt/lustre/d77va.sanityn/nrs_w_oleg351-client.virtnet' removed directory '/mnt/lustre/d77va.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 5, bs: 1048576 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.06406 s, 11.6 MB/s Write runtime is 10 s, speed is 10.000000 vs. 20 Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8809 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 vs. 5 cleanup_tbf /mnt/lustre/d77va.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_mclock_rule=stop ost_r ost.OSS.ost_io.nrs_mclock_rule=stop ost_w ost.OSS.ost_io.nrs_policies=fifo PASS 77va (67s) == sanityn test 77vb: Reserved rate mClock: check UID/GID/JobID/OPCode expression ========================================================== 22:17:44 (1788833864) ost.OSS.ost_io.nrs_policies=mclock jobid+opcode+nid+uid+gid ost.OSS.ost_io.nrs_mclock_rule=start ext_w jobid={dd.500}&opcode={ost_write} resvd_rate=20 rate=20 ost.OSS.ost_io.nrs_mclock_rule=start ext_r jobid={dd.500}&opcode={ost_read} resvd_rate=10 rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77vb.sanityn/nrs_r_oleg351-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.633975 s, 26.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0700537 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.048168 s, 21.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113384 s, 9.2 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.055491 s, 18.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152953 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1359 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0756425 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.244901 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.279287 s, 3.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.286448 s, 3.7 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20658 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.310475 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174538 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212256 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.248984 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.300328 s, 3.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9f8d866d9800.lock_unused_count=1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0866124 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0101662 s, 103 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.030148 s, 34.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156998 s, 66.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0908335 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0904689 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0399824 s, 26.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0710616 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00828094 s, 127 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0602694 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115075 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131012 s, 80.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0565861 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0711331 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134253 s, 78.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.022741 s, 46.1 MB/s removed '/mnt/lustre/d77vb.sanityn/nrs_r_oleg351-client.virtnet' removed '/mnt/lustre/d77vb.sanityn/nrs_w_oleg351-client.virtnet' removed directory '/mnt/lustre/d77vb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10, bs: 1048576 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77vb.sanityn/tbf] [bs=1048576] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.79347 s, 11.9 MB/s Write runtime is 10 s, speed is 10.000000 vs. 20 Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77vb.sanityn/tbf] [of=/dev/null] [bs=1048576] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0304 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 vs. 10 cleanup_tbf /mnt/lustre/d77vb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_mclock_rule=stop ext_w ost.OSS.ost_io.nrs_mclock_rule=stop ext_r ost.OSS.ost_io.nrs_mclock_rule=start ext_w nid={0@lo 192.168.203.51@tcp}&opcode={ost_write} resvd_rate=20 rate=20 ost.OSS.ost_io.nrs_mclock_rule=start ext_r nid={0@lo 192.168.203.51@tcp}&opcode={ost_read} resvd_rate=10 rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.857338 s, 19.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158387 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206335 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222594 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.275464 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.337522 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.33515 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.411636 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.421442 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.481793 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.544476 s, 1.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.529125 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.417812 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.480339 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.525214 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.602826 s, 1.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9f8d866d9800.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff9f8d866d9800.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.728095 s, 1.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00232667 s, 451 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00249374 s, 420 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00500779 s, 209 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0116596 s, 89.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0072291 s, 145 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.006971 s, 150 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107608 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206929 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145375 s, 72.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.281884 s, 3.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0813508 s, 12.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158373 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0729953 s, 14.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0429049 s, 24.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0565959 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00780447 s, 134 MB/s removed '/mnt/lustre/d77vb.sanityn/nrs_r_oleg351-client.virtnet' removed '/mnt/lustre/d77vb.sanityn/nrs_w_oleg351-client.virtnet' removed directory '/mnt/lustre/d77vb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10, bs: 1048576 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0743 s, 10.4 MB/s Write runtime is 11 s, speed is 9.090909 vs. 20 Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.92941 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 vs. 10 cleanup_tbf /mnt/lustre/d77vb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_mclock_rule=stop ext_w ost.OSS.ost_io.nrs_mclock_rule=stop ext_r ost.OSS.ost_io.nrs_mclock_rule=start ext nid={0@lo 192.168.203.51@tcp}&jobid={dd.500} resvd_rate=20 rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77vb.sanityn/nrs_r_oleg351-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.25689 s, 13.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152793 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113848 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15947 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113354 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189184 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196727 s, 5.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.249612 s, 4.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.449623 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.267368 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.2853 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.281625 s, 3.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.487757 s, 2.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.324913 s, 3.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.345234 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.516555 s, 2.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.396795 s, 2.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9f8d866d9800.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff9f8d866d9800.lock_unused_count=5 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.086205 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106208 s, 9.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0648891 s, 16.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0429831 s, 24.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0315485 s, 33.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111073 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0397068 s, 26.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0819873 s, 12.8 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0591614 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0526003 s, 19.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0530773 s, 19.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0698308 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0956637 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0607931 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0698076 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153645 s, 6.8 MB/s removed '/mnt/lustre/d77vb.sanityn/nrs_r_oleg351-client.virtnet' removed '/mnt/lustre/d77vb.sanityn/nrs_w_oleg351-client.virtnet' removed directory '/mnt/lustre/d77vb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20, bs: 1048576 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77vb.sanityn/tbf] [bs=1048576] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.8413 s, 9.7 MB/s Write runtime is 12 s, speed is 8.333333 vs. 20 Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77vb.sanityn/tbf] [of=/dev/null] [bs=1048576] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.7319 s, 18.3 MB/s Read runtime is 7 s, speed is 14.285714 vs. 20 cleanup_tbf /mnt/lustre/d77vb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_mclock_rule=stop ext ost.OSS.ost_io.nrs_mclock_rule=start ext_a jobid={dd.500},opcode={ost_write} resvd_rate=20 rate=20 ost.OSS.ost_io.nrs_mclock_rule=start ext_b jobid={dd.500},opcode={ost_read} resvd_rate=10 rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77vb.sanityn/nrs_r_oleg351-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.851012 s, 19.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0450878 s, 23.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10932 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122304 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21067 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195403 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22846 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.231828 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.25191 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195442 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.219619 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.26126 s, 4.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.323162 s, 3.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.204986 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.245374 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.229732 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.340528 s, 3.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0427015 s, 24.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0442854 s, 23.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0458935 s, 22.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0520844 s, 20.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10444 s, 10.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0650662 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0577407 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.063294 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0680449 s, 15.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0701528 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.072139 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0776742 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0822989 s, 12.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.079171 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0928951 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0980056 s, 10.7 MB/s removed '/mnt/lustre/d77vb.sanityn/nrs_r_oleg351-client.virtnet' removed '/mnt/lustre/d77vb.sanityn/nrs_w_oleg351-client.virtnet' removed directory '/mnt/lustre/d77vb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10, bs: 1048576 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77vb.sanityn/tbf] [bs=1048576] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.215 s, 9.3 MB/s Write runtime is 12 s, speed is 8.333333 vs. 10 Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77vb.sanityn/tbf] [of=/dev/null] [bs=1048576] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.9406 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 vs. 10 cleanup_tbf /mnt/lustre/d77vb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10, bs: 1048576 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.57453 s, 12.2 MB/s Write runtime is 10 s, speed is 10.000000 vs. 20 Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.94059 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 vs. 10 cleanup_tbf /mnt/lustre/d77vb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_mclock_rule=stop ext_a ost.OSS.ost_io.nrs_mclock_rule=stop ext_b ost.OSS.ost_io.nrs_mclock_rule=start ext_ug uid={500}&gid={500} resvd_rate=5 rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77vb.sanityn/nrs_r_oleg351-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.36046 s, 12.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0663185 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13027 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146281 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200386 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174932 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227034 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.283007 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.229043 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.290464 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.271864 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201753 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.27009 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.343244 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.279782 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.231176 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.410788 s, 2.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123393 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124495 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112348 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0573253 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129688 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.069071 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114697 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136626 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101213 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0955229 s, 11.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0926096 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0932858 s, 11.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162932 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109878 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0934147 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212046 s, 4.9 MB/s removed '/mnt/lustre/d77vb.sanityn/nrs_w_oleg351-client.virtnet' removed '/mnt/lustre/d77vb.sanityn/nrs_r_oleg351-client.virtnet' removed directory '/mnt/lustre/d77vb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5, bs: 1048576 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77vb.sanityn/tbf] [bs=1048576] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.23 s, 5.2 MB/s Write runtime is 21 s, speed is 4.761904 vs. 5 Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77vb.sanityn/tbf] [of=/dev/null] [bs=1048576] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8395 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 vs. 5 cleanup_tbf /mnt/lustre/d77vb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_mclock_rule=stop ext_ug ost.OSS.ost_io.nrs_mclock_rule=start ext_uw uid={500}&opcode={ost_write} resvd_rate=20 rate=20 ost.OSS.ost_io.nrs_mclock_rule=start ext_ur uid={500}&opcode={ost_read} resvd_rate=10 rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77vb.sanityn/nrs_r_oleg351-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.529592 s, 31.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103529 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0997543 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197715 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178648 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210204 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242955 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.270107 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242782 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.267488 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.286859 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.292692 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.309195 s, 3.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.316971 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.315094 s, 3.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.34581 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.325801 s, 3.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9f8d866d9800.lock_unused_count=2 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0474377 s, 22.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0606027 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0465005 s, 22.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120162 s, 8.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0687344 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131272 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102219 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106636 s, 9.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110077 s, 9.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110542 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0936598 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117925 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118502 s, 8.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12219 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124685 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128927 s, 8.1 MB/s removed '/mnt/lustre/d77vb.sanityn/nrs_r_oleg351-client.virtnet' removed '/mnt/lustre/d77vb.sanityn/nrs_w_oleg351-client.virtnet' removed directory '/mnt/lustre/d77vb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10, bs: 1048576 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77vb.sanityn/tbf] [bs=1048576] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.12856 s, 11.5 MB/s Write runtime is 10 s, speed is 10.000000 vs. 20 Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77vb.sanityn/tbf] [of=/dev/null] [bs=1048576] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0435 s, 10.4 MB/s Read runtime is 11 s, speed is 9.090909 vs. 10 cleanup_tbf /mnt/lustre/d77vb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_mclock_rule=stop ext_uw ost.OSS.ost_io.nrs_mclock_rule=stop ext_ur ost.OSS.ost_io.nrs_mclock_rule=start ext_a uid={500},opcode={ost_write} resvd_rate=20 rate=20 ost.OSS.ost_io.nrs_mclock_rule=start ext_b uid={500},opcode={ost_read} resvd_rate=10 rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77vb.sanityn/nrs_r_oleg351-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.795614 s, 21.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.080818 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0839622 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0817932 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.287597 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.316269 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.235369 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.249435 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.320049 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.271511 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.434013 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.309547 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.40677 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.365254 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.361186 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.377168 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.405655 s, 2.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103471 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0749733 s, 14.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0367379 s, 28.5 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0490037 s, 21.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0613982 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0638171 s, 16.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0653572 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.066407 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0714223 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0716905 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.075119 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0788268 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0694082 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0958023 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.091619 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0801399 s, 13.1 MB/s removed '/mnt/lustre/d77vb.sanityn/nrs_w_oleg351-client.virtnet' removed '/mnt/lustre/d77vb.sanityn/nrs_r_oleg351-client.virtnet' removed directory '/mnt/lustre/d77vb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10, bs: 1048576 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77vb.sanityn/tbf] [bs=1048576] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 12.0009 s, 8.7 MB/s Write runtime is 13 s, speed is 7.692307 vs. 10 Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77vb.sanityn/tbf] [of=/dev/null] [bs=1048576] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.95704 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 vs. 10 cleanup_tbf /mnt/lustre/d77vb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10, bs: 1048576 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77vb.sanityn/tbf] [bs=1048576] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.1577 s, 9.4 MB/s Write runtime is 12 s, speed is 8.333333 vs. 20 Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77vb.sanityn/tbf] [of=/dev/null] [bs=1048576] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.9533 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 vs. 10 cleanup_tbf /mnt/lustre/d77vb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Cleanup TBF ost.OSS.ost_io.nrs_mclock_rule=stop ext_a ost.OSS.ost_io.nrs_mclock_rule=stop ext_b ost.OSS.ost_io.nrs_policies=fifo PASS 77vb (449s) == sanityn test 77vc: Reserved rate mClock: type granular combination: nid+opcode ========================================================== 22:25:13 (1788834313) ost.OSS.ost_io.nrs_policies=mclock nid+opcode ost.OSS.ost_io.nrs_mclock_rule=start ext_w nid={0@lo 192.168.203.51@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_mclock_rule=start ext_r nid={0@lo 192.168.203.51@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.638876 s, 26.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0221789 s, 47.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.047892 s, 21.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0771707 s, 13.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0922694 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.278213 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.279263 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.320027 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.304777 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.327003 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.347351 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.345809 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.365041 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.382315 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.42064 s, 2.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9f8d866d9800.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffff9f8d866d9800.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.401773 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.454058 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00951762 s, 110 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.078982 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170736 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0676649 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0382052 s, 27.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01526 s, 68.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0442733 s, 23.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0937062 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00473585 s, 221 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0703891 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116773 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0705966 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0473759 s, 22.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174463 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148557 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00793361 s, 132 MB/s removed '/mnt/lustre/d77vc.sanityn/nrs_r_oleg351-client.virtnet' removed '/mnt/lustre/d77vc.sanityn/nrs_w_oleg351-client.virtnet' removed directory '/mnt/lustre/d77vc.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10, bs: 1048576 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.78575 s, 10.7 MB/s Write runtime is 10 s, speed is 10.000000 vs. 20 Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.98304 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 vs. 10 cleanup_tbf /mnt/lustre/d77vc.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Cleanup TBF ost.OSS.ost_io.nrs_mclock_rule=stop ext_w ost.OSS.ost_io.nrs_mclock_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77vc (59s) == sanityn test 77vd: Reserved rate mClock: rate control in BPS performance mode ========================================================== 22:26:12 (1788834372) ost.OSS.ost_io.nrs_policies=mclock nid+opcode ost.OSS.ost_io.nrs_mclock_rule=start nidrd nid={0@lo 192.168.203.51@tcp}&opcode={ost_read} resvd_bps=5 bps=5 ost.OSS.ost_io.nrs_mclock_rule=start nidwr nid={0@lo 192.168.203.51@tcp}&opcode={ost_write} resvd_bps=15 bps=15 ost.OSS.ost_io.nrs_mclock_rule= regular_requests: CPT 0: nidwr nid={0@lo 192.168.203.51@tcp}&opcode={ost_write}, bps 15-15, weight 100, seq 3, type=nid+opcode, ref 0 nidrd nid={0@lo 192.168.203.51@tcp}&opcode={ost_read}, bps 5-5, weight 100, seq 2, type=nid+opcode, ref 0 default *, limit_iops 10000, weight 100, seq 1, type=nid+opcode, ref 0 high_priority_requests: CPT 0: nidwr nid={0@lo 192.168.203.51@tcp}&opcode={ost_write}, bps 15-15, weight 100, seq 3, type=nid+opcode, ref 0 nidrd nid={0@lo 192.168.203.51@tcp}&opcode={ost_read}, bps 5-5, weight 100, seq 2, type=nid+opcode, ref 0 default *, limit_iops 10000, weight 100, seq 1, type=nid+opcode, ref 0 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.750806 s, 22.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110785 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137387 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0884865 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199091 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147115 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.233486 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183054 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213525 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.226853 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.245125 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.204995 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21301 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211277 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193489 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.290154 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.273976 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0408899 s, 25.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00223808 s, 469 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0979223 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128854 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0761521 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0824824 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0839905 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171531 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0746529 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0928776 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0784202 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0802244 s, 13.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0490432 s, 21.4 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125358 s, 8.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0946967 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0969943 s, 10.8 MB/s removed '/mnt/lustre/d77vd.sanityn/nrs_w_oleg351-client.virtnet' removed '/mnt/lustre/d77vd.sanityn/nrs_r_oleg351-client.virtnet' removed directory '/mnt/lustre/d77vd.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 15, read rate: 5, bs: 4096 Verify the write rate is under TBF control 25600+0 records in 25600+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 1702.05 s, 61.6 kB/s Write runtime is 1703 s, speed is .058719 vs. 15 Verify the read rate is under TBF control 25600+0 records in 25600+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 157.696 s, 665 kB/s Read runtime is 159 s, speed is .628930 vs. 5 cleanup_tbf /mnt/lustre/d77vd.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 15, read rate: 5, bs: 65536 Verify the write rate is under TBF control 1600+0 records in 1600+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 135.383 s, 775 kB/s Write runtime is 136 s, speed is .735294 vs. 15 Verify the read rate is under TBF control 1600+0 records in 1600+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 27.4678 s, 3.8 MB/s Read runtime is 29 s, speed is 3.448275 vs. 5 cleanup_tbf /mnt/lustre/d77vd.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 15, read rate: 5, bs: 131072 Verify the write rate is under TBF control 800+0 records in 800+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 58.4381 s, 1.8 MB/s Write runtime is 60 s, speed is 1.666666 vs. 15 Verify the read rate is under TBF control 800+0 records in 800+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 22.0291 s, 4.8 MB/s Read runtime is 23 s, speed is 4.347826 vs. 5 cleanup_tbf /mnt/lustre/d77vd.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 15, read rate: 5, bs: 262144 Verify the write rate is under TBF control 400+0 records in 400+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 27.9484 s, 3.8 MB/s Write runtime is 29 s, speed is 3.448275 vs. 15 Verify the read rate is under TBF control 400+0 records in 400+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.2214 s, 5.2 MB/s Read runtime is 21 s, speed is 4.761904 vs. 5 cleanup_tbf /mnt/lustre/d77vd.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 15, read rate: 5, bs: 1048576 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.28551 s, 12.7 MB/s Write runtime is 9 s, speed is 11.111111 vs. 15 Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8266 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 vs. 5 cleanup_tbf /mnt/lustre/d77vd.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Cleanup TBF ost.OSS.ost_io.nrs_mclock_rule=stop nidrd ost.OSS.ost_io.nrs_mclock_rule=stop nidwr ost.OSS.ost_io.nrs_policies=fifo PASS 77vd (2309s) == sanityn test 77ve: mClock: rule Reserved and Limit rate setting ========================================================== 23:04:41 (1788836681) ost.OSS.ost_io.nrs_policies=mclock projid reserve=1 limit=1 oleg351-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_mclock_rule=start proj100 projid={100} resvd_rate=100 limit_rate=50: Numerical result out of range oleg351-server: error: set_param: setting 'ost/OSS/ost_io/nrs_mclock_rule'='start proj100 projid={100} resvd_rate=100 limit_rate=50': Numerical result out of range pdsh@oleg351-client: oleg351-server: ssh exited with exit code 34 ost.OSS.ost_io.nrs_mclock_rule=start proj100 projid={100} iops=100 Rule rate: limit_iops 100 ost.OSS.ost_io.nrs_mclock_rule=stop proj100 ost.OSS.ost_io.nrs_mclock_rule=start proj100 projid={100} rate=5MiB/s Rule rate: limit_bps 5 ost.OSS.ost_io.nrs_mclock_rule=stop proj100 ost.OSS.ost_io.nrs_mclock_rule=start proj100 projid={100} limit_rate=300MiB/s Rule rate: limit_bps 300 ost.OSS.ost_io.nrs_mclock_rule=stop proj100 ost.OSS.ost_io.nrs_mclock_rule=start proj100 projid={100} resvd_rate=100 limit_rate=300 Rule rate: iops 100-300 ost.OSS.ost_io.nrs_mclock_rule=stop proj100 ost.OSS.ost_io.nrs_mclock_rule=start proj100 projid={100} resvd_rate=10MiB limit_rate=30MiB Rule rate: bps 10-30 ost.OSS.ost_io.nrs_mclock_rule=stop proj100 ost.OSS.ost_io.nrs_mclock_rule=start proj100 projid={100} resvd_rate=10MiB limit_rate=30MiB resvd_iops=5 limit_iops=25 Rule rate: iops 5-25 bps 10-30 ost.OSS.ost_io.nrs_mclock_rule=stop proj100 ost.OSS.ost_io.nrs_mclock_rule=start proj100 projid={100} bps=5MiB/s Rule rate: limit_bps 5 ost.OSS.ost_io.nrs_mclock_rule=stop proj100 ost.OSS.ost_io.nrs_mclock_rule=start proj100 projid={100} limit_bps=10MiB/s ost.OSS.ost_io.nrs_mclock_rule=change proj100 limit_rate=100 ost.OSS.ost_io.nrs_mclock_rule= regular_requests: CPT 0: proj100 projid={100}, limit_iops 100, seq 9, type=projid, ref 0 default *, limit_iops 10000, seq 1, type=projid, ref 0 high_priority_requests: CPT 0: proj100 projid={100}, limit_iops 100, seq 9, type=projid, ref 0 default *, limit_iops 10000, seq 1, type=projid, ref 0 Rule rate: limit_iops 100 ost.OSS.ost_io.nrs_mclock_rule=stop proj100 ost.OSS.ost_io.nrs_mclock_rule=start proj100 projid={100} resvd_rate=10 limit_rate=300 ost.OSS.ost_io.nrs_mclock_rule=change proj100 limit_rate=200 Rule rate: iops 10-200 ost.OSS.ost_io.nrs_mclock_rule=stop proj100 ost.OSS.ost_io.nrs_mclock_rule=start proj100 projid={100} limit_bps=5000B/s Rule rate: limit_bps 5000B/s ost.OSS.ost_io.nrs_mclock_rule=stop proj100 ost.OSS.ost_io.nrs_mclock_rule=start proj100 projid={100} limit_bps=1536KiB/s Rule rate: limit_bps 1572864B/s Cleanup TBF ost.OSS.ost_io.nrs_mclock_rule=stop proj100 ost.OSS.ost_io.nrs_policies=fifo PASS 77ve (35s) == sanityn test 77vf: mClock: I/O testing with both Reserved and Limit rate control ========================================================== 23:05:16 (1788836716) using SAVE_PROJECT_SUPPORTED=0 ost.OSS.ost_io.nrs_policies=mclock projid reserve=1 limit=1 ost.OSS.ost_io.nrs_mclock_rule=start proj100 projid={100} resvd_rate=5 limit_rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.525236 s, 31.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0657281 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0759932 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0658636 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119737 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127576 s, 8.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.2342 s, 4.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193653 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172738 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183149 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.25843 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.268595 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.225409 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.416478 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.343449 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.399792 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.44593 s, 2.4 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9f8d866d9800.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117483 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112849 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163211 s, 6.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.2027 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16234 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197298 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164971 s, 6.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169316 s, 6.2 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170948 s, 6.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202928 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212118 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178722 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180787 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215475 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179147 s, 5.9 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227708 s, 4.6 MB/s removed '/mnt/lustre/d77vf.sanityn/nrs_r_oleg351-client.virtnet' removed '/mnt/lustre/d77vf.sanityn/nrs_w_oleg351-client.virtnet' removed directory '/mnt/lustre/d77vf.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20, bs: 1048576 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.32555 s, 14.3 MB/s Write runtime is 9 s, speed is 11.111111 vs. 20 Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.03585 s, 20.8 MB/s Read runtime is 6 s, speed is 16.666666 vs. 20 cleanup_tbf /mnt/lustre/d77vf.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Cleanup TBF ost.OSS.ost_io.nrs_mclock_rule=stop proj100 ost.OSS.ost_io.nrs_policies=fifo PASS 77vf (47s) == sanityn test 77vg: mClock: multiple flows with different reserved and limit rates ========================================================== 23:06:04 (1788836764) using SAVE_PROJECT_SUPPORTED=0 100 P /mnt/lustre/d77vg.sanityn/f77vg.sanityn.100 200 P /mnt/lustre2/d77vg.sanityn/f77vg.sanityn.200 ost.OSS.ost_io.nrs_policies=mclock projid reserve=1 limit=1 ost.OSS.ost_io.nrs_mclock_rule=start proj100 projid={100} resvd_rate=5 limit_rate=20 ost.OSS.ost_io.nrs_mclock_rule=start proj200 projid={200} resvd_rate=10 limit_rate=15 cpu_npartitions on ost1 is 1 mClock: Run 2 flows in parallel mClock - file: /mnt/lustre/d77vg.sanityn/f77vg.sanityn.100 mClock - rw: w, resvd: 5, limit: 20 WRITE: resvd=5 limit=20 file=/mnt/lustre/d77vg.sanityn/f77vg.sanityn.100 (np=1) mClock - file: /mnt/lustre2/d77vg.sanityn/f77vg.sanityn.200 mClock - rw: w, resvd: 10, limit: 15 WRITE: resvd=10 limit=15 file=/mnt/lustre2/d77vg.sanityn/f77vg.sanityn.200 (np=1) 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.12748 s, 12.9 MB/s Write runtime is 9 s, speed is 11.111111 IOPS === DONE: rate=11.111111 within 5-20 and np=1 === 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.94143 s, 11.7 MB/s Write runtime is 10 s, speed is 10.000000 IOPS === DONE: rate=10.000000 within 10-15 and np=1 === cpu_npartitions on ost1 is 1 mClock: Run 2 flows in parallel mClock - file: /mnt/lustre/d77vg.sanityn/f77vg.sanityn.100 mClock - rw: r, resvd: 5, limit: 20 READ: resvd=5 limit=20 file=/mnt/lustre/d77vg.sanityn/f77vg.sanityn.100 (np=1) mClock - file: /mnt/lustre2/d77vg.sanityn/f77vg.sanityn.200 mClock - rw: r, resvd: 10, limit: 15 READ: resvd=10 limit=15 file=/mnt/lustre2/d77vg.sanityn/f77vg.sanityn.200 (np=1) 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.08214 s, 20.6 MB/s Read runtime is 6 s, speed is 16.666666 IOPS === DONE: rate=16.666666 within 5-20 and np=1 === 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.62856 s, 15.8 MB/s Read runtime is 8 s, speed is 12.500000 IOPS === DONE: rate=12.500000 within 10-15 and np=1 === cpu_npartitions on ost1 is 1 mClock: Run 2 flows in parallel mClock - file: /mnt/lustre/d77vg.sanityn/f77vg.sanityn.100 mClock - rw: r, resvd: 5, limit: 20 READ: resvd=5 limit=20 file=/mnt/lustre/d77vg.sanityn/f77vg.sanityn.100 (np=1) mClock - file: /mnt/lustre2/d77vg.sanityn/f77vg.sanityn.200 mClock - rw: w, resvd: 10, limit: 15 WRITE: resvd=10 limit=15 file=/mnt/lustre2/d77vg.sanityn/f77vg.sanityn.200 (np=1) 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.29218 s, 19.8 MB/s Read runtime is 7 s, speed is 14.285714 IOPS === DONE: rate=14.285714 within 5-20 and np=1 === 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.41981 s, 14.1 MB/s Write runtime is 9 s, speed is 11.111111 IOPS === DONE: rate=11.111111 within 10-15 and np=1 === cpu_npartitions on ost1 is 1 mClock: Run 2 flows in parallel mClock - file: /mnt/lustre/d77vg.sanityn/f77vg.sanityn.100 mClock - rw: w, resvd: 5, limit: 20 mClock - file: /mnt/lustre2/d77vg.sanityn/f77vg.sanityn.200 mClock - rw: r, resvd: 10, limit: 15 READ: resvd=10 limit=15 file=/mnt/lustre2/d77vg.sanityn/f77vg.sanityn.200 (np=1) WRITE: resvd=5 limit=20 file=/mnt/lustre/d77vg.sanityn/f77vg.sanityn.100 (np=1) 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.69372 s, 15.7 MB/s Read runtime is 7 s, speed is 14.285714 IOPS === DONE: rate=14.285714 within 10-15 and np=1 === 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.92635 s, 15.1 MB/s Write runtime is 8 s, speed is 12.500000 IOPS === DONE: rate=12.500000 within 5-20 and np=1 === ost.OSS.ost_io.nrs_mclock_rule=stop proj100 ost.OSS.ost_io.nrs_mclock_rule=stop proj200 ost.OSS.ost_io.nrs_policies=fifo removed '/mnt/lustre/d77vg.sanityn/f77vg.sanityn.100' removed '/mnt/lustre2/d77vg.sanityn/f77vg.sanityn.200' PASS 77vg (49s) == sanityn test 77vm: mClock: reserve=0 turns the reserve dimension off ========================================================== 23:06:52 (1788836812) oleg351-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=mclock projid reserve=0: Invalid argument oleg351-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='mclock projid reserve=0': Invalid argument pdsh@oleg351-client: oleg351-server: ssh exited with exit code 22 ost.OSS.ost_io.nrs_policies=mclock projid reserve=0 limit=1 ost.OSS.ost_io.nrs_mclock_rule=start lim100 projid={100} limit_rate=100 Rule rate: limit_iops 100 oleg351-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_mclock_rule=start res100 projid={200} resvd_rate=100: Invalid argument oleg351-server: error: set_param: setting 'ost/OSS/ost_io/nrs_mclock_rule'='start res100 projid={200} resvd_rate=100': Invalid argument pdsh@oleg351-client: oleg351-server: ssh exited with exit code 22 ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=mclock projid reserve=0 limit=0 weight=1 Cleanup TBF ost.OSS.ost_io.nrs_mclock_rule=stop lim100 ost.OSS.ost_io.nrs_policies=fifo PASS 77vm (18s) == sanityn test 77vh: mClock: weight share fair queueing with 2 fio flows ========================================================== 23:07:10 (1788836830) using SAVE_PROJECT_SUPPORTED=0 /usr/bin/fio /usr/bin/which: no jq in (/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:) SKIP: sanityn test_77vh jq is not installed removed '/tmp/tmp.uZluLuG64y' removed '/tmp/tmp.L1u7TaXPk3' SKIP 77vh (5s) == sanityn test 77vi: mClock: reserved rate and weight combinations ========================================================== 23:07:15 (1788836835) using SAVE_PROJECT_SUPPORTED=0 /usr/bin/fio /usr/bin/which: no jq in (/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:) SKIP: sanityn test_77vi jq is not installed removed '/tmp/tmp.zd6s32QeT1' removed '/tmp/tmp.aSMytk96J6' SKIP 77vi (6s) == sanityn test 77vn: mClock: a start with no dimension key keeps R|L|W ========================================================== 23:07:22 (1788836842) ost.OSS.ost_io.nrs_policies=mclock projid ost.OSS.ost_io.nrs_mclock_rule=start proj100 projid={100} resvd_rate=100 limit_rate=300 weight=200 ost.OSS.ost_io.nrs_mclock_rule= regular_requests: CPT 0: proj100 projid={100}, iops 100-300, weight 200, seq 2, type=projid, ref 0 default *, limit_iops 10000, weight 100, seq 1, type=projid, ref 0 high_priority_requests: CPT 0: proj100 projid={100}, iops 100-300, weight 200, seq 2, type=projid, ref 0 default *, limit_iops 10000, weight 100, seq 1, type=projid, ref 0 Cleanup TBF ost.OSS.ost_io.nrs_mclock_rule=stop proj100 ost.OSS.ost_io.nrs_policies=fifo PASS 77vn (17s) == sanityn test 77vr: mClock: a reserve only policy schedules the default class ========================================================== 23:07:38 (1788836858) using SAVE_PROJECT_SUPPORTED=0 ost.OSS.ost_io.nrs_policies=mclock projid reserve=1 limit=0 Default rule rate: resvd_iops 10000 ost.OSS.ost_io.nrs_mclock_rule=start proj100 projid={100} resvd_rate=100 8+0 records in 8+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.662854 s, 12.7 MB/s 8+0 records in 8+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.184419 s, 45.5 MB/s Cleanup TBF ost.OSS.ost_io.nrs_mclock_rule=stop proj100 ost.OSS.ost_io.nrs_policies=fifo PASS 77vr (16s) == sanityn test 78: Enable policy and specify tunings right away ========================================================== 23:07:54 (1788836874) ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_orr_quantum=1 ost.OSS.ost_io.nrs_policies=fifo PASS 78 (8s) == sanityn test 79: xattr: intent error ================== 23:08:02 (1788836882) File: /mnt/lustre/d79.sanityn Size: 16384 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115272381366532 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-07 23:08:04.000000000 -0400 Modify: 2026-09-07 23:08:04.000000000 -0400 Change: 2026-09-07 23:08:04.000000000 -0400 Birth: 2026-09-07 23:08:05.000000000 -0400 fail_val=0 fail_loc=0x80000160 fail_val=0 fail_loc=0x80000131 # file: mnt/lustre/d79.sanityn trusted.name1="value1" PASS 79 (17s) == sanityn test 80a: migrate directory when some children is being opened ========================================================== 23:08:19 (1788836899) SKIP: sanityn test_80a needs >= 2 MDTs SKIP 80a (3s) == sanityn test 80b: Accessing directory during migration ========================================================== 23:08:22 (1788836902) SKIP: sanityn test_80b needs >= 2 MDTs SKIP 80b (3s) == sanityn test 81a: rename and stat under striped directory ========================================================== 23:08:25 (1788836905) SKIP: sanityn test_81a needs >= 2 MDTs SKIP 81a (3s) == sanityn test 81b: rename under striped directory doesn't deadlock ========================================================== 23:08:28 (1788836908) SKIP: sanityn test_81b We need at least 2 MDTs for this test SKIP 81b (2s) == sanityn test 81c: rename revoke LOOKUP lock for remote object ========================================================== 23:08:31 (1788836911) SKIP: sanityn test_81c needs >= 4 MDTs SKIP 81c (2s) == sanityn test 81d: parallel rename file cross-dir on same MDT ========================================================== 23:08:33 (1788836913) - open/close 858 (time 1788836927.37 total 10.00 last 85.79) - open/close 1700 (time 1788836937.38 total 20.02 last 84.08) total: 2500 open/close in 29.36 seconds: 85.15 ops/second snapshot_time 1788837021.720947821 secs.nsecs start_time 1788836916.847006701 secs.nsecs elapsed_time 104.873941120 secs.nsecs open 2505 samples [usecs] 123 15105 3360892 6927507086 close 2504 samples [usecs] 137 4387 869443 530835851 mknod 2501 samples [usecs] 564 15005 2957318 5502527644 unlink 2500 samples [usecs] 698 53184 5001368 38594886518 mkdir 2 samples [usecs] 1906 1981 3887 7557197 rmdir 2 samples [usecs] 4749 15915 20664 275840226 rename 2500 samples [usecs] 493 33206 13525478 136058758300 getattr 3761 samples [usecs] 7 5329 217089 167015105 parallel_rename_file 2500 samples [usecs] 493 33206 13525478 136058758300 crossdir_rename 2500 samples [usecs] 493 33206 13525478 136058758300 readpage 23 samples [usecs] 5573 38645 467672 11808236096 PASS 81d (112s) == sanityn test 82: fsetxattr and fgetxattr on orphan files ========================================================== 23:10:25 (1788837025) multiop /mnt/lustre/f82.sanityn vO_ac TMPPIPE=/tmp/multiop_open_wait_pipe.7629 multiop /mnt/lustre2/f82.sanityn vo_Ac TMPPIPE=/tmp/multiop_open_wait_pipe.7629 PASS 82 (6s) == sanityn test 83: access striped directory while it is being created/unlinked ========================================================== 23:10:31 (1788837031) SKIP: sanityn test_83 needs >= 2 MDTs SKIP 83 (2s) == sanityn test 84: 0-nlink race in lu_object_find() ===== 23:10:34 (1788837034) fail_loc=0x8000060b PASS 84 (11s) == sanityn test 85: Lustre API root cache race =========== 23:10:45 (1788837045) Starting: llapi_root_test /mnt/lustre/f85.sanityn /mnt/lustre2/f85.sanityn Starting test test1 at 1788837046 Finishing test test1 at 1788837050 PASS 85 (9s) == sanityn test 90: open/create and unlink striped directory ========================================================== 23:10:54 (1788837054) SKIP: sanityn test_90 needs >= 2 MDTs SKIP 90 (3s) == sanityn test 91: chmod and unlink striped directory === 23:10:58 (1788837058) SKIP: sanityn test_91 needs >= 2 MDTs SKIP 91 (4s) == sanityn test 92: create remote directory under orphan directory ========================================================== 23:11:01 (1788837061) SKIP: sanityn test_92 needs >= 2 MDTs SKIP 92 (3s) == sanityn test 93: alloc_rr should not allocate on same ost ========================================================== 23:11:04 (1788837064) fail_loc=0x00000163 fail_loc=0x0 /mnt/lustre/f93.sanityn-1/file1 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 5891 0x1703 0x280000400 0 5975 0x1757 0x240000400 /mnt/lustre2/f93.sanityn-2/file2 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5976 0x1758 0x240000400 1 5892 0x1704 0x280000400 rc1=0 and rc2=0 PASS 93 (17s) == sanityn test 94: signal vs CP callback race =========== 23:11:21 (1788837081) osc.lustre-OST0000-osc-ffff9f8d82fb1800.idle_timeout=0 osc.lustre-OST0000-osc-ffff9f8d866d9800.idle_timeout=0 osc.lustre-OST0001-osc-ffff9f8d82fb1800.idle_timeout=0 osc.lustre-OST0001-osc-ffff9f8d866d9800.idle_timeout=0 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.299988 s, 27.3 kB/s fail_val=5 fail_loc=0x80000312 fail_val=6 fail_loc=0x80000329 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 2.9057 s, 1.4 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 8000: 334160 Alarm clock dd if=/dev/zero of=$DIR/$tfile conv=notrunc oflag=append bs=4k count=1 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0969851 s, 42.2 kB/s osc.lustre-OST0000-osc-ffff9f8d82fb1800.idle_timeout=debug osc.lustre-OST0000-osc-ffff9f8d866d9800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9f8d82fb1800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9f8d866d9800.idle_timeout=debug PASS 94 (16s) == sanityn test 95a: Check readpage() on a page that was removed from page cache ========================================================== 23:11:38 (1788837098) llite.lustre-ffff9f8d82fb1800.fast_read=0 llite.lustre-ffff9f8d866d9800.fast_read=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.064846 s, 32.3 MB/s fail_loc=0x80001421 fail_val=7 fail_loc=0x80001422 fail_val=10 llite.lustre-ffff9f8d82fb1800.fast_read=1 llite.lustre-ffff9f8d866d9800.fast_read=1 PASS 95a (19s) == sanityn test 95b: Check readpage() on a page that is no longer uptodate ========================================================== 23:11:57 (1788837117) llite.lustre-ffff9f8d82fb1800.fast_read=0 llite.lustre-ffff9f8d866d9800.fast_read=0 1+0 records in 1+0 records out 12288 bytes (12 kB, 12 KiB) copied, 0.00919637 s, 1.3 MB/s fail_loc=0x80001424 fail_val=5 fail_loc=0 llite.lustre-ffff9f8d82fb1800.fast_read=1 llite.lustre-ffff9f8d866d9800.fast_read=1 PASS 95b (11s) == sanityn test 100a: DoM: glimpse RPCs for stat without IO lock (DoM only file) ========================================================== 23:12:08 (1788837128) SKIP: sanityn test_100a Reserved for glimpse-ahead SKIP 100a (3s) == sanityn test 100b: DoM: no glimpse RPC for stat with IO lock (DoM only file) ========================================================== 23:12:11 (1788837131) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00446767 s, 917 kB/s /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK PASS 100b (5s) == sanityn test 100c: DoM: write vs stat without IO lock (combined file) ========================================================== 23:12:16 (1788837136) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.164227 s, 12.8 MB/s /mnt/lustre/d100c.sanityn/dom has type file OK /mnt/lustre/d100c.sanityn/dom has size 2097152 OK PASS 100c (6s) == sanityn test 100d: DoM: write+truncate vs stat without IO lock (combined file) ========================================================== 23:12:22 (1788837142) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.131402 s, 16.0 MB/s /mnt/lustre/d100d.sanityn/dom has type file OK /mnt/lustre/d100d.sanityn/dom has size 4096 OK PASS 100d (6s) == sanityn test 100e: DoM: read on open and file size ==== 23:12:28 (1788837148) 1+0 records in 1+0 records out 12000 bytes (12 kB, 12 KiB) copied, 0.0206344 s, 582 kB/s PASS 100e (9s) == sanityn test 101a: Discard DoM data on unlink ========= 23:12:37 (1788837157) /mnt/lustre/f101a.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0049336 s, 830 kB/s PASS 101a (7s) == sanityn test 101b: Discard DoM data on rename ========= 23:12:45 (1788837165) /mnt/lustre/f101b.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0081299 s, 504 kB/s PASS 101b (6s) == sanityn test 101c: Discard DoM data on close-unlink === 23:12:52 (1788837172) /mnt/lustre/f101c.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00402135 s, 1.0 MB/s PASS 101c (6s) == sanityn test 102: Test open by handle of unlinked file ========================================================== 23:12:58 (1788837178) file: /mnt/lustre/f102.sanityn fh_bytes: 32 fh_type: 151 fh_data: a1 13 00 00 02 00 00 00 e5 0a 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 file's parent FID is [0:0x0:0x0] file FID is [0x2000013a1:0xae5:0x0] access via mount point '/mnt/lustre2' - OK access after unlink - OK check_fhandle_syscalls test Passed! PASS 102 (8s) == sanityn test 103: Test size correctness with lockahead ========================================================== 23:13:06 (1788837186) fail_loc=0x415 fail_loc=0x415 fail_val=2 Incorrect size expected (no glimpse fix): Starting test test23 at 1788837189 lockahead_test: lockahead_test.c:1171: test23: assertion 'sb.st_size == sb2.st_size' failed: size on /mnt/lustre/d103.sanityn/f103.sanityn and /mnt/lustre2/d103.sanityn/f103.sanityn differs: 0 vs 1048576 fail_loc=0 fail_loc=0x214 fail_val=2 Starting test test23 at 1788837193 Finishing test test23 at 1788837193 PASS 103 (15s) == sanityn test 104: Verify that MDS stores atime/mtime/ctime during close ========================================================== 23:13:21 (1788837201) SKIP: sanityn test_104 ldiskfs only test SKIP 104 (3s) == sanityn test 105: Glimpse and lock cancel race ======== 23:13:24 (1788837204) fail_loc=0x416 PASS 105 (22s) == sanityn test 106a: Verify the btime via statx() ======= 23:13:46 (1788837226) SKIP: sanityn test_106a Test only for ldiskfs and statx() supported SKIP 106a (3s) == sanityn test 106b: Glimpse RPCs test for statx ======== 23:13:49 (1788837229) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.012996 s, 78.8 kB/s File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115272381369068 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-07 23:13:51.000000000 -0400 Modify: 2026-09-07 23:13:51.000000000 -0400 Change: 2026-09-07 23:13:51.000000000 -0400 Birth: 2026-09-07 23:13:52.000000000 -0400 /mnt/lustre/f106b.sanityn 144115272381369068 -rw-r--r-- 1 0 0 1788837232 1788837231 1788837231 File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115272381369068 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-07 23:13:51.000000000 -0400 Modify: 2026-09-07 23:13:51.000000000 -0400 Change: 2026-09-07 23:13:51.000000000 -0400 Birth: 2026-09-07 23:13:52.000000000 -0400 1788837231 1024 1 PASS 106b (7s) == sanityn test 106c: Verify statx attributes mask ======= 23:13:56 (1788837236) PASS 106c (7s) == sanityn test 107a: Basic grouplock conflict =========== 23:14:03 (1788837243) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.320648 s, 32.7 MB/s /mnt/lustre/f107a.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5985 0x1761 0x240000400 multiop /mnt/lustre/f107a.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7629 multiop /mnt/lustre2/f107a.sanityn vO_G16022000r10g16022000c TMPPIPE=/tmp/multiop_open_wait_pipe.7629 First grouplock blocks second one PASS 107a (9s) == sanityn test 107b: Grouplock is added to the head of waiting list ========================================================== 23:14:12 (1788837252) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.221592 s, 47.3 MB/s /mnt/lustre/f107b.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 5899 0x170b 0x280000400 multiop /mnt/lustre/f107b.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7629 Grouplock blocks IO First grouplock blocks second one Second grouplock blocks IO PASS 107b (13s) == sanityn test 108a: lseek: parallel updates ============ 23:14:25 (1788837265) 1+0 records in 1+0 records out 10000 bytes (10 kB, 9.8 KiB) copied, 0.00993554 s, 1.0 MB/s llite.lustre-ffff9f8d82fb1800.hybrid_io=0 llite.lustre-ffff9f8d866d9800.hybrid_io=0 fail_val=4 fail_loc=0x80000414 1+0 records in 1+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 4.4399 s, 1.9 MB/s PASS 108a (11s) == sanityn test 109: Race with several mount instances on 1 node ========================================================== 23:14:36 (1788837276) 192.168.203.151@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg351-client.virtnet /mnt/lustre (opts:) 192.168.203.151@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg351-client.virtnet /mnt/lustre2 (opts:) Starting race between client mounts (duration 30) Iteration 0 Starting client: oleg351-client.virtnet: -o user_xattr,flock 192.168.203.151@tcp:/lustre /mnt/lustre Starting client: oleg351-client.virtnet: -o user_xattr,flock 192.168.203.151@tcp:/lustre /mnt/lustre2 192.168.203.151@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.151@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg351-client.virtnet /mnt/lustre (opts:) Stopping client oleg351-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg351-server' oleg351-server: oleg351-server.virtnet: executing load_modules_local oleg351-server: kmemleak disabled oleg351-server: Loading modules from /home/green/git/lustre-release/lustre oleg351-server: detected 4 online CPUs by sysfs oleg351-server: MODOPTS_LIBCFS= oleg351-server: Force libcfs to create 2 CPU partitions oleg351-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Iteration 1 Starting client: oleg351-client.virtnet: -o user_xattr,flock 192.168.203.151@tcp:/lustre /mnt/lustre2 Starting client: oleg351-client.virtnet: -o user_xattr,flock 192.168.203.151@tcp:/lustre /mnt/lustre 192.168.203.151@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.151@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg351-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg351-client.virtnet /mnt/lustre (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg351-server' oleg351-server: oleg351-server.virtnet: executing load_modules_local oleg351-server: kmemleak disabled oleg351-server: Loading modules from /home/green/git/lustre-release/lustre oleg351-server: detected 4 online CPUs by sysfs oleg351-server: MODOPTS_LIBCFS= oleg351-server: Force libcfs to create 2 CPU partitions oleg351-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Starting client: oleg351-client.virtnet: -o user_xattr,flock 192.168.203.151@tcp:/lustre /mnt/lustre Starting client: oleg351-client.virtnet: -o user_xattr,flock 192.168.203.151@tcp:/lustre /mnt/lustre2 PASS 109 (62s) == sanityn test 110: do not grant another lock on resend ========================================================== 23:15:38 (1788837338) fail_loc=0x525 fail_val=3 fail_loc=0x534 File: /mnt/lustre/f110.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115272414920707 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-07 23:15:40.000000000 -0400 Modify: 2026-09-07 23:15:40.000000000 -0400 Change: 2026-09-07 23:15:40.000000000 -0400 Birth: 2026-09-07 23:15:41.000000000 -0400 fail_loc=0 fail_val=0 Connected clients: oleg351-client.virtnet oleg351-client.virtnet PASS 110 (167s) == sanityn test 111: A racy rename/link an open file should not cause fs corruption ========================================================== 23:18:26 (1788837506) SKIP: sanityn test_111 needs >= 2 MDTs SKIP 111 (3s) == sanityn test 112: update max-inherit in default LMV === 23:18:29 (1788837509) SKIP: sanityn test_112 We need at least 2 MDTs for this test SKIP 112 (4s) == sanityn test 113: check servers of specified fs ======= 23:18:33 (1788837513) lustre-OST0000-osc-ffff9f8d8e7a6800 active. lustre-OST0001-osc-ffff9f8d8e7a6800 active. lustre-OST0000-osc-ffff9f8d8e7a6800 active. lustre-OST0001-osc-ffff9f8d8e7a6800 active. MGC192.168.203.151@tcp active. PASS 113 (8s) == sanityn test 114: implicit default LMV inherit ======== 23:18:41 (1788837521) SKIP: sanityn test_114 We need at least 2 MDTs for this test SKIP 114 (4s) == sanityn test 115: ldiskfs doesn't check direntry for uniqueness ========================================================== 23:18:45 (1788837525) SKIP: sanityn test_115 ldiskfs only test SKIP 115 (3s) == sanityn test 116: DNE: Set default LMV layout from a remote client ========================================================== 23:18:49 (1788837529) SKIP: sanityn test_116 needs >= 2 MDTs SKIP 116 (3s) == sanityn test 121: trunc append race =================== 23:18:52 (1788837532) 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0449989 s, 455 kB/s fail_loc=0x80001436 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.113332 s, 181 kB/s 0000000 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0005000 0000 0000 0000 0000 0000 0000 0000 0000 * 000c800 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0011800 PASS 121 (10s) == sanityn test 122: directory size is consistent across mounts ========================================================== 23:19:02 (1788837542) PASS 122 (11s) == sanityn test 200: service remains healthy while able to process request ========================================================== 23:19:13 (1788837553) at_max=10 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=12 fail_loc=0x1e1 fail_val=20 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1068 s, 0.2 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.2019 s, 0.2 kB/s 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1851 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1835 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1527 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1121 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.129 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1773 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.2256 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.0395 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.0382 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 39.988 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.0102 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.0169 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.036 s, 0.1 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1741 s, 0.1 kB/s 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.0494 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.2105 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.1032 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2283 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.1737 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.1065 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2229 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2322 s, 0.1 kB/s at_max=600 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=116 oleg351-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9f8d8e7a6800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9f8d8e7a6800.ost_server_uuid in FULL state after 0 sec PASS 200 (83s) cleanup: ====================================================== == sanityn test complete, duration 14225 sec ============= 23:20:37 (1788837637) === sanityn: start cleanup 23:20:39 (1788837639) === Stopping clients: oleg351-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg351-client.virtnet /mnt/lustre2 opts: === sanityn: finish cleanup 23:24:51 (1788837891) ===