-----============= acceptance-small: sanityn ============----- Tue Sep 15 00:55:18 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg360-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanityn.*ex': No such file or directory excepting tests: 27 40a skipping tests SLOW=no: 33a === sanityn: start setup 00:55:39 (1789448139) === Starting client oleg360-client.virtnet: -o user_xattr,flock 192.168.203.160@tcp:/lustre /mnt/lustre2 Mount client oleg360-client.virtnet: mount -t lustre -o user_xattr,flock 192.168.203.160@tcp:/lustre /mnt/lustre2 Started clients oleg360-client.virtnet: 192.168.203.160@tcp:/lustre on /mnt/lustre2 type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) oleg360-client.virtnet: executing check_config_client /mnt/lustre oleg360-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg360-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9837139e5800.idle_timeout=debug osc.lustre-OST0000-osc-ffff9837162cc800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9837139e5800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9837162cc800.idle_timeout=debug disable quota as required oleg360-server: oleg360-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanityn: finish setup 00:56:16 (1789448176) === 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.052769 s, 19.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7627] == sanityn test 0: do_node_vp() and do_facet_vp() do the right thing ========================================================== 00:56:19 (1789448179) PASS 0 (8s) == sanityn test 1: Check attribute updates on 2 mount points ========================================================== 00:56:27 (1789448187) /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0777 OK /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0666 OK /mnt/lustre/f1.sanityn: absent OK PASS 1 (7s) == sanityn test 2a: check cached attribute updates on 2 mtpt's ================================================================== 00:56:35 (1789448195) -rw-r--r-- 1 root root 0 Sep 15 00:56 /mnt/lustre2/f2a /mnt/lustre/f2a has type file OK /mnt/lustre/f2a has perms 0777 OK PASS 2a (8s) == sanityn test 2b: check cached attribute updates on 2 mtpt's ================================================================== 00:56:42 (1789448202) -rw-r--r-- 1 root root 0 Sep 15 00:56 /mnt/lustre2/f2b /mnt/lustre2/f2b has type file OK /mnt/lustre2/f2b has perms 0777 OK PASS 2b (7s) == sanityn test 2c: check cached attribute updates on 2 mtpt's root ============================================================= 00:56:49 (1789448209) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0777 OK PASS 2c (6s) == sanityn test 2d: check cached attribute updates on 2 mtpt's root ============================================================= 00:56:55 (1789448215) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0755 OK PASS 2d (8s) == sanityn test 2e: check chmod on root is propagated to others ========================================================== 00:57:03 (1789448223) total 1 -rwxrwxrwx 1 root root 0 Sep 15 00:56 f2a -rwxrwxrwx 1 root root 0 Sep 15 00:56 f2b total 1 -rwxrwxrwx 1 root root 0 Sep 15 00:56 f2a -rwxrwxrwx 1 root root 0 Sep 15 00:56 f2b running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre2/f2e.sanityn] [count=1] 1+0 records in 1+0 records out 512 bytes copied, 0.0251338 s, 20.4 kB/s PASS 2e (7s) == sanityn test 2f: check attr/owner updates on DNE with 2 mtpt's ========================================================== 00:57:10 (1789448230) SKIP: sanityn test_2f needs >= 2 MDTs SKIP 2f (3s) == sanityn test 2g: check blocks update on sync write ==== 00:57:13 (1789448233) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.420541 s, 5.0 MB/s /mnt/lustre/f2g.sanityn has 4103 blocks /mnt/lustre2/f2g.sanityn has 4103 blocks PASS 2g (6s) == sanityn test 3: symlink on one mtpt, readlink on another ===================================================================== 00:57:20 (1789448240) PASS 3 (7s) == sanityn test 4: fstat validation on multiple mount points ==================================================================== 00:57:26 (1789448246) PASS 4 (7s) == sanityn test 5: create a file on one mount, truncate it on the other ========================================================== 00:57:33 (1789448253) /mnt/lustre/f5 has type file OK /mnt/lustre/f5 has size 100 OK PASS 5 (7s) == sanityn test 6: remove of open file on other node ============================================================================ 00:57:40 (1789448260) opening writing unlinking /mnt/lustre2/f6.sanityn accessing (1) seeking (1) accessing (2) fstat... reading comparing data truncating seeking (2) writing again seeking (3) reading again comparing data again closing SUCCESS - goto beer PASS 6 (7s) == sanityn test 7: remove of open directory on other node ======================================================================= 00:57:47 (1789448267) creating directory /mnt/lustre/d7 opening directory unlinking /mnt/lustre/d7 Ok, everything goes well. PASS 7 (6s) == sanityn test 8: remove of open special file on other node ==================================================================== 00:57:53 (1789448273) creating special file /mnt/lustre/f8.sanityn opening file unlinking /mnt/lustre/f8.sanityn Ok, everything goes well. PASS 8 (7s) == sanityn test 9a: append of file with sub-page size on multiple mounts ========================================================== 00:58:00 (1789448280) PASS 9a (7s) == sanityn test 9b: append to striped sparse file ======== 00:58:07 (1789448287) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0670099 s, 15.6 MB/s 3+0 records in 3+0 records out 3 bytes copied, 0.0867859 s, 0.0 kB/s Data read (expecting 'foo'): foo PASS 9b (7s) == sanityn test 10a: write of file with sub-page size on multiple mounts ========================================================== 00:58:14 (1789448294) 1+0 records in 1+0 records out 1 byte copied, 0.00550038 s, 0.2 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0104949 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0203845 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0108868 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00863773 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0103141 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00885669 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00788787 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0164026 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0101684 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00896606 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0119975 s, 0.1 kB/s PASS 10a (8s) == sanityn test 10b: write of file with sub-page size on multiple mounts ========================================================== 00:58:22 (1789448302) 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.0163195 s, 188 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0523909 s, 78.2 kB/s 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.000987415 s, 3.1 MB/s PASS 10b (7s) == sanityn test 11: execution of file opened for write should return error ============================================================== 00:58:29 (1789448309) multiop /mnt/lustre/d11/f vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7627 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 298: /mnt/lustre2/d11/f: Text file busy PASS 11 (6s) == sanityn test 12: test lock ordering (link, stat, unlink) ========================================================== 00:58:36 (1789448316) warning: '-runtime' deprecated, use '-t runtime' instead - link 313 (time 1789448328.32 total 10.02 last 31.24) - link 613 (time 1789448338.34 total 20.03 last 29.96) - link 945 (time 1789448348.34 total 30.04 last 33.18) - link 1273 (time 1789448358.36 total 40.06 last 32.73) - link 1635 (time 1789448368.39 total 50.08 last 36.12) - link 1971 (time 1789448378.39 total 60.09 last 33.58) - link 2290 (time 1789448388.41 total 70.10 last 31.85) - link 2602 (time 1789448398.44 total 80.14 last 31.10) - link 2928 (time 1789448408.47 total 90.17 last 32.50) total: 3274 link in 100.01 seconds: 32.74 ops/second using seed 3507624239 running for 100 seconds - stat 4734 (time 1789448330 ; total 11 ; last 11) - stat 10000 (time 1789448340 ; total 21 ; last 10) - stat 16067 (time 1789448351 ; total 32 ; last 11) - stat 20000 (time 1789448358 ; total 39 ; last 7) - stat 25627 (time 1789448369 ; total 50 ; last 11) - stat 28397 (time 1789448380 ; total 61 ; last 11) - stat 30000 (time 1789448385 ; total 66 ; last 5) - stat 32015 (time 1789448396 ; total 77 ; last 11) - stat 34864 (time 1789448407 ; total 88 ; last 11) - stat 37610 (time 1789448418 ; total 99 ; last 11) total: 38031 stats in 100 seconds: 380.309998 stats/second - unlinked 0 (time 1789448369 ; total 0 ; last 0) unlink(/mnt/lustre2/lockdir/lockfile3274) error: No such file or directory total: 3273 unlinks in 88 seconds: 37.193180 unlinks/second /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 77: kill: (21430) - No such process /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 78: kill: (21432) - No such process PASS 12 (148s) == sanityn test 13: test directory page revocation ======= 01:01:03 (1789448463) f13 f13 PASS 13 (9s) == sanityn test 14aa: execution of file open for write returns -ETXTBSY ========================================================== 01:01:12 (1789448472) multiop /mnt/lustre/d14aa.sanityn/f14aa.sanityn vOw_c TMPPIPE=/tmp/multiop_open_wait_pipe.7627 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 333: /mnt/lustre2/d14aa.sanityn/f14aa.sanityn: Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5116: 22629 Terminated $MULTIOP_PROG $FILE v$ARGS > $TMPPIPE (wd: ~) PASS 14aa (7s) == sanityn test 14ab: open(RDWR) of executing file returns -ETXTBSY ========================================================== 01:01:19 (1789448479) open(O_RDWR|O_CREAT): Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5116: 23208 Terminated $DIR1/$tdir/sleep 60 (wd: ~) PASS 14ab (8s) == sanityn test 14b: truncate of executing file returns -ETXTBSY ================================================================ 01:01:28 (1789448488) truncate: cannot truncate '/mnt/lustre2/d14b.sanityn/sleep' to length 60: Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 348: 23786 Terminated $DIR1/$tdir/sleep 60 PASS 14b (8s) == sanityn test 14c: open(O_TRUNC) of executing file return -ETXTBSY ============================================================ 01:01:35 (1789448495) cp: cannot create regular file '/mnt/lustre2/d14c.sanityn/sleep': Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 360: 24368 Terminated $DIR1/$tdir/sleep 60 PASS 14c (7s) == sanityn test 14d: chmod of executing file is still possible ================================================================== 01:01:42 (1789448502) chmod /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 371: 24950 Terminated $DIR1/$tdir/sleep 60 PASS 14d (9s) == sanityn test 15: test out-of-space with multiple writers ===================================================================== 01:01:51 (1789448511) PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin: MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky sleep 5 for ZFS MDS Waiting for MDT destroys to complete fallocate on zfs doesn't consume space STRIPECOUNT=2 ORIGFREE=7531520 MAXFREE=800000 SKIP: oos2 test_15 oos2.sh: 7531520KB free > 800000KB, increase MAXFREE (or reduce fs size) sleep 5 for ZFS MDS Waiting for MDT destroys to complete SKIP 15 (51s) == sanityn test 16a: 500 iterations of dual-mount fsx ==== 01:02:42 (1789448562) keep default fallocate mode: 2 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.529791 s, 15.8 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.463547 s, 18.1 MB/s Chance of close/open is 1 in 50 Seed set to 8567 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported truncating to largest ever: 0x4d5ee7 000003[0] 1789448567.674601 trunc from 00000000 to 0x4d5ee6 (0x4d5ee7 bytes) truncating to largest ever: 0x663c14 000005[0] 1789448567.767629 trunc from 0x4d5ee7 to 0x663c13 (0x18dd2d bytes) 000006[0] 1789448567.793577 mapwrite 0x3fbab6 thru 0x407b2d (0xc078 bytes) 000007[1] 1789448567.983249 mapwrite 0x86fb2a thru 0x87968c (0x9b63 bytes) 000008[1] 1789448568.292653 read 0x11340c thru 0x11d139 (0x9d2e bytes) 000010[1] 1789448568.332513 punch from 0x6fe225 to 0x7016ec (0x34c8 bytes) 000011[1] 1789448568.355101 punch from 0x6dd6c2 to 0x6eb239 (0xdb78 bytes) 000012[1] 1789448568.380083 mapwrite 0x464f6f thru 0x472467 (0xd4f9 bytes) 000015[0] 1789448568.574153 mapread 0x127f89 thru 0x1370a9 (0xf121 bytes) 000016[1] 1789448568.659983 mapwrite 0x231bbc thru 0x23eb1b (0xcf60 bytes) 000018[0] 1789448568.940882 mapwrite 0x636a25 thru 0x639a2a (0x3006 bytes) 000019[0] 1789448569.060701 mapwrite 0x24ca05 thru 0x24f0d2 (0x26ce bytes) 000020[0] 1789448569.174337 trunc from 0x87968d to 0x1d8e18 (0x6a0874 bytes) 000022[0] 1789448569.257712 punch from 0x16dc04 to 0x177d96 (0xa193 bytes) truncating to largest ever: 0x9011bc 000023[1] 1789448569.287165 trunc from 0x1d8e19 to 0x9011bb (0x7283a3 bytes) 000024[0] 1789448569.337164 mapwrite 0x4657c6 thru 0x4742dd (0xeb18 bytes) 000027[1] 1789448569.516916 mapread 0x79a50d thru 0x7a04e2 (0x5fd6 bytes) 000028[1] 1789448569.567766 write 0x2739b2 thru 0x273b1c (0x16b bytes) 000030[1] 1789448569.599398 mapwrite 0x198866 thru 0x1a3ce0 (0xb47b bytes) 000031[1] 1789448569.758512 mapread 0x51b939 thru 0x52abab (0xf273 bytes) 000032[0] 1789448569.813077 mapread 0x1873df thru 0x1917f7 (0xa419 bytes) 000033[1] 1789448569.873362 punch from 0x28b998 to 0x29510d (0x9776 bytes) 000034[0] 1789448569.899858 write 0x1b8f44 thru 0x1bdfbf (0x507c bytes) 000035[1] 1789448569.969684 punch from 0x8b7eab to 0x8c56ac (0xd802 bytes) 000041[1] 1789448570.084245 mapwrite 0x78a75e thru 0x7905d7 (0x5e7a bytes) 000044[0] 1789448570.255509 mapread 0x0fa0be thru 0x0fdb6d (0x3ab0 bytes) 000047[1] 1789448570.328747 write 0x1430c3 thru 0x149db6 (0x6cf4 bytes) 000048[1] 1789448570.362391 mapread 0x7bc5ab thru 0x7c69d8 (0xa42e bytes) 000050[0] 1789448570.437971 write 0x9a1261 thru 0x9a8498 (0x7238 bytes) 000051[0] 1789448570.477721 punch from 0x8f2383 to 0x8f6aee (0x476c bytes) 000053[1] 1789448570.533463 read 0x744845 thru 0x7530d5 (0xe891 bytes) 000054[0] 1789448570.577015 trunc from 0x9a8499 to 0x777902 (0x230b96 bytes) 000055[0] 1789448570.622536 read 0x2758f4 thru 0x27b396 (0x5aa3 bytes) 000056[0] 1789448570.662418 read 0x7719e5 thru 0x777902 (0x5f1e bytes) 000057[1] 1789448570.696772 read 0x02f597 thru 0x0368cb (0x7335 bytes) 000058[1] 1789448570.739304 read 0x554606 thru 0x558000 (0x39fb bytes) 000062[1] 1789448570.771149 read 0x566d7e thru 0x570445 (0x96c8 bytes) 000063[1] 1789448570.779436 mapwrite 0x0ac8ea thru 0x0b3270 (0x6987 bytes) 000067[0] 1789448571.002880 mapwrite 0x8dba2c thru 0x8e6607 (0xabdc bytes) 000068[1] 1789448571.184888 read 0x58072d thru 0x58a264 (0x9b38 bytes) 000069[0] 1789448571.219721 punch from 0x7b1c7d to 0x7b6f21 (0x52a5 bytes) 000070[1] 1789448571.245077 write 0x94e33b thru 0x953c8e (0x5954 bytes) 000071[1] 1789448571.302083 read 0x26a086 thru 0x274e98 (0xae13 bytes) 000073[0] 1789448571.357253 write 0x905a09 thru 0x90cdb2 (0x73aa bytes) 000074[0] 1789448571.401409 read 0x4abfe6 thru 0x4ade5d (0x1e78 bytes) 000075[0] 1789448571.430024 write 0x5799fa thru 0x58865a (0xec61 bytes) 000077[0] 1789448571.463106 read 0x90abbf thru 0x919a58 (0xee9a bytes) 000078[0] 1789448571.489139 trunc from 0x953c8f to 0x533cb8 (0x41ffd6 bytes) 000082[1] 1789448571.564984 punch from 0x1d40bf to 0x1d7898 (0x37da bytes) 000083[0] 1789448571.589718 write 0x14fb61 thru 0x15c99c (0xce3c bytes) 000084[0] 1789448571.647196 trunc from 0x533cb9 to 0x5c1595 (0x8d8dd bytes) 000085[0] 1789448571.666628 mapread 0x0f0285 thru 0x0f8258 (0x7fd4 bytes) 000086[1] 1789448571.712346 mapread 0x2e2f47 thru 0x2e4af7 (0x1bb1 bytes) 000087[1] 1789448571.774068 write 0x6f456a thru 0x6f547c (0xf13 bytes) 000088[1] 1789448571.803316 read 0x63b749 thru 0x641d90 (0x6648 bytes) 000089[0] 1789448571.833337 write 0x99856e thru 0x9a2682 (0xa115 bytes) 000091[0] 1789448571.917377 mapwrite 0x7574e2 thru 0x759903 (0x2422 bytes) 000092[0] 1789448572.022731 trunc from 0x9a2683 to 0x0be24c (0x8e4436 bytes) 000097[0] 1789448572.085315 read 0x034fb2 thru 0x03b85a (0x68a9 bytes) 000099[1] 1789448572.160233 read 0x056ebf thru 0x0654a7 (0xe5e9 bytes) 000100[0] 1789448572.218700 trunc from 0x0be24d to 0x5cee5b (0x510c0f bytes) 000105[1] 1789448572.342538 read 0x09b126 thru 0x09f034 (0x3f0f bytes) 000111[0] 1789448572.455488 mapwrite 0x01f448 thru 0x02054a (0x1103 bytes) 000113[0] 1789448572.682955 mapwrite 0x73a40e thru 0x7493d7 (0xefca bytes) 000114[0] 1789448572.872492 trunc from 0x7493d8 to 0x6cfdf5 (0x795e2 bytes) 000116[1] 1789448572.895884 punch from 0x2fd1ed to 0x309200 (0xc014 bytes) 000118[1] 1789448573.030543 write 0x8fb706 thru 0x9026bf (0x6fba bytes) 000119[0] 1789448573.062503 read 0x895bf6 thru 0x8a07f9 (0xac04 bytes) 000120[0] 1789448573.134225 write 0x64ac01 thru 0x64fa49 (0x4e49 bytes) 000121[1] 1789448573.171379 write 0x06fc20 thru 0x071835 (0x1c16 bytes) 000122[0] 1789448573.241811 mapread 0x8021a1 thru 0x802af8 (0x958 bytes) 000123[0] 1789448573.304520 trunc from 0x9026c0 to 0x50533a (0x3fd385 bytes) 000125[0] 1789448573.415991 mapread 0x192131 thru 0x195618 (0x34e8 bytes) 000126[0] 1789448573.504682 mapwrite 0x2e84d9 thru 0x2ed74f (0x5277 bytes) 000127[0] 1789448573.621264 write 0x576589 thru 0x578ba3 (0x261b bytes) 000129[0] 1789448573.667424 trunc from 0x578ba4 to 0x467ec5 (0x110cde bytes) 000131[0] 1789448573.699689 read 0x2c27fd thru 0x2c79d7 (0x51db bytes) 000132[0] 1789448573.707383 read 0x23fb25 thru 0x24d661 (0xdb3d bytes) 000133[1] 1789448573.745356 write 0x570980 thru 0x57ac5e (0xa2df bytes) 000137[1] 1789448573.834875 punch from 0x3ace01 to 0x3afafe (0x2cfe bytes) 000138[1] 1789448573.897889 punch from 0x0a7fd4 to 0x0aaa59 (0x2a86 bytes) truncating to largest ever: 0x913f21 000139[0] 1789448573.923568 trunc from 0x57ac5f to 0x913f20 (0x3992c2 bytes) 000140[1] 1789448574.043721 mapwrite 0x08b4a8 thru 0x0993ba (0xdf13 bytes) 000141[1] 1789448574.331349 read 0x305e80 thru 0x30f3fb (0x957c bytes) 000142[1] 1789448574.409764 mapread 0x821db0 thru 0x82b3cb (0x961c bytes) 000143[0] 1789448574.480385 mapread 0x30b014 thru 0x316240 (0xb22d bytes) 000144[1] 1789448574.572829 write 0x686d7e thru 0x696d4f (0xffd2 bytes) 000147[0] 1789448574.673352 trunc from 0x913f21 to 0x3cb59a (0x548986 bytes) 000150[1] 1789448574.831059 mapread 0x1f4353 thru 0x1fa64f (0x62fd bytes) 000151[0] 1789448574.982892 mapread 0x220b88 thru 0x226e92 (0x630b bytes) truncating to largest ever: 0x968c4b 000153[1] 1789448575.059554 trunc from 0x3cb59b to 0x968c4a (0x59d6b0 bytes) 000154[1] 1789448575.198606 mapread 0x373b2d thru 0x38284b (0xed1f bytes) 000157[0] 1789448575.311791 trunc from 0x968c4b to 0x6f0bb0 (0x27809a bytes) 000161[1] 1789448575.389603 mapwrite 0x3187c8 thru 0x327347 (0xeb80 bytes) 000163[1] 1789448575.499308 read 0x460d4d thru 0x46c4fb (0xb7af bytes) 000166[0] 1789448575.554279 mapwrite 0x2886b0 thru 0x288c00 (0x551 bytes) 000168[1] 1789448575.626702 punch from 0x32ef2a to 0x33a236 (0xb30d bytes) 000170[0] 1789448575.645642 write 0x1dcfde thru 0x1e2798 (0x57bb bytes) 000174[1] 1789448575.689986 trunc from 0x6f0bb1 to 0x039917 (0x6b7299 bytes) 000177[1] 1789448575.795652 punch from 0x00a11f to 0x014e07 (0xace9 bytes) 000178[0] 1789448575.823048 punch from 0x0004db to 0x002c3d (0x2763 bytes) 000180[1] 1789448575.889442 mapwrite 0x24feaf thru 0x25f60a (0xf75c bytes) 000181[0] 1789448576.258086 trunc from 0x25f60b to 0x7335f9 (0x4d3fef bytes) 000182[0] 1789448576.368938 read 0x160bbb thru 0x16a783 (0x9bc9 bytes) 000183[1] 1789448576.390560 mapread 0x15ab9b thru 0x160d8a (0x61f0 bytes) 000185[1] 1789448576.467066 trunc from 0x7335fa to 0x72d5a0 (0x6059 bytes) 000186[1] 1789448576.550033 trunc from 0x72d5a1 to 0x879404 (0x14be64 bytes) 000187[0] 1789448576.612544 punch from 0x253656 to 0x258566 (0x4f11 bytes) 000190[1] 1789448576.688527 punch from 0x396d6e to 0x39b7dd (0x4a70 bytes) 000191[0] 1789448576.705439 read 0x5e4ae6 thru 0x5ea164 (0x567f bytes) 000192[1] 1789448576.747907 mapread 0x83ef01 thru 0x843812 (0x4912 bytes) 000196[0] 1789448576.882339 punch from 0x17f1a6 to 0x18b483 (0xc2de bytes) 000198[0] 1789448576.946934 mapwrite 0x74d283 thru 0x75035f (0x30dd bytes) 000199[0] 1789448577.093974 trunc from 0x879405 to 0x96078d (0xe7389 bytes) 000201[0] 1789448577.131673 mapwrite 0x592ffd thru 0x597eaf (0x4eb3 bytes) 000202[0] 1789448577.199547 trunc from 0x96078e to 0x387147 (0x5d9646 bytes) 000204[0] 1789448577.244252 punch from 0x21a65d to 0x229259 (0xebfd bytes) 000206[0] 1789448577.248116 mapread 0x14d38a thru 0x14e7ab (0x1422 bytes) 000207[0] 1789448577.259360 write 0x920cd4 thru 0x9228c8 (0x1bf5 bytes) 000209[1] 1789448577.278299 write 0x64b6ed thru 0x65a7c5 (0xf0d9 bytes) 000210[1] 1789448577.301568 trunc from 0x9228c9 to 0x2cf33a (0x65358e bytes) 000211[0] 1789448577.327470 read 0x26d693 thru 0x26e5ee (0xf5c bytes) 000212[1] 1789448577.369044 read 0x07b730 thru 0x08b5e9 (0xfeba bytes) 000213[1] 1789448577.402531 read 0x193762 thru 0x1a17d0 (0xe06f bytes) 000215[0] 1789448577.437727 write 0x815e4b thru 0x81cadf (0x6c95 bytes) 000216[0] 1789448577.496377 mapwrite 0x66acbc thru 0x66c64c (0x1991 bytes) 000218[0] 1789448577.675824 punch from 0x539be9 to 0x53e6ad (0x4ac5 bytes) 000219[1] 1789448577.684560 punch from 0x688da4 to 0x692fa6 (0xa203 bytes) 000220[1] 1789448577.714216 read 0x0c3763 thru 0x0d1d1d (0xe5bb bytes) 000221[0] 1789448577.742141 write 0x108133 thru 0x111791 (0x965f bytes) 000223[1] 1789448577.812989 trunc from 0x81cae0 to 0x2ff5d5 (0x51d50a bytes) 000224[0] 1789448577.932508 write 0x507997 thru 0x50f979 (0x7fe3 bytes) 000227[1] 1789448578.027512 trunc from 0x50f97a to 0x05f2b3 (0x4b06c6 bytes) 000229[0] 1789448578.191461 mapread 0x0319c7 thru 0x032b12 (0x114c bytes) 000230[0] 1789448578.290617 mapread 0x0156d5 thru 0x022b84 (0xd4b0 bytes) 000231[0] 1789448578.364091 read 0x01a478 thru 0x025cd1 (0xb85a bytes) 000232[0] 1789448578.399624 mapwrite 0x1be6ed thru 0x1c5c60 (0x7574 bytes) 000233[1] 1789448578.573288 punch from 0x03fba9 to 0x040b35 (0xf8d bytes) 000237[0] 1789448578.693266 mapwrite 0x459910 thru 0x45c689 (0x2d7a bytes) 000238[0] 1789448578.942094 punch from 0x08c2de to 0x092cb1 (0x69d4 bytes) 000243[0] 1789448579.000252 trunc from 0x45c68a to 0x73990b (0x2dd282 bytes) 000244[0] 1789448579.027144 punch from 0x43f50e to 0x441979 (0x246c bytes) 000245[0] 1789448579.033350 read 0x04fd2b thru 0x054c7c (0x4f52 bytes) 000246[0] 1789448579.071965 read 0x68b9b5 thru 0x698646 (0xcc92 bytes) 000252[0] 1789448579.085240 mapread 0x3c32a8 thru 0x3cf418 (0xc171 bytes) 000253[1] 1789448579.086826 trunc from 0x73990c to 0x9072e6 (0x1cd9db bytes) 000254[0] 1789448579.137242 read 0x8eb433 thru 0x8ebf9b (0xb69 bytes) 000255[0] 1789448579.169527 write 0x0f4fea thru 0x10084a (0xb861 bytes) 000256[1] 1789448579.206539 punch from 0x2536ac to 0x262d43 (0xf698 bytes) 000258[1] 1789448579.261685 mapread 0x4576a2 thru 0x45d838 (0x6197 bytes) 000259[1] 1789448579.307186 mapwrite 0x700879 thru 0x70dbfc (0xd384 bytes) 000264[0] 1789448579.425649 read 0x1b3e2d thru 0x1be1aa (0xa37e bytes) 000268[1] 1789448579.534323 trunc from 0x9072e7 to 0x638e81 (0x2ce465 bytes) 000269[1] 1789448579.572237 read 0x37240c thru 0x38015b (0xdd50 bytes) 000270[1] 1789448579.587029 mapread 0x261e39 thru 0x264e5d (0x3025 bytes) 000271[1] 1789448579.713083 mapwrite 0x570685 thru 0x57c5ce (0xbf4a bytes) 000273[1] 1789448579.902632 mapread 0x0eb12c thru 0x0ec5f2 (0x14c7 bytes) 000274[1] 1789448579.934207 mapread 0x1c137e thru 0x1cec9d (0xd920 bytes) 000275[0] 1789448579.994829 trunc from 0x638e82 to 0x7c9677 (0x1907f6 bytes) 000276[0] 1789448580.088922 trunc from 0x7c9678 to 0x70beb4 (0xbd7c3 bytes) 000278[1] 1789448580.122999 mapread 0x590dcd thru 0x5998b8 (0x8aec bytes) 000279[0] 1789448580.277572 write 0x6a6c0f thru 0x6ae0b1 (0x74a3 bytes) 000280[0] 1789448580.326637 mapread 0x0fd869 thru 0x108300 (0xaa98 bytes) 000281[1] 1789448580.387467 write 0x33bb70 thru 0x34864a (0xcadb bytes) 000282[1] 1789448580.422614 read 0x39d9ad thru 0x3a9754 (0xbda8 bytes) 000283[0] 1789448580.450936 trunc from 0x70beb5 to 0x8dfaef (0x1d3c3b bytes) 000284[0] 1789448580.588143 mapwrite 0x5f6f10 thru 0x60172d (0xa81e bytes) 000285[0] 1789448580.732914 trunc from 0x8dfaf0 to 0x46b0ae (0x474a41 bytes) 000286[0] 1789448580.754498 punch from 0x048fd5 to 0x057153 (0xe17f bytes) 000288[0] 1789448580.759577 read 0x07fa44 thru 0x08cf01 (0xd4be bytes) 000289[0] 1789448580.792334 mapread 0x0783b9 thru 0x078f54 (0xb9c bytes) 000292[1] 1789448580.824598 write 0x489a58 thru 0x495081 (0xb62a bytes) 000293[0] 1789448580.860747 punch from 0x0c6389 to 0x0c73f5 (0x106d bytes) 000294[1] 1789448580.879892 punch from 0x3df8b2 to 0x3e1770 (0x1ebf bytes) 000297[1] 1789448580.939706 trunc from 0x495082 to 0x5cf1bc (0x13a13b bytes) 000298[1] 1789448580.966051 punch from 0x134670 to 0x134794 (0x125 bytes) 000299[1] 1789448580.997334 punch from 0x34c33f to 0x357191 (0xae53 bytes) 000300[1] 1789448581.020896 read 0x1e7b44 thru 0x1e865a (0xb17 bytes) 000304[1] 1789448581.027946 read 0x5a7548 thru 0x5af404 (0x7ebd bytes) 000305[1] 1789448581.037018 mapread 0x1f3bf0 thru 0x1f3df3 (0x204 bytes) 000306[0] 1789448581.043015 trunc from 0x5cf1bd to 0x2fec1e (0x2d059e bytes) 000307[1] 1789448581.118577 write 0x480c37 thru 0x48bb16 (0xaee0 bytes) 000308[0] 1789448581.194107 punch from 0x45b369 to 0x45dba7 (0x283f bytes) 000312[0] 1789448581.309618 trunc from 0x48bb17 to 0x7203fb (0x2948e5 bytes) 000314[1] 1789448581.390948 mapread 0x036020 thru 0x036efb (0xedc bytes) 000315[1] 1789448581.463884 punch from 0x13664f to 0x13fb98 (0x954a bytes) 000317[0] 1789448581.532948 read 0x6376b2 thru 0x6473d5 (0xfd24 bytes) 000319[1] 1789448581.591107 read 0x4d9837 thru 0x4e48e0 (0xb0aa bytes) 000320[0] 1789448581.628637 trunc from 0x7203fc to 0x77e515 (0x5e11a bytes) 000321[1] 1789448581.751254 punch from 0x324dc5 to 0x3285f5 (0x3831 bytes) 000322[0] 1789448581.831954 punch from 0x078073 to 0x07d4e9 (0x5477 bytes) 000323[0] 1789448581.885692 read 0x496457 thru 0x49f92c (0x94d6 bytes) 000324[1] 1789448581.919886 mapwrite 0x5080a3 thru 0x50f7af (0x770d bytes) 000325[1] 1789448582.077928 write 0x876909 thru 0x876cb4 (0x3ac bytes) 000327[1] 1789448582.106653 mapread 0x05e847 thru 0x061670 (0x2e2a bytes) 000328[1] 1789448582.155037 mapwrite 0x860007 thru 0x862451 (0x244b bytes) 000329[0] 1789448582.278566 mapread 0x106d77 thru 0x111ec6 (0xb150 bytes) 000330[1] 1789448582.419412 mapread 0x58077c thru 0x58612a (0x59af bytes) 000332[1] 1789448582.501569 mapread 0x43e526 thru 0x44e08f (0xfb6a bytes) 000335[1] 1789448582.574067 mapwrite 0x06f49d thru 0x07ccb0 (0xd814 bytes) 000336[1] 1789448582.752153 mapwrite 0x48a48a thru 0x48c499 (0x2010 bytes) 000338[1] 1789448582.871078 punch from 0x12460c to 0x12bdfa (0x77ef bytes) 000339[0] 1789448582.899262 mapwrite 0x50d275 thru 0x513c48 (0x69d4 bytes) 000341[1] 1789448583.079802 read 0x7dfe61 thru 0x7e0747 (0x8e7 bytes) 000342[0] 1789448583.117320 read 0x2c52e8 thru 0x2cb86f (0x6588 bytes) 000343[0] 1789448583.145538 trunc from 0x876cb5 to 0x73ee1a (0x137e9a bytes) 000345[1] 1789448583.257126 write 0x04483a thru 0x04e6d8 (0x9e9f bytes) 000346[1] 1789448583.345045 trunc from 0x73ee1b to 0x73dffc (0xe1e bytes) 000348[1] 1789448583.438278 write 0x21ef80 thru 0x229032 (0xa0b3 bytes) 000349[1] 1789448583.487056 trunc from 0x73dffd to 0x314f33 (0x4290c9 bytes) 000350[0] 1789448583.536522 mapwrite 0x980da0 thru 0x986fa4 (0x6205 bytes) 000351[0] 1789448583.936100 trunc from 0x986fa5 to 0x54f6ee (0x4378b6 bytes) 000352[1] 1789448583.972737 mapread 0x531b1c thru 0x53c430 (0xa915 bytes) 000353[1] 1789448584.128642 mapread 0x48b2fb thru 0x49a7fc (0xf502 bytes) 000356[0] 1789448584.237779 punch from 0x4a981d to 0x4b7184 (0xd968 bytes) 000358[1] 1789448584.296201 trunc from 0x54f6ef to 0x6b7d1e (0x168630 bytes) 000361[1] 1789448584.393178 read 0x4a18bc thru 0x4b0aa5 (0xf1ea bytes) 000362[1] 1789448584.434932 trunc from 0x6b7d1f to 0x630f95 (0x86d89 bytes) 000364[1] 1789448584.480153 mapwrite 0x0bb1db thru 0x0c6bf1 (0xba17 bytes) 000368[0] 1789448584.643714 mapread 0x37a347 thru 0x386026 (0xbce0 bytes) 000369[0] 1789448584.768628 write 0x481072 thru 0x48c06b (0xaffa bytes) 000371[1] 1789448584.901959 punch from 0x2449fc to 0x24dc88 (0x928d bytes) 000372[0] 1789448584.955036 trunc from 0x630f96 to 0x70fcc3 (0xded2e bytes) 000375[1] 1789448585.047366 mapwrite 0x95199d thru 0x957f80 (0x65e4 bytes) 000377[1] 1789448585.161349 punch from 0x41eea6 to 0x427b54 (0x8caf bytes) 000378[1] 1789448585.171566 read 0x3feb2a thru 0x406461 (0x7938 bytes) 000379[1] 1789448585.232312 mapwrite 0x053bbb thru 0x060a08 (0xce4e bytes) 000384[0] 1789448585.355347 write 0x036571 thru 0x03670f (0x19f bytes) 000385[0] 1789448585.399623 trunc from 0x957f81 to 0x8086f5 (0x14f88b bytes) 000386[0] 1789448585.498263 mapwrite 0x048f35 thru 0x051562 (0x862e bytes) 000388[1] 1789448585.632724 mapread 0x35f57d thru 0x36e0b0 (0xeb34 bytes) 000390[0] 1789448585.758460 trunc from 0x8086f6 to 0x8700f1 (0x679fc bytes) 000391[1] 1789448585.835313 mapread 0x138137 thru 0x13b72b (0x35f5 bytes) 000392[1] 1789448586.044990 mapread 0x70b8d2 thru 0x71521f (0x994e bytes) 000393[1] 1789448586.146864 write 0x4a6bf3 thru 0x4b32b0 (0xc6be bytes) 000395[0] 1789448586.247405 punch from 0x17ee25 to 0x18b363 (0xc53f bytes) 000398[0] 1789448586.356665 trunc from 0x8700f2 to 0x69154b (0x1deba6 bytes) 000399[0] 1789448586.435261 write 0x73e43a thru 0x7422b0 (0x3e77 bytes) 000400[1] 1789448586.444266 mapread 0x3e8e84 thru 0x3f5f97 (0xd114 bytes) 000401[0] 1789448586.584295 read 0x155a0c thru 0x15cb4a (0x713f bytes) 000403[0] 1789448586.635717 trunc from 0x7422b1 to 0x4fe871 (0x243a3f bytes) 000404[0] 1789448586.775435 mapwrite 0x0dca64 thru 0x0eb88f (0xee2c bytes) 000405[1] 1789448586.898476 trunc from 0x4fe872 to 0x86e16d (0x36f8fc bytes) 000406[1] 1789448587.015299 mapread 0x83a5e8 thru 0x83d134 (0x2b4d bytes) 000409[1] 1789448587.068205 mapwrite 0x9a1129 thru 0x9a4ce1 (0x3bb9 bytes) 000411[1] 1789448587.219189 write 0x08b5e3 thru 0x094700 (0x911e bytes) 000412[1] 1789448587.236959 punch from 0x43513e to 0x441e42 (0xcd05 bytes) 000413[0] 1789448587.283090 trunc from 0x9a4ce2 to 0x5db8d4 (0x3c940d bytes) 000414[0] 1789448587.412218 write 0x0a03c2 thru 0x0a6170 (0x5daf bytes) 000416[1] 1789448587.425781 write 0x11b231 thru 0x1220b2 (0x6e82 bytes) 000417[1] 1789448587.486443 mapwrite 0x886cb3 thru 0x88e769 (0x7ab7 bytes) 000418[0] 1789448587.681149 mapread 0x04e2c5 thru 0x059d6d (0xbaa9 bytes) 000420[1] 1789448587.738988 punch from 0x30aa61 to 0x31090f (0x5eaf bytes) 000421[1] 1789448587.787644 mapread 0x6f2532 thru 0x6fb983 (0x9452 bytes) 000422[1] 1789448587.826762 mapwrite 0x9d9e7a thru 0x9e460f (0xa796 bytes) 000424[0] 1789448588.081983 mapwrite 0x53d08c thru 0x5406c1 (0x3636 bytes) 000425[0] 1789448588.419125 mapwrite 0x10556f thru 0x106511 (0xfa3 bytes) 000426[1] 1789448588.505859 read 0x12ed10 thru 0x13527b (0x656c bytes) 000428[1] 1789448588.564708 write 0x7ead8b thru 0x7f92e5 (0xe55b bytes) 000431[0] 1789448588.655057 mapwrite 0x157047 thru 0x164912 (0xd8cc bytes) 000434[1] 1789448588.953438 mapread 0x6afc0e thru 0x6b37d2 (0x3bc5 bytes) 000435[1] 1789448589.006895 trunc from 0x9e4610 to 0x17d649 (0x866fc6 bytes) 000437[0] 1789448589.082627 write 0x77f51c thru 0x7823b2 (0x2e97 bytes) 000438[1] 1789448589.140691 read 0x228425 thru 0x22fe5e (0x7a3a bytes) 000439[1] 1789448589.257915 mapwrite 0x6238bb thru 0x631e4c (0xe592 bytes) 000440[0] 1789448589.478747 punch from 0x25137f to 0x258673 (0x72f5 bytes) 000442[0] 1789448589.563161 punch from 0x254892 to 0x2610c4 (0xc833 bytes) 000443[0] 1789448589.601530 punch from 0x3b401a to 0x3b5898 (0x187f bytes) 000444[1] 1789448589.630092 trunc from 0x7823b3 to 0x0b87c6 (0x6c9bec bytes) 000447[0] 1789448589.741579 trunc from 0x0b87c7 to 0x7addec (0x6f5626 bytes) 000449[1] 1789448589.836189 read 0x752528 thru 0x756394 (0x3e6d bytes) 000450[0] 1789448589.859361 mapwrite 0x7dad11 thru 0x7ea43c (0xf72c bytes) 000451[1] 1789448589.963785 trunc from 0x7ea43d to 0x14ec19 (0x69b823 bytes) 000452[1] 1789448590.005003 read 0x13dd36 thru 0x143077 (0x5342 bytes) 000453[1] 1789448590.034787 trunc from 0x14ec1a to 0x29b3bc (0x14c7a3 bytes) 000454[0] 1789448590.087519 punch from 0x2493ec to 0x24bd78 (0x298d bytes) 000455[1] 1789448590.125084 write 0x17982d thru 0x18913f (0xf913 bytes) 000457[0] 1789448590.243450 read 0x170d20 thru 0x180468 (0xf749 bytes) 000459[1] 1789448590.415402 mapread 0x01ce28 thru 0x0216bf (0x4898 bytes) 000460[1] 1789448590.468705 write 0x818d80 thru 0x825160 (0xc3e1 bytes) 000461[0] 1789448590.506712 write 0x743db1 thru 0x7467f4 (0x2a44 bytes) 000462[0] 1789448590.552954 trunc from 0x825161 to 0x2ada64 (0x5776fc bytes) 000463[0] 1789448590.616502 punch from 0x13d518 to 0x13f70f (0x21f8 bytes) 000464[1] 1789448590.667163 punch from 0x0462f4 to 0x047459 (0x1166 bytes) 000465[1] 1789448590.736360 trunc from 0x2ada65 to 0x5c3c8d (0x316229 bytes) 000467[0] 1789448590.915148 mapread 0x03f76d thru 0x04bd5b (0xc5ef bytes) 000468[1] 1789448591.146573 read 0x0034cb thru 0x005e84 (0x29ba bytes) 000470[1] 1789448591.223246 mapwrite 0x6fd714 thru 0x7086ef (0xafdc bytes) 000472[0] 1789448591.471084 mapread 0x297019 thru 0x2a2590 (0xb578 bytes) 000473[0] 1789448591.729349 read 0x31fe19 thru 0x322748 (0x2930 bytes) 000474[1] 1789448591.746801 mapread 0x1c06ae thru 0x1c999a (0x92ed bytes) 000477[1] 1789448591.869343 mapread 0x3e7257 thru 0x3eefd1 (0x7d7b bytes) 000478[0] 1789448591.893114 read 0x50ad03 thru 0x511215 (0x6513 bytes) 000479[1] 1789448591.939071 write 0x5e98ee thru 0x5ec90f (0x3022 bytes) 000481[1] 1789448592.044925 read 0x1f0b8e thru 0x200b67 (0xffda bytes) 000485[1] 1789448592.098956 mapread 0x2468e9 thru 0x2543e7 (0xdaff bytes) 000486[1] 1789448592.134536 punch from 0x60efad to 0x61c9bc (0xda10 bytes) 000489[1] 1789448592.155373 punch from 0x5488e3 to 0x54e832 (0x5f50 bytes) 000490[0] 1789448592.176127 read 0x59705f thru 0x5986c1 (0x1663 bytes) 000494[1] 1789448592.207993 write 0x83daa2 thru 0x840b9d (0x30fc bytes) 000496[0] 1789448592.266524 write 0x68129c thru 0x684055 (0x2dba bytes) 000497[1] 1789448592.348179 punch from 0x5b072c to 0x5b7187 (0x6a5c bytes) 000498[0] 1789448592.387973 mapread 0x1c8d2c thru 0x1cc96d (0x3c42 bytes) 000499[1] 1789448592.468364 write 0x6be3c0 thru 0x6c9200 (0xae41 bytes) 000500[1] 1789448592.481754 trunc from 0x840b9e to 0x8eeab8 (0xadf1b bytes) All operations completed A-OK! Chance of close/open is 1 in 50 Seed set to 8592 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported truncating to largest ever: 0x2e7d99 000001[1] 1789448593.470179 trunc from 00000000 to 0x2e7d98 (0x2e7d99 bytes) 000003[1] 1789448593.493927 mapwrite 0x668000 thru 0x669620 (0x1621 bytes) 000004[0] 1789448593.668979 punch from 0x1eb5e7 to 0x1edd32 (0x274c bytes) 000005[1] 1789448593.723562 read 0x18a000 thru 0x195204 (0xb205 bytes) 000006[1] 1789448593.782501 write 0x1eb000 thru 0x1eef71 (0x3f72 bytes) 000008[0] 1789448593.843978 mapwrite 0x423000 thru 0x4272ff (0x4300 bytes) truncating to largest ever: 0x98cd83 000009[0] 1789448593.977194 trunc from 0x669621 to 0x98cd82 (0x323762 bytes) 000010[1] 1789448594.018317 write 0x0b1000 thru 0x0b9aa1 (0x8aa2 bytes) 000011[1] 1789448594.055082 read 0x827000 thru 0x82cb8b (0x5b8c bytes) 000013[0] 1789448594.129507 punch from 0x93b066 to 0x94a53f (0xf4da bytes) 000014[0] 1789448594.178823 punch from 0x1de58f to 0x1e9dd2 (0xb844 bytes) 000015[1] 1789448594.184114 punch from 0x1c1615 to 0x1cbc72 (0xa65e bytes) 000016[0] 1789448594.221119 trunc from 0x98cd83 to 0x13de89 (0x84eef9 bytes) 000021[1] 1789448594.366702 trunc from 0x13de8a to 0x4d642c (0x3985a3 bytes) 000025[0] 1789448594.551760 mapread 0x17a000 thru 0x1858ac (0xb8ad bytes) 000026[1] 1789448594.691462 mapread 0x47d000 thru 0x487d9e (0xad9f bytes) 000027[1] 1789448594.735105 read 0x259000 thru 0x263331 (0xa332 bytes) 000031[0] 1789448594.754460 mapread 0x324000 thru 0x33061a (0xc61b bytes) 000032[0] 1789448594.811149 read 0x33a000 thru 0x33eb56 (0x4b57 bytes) 000033[0] 1789448594.828034 trunc from 0x4d642d to 0x3b2196 (0x124296 bytes) 000036[1] 1789448594.909887 mapread 0x0d9000 thru 0x0dbfb1 (0x2fb2 bytes) 000041[1] 1789448594.988557 mapread 0x25c000 thru 0x26b3e1 (0xf3e2 bytes) 000043[0] 1789448595.021334 mapread 0x371000 thru 0x37cd53 (0xbd54 bytes) 000044[0] 1789448595.050937 mapwrite 0x4d4000 thru 0x4e3e13 (0xfe14 bytes) 000045[0] 1789448595.274031 trunc from 0x4e3e14 to 0x160fcb (0x382e48 bytes) 000047[1] 1789448595.312651 trunc from 0x160fcc to 0x7a5697 (0x6446cc bytes) 000049[0] 1789448595.469458 mapwrite 0x9f2000 thru 0x9fb96e (0x996f bytes) 000050[1] 1789448595.622819 mapwrite 0x9b7000 thru 0x9c6cfa (0xfcfb bytes) 000053[1] 1789448595.791399 punch from 0x39077f to 0x39ed7a (0xe5fc bytes) 000058[1] 1789448595.905129 trunc from 0x9fb96f to 0x760830 (0x29b13e bytes) 000059[0] 1789448595.937687 mapwrite 0x5aa000 thru 0x5b1c00 (0x7c01 bytes) 000061[1] 1789448596.098267 trunc from 0x760831 to 0x2540a2 (0x50c78e bytes) 000063[0] 1789448596.157535 trunc from 0x2540a3 to 0x311460 (0xbd3be bytes) 000064[0] 1789448596.216507 write 0x55d000 thru 0x55facb (0x2acc bytes) 000065[1] 1789448596.223746 punch from 0x0b3bd9 to 0x0c168f (0xdab7 bytes) 000066[1] 1789448596.284827 mapread 0x493000 thru 0x4a066a (0xd66b bytes) 000067[0] 1789448596.357339 trunc from 0x55facc to 0x4b1e47 (0xadc84 bytes) 000069[1] 1789448596.459671 read 0x163000 thru 0x1650dd (0x20de bytes) 000070[1] 1789448596.503813 read 0x44d000 thru 0x452720 (0x5721 bytes) 000071[0] 1789448596.517093 mapread 0x057000 thru 0x058c74 (0x1c75 bytes) 000072[1] 1789448596.537397 punch from 0x25fee2 to 0x265988 (0x5aa7 bytes) 000073[1] 1789448596.565728 punch from 0x0c2d48 to 0x0ca2db (0x7594 bytes) 000074[1] 1789448596.572040 punch from 0x23e1d1 to 0x23fbc7 (0x19f7 bytes) 000076[0] 1789448596.578629 read 0x241000 thru 0x24bb32 (0xab33 bytes) 000077[1] 1789448596.617928 write 0x72b000 thru 0x738ea8 (0xdea9 bytes) 000078[0] 1789448596.641270 punch from 0x544387 to 0x5529be (0xe638 bytes) 000079[0] 1789448596.690941 write 0x81f000 thru 0x820b8f (0x1b90 bytes) 000080[1] 1789448596.711175 read 0x5f9000 thru 0x60648f (0xd490 bytes) 000082[0] 1789448596.736321 mapread 0x520000 thru 0x524d25 (0x4d26 bytes) 000086[1] 1789448596.774651 punch from 0x1274ff to 0x132cfb (0xb7fd bytes) 000087[1] 1789448596.793953 trunc from 0x820b90 to 0x373b8d (0x4ad002 bytes) 000088[0] 1789448596.823746 mapwrite 0x0db000 thru 0x0dbdba (0xdbb bytes) 000089[1] 1789448596.932385 write 0x08d000 thru 0x090015 (0x3016 bytes) 000090[1] 1789448596.997410 punch from 0x075971 to 0x082bf8 (0xd288 bytes) 000091[0] 1789448597.032381 punch from 0x1b04bd to 0x1b9be7 (0x972b bytes) 000092[1] 1789448597.071817 read 0x115000 thru 0x11b9dc (0x69dd bytes) 000094[1] 1789448597.180006 write 0x7c5000 thru 0x7cef85 (0x9f86 bytes) 000095[0] 1789448597.226096 read 0x55a000 thru 0x563719 (0x971a bytes) 000096[0] 1789448597.405791 write 0x242000 thru 0x24cf22 (0xaf23 bytes) 000097[1] 1789448597.468871 mapwrite 0x3d1000 thru 0x3e08b8 (0xf8b9 bytes) 000098[1] 1789448597.784641 mapread 0x5dc000 thru 0x5e3200 (0x7201 bytes) 000100[0] 1789448597.835812 mapread 0x185000 thru 0x18f3d7 (0xa3d8 bytes) 000101[0] 1789448597.944477 write 0x651000 thru 0x6600fd (0xf0fe bytes) 000102[1] 1789448598.015577 mapread 0x66b000 thru 0x670bf9 (0x5bfa bytes) 000106[1] 1789448598.088235 write 0x1ad000 thru 0x1b4f66 (0x7f67 bytes) 000108[1] 1789448598.133984 trunc from 0x7cef86 to 0x58f893 (0x23f6f2 bytes) 000109[0] 1789448598.165917 write 0x88f000 thru 0x89e02c (0xf02d bytes) 000110[1] 1789448598.255585 trunc from 0x89e02d to 0x88bccf (0x1235d bytes) 000111[1] 1789448598.335979 mapwrite 0x82f000 thru 0x83cba9 (0xdbaa bytes) 000114[0] 1789448598.454789 mapwrite 0x60f000 thru 0x61e3ea (0xf3eb bytes) 000116[1] 1789448598.744990 mapread 0x209000 thru 0x20c33b (0x333c bytes) 000118[1] 1789448598.813412 mapread 0x556000 thru 0x5569b5 (0x9b6 bytes) 000119[0] 1789448598.819058 mapread 0x12c000 thru 0x132970 (0x6971 bytes) 000120[1] 1789448598.863752 read 0x828000 thru 0x82e8e6 (0x68e7 bytes) 000122[1] 1789448598.877514 read 0x7f4000 thru 0x7f996d (0x596e bytes) 000124[1] 1789448598.895333 mapwrite 0x6a4000 thru 0x6aa57b (0x657c bytes) 000125[0] 1789448599.028996 write 0x392000 thru 0x39d6b0 (0xb6b1 bytes) 000127[0] 1789448599.098118 trunc from 0x88bcd0 to 0x4bafc8 (0x3d0d07 bytes) 000130[1] 1789448599.164139 trunc from 0x4bafc9 to 0x334f3f (0x186089 bytes) 000131[0] 1789448599.214093 mapwrite 0x4ba000 thru 0x4bfd4c (0x5d4d bytes) 000132[0] 1789448599.473305 punch from 0x32f1f2 to 0x33d2e3 (0xe0f2 bytes) 000133[1] 1789448599.480208 read 0x425000 thru 0x4313ef (0xc3f0 bytes) 000134[0] 1789448599.681602 mapread 0x288000 thru 0x28a588 (0x2589 bytes) 000135[1] 1789448599.710640 mapwrite 0x8b6000 thru 0x8c0607 (0xa608 bytes) 000136[0] 1789448599.995408 trunc from 0x8c0608 to 0x88f938 (0x30ccf bytes) 000137[1] 1789448600.094086 write 0x4f4000 thru 0x501b21 (0xdb22 bytes) 000139[1] 1789448600.133615 trunc from 0x88f939 to 0x766312 (0x129626 bytes) 000142[1] 1789448600.210804 mapread 0x281000 thru 0x28f2fc (0xe2fd bytes) 000143[0] 1789448600.302319 trunc from 0x766313 to 0x678388 (0xedf8a bytes) 000144[0] 1789448600.362643 mapread 0x005000 thru 0x006277 (0x1278 bytes) 000145[0] 1789448600.379995 write 0x9be000 thru 0x9c2634 (0x4635 bytes) 000148[0] 1789448600.455976 read 0x7d5000 thru 0x7d9d7d (0x4d7e bytes) 000149[1] 1789448600.470682 punch from 0x561305 to 0x570824 (0xf520 bytes) 000150[0] 1789448600.500297 read 0x4b5000 thru 0x4c3cce (0xeccf bytes) 000156[1] 1789448600.531804 mapwrite 0x54b000 thru 0x54e37c (0x337d bytes) 000157[1] 1789448600.654752 read 0x831000 thru 0x83c159 (0xb15a bytes) 000159[0] 1789448600.771280 trunc from 0x9c2635 to 0x38cac1 (0x635b73 bytes) 000162[1] 1789448600.894340 trunc from 0x38cac2 to 0x5d329c (0x2467db bytes) 000163[0] 1789448600.949700 punch from 0x268cdc to 0x2776ff (0xea24 bytes) 000164[0] 1789448600.976486 mapwrite 0x325000 thru 0x33309e (0xe09f bytes) 000165[0] 1789448601.213763 punch from 0x4d9516 to 0x4da59d (0x1088 bytes) 000167[0] 1789448601.248498 mapread 0x49c000 thru 0x4a7f91 (0xbf92 bytes) 000169[1] 1789448601.347743 mapread 0x425000 thru 0x429c47 (0x4c48 bytes) 000170[1] 1789448601.439399 trunc from 0x5d329d to 0x6a1ef0 (0xcec54 bytes) 000171[0] 1789448601.501957 mapwrite 0x47e000 thru 0x4857c0 (0x77c1 bytes) 000172[1] 1789448601.673871 mapwrite 0x298000 thru 0x2a6f47 (0xef48 bytes) 000174[1] 1789448601.847326 mapread 0x560000 thru 0x56d5c8 (0xd5c9 bytes) 000182[1] 1789448602.020279 mapwrite 0x775000 thru 0x7816a4 (0xc6a5 bytes) 000183[0] 1789448602.176494 trunc from 0x7816a5 to 0x0b055a (0x6d114a bytes) 000185[0] 1789448602.206034 write 0x5a8000 thru 0x5b7cff (0xfd00 bytes) 000186[1] 1789448602.215296 read 0x46c000 thru 0x47144c (0x544d bytes) 000187[0] 1789448602.248058 trunc from 0x5b7d00 to 0x7a64e4 (0x1ee7e5 bytes) 000188[1] 1789448602.285771 mapread 0x619000 thru 0x621789 (0x878a bytes) 000189[1] 1789448602.326411 trunc from 0x7a64e5 to 0x5976ee (0x20edf6 bytes) 000190[1] 1789448602.386606 write 0x5e7000 thru 0x5f41e6 (0xd1e7 bytes) 000191[1] 1789448602.405003 read 0x5cd000 thru 0x5dbb62 (0xeb63 bytes) 000193[0] 1789448602.469678 mapwrite 0x5a9000 thru 0x5ae021 (0x5022 bytes) 000197[1] 1789448602.711968 trunc from 0x5f41e7 to 0x7f352e (0x1ff348 bytes) 000198[0] 1789448602.793654 trunc from 0x7f352f to 0x1016ab (0x6f1e83 bytes) 000199[0] 1789448602.894317 read 0x07a000 thru 0x08470c (0xa70d bytes) 000204[0] 1789448602.978007 trunc from 0x1016ac to 0x982135 (0x880a8a bytes) 000205[1] 1789448603.019038 read 0x052000 thru 0x05ad25 (0x8d26 bytes) 000206[0] 1789448603.043991 mapwrite 0x536000 thru 0x53b40a (0x540b bytes) 000207[1] 1789448603.218592 read 0x710000 thru 0x719a52 (0x9a53 bytes) 000210[1] 1789448603.281269 write 0x080000 thru 0x0849c1 (0x49c2 bytes) 000211[1] 1789448603.324305 mapread 0x5b1000 thru 0x5c0818 (0xf819 bytes) 000213[1] 1789448603.369092 read 0x336000 thru 0x337717 (0x1718 bytes) 000214[1] 1789448603.389872 trunc from 0x982136 to 0x1b9a8c (0x7c86a9 bytes) 000216[1] 1789448603.414622 punch from 0x12c143 to 0x12d864 (0x1722 bytes) 000217[0] 1789448603.437031 read 0x09a000 thru 0x09c1db (0x21dc bytes) 000218[0] 1789448603.502000 punch from 0x154200 to 0x157af7 (0x38f8 bytes) 000219[0] 1789448603.531809 mapread 0x138000 thru 0x13e651 (0x6652 bytes) 000220[0] 1789448603.566667 mapread 0x127000 thru 0x128d18 (0x1d19 bytes) 000223[0] 1789448603.577598 mapread 0x072000 thru 0x081b1f (0xfb20 bytes) 000224[1] 1789448603.707580 write 0x0b5000 thru 0x0babd7 (0x5bd8 bytes) 000226[1] 1789448603.738821 read 0x129000 thru 0x12cc7f (0x3c80 bytes) 000228[0] 1789448603.779059 punch from 0x17d098 to 0x187f36 (0xae9f bytes) 000229[1] 1789448603.828271 read 0x084000 thru 0x090ec8 (0xcec9 bytes) 000231[1] 1789448603.898780 trunc from 0x1b9a8d to 0x114c9f (0xa4ded bytes) 000232[1] 1789448603.970179 read 0x0ac000 thru 0x0b93fc (0xd3fd bytes) 000236[0] 1789448604.104438 write 0x4b2000 thru 0x4b88e7 (0x68e8 bytes) 000239[0] 1789448604.160989 trunc from 0x4b88e8 to 0x835c79 (0x37d392 bytes) 000240[0] 1789448604.190400 mapwrite 0x8f2000 thru 0x9000e8 (0xe0e9 bytes) 000241[0] 1789448604.490651 punch from 0x1bae62 to 0x1bf160 (0x42ff bytes) 000242[0] 1789448604.501194 punch from 0x387844 to 0x393093 (0xb850 bytes) 000243[0] 1789448604.520664 punch from 0x71f06e to 0x72257c (0x350f bytes) 000245[1] 1789448604.565129 write 0x838000 thru 0x83e396 (0x6397 bytes) 000249[1] 1789448604.762629 punch from 0x568c54 to 0x56c5d7 (0x3984 bytes) 000251[1] 1789448604.819141 read 0x50b000 thru 0x510ccc (0x5ccd bytes) truncating to largest ever: 0x9f385f 000252[0] 1789448604.829523 trunc from 0x9000e9 to 0x9f385e (0xf3776 bytes) 000254[1] 1789448604.967806 write 0x17b000 thru 0x181296 (0x6297 bytes) 000255[0] 1789448605.023075 trunc from 0x9f385f to 0x0d4fd9 (0x91e885 bytes) 000256[1] 1789448605.112775 trunc from 0x0d4fda to 0x3031ac (0x22e1d3 bytes) 000258[0] 1789448605.156645 mapread 0x2f3000 thru 0x300c79 (0xdc7a bytes) 000261[0] 1789448605.254817 write 0x60e000 thru 0x61a2c5 (0xc2c6 bytes) 000262[0] 1789448605.275773 mapread 0x135000 thru 0x1436c7 (0xe6c8 bytes) 000263[1] 1789448605.361266 trunc from 0x61a2c6 to 0x6a0652 (0x8638d bytes) 000265[0] 1789448605.434025 write 0x160000 thru 0x1638b0 (0x38b1 bytes) 000270[0] 1789448605.517475 trunc from 0x6a0653 to 0x68a99c (0x15cb6 bytes) 000271[0] 1789448605.552683 mapread 0x5a4000 thru 0x5b0168 (0xc169 bytes) 000273[0] 1789448605.642827 punch from 0x122cb8 to 0x127648 (0x4991 bytes) 000274[1] 1789448605.660302 trunc from 0x68a99d to 0x09022a (0x5fa772 bytes) 000275[0] 1789448605.706939 write 0x3d3000 thru 0x3e2686 (0xf687 bytes) 000276[0] 1789448605.726085 mapwrite 0x7fa000 thru 0x7fcc98 (0x2c99 bytes) 000278[1] 1789448605.952533 read 0x0fe000 thru 0x1041fe (0x61ff bytes) 000279[1] 1789448605.984207 trunc from 0x7fcc99 to 0x8c75d4 (0xca93c bytes) 000281[0] 1789448606.028237 write 0x432000 thru 0x439358 (0x7359 bytes) 000283[1] 1789448606.056659 mapread 0x61b000 thru 0x61ec22 (0x3c23 bytes) 000284[0] 1789448606.123597 write 0x67d000 thru 0x681d2d (0x4d2e bytes) 000285[0] 1789448606.151659 punch from 0x6ef6e1 to 0x6f729a (0x7bba bytes) 000287[0] 1789448606.189401 mapread 0x6cc000 thru 0x6d00af (0x40b0 bytes) 000289[0] 1789448606.241816 write 0x284000 thru 0x28bc32 (0x7c33 bytes) 000292[1] 1789448606.342825 punch from 0x7da834 to 0x7dbfba (0x1787 bytes) 000294[0] 1789448606.385125 mapwrite 0x1c3000 thru 0x1c3922 (0x923 bytes) 000295[0] 1789448606.561031 mapread 0x70e000 thru 0x718223 (0xa224 bytes) 000297[1] 1789448606.668298 read 0x267000 thru 0x2753c3 (0xe3c4 bytes) 000298[0] 1789448606.716688 punch from 0x6ccc6c to 0x6d1c1a (0x4faf bytes) 000299[1] 1789448606.737976 write 0x716000 thru 0x71cac2 (0x6ac3 bytes) 000301[0] 1789448606.780516 mapwrite 0x08f000 thru 0x09900c (0xa00d bytes) 000302[1] 1789448607.068360 write 0x06a000 thru 0x06ecf0 (0x4cf1 bytes) 000303[1] 1789448607.101542 mapwrite 0x353000 thru 0x359f75 (0x6f76 bytes) 000304[1] 1789448607.167742 write 0x580000 thru 0x5838db (0x38dc bytes) 000305[0] 1789448607.189568 mapread 0x4a0000 thru 0x4a5fb0 (0x5fb1 bytes) 000306[1] 1789448607.283071 trunc from 0x8c75d5 to 0x07e75a (0x848e7a bytes) 000307[0] 1789448607.329344 punch from 0x023550 to 0x02b942 (0x83f3 bytes) 000308[1] 1789448607.346122 read 0x005000 thru 0x00e159 (0x915a bytes) 000310[0] 1789448607.403167 mapwrite 0x2f9000 thru 0x2ffe1a (0x6e1b bytes) 000312[1] 1789448607.626404 trunc from 0x2ffe1b to 0x927fd1 (0x6281b7 bytes) 000313[0] 1789448607.687922 trunc from 0x927fd2 to 0x336c75 (0x5f135c bytes) 000314[0] 1789448607.740410 write 0x174000 thru 0x178dee (0x4def bytes) 000315[1] 1789448607.754339 mapwrite 0x39c000 thru 0x3a6fa4 (0xafa5 bytes) 000319[0] 1789448608.035882 write 0x101000 thru 0x10e224 (0xd225 bytes) 000321[1] 1789448608.115126 write 0x599000 thru 0x59feb2 (0x6eb3 bytes) 000323[0] 1789448608.155666 trunc from 0x59feb3 to 0x8de623 (0x33e771 bytes) 000324[1] 1789448608.215536 mapread 0x749000 thru 0x74d1de (0x41df bytes) 000325[1] 1789448608.333341 write 0x82f000 thru 0x836da1 (0x7da2 bytes) 000326[1] 1789448608.397312 mapwrite 0x766000 thru 0x77159d (0xb59e bytes) 000327[0] 1789448608.570748 read 0x83d000 thru 0x843191 (0x6192 bytes) 000329[0] 1789448608.599118 read 0x280000 thru 0x283f9a (0x3f9b bytes) 000330[0] 1789448608.611791 read 0x8a5000 thru 0x8a8e26 (0x3e27 bytes) 000332[1] 1789448608.622761 mapread 0x52b000 thru 0x539cfd (0xecfe bytes) 000334[0] 1789448608.708053 trunc from 0x8de624 to 0x4e5275 (0x3f93ae bytes) 000335[0] 1789448608.767977 read 0x15a000 thru 0x1618fa (0x78fb bytes) 000336[1] 1789448608.797038 punch from 0x4268da to 0x4271f8 (0x91f bytes) 000337[0] 1789448608.813785 write 0x992000 thru 0x9957ce (0x37cf bytes) 000338[1] 1789448608.845980 mapread 0x0bc000 thru 0x0c1751 (0x5752 bytes) 000339[1] 1789448608.910675 punch from 0x8e593b to 0x8ed53e (0x7c04 bytes) 000340[1] 1789448608.930690 punch from 0x5de111 to 0x5e1dc7 (0x3cb7 bytes) 000341[1] 1789448608.952418 write 0x641000 thru 0x64fd6b (0xed6c bytes) 000342[1] 1789448608.976733 mapwrite 0x351000 thru 0x3572af (0x62b0 bytes) 000343[0] 1789448609.060709 mapread 0x433000 thru 0x439ec6 (0x6ec7 bytes) 000345[1] 1789448609.112075 mapwrite 0x04c000 thru 0x04ddb8 (0x1db9 bytes) 000346[1] 1789448609.222442 write 0x17e000 thru 0x184b4e (0x6b4f bytes) 000347[1] 1789448609.241478 mapwrite 0x930000 thru 0x9312f5 (0x12f6 bytes) 000348[1] 1789448609.350757 write 0x25f000 thru 0x26dfff (0xf000 bytes) 000350[1] 1789448609.368746 mapwrite 0x5b8000 thru 0x5c6b87 (0xeb88 bytes) 000353[1] 1789448609.533104 read 0x682000 thru 0x688116 (0x6117 bytes) 000354[1] 1789448609.544101 punch from 0x88e3ba to 0x8996c2 (0xb309 bytes) 000355[0] 1789448609.549781 mapwrite 0x16c000 thru 0x16d69f (0x16a0 bytes) 000356[1] 1789448609.643007 punch from 0x77a1db to 0x780d8d (0x6bb3 bytes) 000358[0] 1789448609.689526 trunc from 0x9957cf to 0x3e213d (0x5b3691 bytes) 000360[1] 1789448609.783365 write 0x85a000 thru 0x85a934 (0x935 bytes) 000361[1] 1789448609.815136 punch from 0x314534 to 0x31e2b3 (0x9d80 bytes) 000362[1] 1789448609.825509 read 0x0ae000 thru 0x0bbe2c (0xde2d bytes) truncating to largest ever: 0x9fafe5 000365[0] 1789448609.905620 trunc from 0x85a935 to 0x9fafe4 (0x1a06b0 bytes) 000368[1] 1789448609.976405 mapread 0x37f000 thru 0x38e7eb (0xf7ec bytes) 000369[1] 1789448610.059605 mapread 0x3cc000 thru 0x3cd50e (0x150f bytes) 000370[1] 1789448610.089038 read 0x5dd000 thru 0x5ecd65 (0xfd66 bytes) 000371[0] 1789448610.106536 mapread 0x8bc000 thru 0x8c6b27 (0xab28 bytes) 000372[0] 1789448610.142310 mapread 0x9e7000 thru 0x9f40fe (0xd0ff bytes) 000374[1] 1789448610.188429 trunc from 0x9fafe5 to 0x74b51f (0x2afac5 bytes) 000376[0] 1789448610.264679 mapread 0x400000 thru 0x40820d (0x820e bytes) 000378[0] 1789448610.319233 read 0x262000 thru 0x26d1be (0xb1bf bytes) 000379[1] 1789448610.333480 mapread 0x1a7000 thru 0x1ac74b (0x574c bytes) 000382[0] 1789448610.435730 punch from 0x699ce4 to 0x6a7a79 (0xdd96 bytes) 000385[0] 1789448610.539359 trunc from 0x74b520 to 0x8b550d (0x169fee bytes) 000386[0] 1789448610.599791 mapwrite 0x756000 thru 0x75796e (0x196f bytes) 000387[0] 1789448610.732083 trunc from 0x8b550e to 0x1d8a0f (0x6dcafe bytes) 000390[0] 1789448610.841409 trunc from 0x1d8a10 to 0x9b37d4 (0x7dadc5 bytes) 000392[1] 1789448610.914222 trunc from 0x9b37d5 to 0x69c00d (0x3177c7 bytes) 000393[0] 1789448610.995685 mapwrite 0x28f000 thru 0x2962de (0x72df bytes) 000394[1] 1789448611.132568 trunc from 0x69c00e to 0x9c2249 (0x32623c bytes) 000395[1] 1789448611.167992 mapread 0x003000 thru 0x00c924 (0x9925 bytes) 000396[1] 1789448611.222603 write 0x5e9000 thru 0x5e9d9d (0xd9e bytes) 000397[1] 1789448611.268800 read 0x4e7000 thru 0x4e9181 (0x2182 bytes) 000398[0] 1789448611.280997 punch from 0x23c61a to 0x241b39 (0x5520 bytes) 000400[1] 1789448611.388369 punch from 0x7214d5 to 0x722b3f (0x166b bytes) 000401[0] 1789448611.433071 punch from 0x39ca42 to 0x3a4390 (0x794f bytes) 000402[1] 1789448611.467992 write 0x02d000 thru 0x0307e5 (0x37e6 bytes) 000404[0] 1789448611.501625 trunc from 0x9c224a to 0x00d3ad (0x9b4e9c bytes) 000405[1] 1789448611.543982 punch from 0x005cb8 to 0x0081f7 (0x2540 bytes) 000407[0] 1789448611.575439 mapwrite 0x8c7000 thru 0x8d01ce (0x91cf bytes) 000408[1] 1789448611.712702 mapwrite 0x0a9000 thru 0x0b6465 (0xd466 bytes) 000410[1] 1789448611.836257 write 0x58c000 thru 0x590e26 (0x4e27 bytes) 000411[1] 1789448611.877679 trunc from 0x8d01cf to 0x30996e (0x5c6860 bytes) 000412[0] 1789448611.911784 trunc from 0x30996f to 0x1c68a6 (0x1430c8 bytes) 000413[0] 1789448611.955740 read 0x1a7000 thru 0x1b1eaa (0xaeab bytes) 000415[1] 1789448612.018216 mapread 0x0c0000 thru 0x0cf05a (0xf05b bytes) 000416[0] 1789448612.080427 punch from 0x04ce37 to 0x0525a9 (0x5773 bytes) 000417[0] 1789448612.105425 mapread 0x04d000 thru 0x056c74 (0x9c75 bytes) 000421[0] 1789448612.177996 mapread 0x028000 thru 0x0356e8 (0xd6e9 bytes) 000422[0] 1789448612.267597 trunc from 0x1c68a7 to 0x5ccc44 (0x40639e bytes) 000423[1] 1789448612.312117 mapread 0x0ca000 thru 0x0d3fcd (0x9fce bytes) 000425[0] 1789448612.357151 read 0x246000 thru 0x24a09f (0x40a0 bytes) 000429[1] 1789448612.409112 punch from 0x2a0d62 to 0x2a6d5b (0x5ffa bytes) 000431[0] 1789448612.506103 mapwrite 0x8b1000 thru 0x8be309 (0xd30a bytes) 000432[1] 1789448612.715518 read 0x10b000 thru 0x10b17c (0x17d bytes) 000433[1] 1789448612.763709 mapread 0x865000 thru 0x86e622 (0x9623 bytes) 000434[1] 1789448612.815656 write 0x1c9000 thru 0x1c9e8b (0xe8c bytes) 000436[0] 1789448612.878134 punch from 0x0f656f to 0x0faa0e (0x44a0 bytes) 000437[0] 1789448612.904725 write 0x8e7000 thru 0x8f2214 (0xb215 bytes) 000440[0] 1789448613.011169 trunc from 0x8f2215 to 0x2ae128 (0x6440ec bytes) 000441[1] 1789448613.102706 punch from 0x2a326c to 0x2ae128 (0xaebd bytes) 000442[1] 1789448613.151790 trunc from 0x2ae129 to 0x82a970 (0x57c848 bytes) 000443[0] 1789448613.183701 write 0x319000 thru 0x31f243 (0x6244 bytes) 000445[1] 1789448613.287358 write 0x43a000 thru 0x4474e4 (0xd4e5 bytes) 000446[0] 1789448613.374354 trunc from 0x82a971 to 0x0e8552 (0x74241e bytes) 000448[1] 1789448613.473095 punch from 0x052bef to 0x05b2a4 (0x86b6 bytes) 000449[1] 1789448613.493899 mapwrite 0x868000 thru 0x86c760 (0x4761 bytes) 000451[1] 1789448613.637623 write 0x23f000 thru 0x2444c5 (0x54c6 bytes) 000452[0] 1789448613.652359 write 0x9e5000 thru 0x9f3647 (0xe648 bytes) 000453[0] 1789448613.706828 read 0x773000 thru 0x781c10 (0xec11 bytes) 000457[1] 1789448613.762600 mapread 0x7c6000 thru 0x7c8dfb (0x2dfc bytes) 000460[1] 1789448613.813592 read 0x4a8000 thru 0x4b712f (0xf130 bytes) 000461[0] 1789448613.823103 read 0x632000 thru 0x639c2a (0x7c2b bytes) 000462[0] 1789448613.830378 write 0x698000 thru 0x698b37 (0xb38 bytes) 000463[1] 1789448613.863565 mapread 0x2d5000 thru 0x2dc71a (0x771b bytes) 000464[1] 1789448613.944259 trunc from 0x9f3648 to 0x47d9e1 (0x575c66 bytes) 000470[1] 1789448614.018688 trunc from 0x47d9e2 to 0x430b7c (0x4ce65 bytes) 000471[0] 1789448614.041730 punch from 0x38de2c to 0x3975e2 (0x97b7 bytes) 000472[1] 1789448614.060599 trunc from 0x430b7d to 0x881939 (0x450dbd bytes) 000474[0] 1789448614.090899 trunc from 0x88193a to 0x1b6571 (0x6cb3c8 bytes) 000475[1] 1789448614.154596 punch from 0x0d575d to 0x0e0c05 (0xb4a9 bytes) 000476[1] 1789448614.199276 mapread 0x081000 thru 0x084858 (0x3859 bytes) 000477[0] 1789448614.239313 trunc from 0x1b6572 to 0x9a113a (0x7eabc9 bytes) 000480[1] 1789448614.366138 punch from 0x175ada to 0x17db75 (0x809c bytes) 000481[0] 1789448614.392250 read 0x546000 thru 0x54b5e4 (0x55e5 bytes) 000482[1] 1789448614.418764 mapread 0x133000 thru 0x137586 (0x4587 bytes) 000483[1] 1789448614.455140 mapwrite 0x940000 thru 0x94663f (0x6640 bytes) 000487[0] 1789448614.592964 read 0x562000 thru 0x5641b5 (0x21b6 bytes) 000489[0] 1789448614.612258 mapread 0x3ad000 thru 0x3b68b5 (0x98b6 bytes) 000490[0] 1789448614.633807 mapread 0x3de000 thru 0x3e76eb (0x96ec bytes) 000492[0] 1789448614.660592 trunc from 0x9a113b to 0x950a52 (0x506e8 bytes) 000493[0] 1789448614.708083 punch from 0x5d2fe0 to 0x5d428e (0x12af bytes) 000494[1] 1789448614.718245 read 0x03d000 thru 0x040b00 (0x3b01 bytes) 000496[0] 1789448614.776827 punch from 0x7cb641 to 0x7d5dda (0xa79a bytes) 000497[1] 1789448614.828368 write 0x548000 thru 0x554356 (0xc357 bytes) 000500[1] 1789448614.922964 write 0x52e000 thru 0x52f249 (0x124a bytes) All operations completed A-OK! PASS 16a (58s) == sanityn test 16b: 500 iterations of dual-mount fsx at small size ========================================================== 01:03:40 (1789448620) keep default fallocate mode: 2 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.460268 s, 18.2 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.434433 s, 19.3 MB/s Chance of close/open is 1 in 50 Seed set to 8623 fd 0: /mnt/lustre/f16b.sanityn fd 1: /mnt/lustre2/f16b.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported truncating to largest ever: 0x2d49d9 000002[0] 1789448624.834142 trunc from 00000000 to 0x2d49d8 (0x2d49d9 bytes) 000003[1] 1789448624.867319 punch from 0x03b322 to 0x03bf5d (0xc3c bytes) 000007[1] 1789448625.060520 mapwrite 0x08a7ed thru 0x08b6c0 (0xed4 bytes) 000008[1] 1789448625.294813 punch from 0x1751dd to 0x17599d (0x7c1 bytes) 000010[1] 1789448625.361169 trunc from 0x2d49d9 to 0x0f657b (0x1de45d bytes) 000011[1] 1789448625.411595 read 0x02eea1 thru 0x02fbd6 (0xd36 bytes) 000012[1] 1789448625.417339 mapread 0x01f38d thru 0x0206e2 (0x1356 bytes) 000013[0] 1789448625.435219 read 0x0e53f6 thru 0x0e723c (0x1e47 bytes) 000016[1] 1789448625.480903 write 0x48becd thru 0x48d13b (0x126f bytes) 000017[1] 1789448625.500127 write 0x581792 thru 0x5827f4 (0x1063 bytes) 000019[0] 1789448625.519738 mapwrite 0x3ec128 thru 0x3ed56d (0x1446 bytes) 000020[0] 1789448625.733709 read 0x2c1595 thru 0x2c17b4 (0x220 bytes) 000022[0] 1789448625.807744 mapread 0x42462b thru 0x4259b5 (0x138b bytes) 000024[1] 1789448625.889562 punch from 0x1fcac5 to 0x1fd9cc (0xf08 bytes) 000029[0] 1789448626.046523 trunc from 0x5827f5 to 0x01cfe3 (0x565811 bytes) 000030[0] 1789448626.221927 mapwrite 0x539430 thru 0x539f5a (0xb2b bytes) 000031[1] 1789448626.367235 mapwrite 0x0b9074 thru 0x0bac8b (0x1c18 bytes) 000032[0] 1789448626.471243 mapread 0x1b838a thru 0x1b88f8 (0x56f bytes) 000033[0] 1789448626.508325 mapwrite 0x728caf thru 0x72939b (0x6ed bytes) 000034[1] 1789448626.637298 read 0x1ed29a thru 0x1edede (0xc45 bytes) 000035[0] 1789448626.659638 write 0x032522 thru 0x03317a (0xc59 bytes) truncating to largest ever: 0x36c7e2 000036[1] 1789448626.684757 trunc from 0x72939c to 0x36c7e1 (0x3bcbba bytes) 000039[0] 1789448626.770207 write 0x9b6ff4 thru 0x9b88d0 (0x18dd bytes) 000041[1] 1789448626.831662 mapwrite 0x1fa056 thru 0x1fb26d (0x1218 bytes) 000045[0] 1789448627.033138 mapread 0x5a3a30 thru 0x5a5661 (0x1c32 bytes) 000047[0] 1789448627.206280 mapread 0x147b22 thru 0x148d55 (0x1234 bytes) 000048[0] 1789448627.305654 trunc from 0x9b88d1 to 0x0830be (0x935812 bytes) 000049[1] 1789448627.401945 read 0x06e396 thru 0x0700bf (0x1d2a bytes) truncating to largest ever: 0x67b69e 000052[1] 1789448627.494580 trunc from 0x0830bf to 0x67b69d (0x5f85df bytes) 000053[1] 1789448627.575390 read 0x5aa1ba thru 0x5aa4b9 (0x300 bytes) 000054[1] 1789448627.620788 write 0x9c14e5 thru 0x9c207e (0xb9a bytes) 000057[1] 1789448627.684686 mapwrite 0x845804 thru 0x846ccd (0x14ca bytes) 000060[0] 1789448627.896126 punch from 0x0ac7b4 to 0x0ada5e (0x12ab bytes) 000061[0] 1789448627.933941 mapwrite 0x24dd7b thru 0x24f6b8 (0x193e bytes) 000062[0] 1789448628.043485 trunc from 0x9c207f to 0x6298d0 (0x3987ae bytes) 000063[0] 1789448628.090824 mapwrite 0x6d0293 thru 0x6d207e (0x1dec bytes) 000064[1] 1789448628.249608 write 0x961395 thru 0x96248a (0x10f6 bytes) 000065[0] 1789448628.312160 mapread 0x341de0 thru 0x34398f (0x1bb0 bytes) truncating to largest ever: 0x77739f 000066[0] 1789448628.379656 trunc from 0x96248b to 0x77739e (0x1eb0ec bytes) 000067[0] 1789448628.421682 mapread 0x3c52c9 thru 0x3c6bb9 (0x18f1 bytes) 000068[0] 1789448628.467411 punch from 0x5c1671 to 0x5c266f (0xfff bytes) 000069[1] 1789448628.494798 mapwrite 0x7be4f3 thru 0x7bfd97 (0x18a5 bytes) 000071[0] 1789448628.605459 mapread 0x5b3f07 thru 0x5b5b2e (0x1c28 bytes) 000072[0] 1789448628.691471 write 0x76169d thru 0x76218b (0xaef bytes) 000074[0] 1789448628.784197 mapwrite 0x629843 thru 0x629e7b (0x639 bytes) truncating to largest ever: 0x8c6660 000075[1] 1789448628.951392 trunc from 0x7bfd98 to 0x8c665f (0x1068c8 bytes) 000076[1] 1789448629.044756 mapread 0x5cc1a6 thru 0x5ccb17 (0x972 bytes) 000079[0] 1789448629.093118 read 0x053d13 thru 0x055639 (0x1927 bytes) 000080[1] 1789448629.171764 write 0x237ab7 thru 0x2383b3 (0x8fd bytes) 000082[1] 1789448629.226773 mapwrite 0x1d2d00 thru 0x1d2ee4 (0x1e5 bytes) 000086[0] 1789448629.380773 punch from 0x43ab97 to 0x43ba12 (0xe7c bytes) 000088[1] 1789448629.465079 mapwrite 0x584787 thru 0x584efc (0x776 bytes) 000090[0] 1789448629.590069 mapread 0x7680bc thru 0x768510 (0x455 bytes) 000091[0] 1789448629.644524 read 0x8758ad thru 0x876f46 (0x169a bytes) 000092[0] 1789448629.676685 mapwrite 0x2df2d2 thru 0x2df9c2 (0x6f1 bytes) 000097[0] 1789448629.841697 mapread 0x6e21d5 thru 0x6e3819 (0x1645 bytes) 000099[1] 1789448629.895217 mapwrite 0x453b73 thru 0x454565 (0x9f3 bytes) 000102[1] 1789448630.082734 punch from 0x241dcd to 0x243615 (0x1849 bytes) 000106[0] 1789448630.128952 mapread 0x517db3 thru 0x518e81 (0x10cf bytes) 000107[0] 1789448630.182088 write 0x8facd1 thru 0x8fb46d (0x79d bytes) 000108[0] 1789448630.221941 write 0x2b53b5 thru 0x2b704e (0x1c9a bytes) 000109[1] 1789448630.235721 punch from 0x5642fe to 0x564fd7 (0xcda bytes) 000110[0] 1789448630.282276 write 0x5de863 thru 0x5e020c (0x19aa bytes) 000111[1] 1789448630.328466 trunc from 0x8fb46e to 0x015404 (0x8e6069 bytes) 000112[0] 1789448630.386549 read 0x0074e7 thru 0x00850f (0x1029 bytes) 000113[0] 1789448630.408962 trunc from 0x015405 to 0x3305f7 (0x31b1f3 bytes) 000115[1] 1789448630.516491 trunc from 0x3305f8 to 0x0e0a24 (0x24fbd3 bytes) 000116[0] 1789448630.598301 read 0x0ca3c0 thru 0x0ca755 (0x396 bytes) 000121[0] 1789448630.722817 write 0x722cc8 thru 0x72328b (0x5c4 bytes) 000122[0] 1789448630.736802 read 0x269f85 thru 0x26bced (0x1d69 bytes) 000124[0] 1789448630.767099 punch from 0x14e447 to 0x14e7da (0x394 bytes) 000125[0] 1789448630.785225 write 0x8a1b02 thru 0x8a23d1 (0x8d0 bytes) 000127[0] 1789448630.831826 mapread 0x25f332 thru 0x261012 (0x1ce1 bytes) 000128[0] 1789448630.852617 punch from 0x7f5d01 to 0x7f70e2 (0x13e2 bytes) 000129[0] 1789448630.860826 punch from 0x37e40c to 0x37fb12 (0x1707 bytes) 000131[1] 1789448630.894766 mapwrite 0x8c3840 thru 0x8c51e4 (0x19a5 bytes) 000132[0] 1789448631.037274 punch from 0x812d37 to 0x81497a (0x1c44 bytes) 000136[1] 1789448631.136258 mapwrite 0x08d453 thru 0x08d4b7 (0x65 bytes) 000137[0] 1789448631.226600 trunc from 0x8c51e5 to 0x1de73e (0x6e6aa6 bytes) 000139[0] 1789448631.298890 trunc from 0x1de73f to 0x5b385a (0x3d511c bytes) 000141[0] 1789448631.357461 read 0x3d636a thru 0x3d7683 (0x131a bytes) 000147[1] 1789448631.395914 mapread 0x351320 thru 0x35307f (0x1d60 bytes) 000148[1] 1789448631.487295 write 0x7a9433 thru 0x7a98cf (0x49d bytes) 000149[1] 1789448631.532404 mapwrite 0x61d134 thru 0x61d56f (0x43c bytes) 000151[1] 1789448631.639776 trunc from 0x7a98d0 to 0x868b40 (0xbf271 bytes) 000152[1] 1789448631.741772 punch from 0x63ce82 to 0x63df47 (0x10c6 bytes) 000153[0] 1789448631.806561 mapread 0x0f2dba thru 0x0f4816 (0x1a5d bytes) 000154[1] 1789448631.912014 write 0x91c6b8 thru 0x91cb5c (0x4a5 bytes) 000155[0] 1789448632.009286 write 0x28a67f thru 0x28a6bf (0x41 bytes) 000159[1] 1789448632.195119 write 0x1fcc55 thru 0x1fe0e1 (0x148d bytes) 000163[1] 1789448632.252722 read 0x8871a1 thru 0x888de9 (0x1c49 bytes) 000167[0] 1789448632.272604 punch from 0x45e49d to 0x45ee47 (0x9ab bytes) 000169[0] 1789448632.336681 mapread 0x800051 thru 0x8013f4 (0x13a4 bytes) 000171[0] 1789448632.383087 punch from 0x6ab053 to 0x6aca86 (0x1a34 bytes) 000173[1] 1789448632.401290 trunc from 0x91cb5d to 0x7fcf4a (0x11fc12 bytes) 000175[0] 1789448632.466951 read 0x71498c thru 0x7162e2 (0x1957 bytes) 000176[0] 1789448632.502208 write 0x12dc62 thru 0x12ed13 (0x10b2 bytes) 000177[1] 1789448632.532708 mapread 0x0265b1 thru 0x027b57 (0x15a7 bytes) truncating to largest ever: 0x8e8eda 000178[0] 1789448632.581477 trunc from 0x7fcf4b to 0x8e8ed9 (0xebf8f bytes) 000179[1] 1789448632.660245 trunc from 0x8e8eda to 0x32d355 (0x5bbb84 bytes) 000180[0] 1789448632.767888 punch from 0x23162d to 0x2327fc (0x11d0 bytes) 000182[1] 1789448632.873588 mapwrite 0x81bab1 thru 0x81d81e (0x1d6e bytes) 000183[1] 1789448633.204927 write 0x4e167a thru 0x4e31a8 (0x1b2f bytes) 000184[0] 1789448633.241779 mapwrite 0x0be3fd thru 0x0bfa8c (0x1690 bytes) 000186[0] 1789448633.544498 trunc from 0x81d81f to 0x759fe4 (0xc383a bytes) 000187[1] 1789448633.633122 mapread 0x033596 thru 0x0345bf (0x102a bytes) 000188[1] 1789448633.683410 mapread 0x10e736 thru 0x10eb05 (0x3d0 bytes) 000189[0] 1789448633.717674 trunc from 0x759fe5 to 0x201954 (0x558690 bytes) 000190[1] 1789448633.782789 read 0x15cc1f thru 0x15cc79 (0x5b bytes) 000193[0] 1789448633.910114 mapread 0x0db722 thru 0x0dbb1a (0x3f9 bytes) 000194[1] 1789448633.921787 mapwrite 0x7ebdcf thru 0x7edcc2 (0x1ef4 bytes) 000195[0] 1789448634.284691 write 0x44f589 thru 0x450a3b (0x14b3 bytes) 000196[0] 1789448634.393090 read 0x53db83 thru 0x53ef4f (0x13cd bytes) 000198[1] 1789448634.478160 read 0x729bf5 thru 0x729ddd (0x1e9 bytes) 000199[0] 1789448634.499173 read 0x4c8f5d thru 0x4ca85f (0x1903 bytes) 000200[1] 1789448634.514131 mapwrite 0x1e51e4 thru 0x1e65f4 (0x1411 bytes) 000201[1] 1789448634.576069 read 0x0442d9 thru 0x044fea (0xd12 bytes) 000203[1] 1789448634.584118 mapwrite 0x058577 thru 0x05a363 (0x1ded bytes) 000204[1] 1789448634.649815 read 0x7c52f7 thru 0x7c5b7b (0x885 bytes) 000208[1] 1789448634.693486 mapread 0x21f58a thru 0x220e5d (0x18d4 bytes) 000209[0] 1789448634.709740 trunc from 0x7edcc3 to 0x4afc95 (0x33e02d bytes) 000210[1] 1789448634.822592 mapread 0x191630 thru 0x1929ae (0x137f bytes) 000211[0] 1789448634.869797 trunc from 0x4afc96 to 0x518cee (0x69059 bytes) 000215[1] 1789448635.025646 write 0x77ab2a thru 0x77af6d (0x444 bytes) 000216[0] 1789448635.066681 write 0x62d448 thru 0x62e3dc (0xf95 bytes) 000217[0] 1789448635.148395 write 0x2cdff6 thru 0x2ceb96 (0xba1 bytes) 000218[0] 1789448635.243348 read 0x1e5ca0 thru 0x1e7017 (0x1378 bytes) 000219[0] 1789448635.254744 punch from 0x2b2b8d to 0x2b3817 (0xc8b bytes) 000220[1] 1789448635.266536 punch from 0x528903 to 0x5289f0 (0xee bytes) 000221[1] 1789448635.306417 write 0x96fdd1 thru 0x971bcb (0x1dfb bytes) 000222[1] 1789448635.341239 mapread 0x4a5492 thru 0x4a66bf (0x122e bytes) 000223[1] 1789448635.354429 mapwrite 0x3b94e6 thru 0x3b9d6c (0x887 bytes) 000228[0] 1789448635.526750 read 0x1672db thru 0x16874c (0x1472 bytes) 000230[0] 1789448635.582258 mapwrite 0x94ad9a thru 0x94b051 (0x2b8 bytes) 000231[1] 1789448635.684784 mapread 0x49c1b8 thru 0x49cecf (0xd18 bytes) 000232[0] 1789448635.738629 trunc from 0x971bcc to 0x857520 (0x11a6ab bytes) 000233[0] 1789448635.807732 mapread 0x03df04 thru 0x03f4b7 (0x15b4 bytes) 000234[0] 1789448635.850132 mapwrite 0x4ba891 thru 0x4baf1f (0x68f bytes) 000236[0] 1789448635.937237 read 0x49a351 thru 0x49a645 (0x2f5 bytes) 000237[0] 1789448635.949501 trunc from 0x857521 to 0x8aa8fb (0x533db bytes) 000239[1] 1789448636.027580 mapwrite 0x8297e4 thru 0x82a0ce (0x8eb bytes) 000242[1] 1789448636.222346 mapread 0x54805e thru 0x5493f8 (0x139b bytes) 000244[1] 1789448636.299720 trunc from 0x8aa8fc to 0x084d7b (0x825b80 bytes) 000245[1] 1789448636.361245 mapwrite 0x413bb3 thru 0x414d76 (0x11c4 bytes) 000248[1] 1789448636.462110 read 0x1dfe71 thru 0x1e150c (0x169c bytes) 000249[1] 1789448636.525360 mapwrite 0x170711 thru 0x1710bb (0x9ab bytes) 000250[1] 1789448636.670733 punch from 0x3eaadf to 0x3ebc1d (0x113f bytes) 000252[1] 1789448636.715218 punch from 0x238940 to 0x239068 (0x729 bytes) 000254[1] 1789448636.836397 trunc from 0x414d77 to 0x82bb09 (0x416d93 bytes) 000256[0] 1789448636.884075 mapwrite 0x721c08 thru 0x72217d (0x576 bytes) 000257[0] 1789448637.081107 mapread 0x5e4036 thru 0x5e5a82 (0x1a4d bytes) 000258[1] 1789448637.151398 mapread 0x314701 thru 0x31523b (0xb3b bytes) 000259[0] 1789448637.214787 mapwrite 0x0d00b3 thru 0x0d190a (0x1858 bytes) 000260[1] 1789448637.382152 write 0x6dfa9b thru 0x6e0841 (0xda7 bytes) 000262[0] 1789448637.486477 read 0x02d25a thru 0x02ef29 (0x1cd0 bytes) 000263[0] 1789448637.505634 mapread 0x4cfaa4 thru 0x4d15da (0x1b37 bytes) 000264[0] 1789448637.547707 write 0x604836 thru 0x604b15 (0x2e0 bytes) 000265[1] 1789448637.586208 punch from 0x1b509a to 0x1b6e61 (0x1dc8 bytes) 000267[1] 1789448637.682615 mapread 0x55deb2 thru 0x55f3ee (0x153d bytes) 000268[0] 1789448637.743062 mapwrite 0x665889 thru 0x66759b (0x1d13 bytes) 000270[0] 1789448637.924653 write 0x0d43c5 thru 0x0d4f9c (0xbd8 bytes) 000273[0] 1789448637.975443 write 0x656849 thru 0x656e94 (0x64c bytes) 000276[1] 1789448638.000738 mapwrite 0x3d3bbd thru 0x3d5ae0 (0x1f24 bytes) 000278[0] 1789448638.110143 punch from 0x77cf1e to 0x77e40e (0x14f1 bytes) 000284[1] 1789448638.209156 read 0x027672 thru 0x029135 (0x1ac4 bytes) 000285[1] 1789448638.228407 punch from 0x544b1b to 0x544fe7 (0x4cd bytes) 000287[1] 1789448638.334236 trunc from 0x82bb0a to 0x56bc70 (0x2bfe99 bytes) 000288[0] 1789448638.375954 mapread 0x31a8d8 thru 0x31b59e (0xcc7 bytes) 000289[1] 1789448638.467880 write 0x265192 thru 0x26687a (0x16e9 bytes) 000290[0] 1789448638.553977 write 0x513258 thru 0x51520d (0x1fb6 bytes) 000291[0] 1789448638.624730 write 0x4dbad9 thru 0x4ddaa1 (0x1fc9 bytes) 000292[0] 1789448638.651202 mapread 0x040782 thru 0x0411dc (0xa5b bytes) 000293[0] 1789448638.748976 mapread 0x27cb21 thru 0x27dc7a (0x115a bytes) 000294[1] 1789448638.793270 write 0x15cbc6 thru 0x15de56 (0x1291 bytes) 000295[0] 1789448638.840329 write 0x81c52c thru 0x81e2f0 (0x1dc5 bytes) 000296[1] 1789448638.885149 mapwrite 0x6fd161 thru 0x6fec81 (0x1b21 bytes) 000298[0] 1789448639.062740 trunc from 0x81e2f1 to 0x6d0c9e (0x14d652 bytes) 000299[1] 1789448639.156125 punch from 0x561277 to 0x562267 (0xff1 bytes) 000300[0] 1789448639.190165 mapread 0x0993c6 thru 0x099a56 (0x691 bytes) 000301[0] 1789448639.233592 trunc from 0x6d0c9f to 0x75ba92 (0x8adf4 bytes) 000304[1] 1789448639.303792 mapwrite 0x464211 thru 0x465e97 (0x1c87 bytes) 000307[0] 1789448639.483183 read 0x2baa69 thru 0x2bb232 (0x7ca bytes) 000308[1] 1789448639.526665 trunc from 0x75ba93 to 0x55a790 (0x201302 bytes) 000309[1] 1789448639.590434 mapread 0x1941b2 thru 0x194f4d (0xd9c bytes) 000310[1] 1789448639.595561 mapread 0x1c67e2 thru 0x1c86bf (0x1ede bytes) 000312[1] 1789448639.609432 mapread 0x1e90a8 thru 0x1e979e (0x6f7 bytes) 000313[0] 1789448639.621617 mapread 0x24a3c5 thru 0x24a910 (0x54c bytes) 000315[1] 1789448639.705014 write 0x42dc7b thru 0x42f407 (0x178d bytes) 000318[0] 1789448639.722559 trunc from 0x55a791 to 0x548ac0 (0x11cd0 bytes) 000319[0] 1789448639.790300 trunc from 0x548ac1 to 0x3f98f0 (0x14f1d0 bytes) 000321[1] 1789448639.815944 read 0x17d2cf thru 0x17f08e (0x1dc0 bytes) 000322[0] 1789448639.867455 trunc from 0x3f98f1 to 0x2016e8 (0x1f8208 bytes) 000324[0] 1789448639.951592 write 0x8ed778 thru 0x8ee668 (0xef1 bytes) 000325[1] 1789448639.987109 write 0x960b7d thru 0x962932 (0x1db6 bytes) 000326[1] 1789448640.026167 punch from 0x1399c2 to 0x13a142 (0x781 bytes) 000329[0] 1789448640.080032 write 0x29868c thru 0x299624 (0xf99 bytes) 000330[0] 1789448640.138035 punch from 0x01b654 to 0x01c915 (0x12c2 bytes) 000333[0] 1789448640.167503 write 0x283948 thru 0x284767 (0xe20 bytes) 000335[1] 1789448640.216316 punch from 0x182b92 to 0x1843c8 (0x1837 bytes) 000337[1] 1789448640.295559 mapread 0x170bf0 thru 0x170e30 (0x241 bytes) 000338[1] 1789448640.337348 punch from 0x4dc6ab to 0x4dce75 (0x7cb bytes) 000339[1] 1789448640.382716 write 0x06d990 thru 0x06f577 (0x1be8 bytes) 000340[0] 1789448640.410586 write 0x974ef2 thru 0x975c83 (0xd92 bytes) 000342[1] 1789448640.465675 read 0x0d52b6 thru 0x0d71f5 (0x1f40 bytes) 000343[0] 1789448640.502793 punch from 0x573514 to 0x5737ca (0x2b7 bytes) 000347[0] 1789448640.572966 write 0x94db93 thru 0x94f12f (0x159d bytes) 000348[0] 1789448640.597148 mapread 0x08e5f7 thru 0x09008d (0x1a97 bytes) 000350[1] 1789448640.633660 trunc from 0x975c84 to 0x7a77ca (0x1ce4b9 bytes) 000351[1] 1789448640.692669 trunc from 0x7a77cb to 0x6bc2ac (0xeb51e bytes) 000352[1] 1789448640.744108 trunc from 0x6bc2ad to 0x627954 (0x94958 bytes) 000353[0] 1789448640.765906 punch from 0x3e37fb to 0x3e41e5 (0x9eb bytes) 000355[1] 1789448640.815781 mapread 0x2f4f95 thru 0x2f5894 (0x900 bytes) 000357[1] 1789448640.848774 write 0x979c14 thru 0x97a4c6 (0x8b3 bytes) 000359[1] 1789448640.865483 mapwrite 0x777728 thru 0x777bfe (0x4d7 bytes) 000361[1] 1789448640.975329 read 0x39a826 thru 0x39aa87 (0x262 bytes) 000362[0] 1789448640.991785 trunc from 0x97a4c7 to 0x25e833 (0x71bc93 bytes) 000367[1] 1789448641.254259 write 0x89f7fc thru 0x89fc8c (0x491 bytes) 000368[1] 1789448641.327165 mapwrite 0x82d771 thru 0x82dbc4 (0x454 bytes) 000369[1] 1789448641.465084 trunc from 0x89fc8d to 0x479a9d (0x4261ef bytes) 000370[1] 1789448641.489667 write 0x5bf56e thru 0x5c0da9 (0x183c bytes) 000371[0] 1789448641.492731 trunc from 0x5c0daa to 0x130b16 (0x490293 bytes) 000373[1] 1789448641.574596 mapwrite 0x39362c thru 0x3941b7 (0xb8c bytes) 000375[1] 1789448641.705155 read 0x0e4662 thru 0x0e63f4 (0x1d93 bytes) 000376[1] 1789448641.771188 write 0x13a9d6 thru 0x13c428 (0x1a53 bytes) 000377[0] 1789448641.791453 punch from 0x26ca60 to 0x26e38a (0x192b bytes) 000381[0] 1789448641.938046 mapwrite 0x7acf52 thru 0x7ad634 (0x6e3 bytes) 000382[0] 1789448642.109479 write 0x7ef5b5 thru 0x7f106b (0x1ab7 bytes) 000383[1] 1789448642.125710 punch from 0x49296f to 0x4935e5 (0xc77 bytes) 000385[0] 1789448642.205162 trunc from 0x7f106c to 0x412e99 (0x3de1d2 bytes) 000387[1] 1789448642.263562 read 0x361167 thru 0x3622ad (0x1147 bytes) 000388[1] 1789448642.288334 punch from 0x2d2137 to 0x2d26b0 (0x57a bytes) 000389[1] 1789448642.332358 read 0x2306c1 thru 0x230d03 (0x643 bytes) 000391[1] 1789448642.368353 write 0x107ff8 thru 0x109f07 (0x1f10 bytes) 000392[0] 1789448642.392131 mapread 0x22c902 thru 0x22de20 (0x151f bytes) 000393[0] 1789448642.427388 punch from 0x2020e9 to 0x203ea4 (0x1dbc bytes) 000394[1] 1789448642.445737 punch from 0x3aad44 to 0x3ac55a (0x1817 bytes) 000397[0] 1789448642.501629 mapread 0x2004c3 thru 0x2014c0 (0xffe bytes) 000398[1] 1789448642.546113 punch from 0x0b289b to 0x0b3d1b (0x1481 bytes) 000399[1] 1789448642.578514 punch from 0x167254 to 0x16829a (0x1047 bytes) 000401[0] 1789448642.612748 trunc from 0x412e9a to 0x1d3a95 (0x23f404 bytes) 000402[1] 1789448642.670377 mapwrite 0x42e193 thru 0x42f0da (0xf48 bytes) 000403[0] 1789448642.824172 read 0x41dc61 thru 0x41f6f4 (0x1a94 bytes) 000405[1] 1789448642.863290 write 0x8527ab thru 0x853203 (0xa59 bytes) 000406[0] 1789448642.867782 write 0x7b88f5 thru 0x7b8f18 (0x624 bytes) 000407[0] 1789448642.901229 write 0x9b9f41 thru 0x9baecf (0xf8f bytes) 000408[0] 1789448642.949794 trunc from 0x9baed0 to 0x2fd82a (0x6bd6a5 bytes) 000409[0] 1789448643.025470 mapwrite 0x5f79fc thru 0x5f8d1b (0x1320 bytes) 000410[0] 1789448643.173132 read 0x3606e8 thru 0x361b08 (0x1421 bytes) 000411[1] 1789448643.199784 read 0x44847f thru 0x44a0c7 (0x1c49 bytes) 000413[1] 1789448643.241577 punch from 0x4f8f35 to 0x4f9e64 (0xf30 bytes) 000415[0] 1789448643.295135 read 0x30403d thru 0x305b0f (0x1ad3 bytes) 000416[1] 1789448643.320618 punch from 0x4ce6cb to 0x4cf2cb (0xc01 bytes) 000417[1] 1789448643.338176 trunc from 0x5f8d1c to 0x35d0eb (0x29bc30 bytes) 000418[0] 1789448643.372783 punch from 0x218855 to 0x21a2be (0x1a6a bytes) 000419[1] 1789448643.400574 read 0x2c3fd6 thru 0x2c5598 (0x15c3 bytes) 000420[0] 1789448643.510566 trunc from 0x35d0ec to 0x19b0a3 (0x1c2048 bytes) 000422[0] 1789448643.563764 trunc from 0x19b0a4 to 0x7371f3 (0x59c150 bytes) 000424[0] 1789448643.622336 trunc from 0x7371f4 to 0x45f058 (0x2d819b bytes) 000425[0] 1789448643.693645 trunc from 0x45f059 to 0x287d70 (0x1d72e8 bytes) 000426[0] 1789448643.718663 read 0x09dc22 thru 0x09dd46 (0x125 bytes) 000429[0] 1789448643.845424 punch from 0x151c01 to 0x152cb6 (0x10b6 bytes) 000430[1] 1789448643.851780 read 0x1e7d66 thru 0x1e82fa (0x595 bytes) 000432[0] 1789448643.941700 mapwrite 0x0d3c26 thru 0x0d5152 (0x152d bytes) 000436[1] 1789448644.057528 read 0x0c107b thru 0x0c2c87 (0x1c0d bytes) 000437[1] 1789448644.143823 trunc from 0x287d71 to 0x36f601 (0xe7891 bytes) 000439[1] 1789448644.265784 read 0x2b3893 thru 0x2b4d50 (0x14be bytes) 000440[1] 1789448644.413471 trunc from 0x36f602 to 0x1bf353 (0x1b02ae bytes) 000441[1] 1789448644.452319 punch from 0x1856a3 to 0x187521 (0x1e7f bytes) 000444[1] 1789448644.519933 mapwrite 0x0546f6 thru 0x055a87 (0x1392 bytes) 000446[0] 1789448644.636278 mapread 0x00a8b2 thru 0x00b6ed (0xe3c bytes) 000448[0] 1789448644.727574 mapwrite 0x9a720a thru 0x9a877e (0x1575 bytes) 000449[1] 1789448645.200691 mapread 0x2b07d8 thru 0x2b1b2b (0x1354 bytes) 000450[0] 1789448645.333306 mapread 0x678916 thru 0x678ecd (0x5b8 bytes) 000451[0] 1789448645.343303 trunc from 0x9a877f to 0x371f40 (0x63683e bytes) 000454[0] 1789448645.417098 trunc from 0x371f41 to 0x4eb370 (0x179430 bytes) 000455[1] 1789448645.456696 read 0x134958 thru 0x136281 (0x192a bytes) 000456[1] 1789448645.493605 trunc from 0x4eb371 to 0x7361b4 (0x24ae44 bytes) 000457[0] 1789448645.580115 trunc from 0x7361b5 to 0x2bc9b0 (0x479804 bytes) 000458[1] 1789448645.648741 trunc from 0x2bc9b1 to 0x6798ad (0x3bcefd bytes) 000460[0] 1789448645.693739 read 0x315671 thru 0x315917 (0x2a7 bytes) 000461[0] 1789448645.807166 mapread 0x4359a1 thru 0x435cfc (0x35c bytes) 000462[1] 1789448645.868249 trunc from 0x6798ae to 0x12f927 (0x549f86 bytes) 000465[0] 1789448645.977521 punch from 0x0bd8da to 0x0bdb01 (0x228 bytes) 000467[0] 1789448646.021621 mapwrite 0x58cc68 thru 0x58cf54 (0x2ed bytes) 000469[0] 1789448646.199151 read 0x11c95a thru 0x11cc63 (0x30a bytes) 000470[0] 1789448646.238629 write 0x67464e thru 0x6764ae (0x1e61 bytes) 000474[1] 1789448646.264005 trunc from 0x6764af to 0x335912 (0x340b9c bytes) 000475[1] 1789448646.348115 trunc from 0x335913 to 0x4e3ffb (0x1ae6e9 bytes) 000476[0] 1789448646.376011 trunc from 0x4e3ffc to 0x4d3175 (0x10e86 bytes) 000477[1] 1789448646.479315 read 0x20452f thru 0x205297 (0xd69 bytes) 000478[0] 1789448646.520327 read 0x1d4a38 thru 0x1d66da (0x1ca3 bytes) 000479[0] 1789448646.548366 punch from 0x07ae77 to 0x07ae7a (0x4 bytes) 000480[0] 1789448646.603507 write 0x4a4861 thru 0x4a5d59 (0x14f9 bytes) 000484[0] 1789448646.792712 trunc from 0x4d3176 to 0x1f3c6b (0x2df50a bytes) 000485[1] 1789448646.834716 write 0x0d271d thru 0x0d2fd1 (0x8b5 bytes) 000486[0] 1789448646.872582 read 0x197fb7 thru 0x1981c4 (0x20e bytes) 000488[1] 1789448646.961514 mapwrite 0x979ff7 thru 0x97b0c5 (0x10cf bytes) 000489[0] 1789448647.075757 mapwrite 0x564dda thru 0x5669ec (0x1c13 bytes) 000491[0] 1789448647.241262 read 0x1ea24f thru 0x1ea950 (0x702 bytes) 000492[0] 1789448647.264550 write 0x745576 thru 0x7473a4 (0x1e2f bytes) 000493[0] 1789448647.291607 mapread 0x48b287 thru 0x48c2dc (0x1056 bytes) 000495[0] 1789448647.369509 punch from 0x34c9ed to 0x34d739 (0xd4d bytes) 000496[1] 1789448647.389088 read 0x203f07 thru 0x203f24 (0x1e bytes) 000497[1] 1789448647.446713 trunc from 0x97b0c6 to 0x5fdf52 (0x37d173 bytes) 000498[1] 1789448647.537366 read 0x37ccf7 thru 0x37e0e5 (0x13ef bytes) 000500[0] 1789448647.550802 trunc from 0x5fdf53 to 0x7e4c20 (0x1e6cce bytes) All operations completed A-OK! PASS 16b (33s) == sanityn test 16c: verify data consistency on ldiskfs with cache disabled (b=17397) ========================================================== 01:04:13 (1789448653) SKIP: sanityn test_16c dio on ldiskfs only SKIP 16c (4s) == sanityn test 16d: Verify DIO and buffer IO with two clients ========================================================== 01:04:17 (1789448657) 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.552773 s, 15.2 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.556873 s, 15.1 MB/s ldlm.namespaces.MGC192.168.203.160@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff9837139e5800.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff9837162cc800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff9837139e5800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff9837162cc800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff9837139e5800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff9837162cc800.lru_size=clear 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.3812 s, 9.2 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 3.03048 s, 34.6 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 2.46813 s, 42.5 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.03973 s, 20.8 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.76531 s, 12.0 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 3.19988 s, 32.8 MB/s PASS 16d (47s) == sanityn test 16e: Verify size consistency for O_DIRECT write ========================================================== 01:05:04 (1789448704) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.299294 s, 35.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161935 s, 6.5 MB/s PASS 16e (7s) == sanityn test 16f: rw sequential consistency vs drop_caches ========================================================== 01:05:11 (1789448711) TIMEOUT PASS 16f (30s) == sanityn test 16g: mmap rw sequential consistency vs drop_caches ========================================================== 01:05:41 (1789448741) TIMEOUT PASS 16g (30s) == sanityn test 16h: mmap read after truncate file ======= 01:06:11 (1789448771) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16h (8s) == sanityn test 16i: read after truncate file ============ 01:06:19 (1789448779) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16i (9s) == sanityn test 16j: race dio with buffered i/o ========== 01:06:28 (1789448788) 1+0 records in 1+0 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.49603 s, 17.5 MB/s bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.73332 s, 15.1 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.48432 s, 7.5 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.48649 s, 10.5 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.43416 s, 7.6 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.50036 s, 10.5 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.32612 s, 7.9 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.54799 s, 10.3 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.96152 s, 8.9 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 4.25072 s, 6.2 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 4.25739 s, 6.2 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.80553 s, 9.3 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.09268 s, 8.5 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.68442 s, 7.1 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.91403 s, 6.7 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.04609 s, 8.6 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.66904 s, 7.1 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 4.58733 s, 5.7 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 4.75549 s, 5.5 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK PASS 16j (45s) == sanityn test 16k: Parallel FSX and drop caches should not panic ========================================================== 01:07:13 (1789448833) + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.1 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.2 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.3 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.4 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.5 -l 67108864 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read truncating to largest ever: 0x1a7062c truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! + fsx 1 done: rc=0 Tue Sep 15 01:08:04 EDT 2026 All operations completed A-OK! All operations completed A-OK! + fsx 2 done: rc=0 Tue Sep 15 01:08:04 EDT 2026 + fsx 3 done: rc=0 Tue Sep 15 01:08:04 EDT 2026 + fsx 4 done: rc=0 Tue Sep 15 01:08:04 EDT 2026 + fsx 5 done: rc=0 Tue Sep 15 01:08:04 EDT 2026 PASS 16k (64s) == sanityn test 17: resource creation/LVB creation race ========================================================================= 01:08:17 (1789448897) fail_loc=0x8000030a PASS 17 (13s) == sanityn test 18: mmap sanity check =========================================================================================== 01:08:31 (1789448911) mmap test1: basic mmap operation (PASS, 0.25146s) mmap test2: MAP_PRIVATE not write back (PASS, 0.2034s) mmap test3: concurrent mmap ops on two nodes (PASS, 12.409s) mmap test4: c1 write to f1 from mmapped f2, c2 write to f1 from mmapped f1 (PASS, 4.7122s) mmap test5: read/write file to/from the buffer which mmapped to just this file (PASS, 0.16657s) mmap test6: check mmap write/read content on two nodes (PASS, 0.2363s) mmap test7: file i/o with an unmapped buffer (SKIPPED, 0s) mmap test8: SIGBUS for beyond file size (SKIPPED, 0s) mmap test9: SIGBUS for negative file offset (SKIPPED, 0s) mmap test10: multi-thread mmap access (PASS, 1.3646s) mmap test11: multi-thread shared mmap (PASS, 1.9371s) mmap test10: mtime not change for readonly mmap access (PASS, 1.1035s) PASS 18 (32s) == sanityn test 19: test concurrent uncached read races ========================================================================= 01:09:03 (1789448943) oleg360-server: error: get_param: param_path 'osd-*/lustre-OST*/read_cache_enable': No such file or directory pdsh@oleg360-client: oleg360-server: ssh exited with exit code 2 SKIP: sanityn test_19 not cache-capable obdfilter SKIP 19 (5s) == sanityn test 20: test extra readahead page left in cache ============================================================== 01:09:08 (1789448948) PASS 20 (7s) == sanityn test 21: Try to remove mountpoint on another dir ============================================================== 01:09:16 (1789448956) rmdir: removing directory, '/mnt/lustre/d21.sanityn' rmdir: failed to remove '/mnt/lustre/d21.sanityn': Device or resource busy rmdir: removing directory, '/mnt/lustre2/d21.sanityn' Removed mounted directory from another mountpoint, needs to be fixed PASS 21 (6s) == sanityn test 23: others should see updated atime while another read============================================================== 01:09:22 (1789448962) now is 1789448965 starting reads multiop /mnt/lustre/f23.sanityn vor20_c TMPPIPE=/tmp/multiop_open_wait_pipe.7627 new atime is 1789449027 PASS 23 (70s) == sanityn test 24a: lfs df [-ih] [path] test =================================================================================== 01:10:33 (1789449033) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 5120 3764224 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 5120 3764224 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre2 UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 115.3K 403 114.9K 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 114.5K 399 114.1K 1% /mnt/lustre[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.3K 403 114.9K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 399 114.1K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre2 UUID bytes Used Available Use% Mounted on lustre-MDT0000_UUID 2.1G 2.1M 2.1G 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3.6G 5.0M 3.6G 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3.6G 32.0M 3.6G 1% /mnt/lustre[OST:1] filesystem_summary: 7.2G 37.0M 7.2G 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118099 403 117696 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117231 399 116832 1% /mnt/lustre2[OST:1] filesystem_summary: 234914 386 234528 1% /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 5120 3764224 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.3K 403 114.9K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 399 114.1K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 118099 403 117696 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 117231 399 116832 1% /mnt/lustre[OST:1] filesystem_summary: 234914 386 234528 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118099 403 117696 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117231 399 116832 1% /mnt/lustre2[OST:1] filesystem_summary: 234914 386 234528 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 5120 3764224 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 5120 3764224 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre2 PASS 24a (8s) == sanityn test 24b: lfs df should show both filesystems ========================================================================= 01:10:40 (1789449040) PASS 24b (7s) == sanityn test 25a: change ACL on one mountpoint be seen on another ============================================================= 01:10:47 (1789449047) running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] PASS 25a (7s) == sanityn test 25b: change ACL under remote dir on one mountpoint be seen on another ========================================================== 01:10:54 (1789449054) SKIP: sanityn test_25b needs >= 2 MDTs SKIP 25b (3s) == sanityn test 26a: allow mtime to get older ============ 01:10:58 (1789449058) utime: good mknod times 1789449058 <= 1789449059 <= 1789449059 for /mnt/lustre/f26a utime: good mknod times 1789449058 <= 1789449059 <= 1789449059 for /mnt/lustre2/f26a utime: good utime mtimes 100000, atime 200000 utime: updated times for /mnt/lustre2/f26a PASS 26a (9s) == sanityn test 26b: sync mtime between ost and mds ====== 01:11:06 (1789449066) PASS 26b (8s) == sanityn test 26c: set-in-past on open file is not lost on close ========================================================== 01:11:15 (1789449075) multiop /mnt/lustre/f26c.sanityn vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7627 File: /mnt/lustre/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502398 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-09-15 01:11:17.000000000 -0400 Birth: 2026-09-15 01:11:17.000000000 -0400 File: /mnt/lustre2/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502398 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-09-15 01:11:17.000000000 -0400 Birth: 2026-09-15 01:11:17.000000000 -0400 PASS 26c (8s) SKIP: sanityn test_27 skipping excluded test 27 == sanityn test 30: recreate file race =================== 01:11:24 (1789449084) 0 PASS 30 (9s) == sanityn test 31a: voluntary cancel / blocking ast race======================================================================== 01:11:33 (1789449093) fail_loc=0x314 PASS 31a (8s) == sanityn test 31b: voluntary OST cancel / blocking ast race======================================================================== 01:11:41 (1789449101) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg360-server mds-ost sync done. sleep 5 for ZFS MDS Waiting for MDT destroys to complete fail_loc=0x314 fail_loc=0x316 fail_loc=0 fail_loc=0 Filesystem 1K-blocks Used Available Use% Mounted on 192.168.203.160@tcp:/lustre 7542784 38912 7497728 1% /mnt/lustre2 PASS 31b (23s) == sanityn test 31r: open-rename(replace) race =========== 01:12:04 (1789449124) -rw-r--r-- 1 root root 0 Sep 15 01:12 /mnt/lustre/f31r.sanityn.target fail_loc=0x1419 fail_val=3 PASS 31r (9s) == sanityn test 31s: open should not revalidate invalid dentry ========================================================== 01:12:13 (1789449133) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 1205: 43170 Terminated tail -f $tfile PASS 31s (8s) == sanityn test 31t: getattr should not revalidate invalid dentry ========================================================== 01:12:21 (1789449141) PASS 31t (9s) == sanityn test 32b: lockless i/o ======================== 01:12:31 (1789449151) SKIP: sanityn test_32b max_nolock_bytes is removed >= 2.17.53 SKIP 32b (4s) == sanityn test 32c: contention detection ================ 01:12:35 (1789449155) ldlm.namespaces.filter-lustre-OST0000_UUID.contended_locks=2 ldlm.namespaces.filter-lustre-OST0001_UUID.contended_locks=2 ldlm.namespaces.filter-lustre-OST0000_UUID.contention_events=0 ldlm.namespaces.filter-lustre-OST0001_UUID.contention_events=0 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.5217 s, 34.5 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.11506 s, 47.0 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.84059 s, 28.5 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.19682 s, 23.9 MB/s contention_seconds 4, contention_events 18 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.69754 s, 30.9 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.6621 s, 31.5 MB/s ldlm.namespaces.filter-lustre-OST0000_UUID.contention_events=0 ldlm.namespaces.filter-lustre-OST0001_UUID.contention_events=0 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.29433 s, 40.5 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.3587 s, 38.6 MB/s PASS 32c (29s) SKIP: sanityn test_33a skipping SLOW test 33a == sanityn test 33b: COS: cross create/delete, 2 clients, benchmark under remote dir ========================================================== 01:13:05 (1789449185) SKIP: sanityn test_33b Need two or more clients, have 1 SKIP 33b (4s) == sanityn test 33c: Cancel cross-MDT lock should trigger Sync-on-Lock-Cancel ========================================================== 01:13:09 (1789449189) SKIP: sanityn test_33c needs >= 2 MDTs SKIP 33c (3s) == sanityn test 33d: dependent transactions should trigger COS ========================================================== 01:13:12 (1789449192) SKIP: sanityn test_33d needs >= 2 MDTs SKIP 33d (3s) == sanityn test 33e: independent transactions shouldn't trigger COS ========================================================== 01:13:15 (1789449195) SKIP: sanityn test_33e needs >= 2 MDTs SKIP 33e (3s) == sanityn test 34: no lock timeout under IO ============= 01:13:18 (1789449198) fail_loc=0x512 fail_loc=0x512 lock should not expire writing on client1 reading on client2 success fail_loc=0x511 fail_loc=0x511 lock should expire writing on client1 reading on client2 success oleg360-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9837139e5800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9837139e5800.ost_server_uuid in FULL state after 0 sec oleg360-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9837139e5800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9837139e5800.ost_server_uuid in IDLE state after 0 sec oleg360-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9837139e5800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9837139e5800.ost_server_uuid in IDLE state after 0 sec oleg360-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9837139e5800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9837139e5800.ost_server_uuid in IDLE state after 0 sec oleg360-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9837139e5800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9837139e5800.ost_server_uuid in IDLE state after 0 sec oleg360-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9837139e5800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9837139e5800.ost_server_uuid in IDLE state after 0 sec PASS 34 (140s) == sanityn test 35: -EINTR cp_ast vs. bl_ast race does not evict client ========================================================== 01:15:38 (1789449338) fail_loc=0x80000317 Race attempt 0 Wait for 50469 50498 for 60 sec... - open/close 655 (time 1789449355.62 total 10.01 last 65.46) - open/close 1377 (time 1789449365.63 total 20.01 last 72.15) - open/close 2067 (time 1789449375.64 total 30.02 last 68.92) - open/close 2763 (time 1789449385.65 total 40.03 last 69.57) - open/close 3408 (time 1789449395.65 total 50.03 last 64.47) total: 4000 open/close in 58.90 seconds: 67.91 ops/second /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 1855: 50469 Killed createmany -o $MOUNT2/$tdir/a 4000 fail_loc=0x0 Filesystem Size Used Avail Use% Mounted on 192.168.203.160@tcp:/lustre 7.2G 38M 7.2G 1% /mnt/lustre 192.168.203.160@tcp:/lustre 7.2G 38M 7.2G 1% /mnt/lustre2 PASS 35 (73s) == sanityn test 36: handle ESTALE/open-unlink correctly == 01:16:52 (1789449412) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg360-server mds-ost sync done. Waiting for MDT destroys to complete 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.57897 s, 33.2 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7627 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(0) *** before(7499776) after_dd(7446528) after(7499776) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.43515 s, 36.5 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7627 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(1) *** before(7499776) after_dd(7444480) after(7495680) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.53643 s, 34.1 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7627 sleep 5 for ZFS MDS Waiting for MDT destroys to complete *** cycle(2) *** before(7495680) after_dd(7444480) after(7495680) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.47793 s, 35.5 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7627 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(3) *** before(7495680) after_dd(7444480) after(7499776) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.52827 s, 34.3 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7627 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(4) *** before(7499776) after_dd(7444480) after(7499776) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.67699 s, 31.3 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7627 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(5) *** before(7499776) after_dd(7448576) after(7499776) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.41917 s, 36.9 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7627 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(6) *** before(7499776) after_dd(7446528) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.44739 s, 36.2 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7627 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(7) *** before(7497728) after_dd(7448576) after(7499776) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.65774 s, 31.6 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7627 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(8) *** before(7499776) after_dd(7446528) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.57039 s, 33.4 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7627 sleep 5 for ZFS MDS Waiting for MDT destroys to complete *** cycle(9) *** before(7497728) after_dd(7446528) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.29189 s, 40.6 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7627 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(10) *** before(7497728) after_dd(7446528) after(7499776) PASS 36 (334s) == sanityn test 37: check i_size is not updated for directory on close (bug 18695) ======================================================================== 01:22:26 (1789449746) multiop /mnt/lustre/d37.sanityn vD_c TMPPIPE=/tmp/multiop_open_wait_pipe.7627 - create 1159 (time 1789449760.81 total 10.00 last 115.89) - create 2316 (time 1789449770.81 total 20.00 last 115.69) - create 3483 (time 1789449780.81 total 30.00 last 116.67) - create 4471 (time 1789449790.81 total 40.01 last 98.79) - create 5651 (time 1789449800.82 total 50.01 last 117.95) - create 6916 (time 1789449810.83 total 60.02 last 126.36) - create 8136 (time 1789449820.84 total 70.03 last 121.89) - create 9201 (time 1789449830.84 total 80.03 last 106.49) total: 10000 create in 86.90 seconds: 115.08 ops/second PASS 37 (101s) == sanityn test 39a: file mtime does not change after rename ========================================================== 01:24:07 (1789449847) repeat after cancel_lru_locks PASS 39a (8s) == sanityn test 39b: file mtime the same on clients with/out lock ========================================================== 01:24:15 (1789449855) repeat after cancel_lru_locks PASS 39b (8s) == sanityn test 39c: check truncate mtime update ================================================================================ 01:24:23 (1789449863) repeat after cancel_lru_locks PASS 39c (8s) == sanityn test 39d: sync write should update mtime ====== 01:24:31 (1789449871) fail_loc=0x411 fail_loc=0 PASS 39d (7s) SKIP: sanityn test_40a skipping ALWAYS excluded test 40a == sanityn test 40b: pdirops: open|create and others ======================================================================== 01:24:40 (1789449880) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40b (18s) == sanityn test 40c: pdirops: link and others ======================================================================== 01:24:58 (1789449898) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40c (18s) == sanityn test 40d: pdirops: unlink and others ======================================================================== 01:25:16 (1789449916) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40d (17s) == sanityn test 40e: pdirops: rename and others ======================================================================== 01:25:33 (1789449933) No conflict No conflict No conflict No conflict No conflict No conflict PASS 40e (17s) == sanityn test 41a: pdirops: create vs mkdir ======================================================================== 01:25:50 (1789449950) mkdir: cannot create directory '/mnt/lustre2/f41a.sanityn': File exists Conflict PASS 41a (12s) == sanityn test 41b: pdirops: create vs create ======================================================================== 01:26:02 (1789449962) open: File exists Conflict PASS 41b (12s) == sanityn test 41c: pdirops: create vs link ======================================================================== 01:26:14 (1789449974) link: cannot create link '/mnt/lustre2/f41c.sanityn' to '/mnt/lustre2/f41c.sanityn-2': File exists Conflict PASS 41c (13s) == sanityn test 41d: pdirops: create vs unlink ======================================================================== 01:26:27 (1789449987) Conflict PASS 41d (14s) == sanityn test 41e: pdirops: create and rename (tgt) ======================================================================== 01:26:41 (1789450001) Conflict PASS 41e (14s) == sanityn test 41f: pdirops: create and rename (src) ======================================================================== 01:26:55 (1789450015) Conflict PASS 41f (15s) == sanityn test 41g: pdirops: create vs getattr ======================================================================== 01:27:10 (1789450030) Conflict PASS 41g (14s) == sanityn test 41h: pdirops: create vs readdir ======================================================================== 01:27:24 (1789450044) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 2427: lctl set_param -n fail_loc=0 2>/dev/null || true: No such file or directory Conflict PASS 41h (14s) == sanityn test 41i: reint_open: create vs create ======== 01:27:38 (1789450058) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 41i (1031s) == sanityn test 42a: pdirops: mkdir vs mkdir ======================================================================== 01:44:49 (1789451089) mkdir: cannot create directory '/mnt/lustre2/f42a.sanityn': File exists Conflict PASS 42a (12s) == sanityn test 42b: pdirops: mkdir vs create ======================================================================== 01:45:01 (1789451101) open: File exists Conflict PASS 42b (15s) == sanityn test 42c: pdirops: mkdir vs link ======================================================================== 01:45:16 (1789451116) link: cannot create link '/mnt/lustre2/f42c.sanityn' to '/mnt/lustre2/f42c.sanityn-2': File exists Conflict PASS 42c (15s) == sanityn test 42d: pdirops: mkdir vs unlink ======================================================================== 01:45:31 (1789451131) Conflict PASS 42d (13s) == sanityn test 42e: pdirops: mkdir and rename (tgt) ======================================================================== 01:45:45 (1789451145) mv: cannot overwrite directory '/mnt/lustre2/f42e.sanityn' with non-directory Conflict PASS 42e (13s) == sanityn test 42f: pdirops: mkdir and rename (src) ======================================================================== 01:45:58 (1789451158) Conflict PASS 42f (13s) == sanityn test 42g: pdirops: mkdir vs getattr ======================================================================== 01:46:11 (1789451171) Conflict PASS 42g (13s) == sanityn test 42h: pdirops: mkdir vs readdir ======================================================================== 01:46:24 (1789451184) Conflict PASS 42h (14s) == sanityn test 43a: rmdir,mkdir doesn't return -EEXIST ======================================================================== 01:46:38 (1789451198) PASS 43a (86s) == sanityn test 43b: pdirops: unlink vs create ======================================================================== 01:48:04 (1789451284) Conflict PASS 43b (17s) == sanityn test 43c: pdirops: unlink vs link ======================================================================== 01:48:21 (1789451301) Conflict PASS 43c (15s) == sanityn test 43d: pdirops: unlink vs unlink ======================================================================== 01:48:36 (1789451316) rm: cannot remove '/mnt/lustre2/f43d.sanityn': No such file or directory Conflict PASS 43d (13s) == sanityn test 43e: pdirops: unlink and rename (tgt) ======================================================================== 01:48:49 (1789451329) Conflict PASS 43e (13s) == sanityn test 43f: pdirops: unlink and rename (src) ======================================================================== 01:49:02 (1789451342) mv: cannot stat '/mnt/lustre2/f43f.sanityn': No such file or directory Conflict PASS 43f (15s) == sanityn test 43g: pdirops: unlink vs getattr ======================================================================== 01:49:17 (1789451357) stat: cannot statx '/mnt/lustre2/f43g.sanityn': No such file or directory Conflict PASS 43g (13s) == sanityn test 43h: pdirops: unlink vs readdir ======================================================================== 01:49:30 (1789451370) Conflict PASS 43h (13s) == sanityn test 43i: pdirops: unlink vs remote mkdir ===== 01:49:44 (1789451384) SKIP: sanityn test_43i needs >= 2 MDTs SKIP 43i (3s) == sanityn test 43j: racy mkdir return EEXIST ======================================================================== 01:49:47 (1789451387) mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists PASS 43j (128s) == sanityn test 43k: unlink vs create ==================== 01:51:56 (1789451516) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 43k (1240s) == sanityn test 44a: pdirops: rename tgt vs mkdir ======================================================================== 02:12:36 (1789452756) mkdir: cannot create directory '/mnt/lustre2/f44a.sanityn': File exists Conflict Tue Sep 15 02:12:45 EDT 2026 PASS 44a (13s) == sanityn test 44b: pdirops: rename tgt vs create ======================================================================== 02:12:50 (1789452770) open: File exists Conflict PASS 44b (15s) == sanityn test 44c: pdirops: rename tgt vs link ======================================================================== 02:13:06 (1789452786) link: cannot create link '/mnt/lustre2/f44c.sanityn' to '/mnt/lustre2/f44c.sanityn-3': File exists Conflict PASS 44c (16s) == sanityn test 44d: pdirops: rename tgt vs unlink ======================================================================== 02:13:21 (1789452801) Conflict PASS 44d (14s) == sanityn test 44e: pdirops: rename tgt and rename (tgt) ======================================================================== 02:13:35 (1789452815) Conflict PASS 44e (13s) == sanityn test 44f: pdirops: rename tgt and rename (src) ======================================================================== 02:13:48 (1789452828) Conflict PASS 44f (16s) == sanityn test 44g: pdirops: rename tgt vs getattr ======================================================================== 02:14:05 (1789452845) Conflict PASS 44g (15s) == sanityn test 44h: pdirops: rename tgt vs readdir ======================================================================== 02:14:20 (1789452860) Conflict PASS 44h (14s) == sanityn test 44i: pdirops: rename tgt vs remote mkdir ========================================================== 02:14:35 (1789452875) SKIP: sanityn test_44i needs >= 2 MDTs SKIP 44i (4s) == sanityn test 45a: rename,mkdir doesn't return -EEXIST ======================================================================== 02:14:38 (1789452878) PASS 45a (131s) == sanityn test 45b: pdirops: rename src vs create ======================================================================== 02:16:50 (1789453010) Conflict PASS 45b (13s) == sanityn test 45c: pdirops: rename src vs link ======================================================================== 02:17:03 (1789453023) Conflict PASS 45c (16s) == sanityn test 45d: pdirops: rename src vs unlink ======================================================================== 02:17:19 (1789453039) rm: cannot remove '/mnt/lustre2/f45d.sanityn': No such file or directory Conflict PASS 45d (12s) == sanityn test 45e: pdirops: rename src and rename (tgt) ======================================================================== 02:17:32 (1789453052) Conflict PASS 45e (13s) == sanityn test 45f: pdirops: rename src and rename (src) ======================================================================== 02:17:45 (1789453065) mv: cannot stat '/mnt/lustre2/f45f.sanityn': No such file or directory Conflict PASS 45f (11s) == sanityn test 45g: pdirops: rename src vs getattr ======================================================================== 02:17:57 (1789453077) stat: cannot statx '/mnt/lustre2/f45g.sanityn': No such file or directory Conflict PASS 45g (11s) == sanityn test 45h: pdirops: unlink vs readdir ======================================================================== 02:18:08 (1789453088) Conflict PASS 45h (12s) == sanityn test 45i: pdirops: rename src vs remote mkdir ========================================================== 02:18:20 (1789453100) SKIP: sanityn test_45i needs >= 2 MDTs SKIP 45i (3s) == sanityn test 45j: read vs rename ====================== 02:18:23 (1789453103) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 45j (1077s) == sanityn test 46a: pdirops: link vs mkdir ======================================================================== 02:36:20 (1789454180) mkdir: cannot create directory '/mnt/lustre2/f46a.sanityn': File exists Conflict PASS 46a (10s) == sanityn test 46b: pdirops: link vs create ======================================================================== 02:36:31 (1789454191) open: File exists Conflict PASS 46b (12s) == sanityn test 46c: pdirops: link vs link ======================================================================== 02:36:42 (1789454202) link: cannot create link '/mnt/lustre2/f46c.sanityn' to '/mnt/lustre2/f46c.sanityn': File exists Conflict PASS 46c (13s) == sanityn test 46d: pdirops: link vs unlink ======================================================================== 02:36:55 (1789454215) Conflict PASS 46d (11s) == sanityn test 46e: pdirops: link and rename (tgt) ======================================================================== 02:37:07 (1789454227) Conflict PASS 46e (12s) == sanityn test 46f: pdirops: link and rename (src) ======================================================================== 02:37:19 (1789454239) Conflict PASS 46f (11s) == sanityn test 46g: pdirops: link vs getattr ======================================================================== 02:37:30 (1789454250) Conflict PASS 46g (13s) == sanityn test 46h: pdirops: link vs readdir ======================================================================== 02:37:43 (1789454263) Conflict PASS 46h (13s) == sanityn test 46i: pdirops: link vs remote mkdir ======= 02:37:56 (1789454276) SKIP: sanityn test_46i needs >= 2 MDTs SKIP 46i (4s) == sanityn test 47a: pdirops: remote mkdir vs mkdir ====== 02:38:00 (1789454280) SKIP: sanityn test_47a needs >= 2 MDTs SKIP 47a (4s) == sanityn test 47b: pdirops: remote mkdir vs create ===== 02:38:04 (1789454284) SKIP: sanityn test_47b needs >= 2 MDTs SKIP 47b (3s) == sanityn test 47c: pdirops: remote mkdir vs link ======= 02:38:07 (1789454287) SKIP: sanityn test_47c needs >= 2 MDTs SKIP 47c (3s) == sanityn test 47d: pdirops: remote mkdir vs unlink ===== 02:38:11 (1789454291) SKIP: sanityn test_47d needs >= 2 MDTs SKIP 47d (3s) == sanityn test 47e: pdirops: remote mkdir and rename (tgt) ========================================================== 02:38:14 (1789454294) SKIP: sanityn test_47e needs >= 2 MDTs SKIP 47e (3s) == sanityn test 47f: pdirops: remote mkdir and rename (src) ========================================================== 02:38:17 (1789454297) SKIP: sanityn test_47f needs >= 2 MDTs SKIP 47f (3s) == sanityn test 47g: pdirops: remote mkdir vs getattr ==== 02:38:20 (1789454300) SKIP: sanityn test_47g needs >= 2 MDTs SKIP 47g (3s) == sanityn test 50: osc lvb attrs: enqueue vs. CP AST ======================================================================== 02:38:23 (1789454303) 10+0 records in 10+0 records out 10240 bytes (10 kB, 10 KiB) copied, 0.182618 s, 56.1 kB/s fail_loc=0x410 fail_loc=0x0 PASS 50 (12s) == sanityn test 51a: layout lock: refresh layout should work ========================================================== 02:38:36 (1789454316) 0+1 records in 0+1 records out 158 bytes copied, 0.0474054 s, 3.3 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 3690: kill: (218857) - No such process PASS 51a (11s) == sanityn test 51b: layout lock: glimpse should be able to restart if layout changed ========================================================== 02:38:46 (1789454326) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0107002 s, 95.7 kB/s fail_loc=0x1404 fail_val=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0896793 s, 11.7 MB/s 3145728 PASS 51b (21s) == sanityn test 51c: layout lock: IT_LAYOUT blocked and correct layout can be returned ========================================================== 02:39:07 (1789454347) fail_loc=0x172 Setting layout to have 2 stripes ... 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0341728 s, 30.0 kB/s PASS 51c (11s) == sanityn test 51d: layout lock: losing layout lock should clean up memory map region ========================================================== 02:39:19 (1789454359) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0677385 s, 15.5 MB/s Before revoking layout lock: 1024 KB mapped PASS 51d (9s) == sanityn test 51e: lfs getstripe does not break leases, part 2 ========================================================== 02:39:27 (1789454367) /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 2453 0x995 0x240000400 write lease(2) has applied. /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 2453 0x995 0x240000400 read lease(1) has applied. PASS 51e (9s) == sanityn test 54: rename locking ======================= 02:39:37 (1789454377) llite.lustre-ffff9837139e5800.xattr_cache=1 llite.lustre-ffff9837162cc800.xattr_cache=1 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505015 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-15 02:39:39.000000000 -0400 Modify: 2026-09-15 02:39:39.000000000 -0400 Change: 2026-09-15 02:39:39.000000000 -0400 Birth: 2026-09-15 02:39:39.000000000 -0400 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000154 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505018 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-15 02:39:45.000000000 -0400 Modify: 2026-09-15 02:39:46.000000000 -0400 Change: 2026-09-15 02:39:46.000000000 -0400 Birth: 2026-09-15 02:39:46.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505021 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-15 02:39:52.000000000 -0400 Modify: 2026-09-15 02:39:52.000000000 -0400 Change: 2026-09-15 02:39:52.000000000 -0400 Birth: 2026-09-15 02:39:52.000000000 -0400 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505020 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-15 02:39:52.000000000 -0400 Modify: 2026-09-15 02:39:53.000000000 -0400 Change: 2026-09-15 02:39:53.000000000 -0400 Birth: 2026-09-15 02:39:52.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000154 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505023 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-15 02:39:58.000000000 -0400 Modify: 2026-09-15 02:39:59.000000000 -0400 Change: 2026-09-15 02:39:59.000000000 -0400 Birth: 2026-09-15 02:39:58.000000000 -0400 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505024 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-15 02:39:58.000000000 -0400 Modify: 2026-09-15 02:39:59.000000000 -0400 Change: 2026-09-15 02:39:59.000000000 -0400 Birth: 2026-09-15 02:39:58.000000000 -0400 PASS 54 (31s) == sanityn test 55a: rename vs unlink target dir ========= 02:40:08 (1789454408) fail_loc=0x80000156 mv: cannot move '/mnt/lustre/d55a.sanityn/d1/d2' to '/mnt/lustre/d55a.sanityn/d3/d2': No such file or directory PASS 55a (12s) == sanityn test 55b: rename vs unlink source dir ========= 02:40:20 (1789454420) fail_loc=0x80000156 mv: cannot stat '/mnt/lustre/d55b.sanityn/d1/d2': No such file or directory PASS 55b (12s) == sanityn test 55c: rename vs unlink orphan target dir == 02:40:32 (1789454432) fail_loc=0x156 mv: cannot move '/mnt/lustre/d55c.sanityn/d1/d2' to '/mnt/lustre/d55c.sanityn/d3/d2': No such file or directory PASS 55c (17s) == sanityn test 55d: rename file vs link ================= 02:40:49 (1789454449) fail_loc=0x155 mv: '/mnt/lustre/d55d.sanityn/f1' and '/mnt/lustre/d55d.sanityn/d55d.sanityn/f1' are the same file PASS 55d (14s) == sanityn test 55e: rename race AB/BA under the same parent dir ========================================================== 02:41:03 (1789454463) SKIP: sanityn test_55e needs >= 2 MDTs SKIP 55e (3s) == sanityn test 55f: rename: (P1/A -> P2/B) race with (P2/B -> P1/A) ========================================================== 02:41:06 (1789454466) fail_loc=0x155 We survived after P1A_P2B/P2B_P1A race test fail_loc=0x0 PASS 55f (19s) == sanityn test 55g: rename: race with trylock =========== 02:41:26 (1789454486) mdt.lustre-MDT0000.enable_rename_trylock=1 fail_loc=0x155 mv: cannot move '/mnt/lustre2/d55g.sanityn/b1/b2' to a subdirectory of itself, '/mnt/lustre2/d55g.sanityn/a1/a2/a3/a4/b2' We survived race with trylock test fail_loc=0x0 PASS 55g (25s) == sanityn test 56a: test llverdev with single large stripe ========================================================== 02:41:51 (1789454511) run_llverdev /mnt/lustre/f56a.sanityn -p -s 16000000 -c 4k llverdev: /mnt/lustre/f56a.sanityn is 16777216000000 bytes (15625 GB) in size timestamp: 1789454513 chunksize: 4096 size: 16777216000000 write offset: 16384000000kB 0.604347 MB/s read offset: 16384000000kB inf MB/s llverdev: data verified successfully PASS 56a (109s) == sanityn test 56b: test llverdev and partial verify of wide stripe file ========================================================== 02:43:40 (1789454620) run_llverdev /mnt/lustre/f56b.sanityn -p -s 32000000 -c 4k llverdev: /mnt/lustre/f56b.sanityn is 33554432000000 bytes (31250 GB) in size timestamp: 1789454626 chunksize: 4096 size: 33554432000000 write offset: 32768000000kB 1.02583 MB/s read offset: 32768000000kB inf MB/s llverdev: data verified successfully PASS 56b (133s) == sanityn test 60: Verify data_version behaviour ======== 02:45:53 (1789454753) PASS 60 (7s) == sanityn test 70a: cd directory && rm directory ======== 02:46:00 (1789454760) PASS 70a (8s) == sanityn test 70b: remove files after calling rm_entry ========================================================== 02:46:08 (1789454768) PASS 70b (9s) == sanityn test 71a: correct file map just after write operation is finished ========================================================== 02:46:17 (1789454777) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0101086 s, 6.5 MB/s SKIP: sanityn test_71a ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71a (3s) == sanityn test 71b: check fiemap support for stripecount > 1 ========================================================== 02:46:21 (1789454781) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.023077 s, 2.8 MB/s SKIP: sanityn test_71b ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71b (4s) == sanityn test 71c: check FIEMAP_EXTENT_LAST flag with different extents number ========================================================== 02:46:24 (1789454784) SKIP: sanityn test_71c support only ldiskfs ost SKIP 71c (4s) == sanityn test 71d: fiemap corruption test with fm_extent_count=0 ========================================================== 02:46:28 (1789454788) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0128482 s, 5.1 MB/s SKIP: sanityn test_71d ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71d (3s) == sanityn test 72: getxattr/setxattr cache should be consistent between nodes ========================================================== 02:46:31 (1789454791) llite.lustre-ffff9837139e5800.xattr_cache=1 llite.lustre-ffff9837162cc800.xattr_cache=1 getfattr: Removing leading '/' from absolute path names user.attr1="value1" getfattr: Removing leading '/' from absolute path names user.attr1="value2" getfattr: Removing leading '/' from absolute path names getfattr: Removing leading '/' from absolute path names b362740ad371eae580e7f85452e8830b - 870e072da1ebfc20ea8c0cd3d6bdf3b6 - PASS 72 (7s) == sanityn test 73: getxattr should not cause xattr lock cancellation ========================================================== 02:46:38 (1789454798) llite.lustre-ffff9837139e5800.xattr_cache=1 llite.lustre-ffff9837162cc800.xattr_cache=1 getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" PASS 73 (9s) == sanityn test 74: flock deadlock: different mounts ======================================================================== 02:46:47 (1789454807) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0226726 s, 45.2 kB/s 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0226593 s, 45.2 kB/s 231666: get lock2 231667: get lock1 231667: done 231667: get lock2 231666: done 231666: get lock1 fcntl cmd 7 failed: Resource deadlock avoided 231666: cannot get lock1: Resource deadlock avoided 231667: done 231667: exit rc=0 231666: done 231666: exit rc=0 PASS 74 (12s) == sanityn test 75: osc: upcall after unuse lock============================================================================= 02:46:59 (1789454819) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0621458 s, 33.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.080148 s, 13.1 MB/s fail_loc=0x31d /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 4373: 232249 Killed stat -c %s $DIR1/$tfile fail_loc=0 PASS 75 (14s) == sanityn test 76: Verify MDT open_files listing ======== 02:47:13 (1789454833) opening files: ........ closing files: ........ PASS 76 (95s) == sanityn test 77a: check FIFO NRS policy =============== 02:48:48 (1789454928) ost.OSS.ost.nrs_policies=fifo ost.OSS.ost_create.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_out.nrs_policies=fifo ost.OSS.ost_seq.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.464489 s, 36.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0637489 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0599195 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127768 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125229 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108431 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190766 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186622 s, 5.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201362 s, 5.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170215 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171385 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.228744 s, 4.6 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9837139e5800.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.230788 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194733 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192146 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.218602 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.244235 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0269008 s, 39.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00322817 s, 325 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0743497 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00827347 s, 127 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00716015 s, 146 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00669058 s, 157 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00375169 s, 279 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0110429 s, 95.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136586 s, 7.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118556 s, 8.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101937 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00799405 s, 131 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138052 s, 76.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170622 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100986 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0510115 s, 20.6 MB/s removed '/mnt/lustre/d77a.sanityn/nrs_r_oleg360-client.virtnet' removed '/mnt/lustre/d77a.sanityn/nrs_w_oleg360-client.virtnet' removed directory '/mnt/lustre/d77a.sanityn' PASS 77a (10s) == sanityn test 77b: check CRR-N NRS policy ============== 02:48:58 (1789454938) ost.OSS.ost.nrs_policies=crrn ost.OSS.ost_create.nrs_policies=crrn ost.OSS.ost_io.nrs_policies=crrn ost.OSS.ost_out.nrs_policies=crrn ost.OSS.ost_seq.nrs_policies=crrn ost.OSS.ost.nrs_crrn_quantum=1 ost.OSS.ost_create.nrs_crrn_quantum=1 ost.OSS.ost_io.nrs_crrn_quantum=1 ost.OSS.ost_out.nrs_crrn_quantum=1 ost.OSS.ost_seq.nrs_crrn_quantum=1 policy: crr-n, crrn_quantum 1 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.474554 s, 35.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103961 s, 10.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191009 s, 5.5 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194616 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.214688 s, 4.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155124 s, 6.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237265 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151917 s, 6.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146977 s, 7.1 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221512 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199237 s, 5.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0827668 s, 12.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.299964 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113812 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.248893 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197388 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.246308 s, 4.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9837139e5800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9837139e5800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0423182 s, 24.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0266772 s, 39.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0358261 s, 29.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134249 s, 78.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0600711 s, 17.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00928864 s, 113 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0908088 s, 11.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0585105 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143762 s, 72.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0593278 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0523955 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00934816 s, 112 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0190198 s, 55.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127799 s, 82.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142434 s, 73.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0956703 s, 11.0 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg360-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg360-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost.nrs_crrn_quantum=64 ost.OSS.ost_create.nrs_crrn_quantum=64 ost.OSS.ost_io.nrs_crrn_quantum=64 ost.OSS.ost_out.nrs_crrn_quantum=64 ost.OSS.ost_seq.nrs_crrn_quantum=64 policy: crr-n, crrn_quantum 64 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.42027 s, 39.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0386534 s, 27.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0998341 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0786181 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102914 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11067 s, 9.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203271 s, 5.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171123 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.204301 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156841 s, 6.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195562 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209973 s, 5.0 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9837139e5800.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164846 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222978 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194406 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.214281 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.289921 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0534225 s, 19.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9837139e5800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9837139e5800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.017883 s, 58.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00226313 s, 463 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152548 s, 68.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0042185 s, 249 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116982 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146858 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00628149 s, 167 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0453918 s, 23.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00913242 s, 115 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0361131 s, 29.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0251493 s, 41.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0429023 s, 24.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0698507 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0344245 s, 30.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117361 s, 8.9 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg360-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg360-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77b (11s) == sanityn test 77c: check ORR NRS policy ================ 02:49:09 (1789454949) ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_orr_quantum=1 ost.OSS.ost_io.nrs_orr_offset_type=physical ost.OSS.ost_io.nrs_orr_supported=reads policy: orr, orr_quantum 1, orr_offset_type physical, orr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.495396 s, 33.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0607584 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0730854 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.049528 s, 21.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141817 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15022 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130021 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122825 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0842772 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0713097 s, 14.7 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9837139e5800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.243369 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.289257 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.240786 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.278171 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.316343 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.274608 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.295843 s, 3.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0437905 s, 23.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0439597 s, 23.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0696871 s, 15.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0391015 s, 26.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0204326 s, 51.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0352349 s, 29.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out ldlm.namespaces.lustre-OST0000-osc-ffff9837139e5800.lock_unused_count=7 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0317814 s, 33.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0508125 s, 20.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0476573 s, 22.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0325167 s, 32.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0500701 s, 20.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0584305 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.049639 s, 21.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0586302 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0642327 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0593323 s, 17.7 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg360-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg360-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=writes ost.OSS.ost_io.nrs_orr_quantum=64 policy: orr, orr_quantum 64, orr_offset_type physical, orr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.459258 s, 36.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0998391 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0267714 s, 39.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116729 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135839 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205819 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128569 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.233725 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.247382 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.287829 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188114 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200166 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.281073 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.229931 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.305747 s, 3.4 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9837139e5800.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.372911 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.477521 s, 2.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0830678 s, 12.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0703937 s, 14.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0683174 s, 15.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0684424 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0660105 s, 15.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0904072 s, 11.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0686653 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0967783 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0755278 s, 13.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0942839 s, 11.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0709008 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101094 s, 10.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105171 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107944 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127276 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127112 s, 8.2 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg360-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg360-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=reads_and_writes ost.OSS.ost_io.nrs_orr_offset_type=logical policy: orr, orr_quantum 64, orr_offset_type logical, orr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.506135 s, 33.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0795938 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0898556 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0945382 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116638 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153219 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174172 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0809503 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10522 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0872942 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175309 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.349517 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.326159 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.262494 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.327398 s, 3.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9837139e5800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9837139e5800.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.403711 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.385603 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00969227 s, 108 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206677 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13384 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146056 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00380014 s, 276 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109785 s, 9.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129394 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0969821 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125619 s, 8.3 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0952581 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0804849 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0646678 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0694401 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0515581 s, 20.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0338247 s, 31.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0259669 s, 40.4 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg360-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg360-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77c (15s) == sanityn test 77d: check TRR nrs policy ================ 02:49:24 (1789454964) ost.OSS.ost_io.nrs_policies=trr ost.OSS.ost_io.nrs_trr_quantum=1 ost.OSS.ost_io.nrs_trr_offset_type=physical ost.OSS.ost_io.nrs_trr_supported=reads policy: trr, trr_quantum 1, trr_offset_type physical, trr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.463251 s, 36.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133656 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113515 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163486 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177427 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178274 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199701 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174504 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186059 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216913 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148655 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164064 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167373 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209439 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.238028 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188765 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.218307 s, 4.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0802075 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0390453 s, 26.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0886973 s, 11.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0863904 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0293191 s, 35.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0303914 s, 34.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00246309 s, 426 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106439 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0090606 s, 116 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0810916 s, 12.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0115539 s, 90.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12513 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123923 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0312748 s, 33.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0194293 s, 54.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128761 s, 8.1 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg360-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg360-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=writes ost.OSS.ost_io.nrs_trr_quantum=64 policy: trr, trr_quantum 64, trr_offset_type physical, trr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.551387 s, 30.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0580686 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.055832 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103051 s, 10.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105814 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112693 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0614834 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0247456 s, 42.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12616 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179893 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20497 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132557 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173894 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160149 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0812191 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137512 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.217132 s, 4.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.041814 s, 25.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0389774 s, 26.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0696597 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0450233 s, 23.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0499515 s, 21.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0743911 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0553199 s, 19.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0845495 s, 12.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0839676 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.056069 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00591171 s, 177 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0597907 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0661412 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0620238 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0761105 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0821002 s, 12.8 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg360-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg360-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=reads_and_writes ost.OSS.ost_io.nrs_trr_offset_type=logical policy: trr, trr_quantum 64, trr_offset_type logical, trr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.497889 s, 33.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0847171 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10945 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0912848 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128109 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146854 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.259107 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128153 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.31507 s, 3.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165926 s, 6.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197751 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.285758 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201607 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.348049 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150382 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.249627 s, 4.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9837139e5800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9837139e5800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.314261 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0530659 s, 19.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0476435 s, 22.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00833817 s, 126 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0559206 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112471 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0452733 s, 23.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00817593 s, 128 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0895507 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00791832 s, 132 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00748064 s, 140 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145906 s, 71.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00903891 s, 116 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00954895 s, 110 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153755 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0721204 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0821853 s, 12.8 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg360-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg360-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77d (16s) == sanityn test 77e: check TBF NID nrs policy ============ 02:49:40 (1789454980) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_tbf_rule=start localhost nid={0@lo} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start clients nid={192.168.203.60@tcp} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start others nid={*.*.*.*@tcp} rate=50 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.38471 s, 43.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0480556 s, 21.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0773817 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119314 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157838 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23781 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202485 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191164 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.239798 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.287322 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.26791 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.260508 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.317987 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.299396 s, 3.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.322989 s, 3.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.317238 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.278857 s, 3.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9837139e5800.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff9837139e5800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176144 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132861 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125593 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129681 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135221 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138495 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140245 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207674 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142827 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155334 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147737 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157807 s, 6.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198512 s, 5.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162905 s, 6.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169293 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190108 s, 5.5 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg360-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg360-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change localhost rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change clients rate=101 ost.OSS.ost_io.nrs_tbf_rule=change others rate=51 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.464556 s, 36.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0315801 s, 33.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.064797 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0309931 s, 33.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105431 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183467 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149532 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.246125 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203724 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.236796 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.250797 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.301387 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.28806 s, 3.6 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9837139e5800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.32682 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.362679 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.36017 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.429993 s, 2.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9837139e5800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0124597 s, 84.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00426304 s, 246 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143294 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125146 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0499011 s, 21.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138445 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133677 s, 7.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134981 s, 77.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137388 s, 76.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00674697 s, 155 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0598383 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0610892 s, 17.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015661 s, 67.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0278573 s, 37.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0680555 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00624527 s, 168 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg360-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg360-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop localhost ost.OSS.ost_io.nrs_tbf_rule=stop clients ost.OSS.ost_io.nrs_tbf_rule=stop others 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.396649 s, 42.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0429777 s, 24.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.071356 s, 14.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0757481 s, 13.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.059876 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0722824 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0774733 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0907207 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10302 s, 10.2 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9837139e5800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113045 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121449 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.224585 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223047 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.239666 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.272652 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.257637 s, 4.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9837139e5800.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.322135 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170257 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107908 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0841746 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0209539 s, 50.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121723 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00341001 s, 307 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00762857 s, 137 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0550713 s, 19.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0227984 s, 46.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0066188 s, 158 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0112271 s, 93.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00755397 s, 139 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0567471 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0103901 s, 101 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0684985 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0114948 s, 91.2 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg360-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg360-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.559558 s, 30.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0457449 s, 22.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0881051 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0636426 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142277 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135838 s, 7.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9837139e5800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22154 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179415 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.230185 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.253765 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.258505 s, 4.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.261727 s, 4.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.283923 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.243719 s, 4.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.325506 s, 3.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.287803 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.31292 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0665893 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.020913 s, 50.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129553 s, 80.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0300617 s, 34.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0311041 s, 33.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0396371 s, 26.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0971362 s, 10.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0493217 s, 21.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0970603 s, 10.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0539742 s, 19.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0564854 s, 18.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0945816 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0627044 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100523 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0744854 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117295 s, 8.9 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg360-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg360-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' PASS 77e (23s) == sanityn test 77f: check TBF JobID nrs policy ========== 02:50:04 (1789455004) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start runas jobid={iozone.500 dd.500 tiotest.500} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start iozone_runas jobid={iozone.500} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg360-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.502412 s, 33.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0401379 s, 26.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.076351 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0265472 s, 39.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165152 s, 6.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223573 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180879 s, 5.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1905 s, 5.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198437 s, 5.3 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.246864 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.306442 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.342225 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.345469 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.241546 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.335731 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.27545 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.375783 s, 2.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132163 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132316 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123717 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0821561 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121363 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0495938 s, 21.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143314 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0936529 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0832285 s, 12.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0649963 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17588 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15698 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0831115 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0839994 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168422 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0933955 s, 11.2 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg360-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg360-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change runas rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change iozone_runas rate=101 ost.OSS.ost_io.nrs_tbf_rule=change dd_runas rate=51 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg360-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.622067 s, 27.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0741897 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120946 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124555 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223194 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164687 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210294 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.253679 s, 4.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.360615 s, 2.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.34317 s, 3.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups:1+0 records in 1+0 records out 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.313341 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.305202 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.306397 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.360394 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.389908 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.362825 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.451409 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0619681 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00768587 s, 136 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0514292 s, 20.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00532537 s, 197 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113075 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0890562 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127091 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0767469 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0703301 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00261196 s, 401 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138765 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0534147 s, 19.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00750037 s, 140 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148215 s, 7.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9837139e5800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0651106 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0443061 s, 23.7 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg360-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg360-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop runas ost.OSS.ost_io.nrs_tbf_rule=stop iozone_runas ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg360-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.596711 s, 28.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0876202 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102031 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0733875 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.241981 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145703 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155249 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.226046 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184844 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.35894 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.346701 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.324285 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.399473 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.257991 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.27903 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.250322 s, 4.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9837139e5800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9837139e5800.lock_unused_count=4 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.410776 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0400116 s, 26.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0651239 s, 16.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0448845 s, 23.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0394994 s, 26.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0234416 s, 44.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0662011 s, 15.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0472212 s, 22.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00899678 s, 117 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0273659 s, 38.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114359 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0366532 s, 28.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110901 s, 9.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143719 s, 73.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121821 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141147 s, 74.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9837139e5800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162033 s, 64.7 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg360-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg360-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_policies=fifo running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg360-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.519962 s, 32.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.076362 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0532467 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132361 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0824206 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0967795 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123653 s, 8.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136622 s, 7.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176635 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.280575 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.255483 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23059 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.214431 s, 4.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165039 s, 6.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172953 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207337 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105673 s, 9.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9837139e5800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9837139e5800.lock_unused_count=1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109002 s, 9.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.074749 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126228 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0992324 s, 10.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0916287 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119423 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11193 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135641 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0774349 s, 13.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109806 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0958979 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0986263 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133863 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109636 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112629 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161664 s, 6.5 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg360-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg360-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' PASS 77f (26s) == sanityn test 77g: Change TBF type directly ============ 02:50:31 (1789455031) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 ost.OSS.ost_io.nrs_policies=fifo PASS 77g (13s) == sanityn test 77h: Wrong policy name should report error, not LBUG ========================================================== 02:50:43 (1789455043) oleg360-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=abc: No such file or directory oleg360-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='abc': No such file or directory pdsh@oleg360-client: oleg360-server: ssh exited with exit code 2 oleg360-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc: Invalid argument oleg360-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc': Invalid argument pdsh@oleg360-client: oleg360-server: ssh exited with exit code 22 oleg360-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf reg abc: Invalid argument oleg360-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf reg abc': Invalid argument pdsh@oleg360-client: oleg360-server: ssh exited with exit code 22 oleg360-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc efg: Invalid argument oleg360-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc efg': Invalid argument pdsh@oleg360-client: oleg360-server: ssh exited with exit code 22 PASS 77h (13s) == sanityn test 77i: Change rank of TBF rule ============= 02:50:56 (1789455056) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start before jobid={jobid} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start after jobid={jobid} rate=1000 rank=default ost.OSS.ost_io.nrs_tbf_rule=start target jobid={jobid} rate=1000 rank=after Move before itself ost.OSS.ost_io.nrs_tbf_rule=change target rank=target Move to higher rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=before Move to lower rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=after Move before default ost.OSS.ost_io.nrs_tbf_rule=change target rank=default ost.OSS.ost_io.nrs_policies=fifo PASS 77i (28s) == sanityn test 77j: check TBF-OPCode NRS policy ========= 02:51:24 (1789455084) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf opcode ost.OSS.ost_io.nrs_tbf_rule=start ost_r opcode={ost_read} rate=5 ost.OSS.ost_io.nrs_tbf_rule=start ost_w opcode={ost_write} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.025 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0334769 s, 31.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.067026 s, 15.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0539474 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0594133 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0969288 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137002 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131582 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0380709 s, 27.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127338 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16431 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161276 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15838 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.278039 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.319459 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.244232 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.501589 s, 2.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9837139e5800.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192891 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.345294 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.265418 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.28699 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.248356 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.318697 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.281953 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.332861 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.267015 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.455507 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.353163 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.478886 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.356263 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.342309 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.372939 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.429102 s, 2.4 MB/s removed '/mnt/lustre/d77j.sanityn/nrs_w_oleg360-client.virtnet' removed '/mnt/lustre/d77j.sanityn/nrs_r_oleg360-client.virtnet' removed directory '/mnt/lustre/d77j.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 5 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.5484 s, 9.9 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.6317 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77j.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_r ost.OSS.ost_io.nrs_tbf_rule=stop ost_w ost.OSS.ost_io.nrs_policies=fifo PASS 77j (73s) == sanityn test 77ja: check TBF-UID/GID NRS policy ======= 02:52:37 (1789455157) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg360-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg360-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg360-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg360-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg360-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.33638 s, 12.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141101 s, 7.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170252 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189129 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173069 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165548 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122311 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131877 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.238625 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0877935 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140676 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0785419 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.358928 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.347669 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.309719 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.256211 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.395569 s, 2.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0959954 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0742453 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0743652 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0796635 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0678632 s, 15.5 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0524171 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102242 s, 10.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0662817 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0966541 s, 10.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131245 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148363 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116114 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.09664 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121971 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105188 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139235 s, 7.5 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg360-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg360-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8918 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.0319 s, 5.2 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg360-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg360-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg360-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg360-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg360-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.3641 s, 12.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.093902 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0888612 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0318263 s, 32.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19797 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.245366 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162548 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.258886 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.241827 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.323816 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.307748 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.361765 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.315461 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168001 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0573538 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131725 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.469322 s, 2.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0974724 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0688011 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0978144 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125201 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106097 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0987319 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13619 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0868729 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151333 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148422 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11228 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16548 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124669 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121046 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125814 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12762 s, 8.2 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg360-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg360-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.2407 s, 5.2 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8202 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77ja (160s) == sanityn test 77jb: check TBF-UID/GID NRS policy on files that don't belong to us ========================================================== 02:55:17 (1789455317) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg360-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg360-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg360-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg360-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg360-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.33769 s, 12.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11313 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116149 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220974 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182038 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.280415 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.321882 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.268932 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.322365 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.310061 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.284013 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.309763 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.343907 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.345256 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.39027 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.378131 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.437578 s, 2.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0606681 s, 17.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0463659 s, 22.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0673181 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0667895 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0445872 s, 23.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.040162 s, 26.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0818684 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0538375 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0873805 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.089168 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0855699 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0906706 s, 11.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0808632 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.062815 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0913301 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0838475 s, 12.5 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg360-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg360-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.331 s, 5.2 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.821 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg360-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg360-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg360-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg360-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg360-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.30363 s, 12.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0801465 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101214 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157827 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0421252 s, 24.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0723015 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178479 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0595 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0755655 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105036 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0908291 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116124 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193107 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188742 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.246663 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145369 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126877 s, 8.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0629421 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0623961 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0735003 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0710505 s, 14.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0724794 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.046048 s, 22.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0897213 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0810046 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0623989 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0811367 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0907608 s, 11.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0892253 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0822951 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10997 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0908614 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121201 s, 8.7 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg360-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg360-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.3512 s, 5.2 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.952 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77jb (162s) == sanityn test 77k: check TBF policy with UID/GID/JobID/OPCode expression ========================================================== 02:58:00 (1789455480) ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg360-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.673658 s, 24.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0826506 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117758 s, 8.9 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0999871 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0537988 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155905 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.252776 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.294617 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205317 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.334779 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.254 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.323537 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182797 s, 5.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9837139e5800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9837139e5800.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.291376 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.282917 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.373046 s, 2.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.507995 s, 2.1 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9837139e5800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01408 s, 74.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0625685 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0279358 s, 37.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165358 s, 63.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0565333 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0368565 s, 28.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0029202 s, 359 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0114478 s, 91.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139501 s, 75.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0907396 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0777419 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0114719 s, 91.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153108 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153203 s, 68.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0741264 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0276431 s, 37.9 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg360-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg360-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.4623 s, 10.0 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.93799 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.203.60@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.203.60@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.753975 s, 22.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0539666 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0861904 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172219 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21308 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20941 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0693921 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182392 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.38311 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.316755 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.298785 s, 3.5 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.381573 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.388504 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.415497 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.310244 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0521497 s, 20.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135441 s, 7.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9837139e5800.lock_unused_count=3 ldlm.namespaces.lustre-OST0001-osc-ffff9837139e5800.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127673 s, 8.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0120992 s, 86.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00687578 s, 153 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0666855 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0198125 s, 52.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016687 s, 62.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126914 s, 82.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136317 s, 76.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0194746 s, 53.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0992772 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109274 s, 9.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9837139e5800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0826308 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0202022 s, 51.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0880499 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184898 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143456 s, 73.1 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg360-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg360-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.4407 s, 10.0 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.99455 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.203.60@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg360-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.596484 s, 28.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103855 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100736 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129902 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134913 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.095011 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0947858 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188154 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.257199 s, 4.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184499 s, 5.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.271821 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.235839 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119916 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.268993 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.374317 s, 2.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9837139e5800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9837139e5800.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.342051 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.365315 s, 2.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0826033 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0663461 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0699646 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0699928 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0344703 s, 30.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0801793 s, 13.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0449031 s, 23.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0601788 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0530204 s, 19.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134736 s, 7.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.09639 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122996 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0752469 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0909554 s, 11.5 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12397 s, 8.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0871993 s, 12.0 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg360-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg360-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.63718 s, 10.9 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.18147 s, 20.2 MB/s Read runtime is 7 s, speed is 14.285714 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg360-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.730149 s, 23.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0671349 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0771141 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18076 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213618 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134343 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169234 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0723491 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.228961 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.245395 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.246379 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130845 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.315885 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129447 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131011 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0791913 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129598 s, 8.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.056143 s, 18.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0692229 s, 15.1 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0539635 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0605906 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0598508 s, 17.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.070119 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0907583 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0715281 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100513 s, 10.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10984 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114549 s, 9.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120634 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146858 s, 7.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146533 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120537 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123687 s, 8.5 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg360-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg360-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.974 s, 9.6 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.91704 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.92567 s, 10.6 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.98147 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg360-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.00676 s, 16.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119885 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0982081 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154938 s, 6.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.274535 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.287605 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222301 s, 4.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.262038 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.306757 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.25913 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.445676 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.434931 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.503038 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.581466 s, 1.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.51454 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.612113 s, 1.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.598374 s, 1.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0603209 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11904 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0673002 s, 15.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11159 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0703576 s, 14.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0734444 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0771644 s, 13.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0797214 s, 13.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0742959 s, 14.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0820523 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0872079 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0816476 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.086767 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0944466 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0926046 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107682 s, 9.7 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg360-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg360-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.1618 s, 5.2 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.6572 s, 5.3 MB/s Read runtime is 20 s, speed is 5.000000 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_ug ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg360-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.781232 s, 21.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0636425 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0995267 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0581333 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105718 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132217 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143893 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0629209 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124347 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162288 s, 6.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9837139e5800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9837139e5800.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.272596 s, 3.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.2624 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200647 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.231476 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242808 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.269705 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.322112 s, 3.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0393936 s, 26.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0478597 s, 21.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.067379 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.036355 s, 28.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0522785 s, 20.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.047764 s, 22.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0336084 s, 31.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0705685 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0501313 s, 20.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0575113 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.068595 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0577756 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0865439 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0689608 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0737192 s, 14.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9837139e5800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0792503 s, 13.2 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg360-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg360-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.06137 s, 11.6 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.99707 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg360-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.656391 s, 25.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0444019 s, 23.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.066447 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0699501 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0754888 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136124 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105218 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163765 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180062 s, 5.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200917 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172004 s, 6.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.229532 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208509 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.268451 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.257148 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.228187 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.277927 s, 3.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.075187 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0759347 s, 13.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.065435 s, 16.0 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0911235 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0916273 s, 11.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.062396 s, 16.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.085962 s, 12.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0601232 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102045 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0559844 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0712427 s, 14.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0805123 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109755 s, 9.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113182 s, 9.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149775 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0915476 s, 11.5 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg360-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg360-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.4345 s, 9.2 MB/s Write runtime is 13 s, speed is 7.692307 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.98905 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.0414 s, 9.5 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.91618 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_policies=fifo PASS 77k (450s) == sanityn test 77kb: Check different granular TBF type combination: jobid+opcode ========================================================== 03:05:30 (1789455930) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/nrs_r_oleg360-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.512416 s, 32.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0674225 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111495 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.246329 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.214151 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.312964 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.232932 s, 4.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.301943 s, 3.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.322056 s, 3.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.270486 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.312818 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.340812 s, 3.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.332284 s, 3.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.351881 s, 3.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.420802 s, 2.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.365664 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.596964 s, 1.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9837139e5800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9837139e5800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182075 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139325 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242802 s, 4.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141805 s, 7.4 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.278665 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140182 s, 7.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179845 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194187 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132417 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127464 s, 8.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0509066 s, 20.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121658 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137774 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109926 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128339 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100984 s, 10.4 MB/s removed '/mnt/lustre/d77kb.sanityn/nrs_w_oleg360-client.virtnet' removed '/mnt/lustre/d77kb.sanityn/nrs_r_oleg360-client.virtnet' removed directory '/mnt/lustre/d77kb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.77574 s, 11.9 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.89889 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kb (54s) == sanityn test 77kc: Check different granular TBF type combination: nid+opcode ========================================================== 03:06:24 (1789455984) ost.OSS.ost_io.nrs_policies=tbf nid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.203.60@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.203.60@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.764854 s, 21.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135335 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11422 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161161 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.293287 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.345604 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.365865 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.362382 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.368593 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.364888 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.319295 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.310895 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.41107 s, 2.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9837139e5800.lock_unused_count=5 ldlm.namespaces.lustre-OST0001-osc-ffff9837139e5800.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.346588 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.476719 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.395896 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.348735 s, 3.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207395 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166923 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150436 s, 7.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18718 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160959 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172233 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166547 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174903 s, 6.0 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187582 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161298 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172747 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190607 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181552 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.257492 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190739 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187001 s, 5.6 MB/s removed '/mnt/lustre/d77kc.sanityn/nrs_w_oleg360-client.virtnet' removed '/mnt/lustre/d77kc.sanityn/nrs_r_oleg360-client.virtnet' removed directory '/mnt/lustre/d77kc.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.48983 s, 11.0 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.96819 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kc.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kc (58s) == sanityn test 77kd: Check different granular TBF type combination: nid+jobid ========================================================== 03:07:22 (1789456042) ost.OSS.ost_io.nrs_policies=tbf nid+jobid ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.203.60@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/nrs_r_oleg360-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.53991 s, 31.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0538862 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.055951 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0809523 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0951296 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.082459 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106796 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189557 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.250752 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220667 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.296852 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.288569 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.255855 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.266567 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.295316 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.239654 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.317982 s, 3.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9837139e5800.lock_unused_count=3 ldlm.namespaces.lustre-OST0001-osc-ffff9837139e5800.lock_unused_count=1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0452273 s, 23.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0509694 s, 20.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0494712 s, 21.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0704792 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0984071 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0783705 s, 13.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0452061 s, 23.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0529481 s, 19.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0717032 s, 14.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0599774 s, 17.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0962968 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0620201 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100505 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0952774 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0723966 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0922048 s, 11.4 MB/s removed '/mnt/lustre/d77kd.sanityn/nrs_r_oleg360-client.virtnet' removed '/mnt/lustre/d77kd.sanityn/nrs_w_oleg360-client.virtnet' removed directory '/mnt/lustre/d77kd.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.0077 s, 9.5 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kd.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.58815 s, 18.8 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kd.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kd (50s) == sanityn test 77ke: Check different granular TBF type combination: jobid+opcode ========================================================== 03:08:12 (1789456092) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/nrs_r_oleg360-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.863076 s, 19.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0740132 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126638 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134351 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207651 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200043 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169319 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.247483 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.307293 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.236496 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.313477 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.344393 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.278937 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.34717 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.346478 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.364782 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.321152 s, 3.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0967317 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0614666 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0241866 s, 43.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0820075 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0637671 s, 16.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0906652 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0645171 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134758 s, 7.8 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0928291 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135969 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0871599 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111481 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102421 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10967 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130242 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137591 s, 7.6 MB/s removed '/mnt/lustre/d77ke.sanityn/nrs_r_oleg360-client.virtnet' removed '/mnt/lustre/d77ke.sanityn/nrs_w_oleg360-client.virtnet' removed directory '/mnt/lustre/d77ke.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.7564 s, 8.9 MB/s Write runtime is 13 s, speed is 7.692307 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ke.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.1661 s, 10.3 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.64899 s, 10.9 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.91291 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77ke (97s) == sanityn test 77kf: Check different granular TBF type combination: uid+gid ========================================================== 03:09:49 (1789456189) ost.OSS.ost_io.nrs_policies=tbf uid+gid ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/nrs_r_oleg360-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.06052 s, 15.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0706179 s, 14.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.093823 s, 11.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0643362 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133084 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184413 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0739507 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159332 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0481518 s, 21.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183097 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113145 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0462713 s, 22.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237779 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208141 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.347371 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.262162 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.247697 s, 4.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0208177 s, 50.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0376915 s, 27.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0521725 s, 20.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00918427 s, 114 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0795976 s, 13.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0331646 s, 31.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0456438 s, 23.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0121714 s, 86.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013377 s, 78.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0585188 s, 17.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0518151 s, 20.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167501 s, 62.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00413329 s, 254 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00900347 s, 116 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.010471 s, 100 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0240784 s, 43.5 MB/s removed '/mnt/lustre/d77kf.sanityn/nrs_w_oleg360-client.virtnet' removed '/mnt/lustre/d77kf.sanityn/nrs_r_oleg360-client.virtnet' removed directory '/mnt/lustre/d77kf.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.1148 s, 5.2 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kf.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.7834 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77kf.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kf (74s) == sanityn test 77kg: check different granular TBF type combination: uid+opcode ========================================================== 03:11:03 (1789456263) ost.OSS.ost_io.nrs_policies=tbf uid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg360-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.874041 s, 19.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0598111 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0616669 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10784 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102228 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175409 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202468 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163796 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21626 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190315 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.281976 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208318 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.219718 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.296496 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.304288 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.328186 s, 3.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9837139e5800.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff9837139e5800.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.374389 s, 2.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0478766 s, 21.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0419711 s, 25.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0344859 s, 30.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0398467 s, 26.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0583847 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0745283 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0819374 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0633788 s, 16.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0854549 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0800753 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0600066 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0648845 s, 16.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9837139e5800.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0634188 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.08937 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0716457 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107269 s, 9.8 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg360-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg360-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.84974 s, 11.8 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.94622 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg360-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.746068 s, 22.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0364242 s, 28.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0365809 s, 28.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128807 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186132 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208019 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.204445 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220394 s, 4.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.240547 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.265322 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.241916 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.274195 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.272198 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.268177 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.281175 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.307976 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.329749 s, 3.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00638803 s, 164 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0472935 s, 22.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0437756 s, 24.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00360465 s, 291 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0459828 s, 22.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0714713 s, 14.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0518235 s, 20.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0768459 s, 13.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0741297 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.05616 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0837334 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0863965 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107856 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0825987 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.072572 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100432 s, 10.4 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg360-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg360-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 12.1696 s, 8.6 MB/s Write runtime is 13 s, speed is 7.692307 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0141 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.1475 s, 9.4 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.94742 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kg (145s) == sanityn test 77kh: Verify that Project ID support for NRS TBF rule ========================================================== 03:13:28 (1789456408) using SAVE_PROJECT_SUPPORTED=0 Stopping clients: oleg360-client.virtnet /mnt/lustre (opts:) Stopping client oleg360-client.virtnet /mnt/lustre opts: Stopping clients: oleg360-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg360-client.virtnet /mnt/lustre2 opts: Stopping /mnt/lustre-mds1 (opts:-f) on oleg360-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg360-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg360-server sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime) proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) devtmpfs on /dev type devtmpfs (rw,nosuid,mode=755) securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime) tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev) devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000) tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755) tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755) cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd) pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime) bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700) cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio) cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset) cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio) cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices) cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer) cgroup on /sys/fs/cgroup/misc type cgroup (rw,nosuid,nodev,noexec,relatime,misc) cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb) cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma) cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory) cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event) cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids) cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct) none on /sys/kernel/tracing type tracefs (rw,relatime) configfs on /sys/kernel/config type configfs (rw,relatime) /dev/nbd0 on / type ext4 (ro,relatime,stripe=32) rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) mqueue on /dev/mqueue type mqueue (rw,relatime) hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M) debugfs on /sys/kernel/debug type debugfs (rw,relatime) none on /mnt type ramfs (rw,relatime) none on /var/lib/stateless/writable type tmpfs (rw,relatime) /dev/vda on /home/green/git/lustre-release type squashfs (ro,relatime) none on /var/cache/man type tmpfs (rw,relatime) none on /var/log type tmpfs (rw,relatime) none on /var/lib/dbus type tmpfs (rw,relatime) none on /tmp type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) none on /var/tmp type tmpfs (rw,relatime) none on /var/lib/NetworkManager type tmpfs (rw,relatime) none on /var/lib/systemd/random-seed type tmpfs (rw,relatime) none on /var/spool type tmpfs (rw,relatime) none on /var/lib/nfs type tmpfs (rw,relatime) none on /var/lib/gssproxy type tmpfs (rw,relatime) none on /var/lib/logrotate type tmpfs (rw,relatime) none on /etc type tmpfs (rw,relatime) none on /var/lib/rsyslog type tmpfs (rw,relatime) none on /root/.ssh type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) 192.168.200.253:/exports/state/oleg360-client.virtnet on /var/lib/stateless/state type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.203.60,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg360-client.virtnet/boot on /boot type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.203.60,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg360-client.virtnet/etc/kdump.conf on /etc/kdump.conf type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.203.60,local_lock=none,addr=192.168.200.253) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=378180k,mode=700) 192.168.200.253:/exports/testreports/70251/testresults/sanityn-zfs-rocky8.10_x86_64-rocky8.10_x86_64 on /tmp/testlogs type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.203.60,local_lock=none,addr=192.168.200.253) /dev/vda on /usr/sbin/mount.lustre type squashfs (ro,relatime) Checking servers environments Checking clients oleg360-client.virtnet environments Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg360-server' oleg360-server: oleg360-server.virtnet: executing load_modules_local oleg360-server: kmemleak disabled oleg360-server: Loading modules from /home/green/git/lustre-release/lustre oleg360-server: detected 4 online CPUs by sysfs oleg360-server: MODOPTS_LIBCFS= oleg360-server: Force libcfs to create 2 CPU partitions oleg360-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Setup mgs, mdt, osts Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg360-server: oleg360-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg360-client: oleg360-server: ssh exited with exit code 1 Started lustre-MDT0000 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg360-server: oleg360-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg360-client: oleg360-server: ssh exited with exit code 1 Started lustre-OST0000 Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg360-server: oleg360-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg360-client: oleg360-server: ssh exited with exit code 1 Started lustre-OST0001 Starting client: oleg360-client.virtnet: -o user_xattr,flock 192.168.203.160@tcp:/lustre /mnt/lustre Starting client oleg360-client.virtnet: -o user_xattr,flock 192.168.203.160@tcp:/lustre /mnt/lustre Started clients oleg360-client.virtnet: 192.168.203.160@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Starting client: oleg360-client.virtnet: -o user_xattr,flock 192.168.203.160@tcp:/lustre /mnt/lustre2 Starting client oleg360-client.virtnet: -o user_xattr,flock 192.168.203.160@tcp:/lustre /mnt/lustre2 Started clients oleg360-client.virtnet: 192.168.203.160@tcp:/lustre on /mnt/lustre2 type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff98371508c800.idle_timeout=debug osc.lustre-OST0000-osc-ffff9837162ef800.idle_timeout=debug osc.lustre-OST0001-osc-ffff98371508c800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9837162ef800.idle_timeout=debug disable quota as required ost.OSS.ost_io.nrs_policies=tbf projid ost.OSS.ost_io.nrs_tbf_rule=start ext_proj_rw projid={100} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.559004 s, 30.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117737 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122299 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104696 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129659 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187121 s, 5.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.238414 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211628 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.244266 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.235696 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223209 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.29393 s, 3.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.26483 s, 4.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.232737 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.342911 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.290098 s, 3.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff98371508c800.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffff98371508c800.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.360881 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.416713 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.365995 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.419994 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.386431 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.431209 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.373344 s, 2.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.391773 s, 2.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.397143 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.383103 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.471141 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.384651 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.387001 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.403435 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.460569 s, 2.3 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.40719 s, 2.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.409167 s, 2.6 MB/s removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg360-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg360-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.14173 s, 12.9 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.2328 s, 20.0 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_proj_rw ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf projid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w projid={100}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r projid={100}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.651902 s, 25.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113086 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118757 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193887 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1857 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202766 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.256756 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.310248 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.31598 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.261716 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.323155 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.34429 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.351676 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.31437 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.307652 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.309462 s, 3.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff98371508c800.lock_unused_count=3 ldlm.namespaces.lustre-OST0001-osc-ffff98371508c800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.318389 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21612 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242353 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18362 s, 5.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189848 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191154 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194536 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216841 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.218815 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220778 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.219685 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.252354 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.296304 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.24188 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.246191 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.249592 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.249378 s, 4.2 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff98371508c800.lock_unused_count=2 removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg360-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg360-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.80578 s, 10.7 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0052 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77kh (203s) == sanityn test 77ki: Add rule with unsupported TBF type should fail for generic TBF ========================================================== 03:16:52 (1789456612) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 oleg360-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_fail projid={100}&opcode={ost_write} rate=20: Invalid argument oleg360-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_fail projid={100}&opcode={ost_write} rate=20': Invalid argument pdsh@oleg360-client: oleg360-server: ssh exited with exit code 22 ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77ki (18s) == sanityn test 77kj: Verify nodemap support for NRS TBF rule ========================================================== 03:17:10 (1789456630) On MGS 192.168.203.160, active = nodemap.active=1 On MGS 192.168.203.160, TBF = ost.OSS.ost_io.nrs_policies=tbf nodemap ost.OSS.ost_io.nrs_tbf_rule=start ext_nm_rw nodemap={TBF} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.642472 s, 26.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0637496 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0973446 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173373 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.231538 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22393 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207397 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.243158 s, 4.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.243401 s, 4.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.250459 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.311046 s, 3.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.322513 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21612 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.247562 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.270092 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.351002 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.307861 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0795845 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0425815 s, 24.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0741525 s, 14.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0336632 s, 31.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0335416 s, 31.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0319219 s, 32.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0395411 s, 26.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104371 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.062452 s, 16.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0408239 s, 25.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0685326 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0513575 s, 20.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0504099 s, 20.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0585534 s, 17.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0596663 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0708035 s, 14.8 MB/s removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg360-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg360-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.0168 s, 13.1 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.28503 s, 19.8 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete On MGS 192.168.203.160, TBF.id = On MGS 192.168.203.160, active = nodemap.active=0 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.537437 s, 31.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138261 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154473 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130584 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.235721 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.313836 s, 3.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.256029 s, 4.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.341565 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.27112 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.384337 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.261065 s, 4.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.382399 s, 2.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.433737 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.322275 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.480873 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.445051 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.487252 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0563951 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.041729 s, 25.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0891549 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00942503 s, 111 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0543242 s, 19.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114892 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0191445 s, 54.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0100138 s, 105 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0374454 s, 28.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126916 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0361454 s, 29.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0286653 s, 36.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00813963 s, 129 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0368437 s, 28.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.011285 s, 92.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0402377 s, 26.1 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff98371508c800.lock_unused_count=3 removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg360-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg360-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 1 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.80493 s, 11.9 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 2.90254 s, 36.1 MB/s Read runtime is 4 s, speed is 25.000000 IOPS read rate (25.000000) should exceed limit (20 * 1) cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_nm_rw ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf nodemap+opcode oleg360-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_w nodemap={TBF}&opcode={ost_write} rate=20: Invalid argument oleg360-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_w nodemap={TBF}&opcode={ost_write} rate=20': Invalid argument oleg360-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_r nodemap={TBF}&opcode={ost_read} rate=10: Invalid argument oleg360-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_r nodemap={TBF}&opcode={ost_read} rate=10': Invalid argument pdsh@oleg360-client: oleg360-server: ssh exited with exit code 22 On MGS 192.168.203.160, active = nodemap.active=1 On MGS 192.168.203.160, TBF = ost.OSS.ost_io.nrs_tbf_rule=start ext_w nodemap={TBF}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nodemap={TBF}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.569337 s, 29.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0841295 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0966121 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150118 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148449 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.226486 s, 4.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.311115 s, 3.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.30128 s, 3.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.271672 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.258493 s, 4.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.313419 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.331466 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.357188 s, 2.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff98371508c800.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffff98371508c800.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.388054 s, 2.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.388697 s, 2.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.346169 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.424198 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15972 s, 6.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185035 s, 5.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216908 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210477 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195636 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185607 s, 5.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.247217 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.204955 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202285 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210452 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.249463 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221813 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.233128 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211789 s, 5.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.229725 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.303964 s, 3.4 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff98371508c800.lock_unused_count=1 removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg360-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg360-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.06473 s, 11.6 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.941 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo On MGS 192.168.203.160, TBF.id = On MGS 192.168.203.160, active = nodemap.active=0 PASS 77kj (183s) == sanityn test 77l: check the output of NRS policies for generic TBF ========================================================== 03:20:13 (1789456813) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_policies=fifo PASS 77l (11s) == sanityn test 77m: check NRS Delay slows write RPC processing ========================================================== 03:20:24 (1789456824) ost.OSS.ost_io.nrs_policies=delay ost.OSS.ost_io.nrs_delay_min=4 ost.OSS.ost_io.nrs_delay_max=4 ost.OSS.ost_io.nrs_delay_pct=100 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg360-client.virtnet] [bs=1M] [count=1] [oflag=direct] [conv=fdatasync] 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 3.73842 s, 280 kB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg360-client.virtnet] [bs=1M] [count=10] [oflag=direct] [conv=fdatasync] 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 40.1224 s, 261 kB/s ost.OSS.ost_io.nrs_policies=fifo PASS 77m (58s) == sanityn test 77n: check wildcard support for TBF JobID NRS policy ========================================================== 03:21:22 (1789456882) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={*.500} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.58161 s, 28.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0566665 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113703 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165216 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138576 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1037 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17191 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161741 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.281118 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.241033 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.311416 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.321526 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.245926 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.316356 s, 3.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff98371508c800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff98371508c800.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.332596 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.330245 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.350428 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0679542 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.043359 s, 24.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0346349 s, 30.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00708055 s, 148 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0231886 s, 45.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00605323 s, 173 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00271962 s, 386 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00694853 s, 151 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00660534 s, 159 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00631074 s, 166 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00567238 s, 185 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0211763 s, 49.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0115459 s, 90.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130794 s, 80.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0288251 s, 36.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00635 s, 165 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg360-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg360-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77n.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.03806 s, 13.0 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77n.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.12761 s, 20.4 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.*} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.670223 s, 25.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12463 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164258 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19289 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180295 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173099 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.289834 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.229885 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.290308 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.421504 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.456199 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.319431 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.460919 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.349673 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.522138 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.512103 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.537523 s, 2.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff98371508c800.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff98371508c800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.453353 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.461775 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.409276 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.40472 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.415881 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.428812 s, 2.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.423314 s, 2.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.416629 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.426683 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.440486 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.430992 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.428341 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.44643 s, 2.3 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.454574 s, 2.3 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.471199 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.459792 s, 2.3 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg360-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg360-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.89784 s, 10.6 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.43455 s, 19.3 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_policies=fifo PASS 77n (92s) == sanityn test 77o: Changing rank should not panic ====== 03:22:55 (1789456975) mds.MDS.mdt.nrs_policies=tbf nid mds.MDS.mdt.nrs_tbf_rule=start name nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=start name1 nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=change name1 rank=name mds.MDS.mdt.nrs_tbf_rule=stop name mds.MDS.mdt.nrs_policies=fifo PASS 77o (18s) == sanityn test 77q: Parallel TBF rule definitions should not panic ========================================================== 03:23:12 (1789456992) mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_policies=fifo PASS 77q (134s) == sanityn test 77p: Check validity of rule names for TBF policies ========================================================== 03:25:27 (1789457127) mds.MDS.mdt.nrs_policies=tbf oleg360-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_overflo uid={500} rate=500: File name too long oleg360-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_overflo uid={500} rate=500': File name too long pdsh@oleg360-client: oleg360-server: ssh exited with exit code 36 oleg360-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start uid={500} rate=500: Invalid argument oleg360-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start uid={500} rate=500': Invalid argument pdsh@oleg360-client: oleg360-server: ssh exited with exit code 22 oleg360-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_empty: Invalid argument oleg360-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_empty': Invalid argument pdsh@oleg360-client: oleg360-server: ssh exited with exit code 22 oleg360-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p@spec uid={500} rate=500: Invalid argument oleg360-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p@spec uid={500} rate=500': Invalid argument pdsh@oleg360-client: oleg360-server: ssh exited with exit code 22 oleg360-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p.spec uid={500} rate=500: Invalid argument oleg360-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p.spec uid={500} rate=500': Invalid argument pdsh@oleg360-client: oleg360-server: ssh exited with exit code 22 oleg360-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p~spec uid={500} rate=500: Invalid argument oleg360-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p~spec uid={500} rate=500': Invalid argument pdsh@oleg360-client: oleg360-server: ssh exited with exit code 22 oleg360-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p#spec uid={500} rate=500: Invalid argument oleg360-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p#spec uid={500} rate=500': Invalid argument pdsh@oleg360-client: oleg360-server: ssh exited with exit code 22 oleg360-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p/spec uid={500} rate=500: Invalid argument oleg360-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p/spec uid={500} rate=500': Invalid argument pdsh@oleg360-client: oleg360-server: ssh exited with exit code 22 oleg360-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p^spec uid={500} rate=500: Invalid argument oleg360-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p^spec uid={500} rate=500': Invalid argument pdsh@oleg360-client: oleg360-server: ssh exited with exit code 22 oleg360-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p%spec uid={500} rate=500: Invalid argument oleg360-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p%spec uid={500} rate=500': Invalid argument pdsh@oleg360-client: oleg360-server: ssh exited with exit code 22 oleg360-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p*spec uid={500} rate=500: Invalid argument oleg360-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p*spec uid={500} rate=500': Invalid argument pdsh@oleg360-client: oleg360-server: ssh exited with exit code 22 oleg360-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p;spec uid={500} rate=500: Invalid argument oleg360-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p;spec uid={500} rate=500': Invalid argument pdsh@oleg360-client: oleg360-server: ssh exited with exit code 22 oleg360-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p,spec uid={500} rate=500: Invalid argument oleg360-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p,spec uid={500} rate=500': Invalid argument pdsh@oleg360-client: oleg360-server: ssh exited with exit code 22 oleg360-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p?spec uid={500} rate=500: Invalid argument oleg360-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p?spec uid={500} rate=500': Invalid argument pdsh@oleg360-client: oleg360-server: ssh exited with exit code 22 oleg360-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77pspec uid={500} rate=500: Invalid argument oleg360-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p>spec uid={500} rate=500': Invalid argument pdsh@oleg360-client: oleg360-server: ssh exited with exit code 22 oleg360-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p:spec uid={500} rate=500: Invalid argument oleg360-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p:spec uid={500} rate=500': Invalid argument pdsh@oleg360-client: oleg360-server: ssh exited with exit code 22 oleg360-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p+spec uid={500} rate=500: Invalid argument oleg360-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p+spec uid={500} rate=500': Invalid argument pdsh@oleg360-client: oleg360-server: ssh exited with exit code 22 oleg360-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p=spec uid={500} rate=500: Invalid argument oleg360-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p=spec uid={500} rate=500': Invalid argument pdsh@oleg360-client: oleg360-server: ssh exited with exit code 22 oleg360-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p)spec uid={500} rate=500: Invalid argument oleg360-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p)spec uid={500} rate=500': Invalid argument pdsh@oleg360-client: oleg360-server: ssh exited with exit code 22 oleg360-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p(spec uid={500} rate=500: Invalid argument oleg360-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p(spec uid={500} rate=500': Invalid argument pdsh@oleg360-client: oleg360-server: ssh exited with exit code 22 oleg360-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p{spec uid={500} rate=500: Invalid argument oleg360-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p{spec uid={500} rate=500': Invalid argument pdsh@oleg360-client: oleg360-server: ssh exited with exit code 22 oleg360-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p}spec uid={500} rate=500: Invalid argument oleg360-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p}spec uid={500} rate=500': Invalid argument pdsh@oleg360-client: oleg360-server: ssh exited with exit code 22 oleg360-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p|spec uid={500} rate=500: Invalid argument oleg360-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p|spec uid={500} rate=500': Invalid argument pdsh@oleg360-client: oleg360-server: ssh exited with exit code 22 oleg360-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p[spec uid={500} rate=500: Invalid argument oleg360-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p[spec uid={500} rate=500': Invalid argument pdsh@oleg360-client: oleg360-server: ssh exited with exit code 22 oleg360-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p]spec uid={500} rate=500: Invalid argument oleg360-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p]spec uid={500} rate=500': Invalid argument pdsh@oleg360-client: oleg360-server: ssh exited with exit code 22 oleg360-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p!spec uid={500} rate=500: Invalid argument oleg360-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p!spec uid={500} rate=500': Invalid argument pdsh@oleg360-client: oleg360-server: ssh exited with exit code 22 oleg360-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p&spec uid={500} rate=500: Invalid argument oleg360-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p&spec uid={500} rate=500': Invalid argument pdsh@oleg360-client: oleg360-server: ssh exited with exit code 22 oleg360-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p$spec uid={500} rate=500: Invalid argument oleg360-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p$spec uid={500} rate=500': Invalid argument pdsh@oleg360-client: oleg360-server: ssh exited with exit code 22 oleg360-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p`spec uid={500} rate=500: Invalid argument oleg360-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p`spec uid={500} rate=500': Invalid argument pdsh@oleg360-client: oleg360-server: ssh exited with exit code 22 oleg360-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p\spec uid={500} rate=500: Invalid argument oleg360-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p\spec uid={500} rate=500': Invalid argument pdsh@oleg360-client: oleg360-server: ssh exited with exit code 22 mds.MDS.mdt.nrs_policies=fifo PASS 77p (37s) == sanityn test 77r: Change type of tbf policy at run time ========================================================== 03:26:04 (1789457164) mds.MDS.mdt.nrs_policies=tbf uid mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf gid mds.MDS.mdt.nrs_tbf_rule=start md_rule gid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start md_rule jobid={*.500} rate=1 mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=fifo check the number of requests in queue: fifo: 0 crrn: 0 tbf: 0 delay: 0 fifo: 0 crrn: 0 tbf: 0 delay: 0 cleanup 77r 273423 kill md_thread xargs PASS 77r (58s) == sanityn test 77s: Check TBF LRU shrinker ============== 03:27:03 (1789457223) jobid_name=%e.%p jobid_var=TEST_77s_JOBVAR mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start touch jobid={touch.*} rate=1000 mds.MDS.mdt.nrs_tbf_rule=start unlink jobid={unlink.*} rate=1000 touch jobid={touch.*} 1000, ref 9 touch jobid={touch.*} 1000, ref 0 unlink jobid={unlink.*} 1000, ref 9 unlink jobid={unlink.*} 1000, ref 0 mds.MDS.mdt.nrs_policies=fifo jobid_var=procname_uid jobid_name=%e.%u PASS 77s (76s) == sanityn test 78: Enable policy and specify tunings right away ========================================================== 03:28:19 (1789457299) ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_orr_quantum=1 ost.OSS.ost_io.nrs_policies=fifo PASS 78 (11s) == sanityn test 79: xattr: intent error ================== 03:28:30 (1789457310) File: /mnt/lustre/d79.sanityn Size: 16384 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115238826935667 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-15 03:28:33.000000000 -0400 Modify: 2026-09-15 03:28:33.000000000 -0400 Change: 2026-09-15 03:28:33.000000000 -0400 Birth: 2026-09-15 03:28:33.000000000 -0400 fail_val=0 fail_loc=0x80000160 fail_val=0 fail_loc=0x80000131 # file: mnt/lustre/d79.sanityn trusted.name1="value1" PASS 79 (21s) == sanityn test 80a: migrate directory when some children is being opened ========================================================== 03:28:52 (1789457332) SKIP: sanityn test_80a needs >= 2 MDTs SKIP 80a (5s) == sanityn test 80b: Accessing directory during migration ========================================================== 03:28:56 (1789457336) SKIP: sanityn test_80b needs >= 2 MDTs SKIP 80b (4s) == sanityn test 81a: rename and stat under striped directory ========================================================== 03:29:01 (1789457341) SKIP: sanityn test_81a needs >= 2 MDTs SKIP 81a (5s) == sanityn test 81b: rename under striped directory doesn't deadlock ========================================================== 03:29:05 (1789457345) SKIP: sanityn test_81b We need at least 2 MDTs for this test SKIP 81b (6s) == sanityn test 81c: rename revoke LOOKUP lock for remote object ========================================================== 03:29:11 (1789457351) SKIP: sanityn test_81c needs >= 4 MDTs SKIP 81c (4s) == sanityn test 81d: parallel rename file cross-dir on same MDT ========================================================== 03:29:15 (1789457355) - open/close 560 (time 1789457371.99 total 10.01 last 55.96) - open/close 1176 (time 1789457382.01 total 20.03 last 61.45) - open/close 1809 (time 1789457392.03 total 30.05 last 63.15) - open/close 2422 (time 1789457402.03 total 40.06 last 61.29) total: 2500 open/close in 41.17 seconds: 60.72 ops/second snapshot_time 1789457503.765869709 secs.nsecs start_time 1789457359.433598284 secs.nsecs elapsed_time 144.332271425 secs.nsecs open 2505 samples [usecs] 130 24935 4569117 15744709025 close 2505 samples [usecs] 153 10696 1035051 953570509 mknod 2501 samples [usecs] 539 24832 3977407 12524522673 unlink 2500 samples [usecs] 712 110430 5933043 64069683545 mkdir 2 samples [usecs] 1930 5928 7858 38866084 rmdir 2 samples [usecs] 1518 7130 8648 53141224 rename 2500 samples [usecs] 432 82405 19466809 320868263517 getattr 3760 samples [usecs] 7 5434 249450 216111758 parallel_rename_file 2500 samples [usecs] 432 82405 19466809 320868263517 crossdir_rename 2500 samples [usecs] 432 82405 19466809 320868263517 readpage 22 samples [usecs] 6075 59490 481990 15337994518 PASS 81d (154s) == sanityn test 82: fsetxattr and fgetxattr on orphan files ========================================================== 03:31:50 (1789457510) multiop /mnt/lustre/f82.sanityn vO_ac TMPPIPE=/tmp/multiop_open_wait_pipe.7627 multiop /mnt/lustre2/f82.sanityn vo_Ac TMPPIPE=/tmp/multiop_open_wait_pipe.7627 PASS 82 (7s) == sanityn test 83: access striped directory while it is being created/unlinked ========================================================== 03:31:57 (1789457517) SKIP: sanityn test_83 needs >= 2 MDTs SKIP 83 (4s) == sanityn test 84: 0-nlink race in lu_object_find() ===== 03:32:01 (1789457521) fail_loc=0x8000060b PASS 84 (13s) == sanityn test 85: Lustre API root cache race =========== 03:32:14 (1789457534) Starting: llapi_root_test /mnt/lustre/f85.sanityn /mnt/lustre2/f85.sanityn Starting test test1 at 1789457536 Finishing test test1 at 1789457541 PASS 85 (13s) == sanityn test 90: open/create and unlink striped directory ========================================================== 03:32:28 (1789457548) SKIP: sanityn test_90 needs >= 2 MDTs SKIP 90 (4s) == sanityn test 91: chmod and unlink striped directory === 03:32:32 (1789457552) SKIP: sanityn test_91 needs >= 2 MDTs SKIP 91 (4s) == sanityn test 92: create remote directory under orphan directory ========================================================== 03:32:36 (1789457556) SKIP: sanityn test_92 needs >= 2 MDTs SKIP 92 (4s) == sanityn test 93: alloc_rr should not allocate on same ost ========================================================== 03:32:40 (1789457560) fail_loc=0x00000163 fail_loc=0x0 /mnt/lustre/f93.sanityn-1/file1 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5784 0x1698 0x240000400 1 5745 0x1671 0x280000400 /mnt/lustre2/f93.sanityn-2/file2 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 5746 0x1672 0x280000400 0 5785 0x1699 0x240000400 rc1=0 and rc2=0 PASS 93 (19s) == sanityn test 94: signal vs CP callback race =========== 03:32:59 (1789457579) osc.lustre-OST0000-osc-ffff98371508c800.idle_timeout=0 osc.lustre-OST0000-osc-ffff9837162ef800.idle_timeout=0 osc.lustre-OST0001-osc-ffff98371508c800.idle_timeout=0 osc.lustre-OST0001-osc-ffff9837162ef800.idle_timeout=0 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.232844 s, 35.2 kB/s fail_val=5 fail_loc=0x80000312 fail_val=6 fail_loc=0x80000329 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 2.97512 s, 1.4 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 6389: 290551 Alarm clock dd if=/dev/zero of=$DIR/$tfile conv=notrunc oflag=append bs=4k count=1 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.100591 s, 40.7 kB/s osc.lustre-OST0000-osc-ffff98371508c800.idle_timeout=debug osc.lustre-OST0000-osc-ffff9837162ef800.idle_timeout=debug osc.lustre-OST0001-osc-ffff98371508c800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9837162ef800.idle_timeout=debug PASS 94 (16s) == sanityn test 95a: Check readpage() on a page that was removed from page cache ========================================================== 03:33:15 (1789457595) llite.lustre-ffff98371508c800.fast_read=0 llite.lustre-ffff9837162ef800.fast_read=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0432173 s, 48.5 MB/s fail_loc=0x80001421 fail_val=7 fail_loc=0x80001422 fail_val=10 llite.lustre-ffff98371508c800.fast_read=1 llite.lustre-ffff9837162ef800.fast_read=1 PASS 95a (20s) == sanityn test 95b: Check readpage() on a page that is no longer uptodate ========================================================== 03:33:36 (1789457616) llite.lustre-ffff98371508c800.fast_read=0 llite.lustre-ffff9837162ef800.fast_read=0 1+0 records in 1+0 records out 12288 bytes (12 kB, 12 KiB) copied, 0.00525006 s, 2.3 MB/s fail_loc=0x80001424 fail_val=5 fail_loc=0 llite.lustre-ffff98371508c800.fast_read=1 llite.lustre-ffff9837162ef800.fast_read=1 PASS 95b (14s) == sanityn test 100a: DoM: glimpse RPCs for stat without IO lock (DoM only file) ========================================================== 03:33:51 (1789457631) SKIP: sanityn test_100a Reserved for glimpse-ahead SKIP 100a (5s) == sanityn test 100b: DoM: no glimpse RPC for stat with IO lock (DoM only file) ========================================================== 03:33:56 (1789457636) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0148924 s, 275 kB/s /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK PASS 100b (10s) == sanityn test 100c: DoM: write vs stat without IO lock (combined file) ========================================================== 03:34:06 (1789457646) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.215351 s, 9.7 MB/s /mnt/lustre/d100c.sanityn/dom has type file OK /mnt/lustre/d100c.sanityn/dom has size 2097152 OK PASS 100c (9s) == sanityn test 100d: DoM: write+truncate vs stat without IO lock (combined file) ========================================================== 03:34:15 (1789457655) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.207822 s, 10.1 MB/s /mnt/lustre/d100d.sanityn/dom has type file OK /mnt/lustre/d100d.sanityn/dom has size 4096 OK PASS 100d (8s) == sanityn test 100e: DoM: read on open and file size ==== 03:34:24 (1789457664) 1+0 records in 1+0 records out 12000 bytes (12 kB, 12 KiB) copied, 0.00777586 s, 1.5 MB/s PASS 100e (8s) == sanityn test 101a: Discard DoM data on unlink ========= 03:34:31 (1789457671) /mnt/lustre/f101a.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00863945 s, 474 kB/s PASS 101a (7s) == sanityn test 101b: Discard DoM data on rename ========= 03:34:39 (1789457679) /mnt/lustre/f101b.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0105954 s, 387 kB/s PASS 101b (7s) == sanityn test 101c: Discard DoM data on close-unlink === 03:34:46 (1789457686) /mnt/lustre/f101c.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0165374 s, 248 kB/s PASS 101c (10s) == sanityn test 102: Test open by handle of unlinked file ========================================================== 03:34:57 (1789457697) file: /mnt/lustre/f102.sanityn fh_bytes: 32 fh_type: 151 fh_data: d1 0b 00 00 02 00 00 00 54 0f 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 file's parent FID is [0:0x0:0x0] file FID is [0x200000bd1:0xf54:0x0] access via mount point '/mnt/lustre2' - OK access after unlink - OK check_fhandle_syscalls test Passed! PASS 102 (10s) == sanityn test 103: Test size correctness with lockahead ========================================================== 03:35:06 (1789457706) fail_loc=0x415 fail_loc=0x415 fail_val=2 Incorrect size expected (no glimpse fix): Starting test test23 at 1789457710 lockahead_test: lockahead_test.c:1171: test23: assertion 'sb.st_size == sb2.st_size' failed: size on /mnt/lustre/d103.sanityn/f103.sanityn and /mnt/lustre2/d103.sanityn/f103.sanityn differs: 0 vs 1048576 fail_loc=0 fail_loc=0x214 fail_val=2 Starting test test23 at 1789457713 Finishing test test23 at 1789457714 PASS 103 (16s) == sanityn test 104: Verify that MDS stores atime/mtime/ctime during close ========================================================== 03:35:22 (1789457722) SKIP: sanityn test_104 ldiskfs only test SKIP 104 (3s) == sanityn test 105: Glimpse and lock cancel race ======== 03:35:26 (1789457726) fail_loc=0x416 PASS 105 (28s) == sanityn test 106a: Verify the btime via statx() ======= 03:35:54 (1789457754) SKIP: sanityn test_106a Test only for ldiskfs and statx() supported SKIP 106a (4s) == sanityn test 106b: Glimpse RPCs test for statx ======== 03:35:58 (1789457758) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00932017 s, 110 kB/s File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 11 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238826938203 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-15 03:36:00.000000000 -0400 Modify: 2026-09-15 03:36:00.000000000 -0400 Change: 2026-09-15 03:36:00.000000000 -0400 Birth: 2026-09-15 03:36:00.000000000 -0400 /mnt/lustre/f106b.sanityn 144115238826938203 -rw-r--r-- 1 0 0 1789457760 1789457760 1789457760 File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 11 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238826938203 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-15 03:36:00.000000000 -0400 Modify: 2026-09-15 03:36:00.000000000 -0400 Change: 2026-09-15 03:36:00.000000000 -0400 Birth: 2026-09-15 03:36:00.000000000 -0400 1789457760 1024 11 PASS 106b (10s) == sanityn test 106c: Verify statx attributes mask ======= 03:36:08 (1789457768) PASS 106c (9s) == sanityn test 107a: Basic grouplock conflict =========== 03:36:17 (1789457777) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.324115 s, 32.4 MB/s /mnt/lustre/f107a.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 5753 0x1679 0x280000400 multiop /mnt/lustre/f107a.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7627 multiop /mnt/lustre2/f107a.sanityn vO_G16022000r10g16022000c TMPPIPE=/tmp/multiop_open_wait_pipe.7627 First grouplock blocks second one PASS 107a (9s) == sanityn test 107b: Grouplock is added to the head of waiting list ========================================================== 03:36:27 (1789457787) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.294597 s, 35.6 MB/s /mnt/lustre/f107b.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5794 0x16a2 0x240000400 multiop /mnt/lustre/f107b.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7627 Grouplock blocks IO First grouplock blocks second one Second grouplock blocks IO PASS 107b (13s) == sanityn test 108a: lseek: parallel updates ============ 03:36:40 (1789457800) 1+0 records in 1+0 records out 10000 bytes (10 kB, 9.8 KiB) copied, 0.00683841 s, 1.5 MB/s llite.lustre-ffff98371508c800.hybrid_io=0 llite.lustre-ffff9837162ef800.hybrid_io=0 fail_val=4 fail_loc=0x80000414 1+0 records in 1+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 4.74743 s, 1.8 MB/s PASS 108a (13s) == sanityn test 109: Race with several mount instances on 1 node ========================================================== 03:36:53 (1789457813) 192.168.203.160@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg360-client.virtnet /mnt/lustre (opts:) 192.168.203.160@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg360-client.virtnet /mnt/lustre2 (opts:) Starting race between client mounts (duration 30) Iteration 0 Starting client: oleg360-client.virtnet: -o user_xattr,flock 192.168.203.160@tcp:/lustre /mnt/lustre Starting client: oleg360-client.virtnet: -o user_xattr,flock 192.168.203.160@tcp:/lustre /mnt/lustre2 192.168.203.160@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.160@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg360-client.virtnet /mnt/lustre (opts:) Stopping client oleg360-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg360-server' oleg360-server: oleg360-server.virtnet: executing load_modules_local oleg360-server: kmemleak disabled oleg360-server: Loading modules from /home/green/git/lustre-release/lustre oleg360-server: detected 4 online CPUs by sysfs oleg360-server: MODOPTS_LIBCFS= oleg360-server: Force libcfs to create 2 CPU partitions oleg360-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Starting client: oleg360-client.virtnet: -o user_xattr,flock 192.168.203.160@tcp:/lustre /mnt/lustre Starting client: oleg360-client.virtnet: -o user_xattr,flock 192.168.203.160@tcp:/lustre /mnt/lustre2 PASS 109 (49s) == sanityn test 110: do not grant another lock on resend ========================================================== 03:37:43 (1789457863) fail_loc=0x525 fail_val=3 fail_loc=0x534 File: /mnt/lustre/f110.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238860488707 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-15 03:37:45.000000000 -0400 Modify: 2026-09-15 03:37:45.000000000 -0400 Change: 2026-09-15 03:37:45.000000000 -0400 Birth: 2026-09-15 03:37:46.000000000 -0400 fail_loc=0 fail_val=0 Connected clients: oleg360-client.virtnet oleg360-client.virtnet PASS 110 (72s) == sanityn test 111: A racy rename/link an open file should not cause fs corruption ========================================================== 03:38:55 (1789457935) SKIP: sanityn test_111 needs >= 2 MDTs SKIP 111 (4s) == sanityn test 112: update max-inherit in default LMV === 03:38:59 (1789457939) SKIP: sanityn test_112 We need at least 2 MDTs for this test SKIP 112 (3s) == sanityn test 113: check servers of specified fs ======= 03:39:02 (1789457942) lustre-OST0000-osc-ffff98371093d000 active. lustre-OST0001-osc-ffff98371093d000 active. lustre-OST0000-osc-ffff98371093d000 active. lustre-OST0001-osc-ffff98371093d000 active. MGC192.168.203.160@tcp active. PASS 113 (8s) == sanityn test 114: implicit default LMV inherit ======== 03:39:10 (1789457950) SKIP: sanityn test_114 We need at least 2 MDTs for this test SKIP 114 (4s) == sanityn test 115: ldiskfs doesn't check direntry for uniqueness ========================================================== 03:39:14 (1789457954) SKIP: sanityn test_115 ldiskfs only test SKIP 115 (3s) == sanityn test 116: DNE: Set default LMV layout from a remote client ========================================================== 03:39:18 (1789457958) SKIP: sanityn test_116 needs >= 2 MDTs SKIP 116 (3s) == sanityn test 121: trunc append race =================== 03:39:21 (1789457961) 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0285394 s, 718 kB/s fail_loc=0x80001436 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.103179 s, 198 kB/s 0000000 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0005000 0000 0000 0000 0000 0000 0000 0000 0000 * 000c800 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0011800 PASS 121 (10s) == sanityn test 122: directory size is consistent across mounts ========================================================== 03:39:32 (1789457972) PASS 122 (10s) == sanityn test 200: service remains healthy while able to process request ========================================================== 03:39:42 (1789457982) at_max=10 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=9 fail_loc=0x1e1 fail_val=20 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1255 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.204 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.2259 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.2413 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.172 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.2043 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.2246 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1568 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1804 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.2457 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.2159 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1946 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.2967 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.2952 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2869 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2601 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.3768 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.4436 s, 0.1 kB/s at_max=600 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=116 oleg360-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff98371093d000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff98371093d000.ost_server_uuid in FULL state after 0 sec PASS 200 (84s) cleanup: ====================================================== == sanityn test complete, duration 9948 sec ============== 03:41:08 (1789458068) === sanityn: start cleanup 03:41:10 (1789458070) === Stopping clients: oleg360-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg360-client.virtnet /mnt/lustre2 opts: === sanityn: finish cleanup 03:46:12 (1789458372) ===