-----============= acceptance-small: sanityn ============----- Wed Sep 2 19:07:00 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg153-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanityn.*ex': No such file or directory excepting tests: 27 40a skipping tests SLOW=no: 33a === sanityn: start setup 19:07:17 (1788390437) === Starting client oleg153-client.virtnet: -o user_xattr,flock 192.168.201.153@tcp:/lustre /mnt/lustre2 Mount client oleg153-client.virtnet: mount -t lustre -o user_xattr,flock 192.168.201.153@tcp:/lustre /mnt/lustre2 Started clients oleg153-client.virtnet: 192.168.201.153@tcp:/lustre on /mnt/lustre2 type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) oleg153-client.virtnet: executing check_config_client /mnt/lustre oleg153-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg153-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff96b686658000.idle_timeout=debug osc.lustre-OST0000-osc-ffff96b688d5b800.idle_timeout=debug osc.lustre-OST0001-osc-ffff96b686658000.idle_timeout=debug osc.lustre-OST0001-osc-ffff96b688d5b800.idle_timeout=debug disable quota as required oleg153-server: oleg153-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanityn: finish setup 19:07:52 (1788390472) === 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0394054 s, 26.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7616] == sanityn test 0: do_node_vp() and do_facet_vp() do the right thing ========================================================== 19:07:54 (1788390474) PASS 0 (8s) == sanityn test 1: Check attribute updates on 2 mount points ========================================================== 19:08:03 (1788390483) /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0777 OK /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0666 OK /mnt/lustre/f1.sanityn: absent OK PASS 1 (8s) == sanityn test 2a: check cached attribute updates on 2 mtpt's ================================================================== 19:08:10 (1788390490) -rw-r--r-- 1 root root 0 Sep 2 19:08 /mnt/lustre2/f2a /mnt/lustre/f2a has type file OK /mnt/lustre/f2a has perms 0777 OK PASS 2a (7s) == sanityn test 2b: check cached attribute updates on 2 mtpt's ================================================================== 19:08:17 (1788390497) -rw-r--r-- 1 root root 0 Sep 2 19:08 /mnt/lustre2/f2b /mnt/lustre2/f2b has type file OK /mnt/lustre2/f2b has perms 0777 OK PASS 2b (6s) == sanityn test 2c: check cached attribute updates on 2 mtpt's root ============================================================= 19:08:23 (1788390503) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0777 OK PASS 2c (6s) == sanityn test 2d: check cached attribute updates on 2 mtpt's root ============================================================= 19:08:29 (1788390509) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0755 OK PASS 2d (6s) == sanityn test 2e: check chmod on root is propagated to others ========================================================== 19:08:35 (1788390515) total 1 -rwxrwxrwx 1 root root 0 Sep 2 19:08 f2a -rwxrwxrwx 1 root root 0 Sep 2 19:08 f2b total 1 -rwxrwxrwx 1 root root 0 Sep 2 19:08 f2a -rwxrwxrwx 1 root root 0 Sep 2 19:08 f2b running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre2/f2e.sanityn] [count=1] 1+0 records in 1+0 records out 512 bytes copied, 0.0241043 s, 21.2 kB/s PASS 2e (7s) == sanityn test 2f: check attr/owner updates on DNE with 2 mtpt's ========================================================== 19:08:42 (1788390522) SKIP: sanityn test_2f needs >= 2 MDTs SKIP 2f (3s) == sanityn test 2g: check blocks update on sync write ==== 19:08:46 (1788390526) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.365462 s, 5.7 MB/s /mnt/lustre/f2g.sanityn has 4103 blocks /mnt/lustre2/f2g.sanityn has 4103 blocks PASS 2g (7s) == sanityn test 3: symlink on one mtpt, readlink on another ===================================================================== 19:08:53 (1788390533) PASS 3 (6s) == sanityn test 4: fstat validation on multiple mount points ==================================================================== 19:09:00 (1788390540) PASS 4 (7s) == sanityn test 5: create a file on one mount, truncate it on the other ========================================================== 19:09:07 (1788390547) /mnt/lustre/f5 has type file OK /mnt/lustre/f5 has size 100 OK PASS 5 (5s) == sanityn test 6: remove of open file on other node ============================================================================ 19:09:12 (1788390552) opening writing unlinking /mnt/lustre2/f6.sanityn accessing (1) seeking (1) accessing (2) fstat... reading comparing data truncating seeking (2) writing again seeking (3) reading again comparing data again closing SUCCESS - goto beer PASS 6 (6s) == sanityn test 7: remove of open directory on other node ======================================================================= 19:09:18 (1788390558) creating directory /mnt/lustre/d7 opening directory unlinking /mnt/lustre/d7 Ok, everything goes well. PASS 7 (6s) == sanityn test 8: remove of open special file on other node ==================================================================== 19:09:24 (1788390564) creating special file /mnt/lustre/f8.sanityn opening file unlinking /mnt/lustre/f8.sanityn Ok, everything goes well. PASS 8 (7s) == sanityn test 9a: append of file with sub-page size on multiple mounts ========================================================== 19:09:31 (1788390571) PASS 9a (8s) == sanityn test 9b: append to striped sparse file ======== 19:09:39 (1788390579) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0601936 s, 17.4 MB/s 3+0 records in 3+0 records out 3 bytes copied, 0.0597693 s, 0.1 kB/s Data read (expecting 'foo'): foo PASS 9b (7s) == sanityn test 10a: write of file with sub-page size on multiple mounts ========================================================== 19:09:46 (1788390586) 1+0 records in 1+0 records out 1 byte copied, 0.0152864 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0259635 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0246899 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0102659 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00852781 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00829819 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00901832 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00711393 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00804502 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00887001 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.011149 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0270753 s, 0.0 kB/s PASS 10a (7s) == sanityn test 10b: write of file with sub-page size on multiple mounts ========================================================== 19:09:53 (1788390593) 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.00719825 s, 427 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0253784 s, 161 kB/s 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.000276676 s, 11.1 MB/s PASS 10b (7s) == sanityn test 11: execution of file opened for write should return error ============================================================== 19:10:00 (1788390600) multiop /mnt/lustre/d11/f vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7616 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 298: /mnt/lustre2/d11/f: Text file busy PASS 11 (7s) == sanityn test 12: test lock ordering (link, stat, unlink) ========================================================== 19:10:07 (1788390607) warning: '-runtime' deprecated, use '-t runtime' instead - link 354 (time 1788390619.64 total 10.00 last 35.40) - link 655 (time 1788390629.69 total 20.05 last 29.95) - link 1000 (time 1788390639.70 total 30.06 last 34.46) - link 1373 (time 1788390649.71 total 40.07 last 37.27) - link 1745 (time 1788390659.71 total 50.08 last 37.19) - link 2127 (time 1788390669.72 total 60.08 last 38.17) - link 2424 (time 1788390679.76 total 70.12 last 29.58) - link 2712 (time 1788390689.77 total 80.13 last 28.77) - link 3037 (time 1788390699.82 total 90.18 last 32.34) total: 3419 link in 100.01 seconds: 34.19 ops/second using seed 2896141376 running for 100 seconds - stat 5473 (time 1788390621 ; total 11 ; last 11) - stat 10000 (time 1788390628 ; total 18 ; last 7) - stat 16250 (time 1788390639 ; total 29 ; last 11) - stat 20000 (time 1788390644 ; total 34 ; last 5) - stat 26231 (time 1788390655 ; total 45 ; last 11) - stat 30000 (time 1788390662 ; total 52 ; last 7) - stat 33171 (time 1788390673 ; total 63 ; last 11) - stat 35982 (time 1788390684 ; total 74 ; last 11) - stat 38896 (time 1788390695 ; total 85 ; last 11) - stat 40000 (time 1788390698 ; total 88 ; last 3) - stat 43260 (time 1788390709 ; total 99 ; last 11) total: 43598 stats in 100 seconds: 435.980011 stats/second - unlinked 0 (time 1788390660 ; total 0 ; last 0) unlink(/mnt/lustre2/lockdir/lockfile3419) error: No such file or directory total: 3418 unlinks in 85 seconds: 40.211765 unlinks/second /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 77: kill: (21414) - No such process /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 78: kill: (21416) - No such process PASS 12 (143s) == sanityn test 13: test directory page revocation ======= 19:12:30 (1788390750) f13 f13 PASS 13 (8s) == sanityn test 14aa: execution of file open for write returns -ETXTBSY ========================================================== 19:12:38 (1788390758) multiop /mnt/lustre/d14aa.sanityn/f14aa.sanityn vOw_c TMPPIPE=/tmp/multiop_open_wait_pipe.7616 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 333: /mnt/lustre2/d14aa.sanityn/f14aa.sanityn: Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 22600 Terminated $MULTIOP_PROG $FILE v$ARGS > $TMPPIPE (wd: ~) PASS 14aa (7s) == sanityn test 14ab: open(RDWR) of executing file returns -ETXTBSY ========================================================== 19:12:45 (1788390765) open(O_RDWR|O_CREAT): Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 23179 Terminated $DIR1/$tdir/sleep 60 (wd: ~) PASS 14ab (6s) == sanityn test 14b: truncate of executing file returns -ETXTBSY ================================================================ 19:12:51 (1788390771) truncate: cannot truncate '/mnt/lustre2/d14b.sanityn/sleep' to length 60: Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 348: 23758 Terminated $DIR1/$tdir/sleep 60 PASS 14b (7s) == sanityn test 14c: open(O_TRUNC) of executing file return -ETXTBSY ============================================================ 19:12:58 (1788390778) cp: cannot create regular file '/mnt/lustre2/d14c.sanityn/sleep': Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 360: 24340 Terminated $DIR1/$tdir/sleep 60 PASS 14c (7s) == sanityn test 14d: chmod of executing file is still possible ================================================================== 19:13:05 (1788390785) chmod /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 371: 24922 Terminated $DIR1/$tdir/sleep 60 PASS 14d (8s) == sanityn test 15: test out-of-space with multiple writers ===================================================================== 19:13:14 (1788390794) PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin: MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky sleep 5 for ZFS MDS Waiting for MDT destroys to complete fallocate on zfs doesn't consume space STRIPECOUNT=2 ORIGFREE=7531520 MAXFREE=800000 SKIP: oos2 test_15 oos2.sh: 7531520KB free > 800000KB, increase MAXFREE (or reduce fs size) sleep 5 for ZFS MDS Waiting for MDT destroys to complete SKIP 15 (46s) == sanityn test 16a: 500 iterations of dual-mount fsx ==== 19:14:00 (1788390840) keep default fallocate mode: 2 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.405645 s, 20.7 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.463443 s, 18.1 MB/s Chance of close/open is 1 in 50 Seed set to 845 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read skipping zero size read 000005[1] 1788390845.893928 write 0x895fcc thru 0x8982d9 (0x230e bytes) truncating to largest ever: 0x2be016 000007[0] 1788390845.918966 trunc from 0x8982da to 0x2be015 (0x5da2c4 bytes) truncating to largest ever: 0x724ef1 000011[1] 1788390846.001218 trunc from 0x2be016 to 0x724ef0 (0x466edb bytes) 000012[0] 1788390846.041056 read 0x351471 thru 0x35d9bb (0xc54b bytes) 000016[0] 1788390846.094824 mapwrite 0x084be2 thru 0x08f0b0 (0xa4cf bytes) 000017[0] 1788390846.215071 read 0x5610b0 thru 0x56eb3d (0xda8e bytes) 000018[1] 1788390846.236317 trunc from 0x724ef1 to 0x3f24a8 (0x332a48 bytes) 000020[0] 1788390846.319567 trunc from 0x3f24a9 to 0x3f3b42 (0x169a bytes) 000021[1] 1788390846.371001 mapread 0x086965 thru 0x08e809 (0x7ea5 bytes) truncating to largest ever: 0x7a89af 000022[0] 1788390846.429618 trunc from 0x3f3b43 to 0x7a89ae (0x3b4e6c bytes) 000023[1] 1788390846.468196 write 0x6d069b thru 0x6d76ad (0x7013 bytes) 000025[1] 1788390846.527063 write 0x78d22b thru 0x795207 (0x7fdd bytes) 000027[1] 1788390846.561837 trunc from 0x7a89af to 0x087902 (0x7210ac bytes) 000029[0] 1788390846.594861 mapwrite 0x1f97fb thru 0x20651e (0xcd24 bytes) 000030[0] 1788390846.740344 write 0x572a8f thru 0x573e35 (0x13a7 bytes) 000031[0] 1788390846.745392 read 0x0450ed thru 0x053a1f (0xe933 bytes) 000032[0] 1788390846.768104 write 0x2e3b02 thru 0x2e995a (0x5e59 bytes) 000033[1] 1788390846.785726 trunc from 0x573e36 to 0x1576c6 (0x41c76f bytes) 000037[0] 1788390846.864340 write 0x8cc145 thru 0x8dba4a (0xf906 bytes) 000039[0] 1788390846.901535 punch from 0x543920 to 0x54a02f (0x6710 bytes) 000040[0] 1788390846.943605 read 0x86ac47 thru 0x873694 (0x8a4e bytes) 000041[1] 1788390846.964334 write 0x995592 thru 0x9a5553 (0xffc2 bytes) 000044[0] 1788390847.011056 punch from 0x191f36 to 0x19cfef (0xb0ba bytes) 000047[1] 1788390847.033964 punch from 0x5bb265 to 0x5c2412 (0x71ae bytes) 000048[1] 1788390847.064942 mapwrite 0x47755a thru 0x4841ab (0xcc52 bytes) 000050[1] 1788390847.402587 trunc from 0x9a5554 to 0x10c1d9 (0x89937a bytes) 000053[1] 1788390847.468964 read 0x0677df thru 0x0770d9 (0xf8fb bytes) 000054[1] 1788390847.474868 read 0x062fcb thru 0x0648ae (0x18e4 bytes) 000055[0] 1788390847.510271 mapread 0x0b7b8b thru 0x0c6a2d (0xeea3 bytes) 000056[1] 1788390847.634538 mapwrite 0x675cb4 thru 0x677b55 (0x1ea2 bytes) 000062[0] 1788390847.879299 mapwrite 0x224b07 thru 0x2262bc (0x17b6 bytes) 000063[1] 1788390848.009843 write 0x258872 thru 0x262569 (0x9cf8 bytes) 000065[1] 1788390848.045236 mapread 0x1b0f95 thru 0x1be58b (0xd5f7 bytes) 000066[1] 1788390848.090517 write 0x7a01bc thru 0x7ad052 (0xce97 bytes) 000067[0] 1788390848.092332 write 0x9cdee7 thru 0x9d64ea (0x8604 bytes) 000070[1] 1788390848.163134 mapwrite 0x514223 thru 0x517ea1 (0x3c7f bytes) truncating to largest ever: 0x91c03b 000072[0] 1788390848.232186 trunc from 0x9d64eb to 0x91c03a (0xba4b0 bytes) 000073[0] 1788390848.298906 mapread 0x3bf82f thru 0x3cbb74 (0xc346 bytes) 000075[0] 1788390848.386147 mapread 0x225ff0 thru 0x22764b (0x165c bytes) 000081[0] 1788390848.477899 punch from 0x5ecb5e to 0x5f7691 (0xab34 bytes) 000083[1] 1788390848.533488 mapwrite 0x772d92 thru 0x77fffb (0xd26a bytes) 000084[1] 1788390848.685975 mapread 0x12498b thru 0x12b25d (0x68d3 bytes) 000085[1] 1788390848.781208 write 0x99ab75 thru 0x9a2fb3 (0x843f bytes) 000086[0] 1788390848.795125 mapread 0x64a7f8 thru 0x64e1cf (0x39d8 bytes) 000089[1] 1788390848.872906 trunc from 0x9a2fb4 to 0x22b0ea (0x777ec9 bytes) 000090[0] 1788390848.967404 mapwrite 0x5516eb thru 0x556bc8 (0x54de bytes) 000094[0] 1788390849.086733 mapread 0x07cb31 thru 0x087c54 (0xb124 bytes) 000096[1] 1788390849.130191 trunc from 0x556bc9 to 0x2e604b (0x270b7d bytes) 000097[1] 1788390849.179916 mapwrite 0x2168aa thru 0x21edb5 (0x850c bytes) 000099[1] 1788390849.250871 write 0x3bfbb0 thru 0x3c4ad6 (0x4f27 bytes) 000101[1] 1788390849.252196 write 0x820d2c thru 0x82eccf (0xdfa4 bytes) 000104[1] 1788390849.256232 mapwrite 0x2589d0 thru 0x263518 (0xab49 bytes) 000106[0] 1788390849.365920 read 0x75cc15 thru 0x75ed63 (0x214f bytes) 000107[0] 1788390849.429253 mapread 0x29a256 thru 0x2a4eb6 (0xac61 bytes) 000110[1] 1788390849.492154 mapread 0x0ec313 thru 0x0f1982 (0x5670 bytes) 000111[0] 1788390849.534986 write 0x7d94cf thru 0x7ded2b (0x585d bytes) 000112[1] 1788390849.586606 punch from 0x4a84a1 to 0x4b7069 (0xebc9 bytes) 000113[0] 1788390849.629134 punch from 0x49844b to 0x4a782c (0xf3e2 bytes) 000114[1] 1788390849.657649 punch from 0x762754 to 0x76eb8d (0xc43a bytes) 000115[1] 1788390849.694526 trunc from 0x82ecd0 to 0x27310d (0x5bbbc2 bytes) 000116[0] 1788390849.753817 read 0x0d49ac thru 0x0db11a (0x676f bytes) 000117[1] 1788390849.812907 read 0x1cf8aa thru 0x1d57ca (0x5f21 bytes) 000119[0] 1788390849.941955 punch from 0x1cf737 to 0x1d1326 (0x1bf0 bytes) 000120[0] 1788390849.987115 punch from 0x14057d to 0x148cd4 (0x8758 bytes) 000121[1] 1788390850.004913 read 0x07242e thru 0x077c4a (0x581d bytes) 000122[0] 1788390850.096971 punch from 0x0fd038 to 0x0feb45 (0x1b0e bytes) 000124[1] 1788390850.149945 trunc from 0x27310e to 0x48cc59 (0x219b4c bytes) 000125[1] 1788390850.207269 punch from 0x10ca02 to 0x117fc6 (0xb5c5 bytes) 000126[0] 1788390850.231314 mapwrite 0x75006a thru 0x750cab (0xc42 bytes) 000127[0] 1788390850.355235 read 0x54cb1a thru 0x554e3d (0x8324 bytes) 000129[0] 1788390850.455708 mapread 0x2272ae thru 0x2363cd (0xf120 bytes) 000130[0] 1788390850.515694 trunc from 0x750cac to 0x2972e5 (0x4b99c6 bytes) 000134[1] 1788390850.627143 mapread 0x015fce thru 0x01caa6 (0x6ad9 bytes) 000136[0] 1788390850.835763 write 0x73fe36 thru 0x7448b5 (0x4a80 bytes) 000138[1] 1788390850.867981 mapread 0x6e2e67 thru 0x6e7e70 (0x500a bytes) 000142[1] 1788390850.957682 punch from 0x4062c9 to 0x410430 (0xa168 bytes) 000143[1] 1788390851.005107 trunc from 0x7448b6 to 0x5d0c42 (0x173c73 bytes) 000147[0] 1788390851.082516 read 0x572b82 thru 0x57457a (0x19f9 bytes) 000150[1] 1788390851.113192 punch from 0x113ffb to 0x11bc7a (0x7c80 bytes) 000151[0] 1788390851.135258 read 0x580850 thru 0x58de5b (0xd60c bytes) 000153[1] 1788390851.148165 mapread 0x41d5b0 thru 0x427114 (0x9b65 bytes) 000154[1] 1788390851.220208 write 0x108451 thru 0x10aa46 (0x25f6 bytes) 000155[1] 1788390851.259517 mapread 0x05cd04 thru 0x067388 (0xa685 bytes) 000156[1] 1788390851.332477 mapwrite 0x22e628 thru 0x233731 (0x510a bytes) 000157[0] 1788390851.446139 mapwrite 0x3fec08 thru 0x401613 (0x2a0c bytes) 000158[1] 1788390851.642764 mapwrite 0x03e487 thru 0x0446c9 (0x6243 bytes) 000159[0] 1788390851.815926 mapread 0x24e69b thru 0x25e4e0 (0xfe46 bytes) 000161[0] 1788390851.888951 write 0x3222dc thru 0x32b585 (0x92aa bytes) 000162[1] 1788390851.920157 read 0x3c08a3 thru 0x3c326a (0x29c8 bytes) 000163[0] 1788390851.949871 mapread 0x0cd7c3 thru 0x0d7ba0 (0xa3de bytes) 000168[1] 1788390851.985670 trunc from 0x5d0c43 to 0x0efa86 (0x4e11bc bytes) 000172[0] 1788390852.100533 punch from 0x0a8ac2 to 0x0a90a5 (0x5e4 bytes) 000173[1] 1788390852.140505 mapread 0x08adeb thru 0x08b220 (0x436 bytes) 000174[1] 1788390852.171535 trunc from 0x0efa87 to 0x3b9866 (0x2c9de0 bytes) 000177[0] 1788390852.208317 mapwrite 0x1b6cb8 thru 0x1c44f2 (0xd83b bytes) 000180[0] 1788390852.368691 trunc from 0x3b9867 to 0x528619 (0x16edb3 bytes) 000181[1] 1788390852.414829 mapread 0x44f09a thru 0x45a4f8 (0xb45f bytes) 000182[0] 1788390852.490417 read 0x0e5549 thru 0x0ec1db (0x6c93 bytes) 000183[0] 1788390852.536693 write 0x105f35 thru 0x1080b0 (0x217c bytes) 000187[1] 1788390852.587051 mapwrite 0x19ba8f thru 0x1a9aff (0xe071 bytes) 000189[1] 1788390852.753708 punch from 0x1625d3 to 0x1705fd (0xe02b bytes) 000190[1] 1788390852.795150 punch from 0x45afdd to 0x4652f1 (0xa315 bytes) 000193[1] 1788390852.809731 write 0x6624dc thru 0x66dd06 (0xb82b bytes) 000194[0] 1788390852.829356 trunc from 0x66dd07 to 0x000cd7 (0x66d02f bytes) 000195[0] 1788390852.899405 mapread 0x000783 thru 0x000cd7 (0x555 bytes) 000196[0] 1788390852.929134 trunc from 0x000cd8 to 0x3c849f (0x3c77c8 bytes) 000197[0] 1788390852.970303 write 0x28089c thru 0x289765 (0x8eca bytes) 000198[0] 1788390852.990760 mapwrite 0x20aef5 thru 0x2192aa (0xe3b6 bytes) 000199[1] 1788390853.098782 punch from 0x049bb1 to 0x058553 (0xe9a3 bytes) 000200[0] 1788390853.129818 write 0x6afc90 thru 0x6bda25 (0xdd96 bytes) 000201[0] 1788390853.147922 mapwrite 0x86f579 thru 0x877ed2 (0x895a bytes) 000202[0] 1788390853.297205 write 0x9d864a thru 0x9e3270 (0xac27 bytes) 000205[0] 1788390853.385399 punch from 0x2a07e0 to 0x2a4d67 (0x4588 bytes) 000207[1] 1788390853.393615 mapwrite 0x4d219a thru 0x4d265e (0x4c5 bytes) 000208[1] 1788390853.540901 read 0x0643e7 thru 0x064e16 (0xa30 bytes) 000210[1] 1788390853.582747 mapwrite 0x05ccc0 thru 0x062ead (0x61ee bytes) truncating to largest ever: 0x93a1b5 000211[0] 1788390853.704933 trunc from 0x9e3271 to 0x93a1b4 (0xa90bc bytes) 000213[0] 1788390853.778274 punch from 0x61bb2a to 0x626a84 (0xaf5b bytes) 000217[1] 1788390853.820168 read 0x20308f thru 0x207b3a (0x4aac bytes) 000220[1] 1788390853.863785 trunc from 0x93a1b5 to 0x22b711 (0x70eaa3 bytes) 000221[0] 1788390853.911817 punch from 0x1f146b to 0x1fae1d (0x99b3 bytes) 000222[0] 1788390853.941764 mapwrite 0x0700e9 thru 0x07086d (0x785 bytes) 000226[0] 1788390854.047318 mapwrite 0x49b88c thru 0x49c57d (0xcf2 bytes) 000229[1] 1788390854.330055 trunc from 0x49c57e to 0x02e266 (0x46e317 bytes) 000231[1] 1788390854.450031 trunc from 0x02e267 to 0x18aa37 (0x15c7d1 bytes) 000232[0] 1788390854.497490 mapread 0x0f9f4f thru 0x0fb367 (0x1419 bytes) 000233[0] 1788390854.592846 mapread 0x0b562e thru 0x0c27b3 (0xd186 bytes) 000235[1] 1788390854.677900 mapwrite 0x14db1f thru 0x14fc43 (0x2125 bytes) 000236[0] 1788390854.760056 mapwrite 0x96a8f5 thru 0x974bf7 (0xa303 bytes) 000237[1] 1788390854.876304 read 0x56fefb thru 0x57f12e (0xf234 bytes) 000239[1] 1788390854.979399 write 0x9dd592 thru 0x9e08c9 (0x3338 bytes) 000240[0] 1788390854.989031 mapwrite 0x1e691e thru 0x1f169a (0xad7d bytes) 000241[1] 1788390855.079062 write 0x362421 thru 0x362b96 (0x776 bytes) 000242[1] 1788390855.128158 mapwrite 0x632be3 thru 0x63cc47 (0xa065 bytes) 000244[0] 1788390855.286312 punch from 0x7467bb to 0x74b4d0 (0x4d16 bytes) 000245[1] 1788390855.317575 punch from 0x933376 to 0x93e01f (0xacaa bytes) 000246[0] 1788390855.330949 mapwrite 0x25bd32 thru 0x261176 (0x5445 bytes) 000247[1] 1788390855.426471 punch from 0x91e156 to 0x92c6a9 (0xe554 bytes) 000248[1] 1788390855.458444 write 0x510c58 thru 0x514904 (0x3cad bytes) 000249[0] 1788390855.475843 mapread 0x8a83aa thru 0x8a8ed1 (0xb28 bytes) 000252[0] 1788390855.524980 trunc from 0x9e08ca to 0x7a0e62 (0x23fa67 bytes) 000257[1] 1788390855.723050 write 0x897fb5 thru 0x8a43b4 (0xc400 bytes) 000259[0] 1788390855.767113 mapread 0x4cfa3d thru 0x4da950 (0xaf14 bytes) 000260[0] 1788390855.816190 mapread 0x854b3c thru 0x855b68 (0x102d bytes) 000263[1] 1788390855.916688 mapread 0x014bfe thru 0x01f1ff (0xa602 bytes) 000265[0] 1788390856.033079 mapwrite 0x8c743e thru 0x8cef5c (0x7b1f bytes) 000266[1] 1788390856.141905 write 0x45a888 thru 0x46759c (0xcd15 bytes) 000267[0] 1788390856.168025 write 0x74ee43 thru 0x753085 (0x4243 bytes) 000268[1] 1788390856.195075 read 0x75eab6 thru 0x769e03 (0xb34e bytes) 000269[0] 1788390856.229693 mapread 0x5f6d83 thru 0x5fd4c9 (0x6747 bytes) 000271[0] 1788390856.295216 mapread 0x47fd14 thru 0x48e7dd (0xeaca bytes) 000272[0] 1788390856.331369 mapwrite 0x590f3c thru 0x59e866 (0xd92b bytes) 000274[1] 1788390856.515324 punch from 0x849f60 to 0x84e255 (0x42f6 bytes) 000275[0] 1788390856.576028 trunc from 0x8cef5d to 0x4088d8 (0x4c6684 bytes) 000277[1] 1788390856.666117 mapwrite 0x1ccdfe thru 0x1cdd92 (0xf95 bytes) 000279[0] 1788390856.757550 punch from 0x1bcd29 to 0x1bd483 (0x75b bytes) 000280[0] 1788390856.788254 write 0x51c268 thru 0x52506c (0x8e05 bytes) 000281[1] 1788390856.790297 mapread 0x455264 thru 0x45761d (0x23ba bytes) 000283[1] 1788390856.844917 trunc from 0x52506d to 0x1f8306 (0x32cd66 bytes) 000286[0] 1788390856.965492 trunc from 0x1f8307 to 0x431e77 (0x239b71 bytes) 000287[1] 1788390857.013582 punch from 0x23503f to 0x23695b (0x191d bytes) 000288[1] 1788390857.044352 trunc from 0x431e78 to 0x4111c0 (0x20cb7 bytes) 000295[0] 1788390857.130825 mapread 0x06876c thru 0x069738 (0xfcd bytes) 000296[1] 1788390857.175169 punch from 0x35f50c to 0x365100 (0x5bf5 bytes) 000298[0] 1788390857.187802 mapwrite 0x385554 thru 0x387fc3 (0x2a70 bytes) 000299[1] 1788390857.289952 mapread 0x258ba6 thru 0x262155 (0x95b0 bytes) 000305[1] 1788390857.362196 mapwrite 0x0142b5 thru 0x01e2b0 (0x9ffc bytes) 000306[0] 1788390857.439068 mapread 0x050a5a thru 0x051f36 (0x14dd bytes) 000309[1] 1788390857.519027 read 0x2bc018 thru 0x2bd9bb (0x19a4 bytes) 000312[0] 1788390857.542739 punch from 0x1e8959 to 0x1eb50f (0x2bb7 bytes) 000315[0] 1788390857.612508 read 0x290fbb thru 0x294114 (0x315a bytes) 000316[0] 1788390857.639120 write 0x1b9dba thru 0x1c750d (0xd754 bytes) 000317[0] 1788390857.660443 mapread 0x31f206 thru 0x320604 (0x13ff bytes) 000318[1] 1788390857.693888 read 0x212caa thru 0x21a93c (0x7c93 bytes) 000320[0] 1788390857.732992 write 0x5fca15 thru 0x601ae6 (0x50d2 bytes) 000321[1] 1788390857.744255 trunc from 0x601ae7 to 0x8d4c42 (0x2d315c bytes) 000323[1] 1788390857.825850 mapwrite 0x787b28 thru 0x792c29 (0xb102 bytes) truncating to largest ever: 0x9bf31f 000324[0] 1788390857.928883 trunc from 0x8d4c43 to 0x9bf31e (0xea6dc bytes) 000326[0] 1788390858.030837 punch from 0x1ecdd2 to 0x1f6539 (0x9768 bytes) 000327[1] 1788390858.036343 read 0x2ee29a thru 0x2f990f (0xb676 bytes) 000328[1] 1788390858.078772 mapread 0x7a67b8 thru 0x7ac6b5 (0x5efe bytes) 000329[1] 1788390858.121668 punch from 0x13bb9f to 0x143192 (0x75f4 bytes) 000330[1] 1788390858.141735 mapwrite 0x13d9b3 thru 0x13fbe3 (0x2231 bytes) 000331[1] 1788390858.198946 write 0x0deec9 thru 0x0e2444 (0x357c bytes) 000335[1] 1788390858.238416 mapwrite 0x5d6fc3 thru 0x5de67b (0x76b9 bytes) 000336[1] 1788390858.303051 write 0x8eeeec thru 0x8f3818 (0x492d bytes) 000337[0] 1788390858.320148 mapwrite 0x06e63d thru 0x071cb5 (0x3679 bytes) 000339[1] 1788390858.488976 mapwrite 0x564711 thru 0x569e76 (0x5766 bytes) 000341[1] 1788390858.582941 punch from 0x50c0e5 to 0x5115de (0x54fa bytes) 000342[0] 1788390858.602099 write 0x27094d thru 0x27ad4d (0xa401 bytes) 000343[1] 1788390858.678057 write 0x29a6de thru 0x2a7fbd (0xd8e0 bytes) 000347[1] 1788390858.799291 read 0x45ca75 thru 0x464c65 (0x81f1 bytes) 000348[0] 1788390858.829855 trunc from 0x9bf31f to 0x1f20f1 (0x7cd22d bytes) 000349[0] 1788390858.916956 read 0x09c33b thru 0x0a87b3 (0xc479 bytes) 000352[0] 1788390858.995003 trunc from 0x1f20f2 to 0x049598 (0x1a8b59 bytes) 000353[1] 1788390859.036980 punch from 0x03cd8f to 0x047bf3 (0xae65 bytes) 000354[0] 1788390859.068641 read 0x000725 thru 0x009610 (0x8eec bytes) 000355[0] 1788390859.100091 trunc from 0x049599 to 0x4b9015 (0x46fa7d bytes) 000356[0] 1788390859.167426 mapread 0x3ef379 thru 0x3fd690 (0xe318 bytes) 000357[0] 1788390859.251141 mapwrite 0x34b9c1 thru 0x34be23 (0x463 bytes) 000359[1] 1788390859.383388 punch from 0x2fca05 to 0x2fff80 (0x357c bytes) 000360[0] 1788390859.424411 read 0x452bb3 thru 0x462b4c (0xff9a bytes) 000361[1] 1788390859.471022 write 0x1bd6a6 thru 0x1c36e9 (0x6044 bytes) 000362[0] 1788390859.493202 mapread 0x1eae8a thru 0x1f897d (0xdaf4 bytes) 000367[0] 1788390859.600074 trunc from 0x4b9016 to 0x24ad0e (0x26e307 bytes) 000369[1] 1788390859.691060 mapwrite 0x6ac619 thru 0x6b4e91 (0x8879 bytes) 000371[0] 1788390859.819052 trunc from 0x6b4e92 to 0x543edc (0x170fb5 bytes) 000372[1] 1788390859.910596 mapwrite 0x11bd0f thru 0x11e62e (0x2920 bytes) 000374[1] 1788390860.004052 mapwrite 0x3bb453 thru 0x3c6f9a (0xbb48 bytes) 000376[0] 1788390860.202071 mapwrite 0x7964dd thru 0x7a4315 (0xde39 bytes) 000377[1] 1788390860.312519 mapread 0x225c91 thru 0x226072 (0x3e2 bytes) 000378[1] 1788390860.440441 mapwrite 0x9d992f thru 0x9dd899 (0x3f6b bytes) 000379[1] 1788390860.663272 mapwrite 0x097372 thru 0x09dcba (0x6949 bytes) 000380[0] 1788390860.764107 read 0x4c0342 thru 0x4ccc33 (0xc8f2 bytes) 000381[1] 1788390860.801214 mapwrite 0x4ce32c thru 0x4d76de (0x93b3 bytes) 000382[0] 1788390860.913148 trunc from 0x9dd89a to 0x76876b (0x27512e bytes) 000385[0] 1788390860.970640 mapread 0x6fe36f thru 0x6fe4a9 (0x13b bytes) 000386[1] 1788390861.026214 read 0x36aa66 thru 0x36c024 (0x15bf bytes) 000387[1] 1788390861.048894 write 0x6a1c9a thru 0x6b0572 (0xe8d9 bytes) 000389[0] 1788390861.087173 mapread 0x5c6c2d thru 0x5d1f06 (0xb2da bytes) 000391[1] 1788390861.160443 mapwrite 0x92c618 thru 0x92fd4e (0x3737 bytes) 000393[1] 1788390861.315207 read 0x8828fa thru 0x883565 (0xc6c bytes) 000394[0] 1788390861.349164 mapread 0x7f7426 thru 0x8058f5 (0xe4d0 bytes) 000397[0] 1788390861.441524 punch from 0x75abc7 to 0x75b051 (0x48b bytes) 000399[1] 1788390861.470852 punch from 0x1762db to 0x179ef9 (0x3c1f bytes) 000401[0] 1788390861.510860 mapread 0x7a3895 thru 0x7a52ff (0x1a6b bytes) 000403[0] 1788390861.555124 mapwrite 0x40e7fe thru 0x4142d4 (0x5ad7 bytes) 000406[0] 1788390861.665827 write 0x650054 thru 0x65f9b5 (0xf962 bytes) 000407[1] 1788390861.689035 read 0x104bcd thru 0x10e34a (0x977e bytes) 000408[1] 1788390861.705303 mapread 0x149a8f thru 0x14ffaa (0x651c bytes) 000409[1] 1788390861.741280 mapwrite 0x0fa635 thru 0x10199f (0x736b bytes) 000413[1] 1788390861.857969 write 0x58a4ce thru 0x595223 (0xad56 bytes) 000414[0] 1788390861.905443 mapwrite 0x55c76f thru 0x55e8e4 (0x2176 bytes) 000416[1] 1788390862.042929 mapread 0x0b01c6 thru 0x0b40ba (0x3ef5 bytes) 000417[0] 1788390862.066337 read 0x569e2b thru 0x575638 (0xb80e bytes) 000421[1] 1788390862.166855 punch from 0x7f055f to 0x7fe2fe (0xdda0 bytes) 000423[0] 1788390862.210009 trunc from 0x92fd4f to 0x45574a (0x4da604 bytes) 000424[1] 1788390862.308860 mapread 0x09a016 thru 0x09baf0 (0x1adb bytes) 000425[1] 1788390862.359297 mapread 0x230e95 thru 0x232b72 (0x1cde bytes) 000427[0] 1788390862.432895 punch from 0x0db4f2 to 0x0e79de (0xc4ed bytes) 000428[0] 1788390862.457309 punch from 0x3d9f26 to 0x3e29bc (0x8a97 bytes) 000429[1] 1788390862.480216 punch from 0x032181 to 0x034998 (0x2818 bytes) 000432[0] 1788390862.541862 mapread 0x08a89c thru 0x096694 (0xbdf9 bytes) 000434[1] 1788390862.602050 mapwrite 0x8bfc0d thru 0x8c75fe (0x79f2 bytes) 000435[1] 1788390862.725771 mapwrite 0x936322 thru 0x9393f4 (0x30d3 bytes) 000438[0] 1788390862.870435 read 0x6fae1a thru 0x6fef42 (0x4129 bytes) 000439[0] 1788390862.896932 mapwrite 0x94820b thru 0x94f200 (0x6ff6 bytes) 000440[1] 1788390862.998289 mapwrite 0x072c27 thru 0x0827d0 (0xfbaa bytes) 000442[1] 1788390863.155103 mapread 0x43de10 thru 0x444ae4 (0x6cd5 bytes) 000443[1] 1788390863.201429 mapread 0x282f4d thru 0x287b16 (0x4bca bytes) 000447[1] 1788390863.237259 read 0x7f6c75 thru 0x80435b (0xd6e7 bytes) 000450[1] 1788390863.277167 write 0x5e595b thru 0x5eb67d (0x5d23 bytes) 000451[0] 1788390863.326088 write 0x1c05bd thru 0x1c3c5b (0x369f bytes) 000453[1] 1788390863.381629 read 0x7b6b40 thru 0x7bf8cd (0x8d8e bytes) 000454[1] 1788390863.410064 punch from 0x12985f to 0x13025e (0x6a00 bytes) 000455[0] 1788390863.435589 mapread 0x2199ef thru 0x22760a (0xdc1c bytes) 000456[0] 1788390863.503985 read 0x882fed thru 0x88e017 (0xb02b bytes) 000458[1] 1788390863.528083 read 0x94916d thru 0x94cb41 (0x39d5 bytes) 000460[1] 1788390863.575894 mapread 0x635cbf thru 0x6443df (0xe721 bytes) 000461[1] 1788390863.603828 mapread 0x2859d8 thru 0x286a76 (0x109f bytes) 000462[1] 1788390863.626257 mapwrite 0x76a847 thru 0x77399c (0x9156 bytes) 000463[0] 1788390863.694165 trunc from 0x94f201 to 0x10f860 (0x83f9a0 bytes) 000465[0] 1788390863.746869 trunc from 0x10f861 to 0x918ffa (0x80979a bytes) 000466[1] 1788390863.764743 write 0x4bba86 thru 0x4bc51f (0xa9a bytes) 000467[0] 1788390863.808333 read 0x3d6224 thru 0x3de1ba (0x7f97 bytes) 000468[0] 1788390863.828803 write 0x5e9d69 thru 0x5f0a0d (0x6ca5 bytes) 000469[1] 1788390863.856214 mapwrite 0x168df6 thru 0x177b6b (0xed76 bytes) 000470[1] 1788390863.957166 punch from 0x354d4c to 0x364619 (0xf8ce bytes) 000471[0] 1788390863.973063 mapwrite 0x1c75f8 thru 0x1c9624 (0x202d bytes) 000472[1] 1788390864.100007 mapwrite 0x101e86 thru 0x105856 (0x39d1 bytes) 000473[1] 1788390864.266417 mapread 0x2146d0 thru 0x21ffc3 (0xb8f4 bytes) 000474[0] 1788390864.348012 punch from 0x79f056 to 0x7a7f4a (0x8ef5 bytes) 000475[1] 1788390864.367926 mapwrite 0x30ed62 thru 0x31d9cb (0xec6a bytes) 000476[0] 1788390864.474210 write 0x1f8e23 thru 0x1f8e27 (0x5 bytes) 000477[0] 1788390864.514963 write 0x7212e8 thru 0x721a27 (0x740 bytes) 000478[1] 1788390864.551302 trunc from 0x918ffb to 0x42627e (0x4f2d7c bytes) 000479[0] 1788390864.614516 mapread 0x2f30b1 thru 0x2fa411 (0x7361 bytes) 000480[1] 1788390864.667947 read 0x023d20 thru 0x0287aa (0x4a8b bytes) 000482[1] 1788390864.683650 mapread 0x1768e3 thru 0x177c1b (0x1339 bytes) 000483[1] 1788390864.709906 mapread 0x36c4f4 thru 0x37babc (0xf5c9 bytes) 000484[0] 1788390864.736154 mapread 0x05f743 thru 0x060574 (0xe32 bytes) 000487[1] 1788390864.798230 trunc from 0x42627f to 0x134f80 (0x2f12fe bytes) 000491[0] 1788390864.870712 write 0x9b68a2 thru 0x9c10d3 (0xa832 bytes) 000494[1] 1788390864.957230 mapread 0x4010bf thru 0x4079c5 (0x6907 bytes) 000498[0] 1788390865.112737 trunc from 0x9c10d4 to 0x3caca2 (0x5f6431 bytes) All operations completed A-OK! Chance of close/open is 1 in 50 Seed set to 865 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read 000003[0] 1788390865.910307 mapwrite 0x1bb000 thru 0x1be26c (0x326d bytes) 000004[1] 1788390866.033253 mapread 0x0d2000 thru 0x0e0cfa (0xecfb bytes) 000005[1] 1788390866.083071 write 0x021000 thru 0x02d99e (0xc99f bytes) 000006[1] 1788390866.099588 mapwrite 0x82b000 thru 0x82d485 (0x2486 bytes) truncating to largest ever: 0x442841 000009[0] 1788390866.251187 trunc from 0x82d486 to 0x442840 (0x3eac45 bytes) 000010[0] 1788390866.306228 punch from 0x238f85 to 0x23ea6b (0x5ae7 bytes) 000011[0] 1788390866.323282 write_OD 0x451000 thru 0x457fff (0x7000 bytes) 000014[0] 1788390866.385199 write_OD 0x001000 thru 0x003fff (0x3000 bytes) 000015[1] 1788390866.450762 trunc from 0x458000 to 0x12ea29 (0x3295d6 bytes) truncating to largest ever: 0x5d2db7 000020[0] 1788390866.571407 trunc from 0x12ea2a to 0x5d2db6 (0x4a438d bytes) 000021[1] 1788390866.605150 mapwrite 0x7b6000 thru 0x7b8931 (0x2932 bytes) 000022[1] 1788390866.674747 mapread 0x028000 thru 0x02a719 (0x271a bytes) 000023[0] 1788390866.724195 mapread 0x368000 thru 0x371d23 (0x9d24 bytes) 000024[0] 1788390866.754896 punch from 0x60b8eb to 0x60d7af (0x1ec5 bytes) 000025[1] 1788390866.780890 read 0x35d000 thru 0x3687cd (0xb7ce bytes) 000026[1] 1788390866.802959 write 0x1e3000 thru 0x1e4690 (0x1691 bytes) 000027[0] 1788390866.821234 read_OD 0x117000 thru 0x11dfff (0x7000 bytes) 000028[1] 1788390866.852538 write 0x3ef000 thru 0x3f4080 (0x5081 bytes) 000029[0] 1788390866.884216 mapread 0x55b000 thru 0x567853 (0xc854 bytes) 000030[0] 1788390866.921727 mapread 0x216000 thru 0x224855 (0xe856 bytes) 000032[0] 1788390866.958983 read_OD 0x071000 thru 0x071fff (0x1000 bytes) truncating to largest ever: 0x65d35e 000033[1] 1788390866.963585 trunc from 0x7b8932 to 0x65d35d (0x15b5d4 bytes) 000034[0] 1788390867.014320 read_OD 0x3d1000 thru 0x3d4fff (0x4000 bytes) 000035[1] 1788390867.040924 mapread 0x419000 thru 0x420fd3 (0x7fd4 bytes) 000037[1] 1788390867.059095 mapwrite 0x849000 thru 0x84d1b5 (0x41b6 bytes) 000038[0] 1788390867.138683 trunc from 0x84d1b6 to 0x50ee74 (0x33e341 bytes) truncating to largest ever: 0x67ca1f 000039[0] 1788390867.172017 trunc from 0x50ee75 to 0x67ca1e (0x16dbaa bytes) 000041[0] 1788390867.186215 mapwrite 0x05f000 thru 0x067fbe (0x8fbf bytes) 000042[1] 1788390867.297031 punch from 0x2bbf0a to 0x2c01e1 (0x42d8 bytes) 000043[0] 1788390867.342039 punch from 0x2ab244 to 0x2b6be1 (0xb99e bytes) 000046[0] 1788390867.370106 mapread 0x5f8000 thru 0x606506 (0xe507 bytes) 000047[1] 1788390867.422291 write 0x7f1000 thru 0x7f8f68 (0x7f69 bytes) 000051[0] 1788390867.470173 mapwrite 0x7f3000 thru 0x7f4cec (0x1ced bytes) 000052[0] 1788390867.585321 read_OD 0x695000 thru 0x699fff (0x5000 bytes) 000053[0] 1788390867.618772 punch from 0x4e9898 to 0x4f44d4 (0xac3d bytes) 000054[0] 1788390867.654178 read_OD 0x667000 thru 0x673fff (0xd000 bytes) 000056[0] 1788390867.680023 mapwrite 0x085000 thru 0x08ab88 (0x5b89 bytes) 000057[0] 1788390867.807639 read_OD 0x072000 thru 0x074fff (0x3000 bytes) 000058[0] 1788390867.825582 mapwrite 0x422000 thru 0x422b09 (0xb0a bytes) 000059[1] 1788390867.903172 mapwrite 0x61e000 thru 0x6226c6 (0x46c7 bytes) 000061[0] 1788390867.999014 punch from 0x043846 to 0x04671a (0x2ed5 bytes) 000063[0] 1788390868.013361 mapwrite 0x937000 thru 0x94009a (0x909b bytes) 000064[0] 1788390868.100705 punch from 0x3e5340 to 0x3e7763 (0x2424 bytes) 000065[0] 1788390868.105162 mapread 0x863000 thru 0x86fdeb (0xcdec bytes) 000066[0] 1788390868.122939 mapread 0x3ee000 thru 0x3f22a6 (0x42a7 bytes) 000069[1] 1788390868.158864 punch from 0x6017e1 to 0x60aefb (0x971b bytes) 000071[0] 1788390868.229656 read_OD 0x624000 thru 0x62bfff (0x8000 bytes) 000072[0] 1788390868.264338 punch from 0x57d0ae to 0x57d779 (0x6cc bytes) 000073[0] 1788390868.293136 mapread 0x696000 thru 0x696a42 (0xa43 bytes) 000074[1] 1788390868.311499 mapread 0x2a0000 thru 0x2ac3c1 (0xc3c2 bytes) 000078[0] 1788390868.360666 read_OD 0x298000 thru 0x2a4fff (0xd000 bytes) 000079[0] 1788390868.371123 mapread 0x0be000 thru 0x0c6f90 (0x8f91 bytes) 000080[0] 1788390868.390875 punch from 0x7f370a to 0x7fe00d (0xa904 bytes) 000081[0] 1788390868.409775 read_OD 0x526000 thru 0x52afff (0x5000 bytes) truncating to largest ever: 0x92cefa 000082[1] 1788390868.434037 trunc from 0x94009b to 0x92cef9 (0x131a1 bytes) 000087[1] 1788390868.534924 trunc from 0x92cefa to 0x0c00b4 (0x86ce45 bytes) 000089[1] 1788390868.573708 mapread 0x05f000 thru 0x06bbb1 (0xcbb2 bytes) 000094[1] 1788390868.635853 mapwrite 0x3d2000 thru 0x3e1786 (0xf787 bytes) 000095[0] 1788390868.737154 mapwrite 0x4a1000 thru 0x4abae2 (0xaae3 bytes) 000096[1] 1788390868.832682 mapread 0x067000 thru 0x075604 (0xe605 bytes) 000097[0] 1788390868.896226 mapread 0x134000 thru 0x13bbb7 (0x7bb8 bytes) 000100[1] 1788390868.922150 mapwrite 0x277000 thru 0x28343c (0xc43d bytes) 000106[1] 1788390869.152121 trunc from 0x4abae3 to 0x18eabf (0x31d023 bytes) 000107[0] 1788390869.218805 punch from 0x13273b to 0x13dd9f (0xb665 bytes) 000109[0] 1788390869.269420 mapwrite 0x358000 thru 0x35c71d (0x471e bytes) 000110[1] 1788390869.415439 punch from 0x265e0d to 0x272146 (0xc33a bytes) 000112[0] 1788390869.469582 mapread 0x02a000 thru 0x0321e3 (0x81e4 bytes) 000114[0] 1788390869.523051 trunc from 0x35c71e to 0x4c101f (0x164902 bytes) 000117[1] 1788390869.574622 mapread 0x415000 thru 0x422a65 (0xda66 bytes) 000118[0] 1788390869.620543 mapread 0x0a8000 thru 0x0aa960 (0x2961 bytes) 000122[0] 1788390869.635123 mapwrite 0x300000 thru 0x305063 (0x5064 bytes) 000123[0] 1788390869.770978 punch from 0x25890f to 0x25ae47 (0x2539 bytes) 000128[1] 1788390869.814669 mapread 0x46a000 thru 0x476027 (0xc028 bytes) 000132[1] 1788390869.872258 punch from 0x0a8a94 to 0x0b2046 (0x95b3 bytes) 000134[0] 1788390869.890132 trunc from 0x4c1020 to 0x796b1f (0x2d5b00 bytes) 000135[1] 1788390869.969811 read 0x542000 thru 0x548e24 (0x6e25 bytes) 000137[0] 1788390869.999729 punch from 0x664322 to 0x66621f (0x1efe bytes) 000139[1] 1788390870.012582 write 0x694000 thru 0x699966 (0x5967 bytes) 000141[0] 1788390870.037228 write_OD 0x92d000 thru 0x938fff (0xc000 bytes) 000143[1] 1788390870.101877 read 0x46c000 thru 0x4786aa (0xc6ab bytes) 000145[0] 1788390870.112765 read_OD 0x482000 thru 0x487fff (0x6000 bytes) 000149[0] 1788390870.122875 mapread 0x796000 thru 0x796c3c (0xc3d bytes) 000151[1] 1788390870.128980 punch from 0x497273 to 0x4a6988 (0xf716 bytes) 000153[1] 1788390870.164765 write 0x6ef000 thru 0x6fb907 (0xc908 bytes) 000154[1] 1788390870.187426 read 0x66f000 thru 0x677bb9 (0x8bba bytes) 000155[0] 1788390870.206994 read_OD 0x337000 thru 0x344fff (0xe000 bytes) 000157[0] 1788390870.239610 punch from 0x35ae97 to 0x363360 (0x84ca bytes) 000158[0] 1788390870.256435 read_OD 0x626000 thru 0x630fff (0xb000 bytes) 000159[1] 1788390870.270066 write 0x362000 thru 0x365339 (0x333a bytes) 000160[0] 1788390870.308026 punch from 0x001787 to 0x0093aa (0x7c24 bytes) 000163[1] 1788390870.335167 punch from 0x230606 to 0x23c148 (0xbb43 bytes) 000167[0] 1788390870.371526 punch from 0x3f1742 to 0x3f7f1f (0x67de bytes) 000168[1] 1788390870.384325 trunc from 0x939000 to 0x1e1e8e (0x757171 bytes) 000170[1] 1788390870.422088 read 0x0a0000 thru 0x0a9bc6 (0x9bc7 bytes) 000172[0] 1788390870.478267 trunc from 0x1e1e8f to 0x089460 (0x158a2e bytes) 000173[1] 1788390870.507266 read 0x072000 thru 0x0742b9 (0x22ba bytes) 000176[0] 1788390870.535126 trunc from 0x089461 to 0x0144b8 (0x74fa8 bytes) 000177[0] 1788390870.576413 punch from 0x012e2c to 0x0144b8 (0x168d bytes) 000178[1] 1788390870.592132 write 0x72c000 thru 0x739c36 (0xdc37 bytes) 000179[1] 1788390870.620195 mapread 0x6da000 thru 0x6dd166 (0x3167 bytes) 000180[0] 1788390870.693668 trunc from 0x739c37 to 0x0191cd (0x720a69 bytes) 000181[1] 1788390870.734012 mapwrite 0x1af000 thru 0x1bc3e3 (0xd3e4 bytes) 000182[0] 1788390870.879772 mapread 0x084000 thru 0x08d79c (0x979d bytes) 000184[1] 1788390870.923569 trunc from 0x1bc3e4 to 0x668cf4 (0x4ac911 bytes) 000185[0] 1788390870.956832 read_OD 0x58a000 thru 0x593fff (0xa000 bytes) 000186[1] 1788390870.983587 punch from 0x5eab60 to 0x5f4d97 (0xa238 bytes) 000187[1] 1788390871.002163 punch from 0x5fa8b5 to 0x6082c1 (0xda0d bytes) 000188[1] 1788390871.006912 mapwrite 0x1a6000 thru 0x1a93f6 (0x33f7 bytes) 000189[0] 1788390871.109576 punch from 0x4039a7 to 0x406d85 (0x33df bytes) 000191[0] 1788390871.125373 write_OD 0x48b000 thru 0x48efff (0x4000 bytes) 000192[0] 1788390871.173611 read_OD 0x387000 thru 0x392fff (0xc000 bytes) 000194[0] 1788390871.188099 write_OD 0x030000 thru 0x034fff (0x5000 bytes) 000195[1] 1788390871.244339 mapwrite 0x187000 thru 0x189e5e (0x2e5f bytes) 000196[0] 1788390871.304231 mapread 0x48b000 thru 0x4939a1 (0x89a2 bytes) 000197[0] 1788390871.355246 mapread 0x0a6000 thru 0x0af94d (0x994e bytes) 000198[1] 1788390871.388941 trunc from 0x668cf5 to 0x1fcdf0 (0x46bf04 bytes) 000199[1] 1788390871.420801 mapread 0x18e000 thru 0x194f34 (0x6f35 bytes) 000202[1] 1788390871.447541 punch from 0x114e62 to 0x123565 (0xe704 bytes) 000203[0] 1788390871.451726 write_OD 0x29b000 thru 0x2a2fff (0x8000 bytes) 000206[1] 1788390871.503669 read 0x21e000 thru 0x227c2d (0x9c2e bytes) 000207[0] 1788390871.559341 mapread 0x0a9000 thru 0x0add1f (0x4d20 bytes) 000209[0] 1788390871.566791 punch from 0x1fa22f to 0x2067cd (0xc59f bytes) 000210[1] 1788390871.589482 read 0x16d000 thru 0x1769a0 (0x99a1 bytes) 000211[1] 1788390871.665871 trunc from 0x2a3000 to 0x090005 (0x212ffa bytes) 000212[1] 1788390871.710830 mapread 0x049000 thru 0x0537a9 (0xa7aa bytes) 000213[1] 1788390871.756075 trunc from 0x090006 to 0x795e92 (0x705e8d bytes) 000214[0] 1788390871.787056 trunc from 0x795e93 to 0x751ddf (0x440b3 bytes) 000216[0] 1788390871.805250 read_OD 0x68a000 thru 0x68ffff (0x6000 bytes) 000217[0] 1788390871.829341 write_OD 0x9aa000 thru 0x9b0fff (0x7000 bytes) 000219[1] 1788390871.895106 mapread 0x965000 thru 0x966b3e (0x1b3f bytes) 000220[1] 1788390872.032178 mapwrite 0x31f000 thru 0x324729 (0x572a bytes) 000221[1] 1788390872.213669 write 0x803000 thru 0x80352e (0x52f bytes) 000223[0] 1788390872.236029 punch from 0x2a0c0f to 0x2a2a13 (0x1e05 bytes) 000224[0] 1788390872.285774 mapread 0x1fb000 thru 0x2008c0 (0x58c1 bytes) 000225[1] 1788390872.316001 mapwrite 0x242000 thru 0x24a5e0 (0x85e1 bytes) 000226[1] 1788390872.527610 write 0x3e0000 thru 0x3e40c8 (0x40c9 bytes) 000228[1] 1788390872.534057 write 0x458000 thru 0x45ae48 (0x2e49 bytes) 000229[0] 1788390872.566060 mapread 0x8af000 thru 0x8b0a06 (0x1a07 bytes) 000232[0] 1788390872.612153 mapread 0x74d000 thru 0x753c08 (0x6c09 bytes) 000233[0] 1788390872.642252 mapread 0x67a000 thru 0x67eb4d (0x4b4e bytes) 000234[0] 1788390872.656181 punch from 0x30d628 to 0x31d241 (0xfc1a bytes) 000236[0] 1788390872.701723 punch from 0x238b04 to 0x240483 (0x7980 bytes) 000239[0] 1788390872.725656 read_OD 0x223000 thru 0x22afff (0x8000 bytes) 000241[1] 1788390872.757105 read 0x8c4000 thru 0x8cf711 (0xb712 bytes) 000242[0] 1788390872.770832 mapread 0x4b8000 thru 0x4c76ea (0xf6eb bytes) 000243[0] 1788390872.818355 punch from 0x971cfe to 0x97981c (0x7b1f bytes) 000244[1] 1788390872.842367 trunc from 0x9b1000 to 0x81e326 (0x192cd9 bytes) 000245[1] 1788390872.914341 punch from 0x04f7c7 to 0x05a3e8 (0xac22 bytes) 000246[1] 1788390872.933582 read 0x72a000 thru 0x733ac4 (0x9ac5 bytes) truncating to largest ever: 0x9a2d95 000247[1] 1788390872.948988 trunc from 0x81e327 to 0x9a2d94 (0x184a6e bytes) 000249[1] 1788390873.051200 read 0x961000 thru 0x969d98 (0x8d99 bytes) 000250[0] 1788390873.067731 write_OD 0x313000 thru 0x321fff (0xf000 bytes) 000251[1] 1788390873.176703 mapread 0x0ac000 thru 0x0b0d3a (0x4d3b bytes) 000252[1] 1788390873.208959 trunc from 0x9a2d95 to 0x33a416 (0x66897e bytes) 000253[1] 1788390873.317485 trunc from 0x33a417 to 0x6e4c57 (0x3aa841 bytes) 000254[0] 1788390873.366889 read_OD 0x3fd000 thru 0x40bfff (0xf000 bytes) 000255[1] 1788390873.409153 write 0x589000 thru 0x58a1bb (0x11bc bytes) 000256[1] 1788390873.440862 write 0x5c9000 thru 0x5d00db (0x70dc bytes) 000257[0] 1788390873.508790 trunc from 0x6e4c58 to 0x6b01a0 (0x34ab7 bytes) 000260[1] 1788390873.661058 trunc from 0x6b01a1 to 0x208410 (0x4a7d90 bytes) 000264[1] 1788390873.809532 write 0x4f9000 thru 0x4fc8e6 (0x38e7 bytes) 000265[0] 1788390873.836203 write_OD 0x954000 thru 0x962fff (0xf000 bytes) 000266[1] 1788390873.904417 mapread 0x466000 thru 0x4757f2 (0xf7f3 bytes) 000267[1] 1788390873.937914 write 0x2c3000 thru 0x2c47a5 (0x17a6 bytes) 000268[1] 1788390873.960098 trunc from 0x963000 to 0x09672d (0x8cc8d2 bytes) 000269[0] 1788390874.008631 read_OD 0x017000 thru 0x01afff (0x4000 bytes) 000270[1] 1788390874.039292 mapwrite 0x386000 thru 0x38af61 (0x4f62 bytes) 000271[0] 1788390874.155722 punch from 0x314dab to 0x316c76 (0x1ecc bytes) 000272[1] 1788390874.184063 write 0x276000 thru 0x27905e (0x305f bytes) 000278[1] 1788390874.289837 mapwrite 0x3a9000 thru 0x3b1832 (0x8833 bytes) 000279[1] 1788390874.423271 trunc from 0x3b1833 to 0x30aac3 (0xa6d6f bytes) 000280[1] 1788390874.466762 trunc from 0x30aac4 to 0x103545 (0x20757e bytes) 000281[1] 1788390874.513099 mapwrite 0x76f000 thru 0x77df49 (0xef4a bytes) 000284[0] 1788390874.698742 read_OD 0x738000 thru 0x742fff (0xb000 bytes) 000286[1] 1788390874.739685 write 0x829000 thru 0x831108 (0x8109 bytes) 000288[1] 1788390874.775074 read 0x20e000 thru 0x2179f8 (0x99f9 bytes) 000291[0] 1788390874.800947 read_OD 0x3b8000 thru 0x3bdfff (0x6000 bytes) 000295[0] 1788390874.906649 trunc from 0x831109 to 0x1c6b9c (0x66a56c bytes) 000300[0] 1788390875.111708 mapwrite 0x62a000 thru 0x633078 (0x9079 bytes) 000301[0] 1788390875.296320 mapread 0x594000 thru 0x5a3cc6 (0xfcc7 bytes) 000302[0] 1788390875.433567 punch from 0x2604c2 to 0x26da4f (0xd58e bytes) 000305[1] 1788390875.497017 mapread 0x304000 thru 0x311368 (0xd369 bytes) 000307[0] 1788390875.588365 trunc from 0x633079 to 0x0f7704 (0x53b974 bytes) 000308[0] 1788390875.688172 trunc from 0x0f7705 to 0x05d6f6 (0x9a00e bytes) 000311[0] 1788390875.764256 punch from 0x050165 to 0x05324e (0x30ea bytes) 000312[1] 1788390875.777288 write 0x36f000 thru 0x371212 (0x2213 bytes) 000313[0] 1788390875.839007 mapread 0x252000 thru 0x25b3c6 (0x93c7 bytes) 000315[0] 1788390875.965430 read_OD 0x257000 thru 0x265fff (0xf000 bytes) 000316[1] 1788390875.990025 punch from 0x1ff0d7 to 0x207a57 (0x8981 bytes) 000318[1] 1788390876.028037 punch from 0x291c12 to 0x291dd8 (0x1c7 bytes) 000319[0] 1788390876.037924 mapread 0x032000 thru 0x032b50 (0xb51 bytes) truncating to largest ever: 0x9e1bc2 000320[1] 1788390876.171142 trunc from 0x371213 to 0x9e1bc1 (0x6709af bytes) 000321[0] 1788390876.285987 mapread 0x440000 thru 0x44dc2e (0xdc2f bytes) 000324[0] 1788390876.438820 write_OD 0x6a4000 thru 0x6a9fff (0x6000 bytes) 000325[1] 1788390876.521170 trunc from 0x9e1bc2 to 0x8eb8a3 (0xf631e bytes) 000326[1] 1788390876.578661 mapwrite 0x27a000 thru 0x288770 (0xe771 bytes) 000327[1] 1788390876.748978 punch from 0x530a00 to 0x53c36a (0xb96b bytes) 000328[0] 1788390876.754783 write_OD 0x054000 thru 0x05bfff (0x8000 bytes) 000329[0] 1788390876.894975 trunc from 0x8eb8a4 to 0x47d70e (0x46e195 bytes) 000330[1] 1788390876.947027 write 0x9c9000 thru 0x9cc48c (0x348d bytes) 000331[0] 1788390877.004349 trunc from 0x9cc48d to 0x386f64 (0x645528 bytes) 000332[1] 1788390877.060844 read 0x272000 thru 0x27879c (0x679d bytes) 000333[0] 1788390877.119143 mapwrite 0x23b000 thru 0x2452cd (0xa2ce bytes) 000336[0] 1788390877.464536 punch from 0x315fe5 to 0x31cf6f (0x6f8b bytes) 000337[1] 1788390877.483012 read 0x0be000 thru 0x0c1467 (0x3468 bytes) 000339[0] 1788390877.614003 write_OD 0x830000 thru 0x838fff (0x9000 bytes) 000340[1] 1788390877.727101 mapwrite 0x529000 thru 0x536c44 (0xdc45 bytes) 000341[1] 1788390878.004742 trunc from 0x839000 to 0x3cfdee (0x469211 bytes) 000342[1] 1788390878.123901 write 0x46c000 thru 0x46f5f5 (0x35f6 bytes) 000344[0] 1788390878.180101 mapread 0x3ff000 thru 0x401b0c (0x2b0d bytes) 000347[0] 1788390878.234898 mapwrite 0x80b000 thru 0x816dfd (0xbdfe bytes) 000348[0] 1788390878.420797 trunc from 0x816dfe to 0x11d79b (0x6f9662 bytes) 000349[0] 1788390878.445546 read_OD 0x0ea000 thru 0x0eefff (0x5000 bytes) 000352[1] 1788390878.484209 read 0x051000 thru 0x0585b6 (0x75b7 bytes) 000353[0] 1788390878.514020 write_OD 0x584000 thru 0x58dfff (0xa000 bytes) 000356[1] 1788390878.594787 punch from 0x163355 to 0x16bb7d (0x8829 bytes) 000357[0] 1788390878.646748 trunc from 0x58e000 to 0x001416 (0x58cbe9 bytes) 000362[0] 1788390878.773090 mapwrite 0x293000 thru 0x29de3b (0xae3c bytes) 000365[0] 1788390878.967792 trunc from 0x29de3c to 0x8f2731 (0x6548f6 bytes) 000367[0] 1788390879.013587 punch from 0x326e82 to 0x331955 (0xaad4 bytes) 000374[1] 1788390879.067718 write 0x072000 thru 0x07540c (0x340d bytes) 000375[0] 1788390879.207969 mapread 0x1a5000 thru 0x1b1308 (0xc309 bytes) 000376[1] 1788390879.458300 mapwrite 0x470000 thru 0x47fc70 (0xfc71 bytes) 000379[0] 1788390879.806997 mapread 0x1ff000 thru 0x1ffb17 (0xb18 bytes) 000381[1] 1788390879.889720 write 0x491000 thru 0x4a04c7 (0xf4c8 bytes) 000382[0] 1788390879.923510 read_OD 0x2dc000 thru 0x2e0fff (0x5000 bytes) 000384[1] 1788390880.021925 trunc from 0x8f2732 to 0x379fc1 (0x578770 bytes) 000389[0] 1788390880.168481 trunc from 0x379fc2 to 0x543717 (0x1c9756 bytes) 000390[1] 1788390880.261515 trunc from 0x543718 to 0x167ccc (0x3dba4b bytes) 000391[1] 1788390880.330806 mapwrite 0x501000 thru 0x506038 (0x5039 bytes) 000392[0] 1788390880.488090 read_OD 0x2e4000 thru 0x2eefff (0xb000 bytes) 000394[0] 1788390880.528586 punch from 0x454cb0 to 0x45ca83 (0x7dd4 bytes) 000395[0] 1788390880.575422 mapread 0x203000 thru 0x2075a1 (0x45a2 bytes) 000397[1] 1788390880.707923 trunc from 0x506039 to 0x716c40 (0x210c08 bytes) 000400[1] 1788390880.878016 mapwrite 0x1a9000 thru 0x1af7df (0x67e0 bytes) 000403[0] 1788390881.147908 read_OD 0x18f000 thru 0x19bfff (0xd000 bytes) 000405[0] 1788390881.195436 mapwrite 0x9a9000 thru 0x9a9bc6 (0xbc7 bytes) 000406[0] 1788390881.359742 mapwrite 0x65d000 thru 0x66a016 (0xd017 bytes) 000408[1] 1788390881.504547 trunc from 0x9a9bc7 to 0x519f12 (0x48fcb4 bytes) 000410[1] 1788390881.571856 punch from 0x2e27ae to 0x2e8211 (0x5a64 bytes) 000411[1] 1788390881.589178 punch from 0x14454f to 0x15115f (0xcc11 bytes) 000412[0] 1788390881.605828 mapread 0x3f1000 thru 0x3f8692 (0x7693 bytes) 000413[1] 1788390881.652723 mapwrite 0x617000 thru 0x61d612 (0x6613 bytes) 000414[0] 1788390881.828336 mapread 0x31a000 thru 0x31ff33 (0x5f34 bytes) 000415[0] 1788390881.892430 read_OD 0x5b0000 thru 0x5b1fff (0x2000 bytes) 000419[0] 1788390881.904962 punch from 0x22ecb4 to 0x23ead9 (0xfe26 bytes) 000420[0] 1788390881.934891 mapread 0x0a1000 thru 0x0a5166 (0x4167 bytes) 000421[1] 1788390881.980843 punch from 0x3b9f9d to 0x3c42e0 (0xa344 bytes) 000422[0] 1788390882.007248 trunc from 0x61d613 to 0x78feda (0x1728c8 bytes) 000425[1] 1788390882.065106 mapwrite 0x45d000 thru 0x464b48 (0x7b49 bytes) 000426[0] 1788390882.206157 punch from 0x77ea70 to 0x7850b2 (0x6643 bytes) 000428[1] 1788390882.220436 trunc from 0x78fedb to 0x3029dd (0x48d4fd bytes) 000429[1] 1788390882.265002 mapwrite 0x13e000 thru 0x14a7e7 (0xc7e8 bytes) 000431[0] 1788390882.451252 write_OD 0x33c000 thru 0x343fff (0x8000 bytes) 000432[1] 1788390882.543605 mapwrite 0x7a7000 thru 0x7b42bc (0xd2bd bytes) 000433[1] 1788390882.792680 trunc from 0x7b42bd to 0x033bf6 (0x7806c6 bytes) 000434[1] 1788390882.811963 mapread 0x01b000 thru 0x02724a (0xc24b bytes) 000435[0] 1788390882.896851 punch from 0x003002 to 0x00b4bf (0x84be bytes) 000436[0] 1788390882.909659 write_OD 0x1ff000 thru 0x204fff (0x6000 bytes) 000438[0] 1788390882.982691 mapwrite 0x3c5000 thru 0x3c5fc0 (0xfc1 bytes) 000439[0] 1788390883.064991 trunc from 0x3c5fc1 to 0x177411 (0x24ebaf bytes) 000440[0] 1788390883.083298 write_OD 0x1d5000 thru 0x1d9fff (0x5000 bytes) 000441[0] 1788390883.133067 punch from 0x0e2f71 to 0x0f2828 (0xf8b8 bytes) 000442[0] 1788390883.183715 trunc from 0x1da000 to 0x897b73 (0x6bdb74 bytes) 000443[1] 1788390883.230235 read 0x1e0000 thru 0x1e224e (0x224f bytes) 000445[1] 1788390883.376496 trunc from 0x897b74 to 0x179bdd (0x71df96 bytes) 000448[1] 1788390883.464376 write 0x468000 thru 0x4712ac (0x92ad bytes) 000450[1] 1788390883.470287 mapwrite 0x694000 thru 0x698046 (0x4047 bytes) 000452[0] 1788390883.629859 write_OD 0x033000 thru 0x039fff (0x7000 bytes) 000455[0] 1788390883.696943 mapwrite 0x796000 thru 0x7a5984 (0xf985 bytes) 000456[0] 1788390883.856679 punch from 0x29b2ca to 0x29e9f0 (0x3727 bytes) 000457[0] 1788390883.868540 punch from 0x446b54 to 0x4522c3 (0xb770 bytes) 000458[1] 1788390883.893001 mapread 0x020000 thru 0x025578 (0x5579 bytes) 000459[1] 1788390883.987087 mapread 0x2b5000 thru 0x2c2250 (0xd251 bytes) 000460[1] 1788390884.129918 write 0x40d000 thru 0x40d13c (0x13d bytes) 000461[1] 1788390884.186773 trunc from 0x7a5985 to 0x658d01 (0x14cc83 bytes) 000462[1] 1788390884.219854 punch from 0x344c23 to 0x348605 (0x39e3 bytes) 000463[0] 1788390884.233632 punch from 0x00b3e4 to 0x013bab (0x87c8 bytes) 000464[1] 1788390884.277294 trunc from 0x658d02 to 0x2379a4 (0x42135d bytes) truncating to largest ever: 0x9f8e86 000465[0] 1788390884.342873 trunc from 0x2379a5 to 0x9f8e85 (0x7c14e1 bytes) 000466[1] 1788390884.413743 read 0x57f000 thru 0x58dce1 (0xece2 bytes) 000467[0] 1788390884.443238 read_OD 0x234000 thru 0x23cfff (0x9000 bytes) 000469[1] 1788390884.461972 mapread 0x370000 thru 0x375821 (0x5822 bytes) 000470[0] 1788390884.499957 punch from 0x7099bc to 0x70dbe2 (0x4227 bytes) 000472[0] 1788390884.573062 write_OD 0x776000 thru 0x77ffff (0xa000 bytes) 000473[1] 1788390884.634991 trunc from 0x9f8e86 to 0x597caf (0x4611d6 bytes) 000474[1] 1788390884.692846 mapwrite 0x5c0000 thru 0x5c4b6b (0x4b6c bytes) 000475[0] 1788390884.813286 trunc from 0x5c4b6c to 0x56cb20 (0x5804b bytes) 000479[0] 1788390884.903967 mapwrite 0x692000 thru 0x6962af (0x42b0 bytes) 000484[1] 1788390885.142350 read 0x57b000 thru 0x57ca9c (0x1a9d bytes) 000485[0] 1788390885.166498 mapread 0x363000 thru 0x36fb29 (0xcb2a bytes) 000487[0] 1788390885.192231 mapread 0x39e000 thru 0x3a29cb (0x49cc bytes) 000488[0] 1788390885.200407 mapwrite 0x2cd000 thru 0x2d5599 (0x859a bytes) 000491[1] 1788390885.351959 mapread 0x64a000 thru 0x6526b6 (0x86b7 bytes) 000495[0] 1788390885.384508 write_OD 0x6ef000 thru 0x6f0fff (0x2000 bytes) 000498[1] 1788390885.489127 trunc from 0x6f1000 to 0x2f1995 (0x3ff66a bytes) All operations completed A-OK! PASS 16a (50s) == sanityn test 16b: 500 iterations of dual-mount fsx at small size ========================================================== 19:14:50 (1788390890) keep default fallocate mode: 2 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.348802 s, 24.0 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.316052 s, 26.5 MB/s Chance of close/open is 1 in 50 Seed set to 893 fd 0: /mnt/lustre/f16b.sanityn fd 1: /mnt/lustre2/f16b.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported truncating to largest ever: 0x815e78 000002[1] 1788390894.588154 trunc from 00000000 to 0x815e77 (0x815e78 bytes) 000003[1] 1788390894.652100 read 0x701c36 thru 0x702cbc (0x1087 bytes) 000005[1] 1788390894.702886 mapread 0x12ae39 thru 0x12c5bf (0x1787 bytes) 000008[1] 1788390894.810729 punch from 0x66f0da to 0x66fe0a (0xd31 bytes) 000011[1] 1788390894.858909 trunc from 0x815e78 to 0x5284ee (0x2ed989 bytes) 000012[1] 1788390894.901052 read 0x0bbfd6 thru 0x0bcbec (0xc17 bytes) 000013[0] 1788390894.911230 mapread 0x39f15b thru 0x39f808 (0x6ae bytes) 000014[1] 1788390895.009289 trunc from 0x5284ef to 0x7b02ed (0x287dff bytes) 000015[1] 1788390895.054020 trunc from 0x7b02ee to 0x72112d (0x8f1c0 bytes) 000016[0] 1788390895.111545 trunc from 0x72112e to 0x24fe76 (0x4d12b7 bytes) 000018[0] 1788390895.160296 read 0x0cced6 thru 0x0cda2a (0xb55 bytes) 000019[1] 1788390895.191375 write 0x5ac3c6 thru 0x5ac7c2 (0x3fd bytes) 000024[1] 1788390895.271985 mapread 0x31f8f8 thru 0x31ff19 (0x622 bytes) 000025[0] 1788390895.308175 write 0x3a929e thru 0x3aa592 (0x12f5 bytes) 000027[1] 1788390895.345313 write 0x762de5 thru 0x76385d (0xa79 bytes) 000028[0] 1788390895.358935 read 0x0ae629 thru 0x0afcfd (0x16d5 bytes) 000029[1] 1788390895.375566 punch from 0x3682c1 to 0x3683de (0x11e bytes) 000030[0] 1788390895.439653 punch from 0x26d64f to 0x26e2f0 (0xca2 bytes) 000032[1] 1788390895.478581 trunc from 0x76385e to 0x60a5ed (0x159270 bytes) 000033[0] 1788390895.530757 trunc from 0x60a5ee to 0x17e35c (0x48c291 bytes) 000034[1] 1788390895.618019 mapread 0x0f27bd thru 0x0f38e0 (0x1124 bytes) 000037[0] 1788390895.673549 trunc from 0x17e35d to 0x21c8a8 (0x9e54c bytes) 000038[1] 1788390895.731877 mapwrite 0x518e40 thru 0x519bef (0xdb0 bytes) 000039[1] 1788390895.885706 mapwrite 0x04aecd thru 0x04b899 (0x9cd bytes) 000040[1] 1788390895.983751 mapwrite 0x6af694 thru 0x6afdee (0x75b bytes) 000041[0] 1788390896.098813 read 0x5bae65 thru 0x5bbf5c (0x10f8 bytes) 000042[1] 1788390896.143752 trunc from 0x6afdef to 0x01992d (0x6964c1 bytes) 000043[1] 1788390896.217867 punch from 0x0017e7 to 0x002455 (0xc6f bytes) 000044[1] 1788390896.230808 mapread 0x002df8 thru 0x004831 (0x1a3a bytes) 000046[0] 1788390896.359281 mapread 0x00b78b thru 0x00c1f1 (0xa67 bytes) 000050[1] 1788390896.505741 trunc from 0x01992e to 0x1738fc (0x159fcf bytes) 000056[1] 1788390896.625846 trunc from 0x1738fd to 0x139b51 (0x39dab bytes) 000058[0] 1788390896.658130 punch from 0x07d2fc to 0x07ef41 (0x1c46 bytes) 000059[1] 1788390896.696338 punch from 0x09cb1f to 0x09e511 (0x19f3 bytes) 000063[1] 1788390896.710152 write 0x23ba1b thru 0x23ca2b (0x1011 bytes) 000064[0] 1788390896.712133 mapwrite 0x32b634 thru 0x32d291 (0x1c5e bytes) 000070[1] 1788390896.928273 mapwrite 0x0f11af thru 0x0f139b (0x1ed bytes) 000071[0] 1788390897.017791 trunc from 0x32d292 to 0x7e5ff0 (0x4b8d5f bytes) 000073[1] 1788390897.078992 trunc from 0x7e5ff1 to 0x665031 (0x180fbf bytes) 000076[0] 1788390897.152092 trunc from 0x665032 to 0x4b929b (0x1abd96 bytes) 000077[0] 1788390897.211006 write 0x0a9844 thru 0x0ab5e5 (0x1da2 bytes) 000078[1] 1788390897.222507 punch from 0x3d1586 to 0x3d3504 (0x1f7f bytes) 000080[1] 1788390897.263438 punch from 0x471c16 to 0x473a04 (0x1def bytes) 000082[0] 1788390897.274284 read 0x16d6d1 thru 0x16f107 (0x1a37 bytes) 000085[1] 1788390897.324894 mapread 0x0b224d thru 0x0b25b2 (0x366 bytes) 000088[0] 1788390897.344580 mapwrite 0x16311e thru 0x16425e (0x1141 bytes) truncating to largest ever: 0x941edd 000091[1] 1788390897.477133 trunc from 0x4b929c to 0x941edc (0x488c41 bytes) 000092[0] 1788390897.559747 trunc from 0x941edd to 0x3808a1 (0x5c163b bytes) 000093[1] 1788390897.621589 write 0x678280 thru 0x678fe6 (0xd67 bytes) 000094[1] 1788390897.655156 write 0x0ae552 thru 0x0af7fc (0x12ab bytes) 000095[0] 1788390897.675682 mapread 0x57c867 thru 0x57ceb5 (0x64f bytes) 000096[1] 1788390897.709421 punch from 0x0ecbae to 0x0edcbc (0x110f bytes) 000099[0] 1788390897.749830 write 0x240ae1 thru 0x2413ff (0x91f bytes) 000100[1] 1788390897.773307 write 0x29c6f2 thru 0x29d226 (0xb35 bytes) 000106[0] 1788390897.824348 read 0x195fc2 thru 0x1975e7 (0x1626 bytes) 000107[1] 1788390897.842003 write 0x562bfe thru 0x564bac (0x1faf bytes) 000108[0] 1788390897.875873 punch from 0x5bf2ae to 0x5bf3c1 (0x114 bytes) 000109[0] 1788390897.906774 write 0x283bbe thru 0x284a95 (0xed8 bytes) 000110[0] 1788390897.926513 read 0x1892d3 thru 0x18a936 (0x1664 bytes) 000111[1] 1788390897.930332 punch from 0x27833b to 0x279599 (0x125f bytes) 000114[0] 1788390897.999337 trunc from 0x678fe7 to 0x8af813 (0x23682d bytes) 000115[1] 1788390898.034606 trunc from 0x8af814 to 0x6d2618 (0x1dd1fb bytes) 000117[1] 1788390898.083994 mapwrite 0x6f92d1 thru 0x6fa7b9 (0x14e9 bytes) 000118[1] 1788390898.171113 mapread 0x08aea0 thru 0x08c933 (0x1a94 bytes) 000120[0] 1788390898.215243 mapread 0x53d672 thru 0x53d7dd (0x16c bytes) 000121[1] 1788390898.253328 mapread 0x010ccd thru 0x012689 (0x19bd bytes) 000122[0] 1788390898.277892 mapwrite 0x3858c2 thru 0x3867a9 (0xee8 bytes) 000123[1] 1788390898.349640 mapread 0x6727f6 thru 0x6737d7 (0xfe2 bytes) 000127[1] 1788390898.400141 mapwrite 0x223309 thru 0x2244d2 (0x11ca bytes) 000128[1] 1788390898.458261 read 0x6b6b5a thru 0x6b7309 (0x7b0 bytes) 000129[0] 1788390898.464817 write 0x605ce0 thru 0x6074f7 (0x1818 bytes) 000130[0] 1788390898.517140 read 0x157b7b thru 0x157f88 (0x40e bytes) 000131[0] 1788390898.536434 mapwrite 0x8972a2 thru 0x899162 (0x1ec1 bytes) 000132[1] 1788390898.622547 mapwrite 0x1372b0 thru 0x138c64 (0x19b5 bytes) 000133[1] 1788390898.706085 punch from 0x37f0ed to 0x37fd49 (0xc5d bytes) 000135[0] 1788390898.740109 punch from 0x33543d to 0x3373bb (0x1f7f bytes) 000136[1] 1788390898.753873 trunc from 0x899163 to 0x462d75 (0x4363ed bytes) 000137[1] 1788390898.807913 trunc from 0x462d76 to 0x9325f9 (0x4cf884 bytes) 000139[0] 1788390898.839330 punch from 0x6998ff to 0x699a23 (0x125 bytes) 000140[1] 1788390898.862523 trunc from 0x9325fa to 0x0f3f44 (0x83e6b5 bytes) 000141[1] 1788390898.887904 mapread 0x08c245 thru 0x08d653 (0x140f bytes) 000142[1] 1788390898.899291 write 0x5452a6 thru 0x54654d (0x12a8 bytes) 000143[1] 1788390898.927558 punch from 0x4c3fc0 to 0x4c43e5 (0x426 bytes) 000144[1] 1788390898.949961 mapread 0x04b72c thru 0x04cfd9 (0x18ae bytes) 000145[1] 1788390898.957280 read 0x4bd58e thru 0x4be40d (0xe80 bytes) 000147[1] 1788390898.985064 mapread 0x24ef05 thru 0x24f725 (0x821 bytes) 000154[1] 1788390899.048806 mapread 0x347e41 thru 0x348673 (0x833 bytes) 000155[1] 1788390899.073921 punch from 0x3ed4a5 to 0x3ef0ee (0x1c4a bytes) 000156[0] 1788390899.100028 mapwrite 0x034311 thru 0x03469f (0x38f bytes) 000157[1] 1788390899.179915 mapread 0x22af06 thru 0x22c95d (0x1a58 bytes) 000159[0] 1788390899.214403 mapread 0x091de9 thru 0x091f0e (0x126 bytes) truncating to largest ever: 0x9f1ff8 000160[0] 1788390899.235073 trunc from 0x54654e to 0x9f1ff7 (0x4abaaa bytes) 000163[1] 1788390899.370327 mapread 0x99b26a thru 0x99ce8c (0x1c23 bytes) 000166[0] 1788390899.450428 punch from 0x878b9d to 0x8796f8 (0xb5c bytes) 000167[1] 1788390899.517173 trunc from 0x9f1ff8 to 0x771490 (0x280b67 bytes) 000168[0] 1788390899.574982 mapread 0x72b117 thru 0x72c0ca (0xfb4 bytes) 000170[0] 1788390899.621319 punch from 0x48d6c2 to 0x48e3ac (0xceb bytes) 000172[0] 1788390899.671172 mapwrite 0x22e325 thru 0x22e89f (0x57b bytes) 000173[1] 1788390899.726208 mapread 0x4a1504 thru 0x4a2fc0 (0x1abd bytes) 000179[1] 1788390899.784627 mapread 0x3f4603 thru 0x3f49a3 (0x3a1 bytes) 000180[0] 1788390899.809722 write 0x4e2c98 thru 0x4e3bd7 (0xf40 bytes) 000183[0] 1788390899.835155 punch from 0x3da82d to 0x3dc694 (0x1e68 bytes) 000186[1] 1788390899.883126 mapwrite 0x9d12ff thru 0x9d1626 (0x328 bytes) 000190[0] 1788390899.997324 read 0x92d049 thru 0x92e2a1 (0x1259 bytes) 000191[0] 1788390900.015132 write 0x214e1c thru 0x21626c (0x1451 bytes) 000193[1] 1788390900.078296 read 0x13d1e4 thru 0x13dd36 (0xb53 bytes) 000196[0] 1788390900.139567 write 0x837a8b thru 0x83917b (0x16f1 bytes) 000197[1] 1788390900.173071 mapread 0x842c2b thru 0x843ded (0x11c3 bytes) 000198[1] 1788390900.201677 trunc from 0x9d1627 to 0x67599c (0x35bc8a bytes) 000200[0] 1788390900.264311 write 0x7ee032 thru 0x7eea91 (0xa60 bytes) 000202[1] 1788390900.316372 write 0x12361e thru 0x1245c3 (0xfa6 bytes) 000204[0] 1788390900.357032 read 0x4f4e4a thru 0x4f69af (0x1b66 bytes) 000205[1] 1788390900.380911 write 0x09da96 thru 0x09e8ae (0xe19 bytes) 000206[1] 1788390900.401315 trunc from 0x7eea92 to 0x02545e (0x7c9633 bytes) 000208[1] 1788390900.481296 read 0x007c4f thru 0x009a67 (0x1e19 bytes) 000210[1] 1788390900.516835 punch from 0x014779 to 0x0151bb (0xa43 bytes) 000212[1] 1788390900.542696 mapread 0x0154ff thru 0x016438 (0xf3a bytes) 000213[0] 1788390900.547343 trunc from 0x02545f to 0x90bcbf (0x8e6861 bytes) 000214[1] 1788390900.592696 punch from 0x30715e to 0x308413 (0x12b6 bytes) 000215[0] 1788390900.622022 read 0x102b96 thru 0x104426 (0x1891 bytes) 000216[0] 1788390900.634616 trunc from 0x90bcc0 to 0x40a943 (0x50137c bytes) 000220[0] 1788390900.698102 mapread 0x31a1b5 thru 0x31b7b0 (0x15fc bytes) 000221[1] 1788390900.707843 punch from 0x02f01d to 0x02f2c9 (0x2ad bytes) 000222[0] 1788390900.732983 trunc from 0x40a944 to 0x927810 (0x51cecd bytes) 000225[1] 1788390900.796051 trunc from 0x927811 to 0x6f5ad7 (0x231d39 bytes) 000226[0] 1788390900.862837 write 0x4f1f85 thru 0x4f31e1 (0x125d bytes) 000228[0] 1788390900.938330 mapread 0x56213a thru 0x5624f0 (0x3b7 bytes) 000232[1] 1788390901.028603 mapwrite 0x5e187b thru 0x5e245c (0xbe2 bytes) 000233[0] 1788390901.345080 read 0x219207 thru 0x219c9a (0xa94 bytes) 000234[0] 1788390901.442077 punch from 0x064e97 to 0x065267 (0x3d1 bytes) 000235[1] 1788390901.480120 mapread 0x1b198d thru 0x1b2507 (0xb7b bytes) 000237[0] 1788390901.511606 read 0x26dd1b thru 0x26efec (0x12d2 bytes) 000238[0] 1788390901.523746 mapwrite 0x00c74f thru 0x00def8 (0x17aa bytes) 000239[0] 1788390901.659290 mapwrite 0x2af647 thru 0x2b1378 (0x1d32 bytes) 000241[1] 1788390901.739419 punch from 0x05a91e to 0x05bd9b (0x147e bytes) 000243[0] 1788390901.796157 write 0x954b04 thru 0x9554cb (0x9c8 bytes) 000244[1] 1788390901.816098 punch from 0x48cf55 to 0x48e875 (0x1921 bytes) 000245[1] 1788390901.834599 punch from 0x268490 to 0x268943 (0x4b4 bytes) 000246[0] 1788390901.874279 write 0x6dd594 thru 0x6dde5b (0x8c8 bytes) 000247[1] 1788390901.916591 read 0x81be6b thru 0x81c99b (0xb31 bytes) 000249[0] 1788390901.946276 write 0x52ffab thru 0x531885 (0x18db bytes) 000250[1] 1788390901.982410 mapread 0x55e333 thru 0x55edfe (0xacc bytes) 000251[0] 1788390902.028135 mapread 0x20a596 thru 0x20a93a (0x3a5 bytes) 000252[0] 1788390902.098431 trunc from 0x9554cc to 0x691cda (0x2c37f1 bytes) 000253[1] 1788390902.173934 read 0x655dd5 thru 0x656db6 (0xfe2 bytes) 000255[0] 1788390902.237363 mapread 0x5c3018 thru 0x5c35b3 (0x59c bytes) 000257[0] 1788390902.261014 trunc from 0x691cdb to 0x23f83e (0x45249c bytes) 000258[0] 1788390902.293338 punch from 0x1d4094 to 0x1d48f8 (0x865 bytes) 000260[1] 1788390902.324221 mapread 0x1a0716 thru 0x1a15fb (0xee6 bytes) 000262[0] 1788390902.364954 mapread 0x05e446 thru 0x0600d4 (0x1c8f bytes) 000263[0] 1788390902.417106 mapread 0x021851 thru 0x022cab (0x145b bytes) 000264[0] 1788390902.445113 mapwrite 0x8d05e7 thru 0x8d080a (0x224 bytes) 000265[0] 1788390902.674136 read 0x27d1b3 thru 0x27d78f (0x5dd bytes) 000266[0] 1788390902.684417 mapread 0x76bcc1 thru 0x76c6eb (0xa2b bytes) 000268[0] 1788390902.721221 mapwrite 0x085728 thru 0x086b7b (0x1454 bytes) 000269[0] 1788390902.830537 mapwrite 0x85cf6f thru 0x85d47a (0x50c bytes) 000270[0] 1788390902.917681 mapwrite 0x64101c thru 0x642b45 (0x1b2a bytes) 000274[1] 1788390903.169398 read 0x4f0f4f thru 0x4f1563 (0x615 bytes) 000277[1] 1788390903.307583 write 0x8a0036 thru 0x8a1558 (0x1523 bytes) 000279[0] 1788390903.389129 write 0x3a0f95 thru 0x3a24e2 (0x154e bytes) 000280[1] 1788390903.500425 read 0x72e3ca thru 0x73015f (0x1d96 bytes) 000283[0] 1788390903.508535 trunc from 0x8d080b to 0x5b0696 (0x320174 bytes) 000284[1] 1788390903.552324 mapwrite 0x749258 thru 0x74a737 (0x14e0 bytes) 000286[1] 1788390903.711954 read 0x0e1fc1 thru 0x0e3586 (0x15c6 bytes) 000288[0] 1788390903.735291 write 0x4e1593 thru 0x4e2b75 (0x15e3 bytes) 000291[0] 1788390903.809203 trunc from 0x74a738 to 0x05eb58 (0x6ebbdf bytes) 000292[0] 1788390903.874210 read 0x032a8d thru 0x0348e0 (0x1e54 bytes) 000293[0] 1788390903.884316 trunc from 0x05eb59 to 0x297de8 (0x239290 bytes) 000295[1] 1788390903.935885 mapwrite 0x04c83a thru 0x04d1b9 (0x980 bytes) 000296[1] 1788390904.047282 punch from 0x1cb92b to 0x1cbd7e (0x454 bytes) 000299[0] 1788390904.117105 mapwrite 0x8cf950 thru 0x8d175d (0x1e0e bytes) 000302[1] 1788390904.415925 write 0x55e671 thru 0x55f8fe (0x128e bytes) 000304[0] 1788390904.500844 punch from 0x269918 to 0x26a4ad (0xb96 bytes) 000305[1] 1788390904.519301 mapread 0x1171e9 thru 0x11899b (0x17b3 bytes) 000306[1] 1788390904.564526 read 0x892052 thru 0x8939c4 (0x1973 bytes) 000307[1] 1788390904.587760 trunc from 0x8d175e to 0x72e0ff (0x1a365e bytes) 000308[0] 1788390904.633832 punch from 0x6cfde5 to 0x6d1589 (0x17a5 bytes) 000309[1] 1788390904.704682 trunc from 0x72e100 to 0x2a7fca (0x486135 bytes) 000311[1] 1788390904.808966 mapread 0x112d26 thru 0x1134b6 (0x791 bytes) 000312[1] 1788390904.836239 mapwrite 0x85c5d0 thru 0x85ca7c (0x4ad bytes) 000313[1] 1788390904.913947 punch from 0x61d38e to 0x61e265 (0xed8 bytes) 000315[0] 1788390904.946236 trunc from 0x85ca7d to 0x35158f (0x50b4ed bytes) 000317[0] 1788390905.005843 read 0x2d6140 thru 0x2d6da7 (0xc68 bytes) 000318[1] 1788390905.093875 mapwrite 0x642b84 thru 0x6433ae (0x82b bytes) 000319[0] 1788390905.184316 punch from 0x051bfd to 0x052708 (0xb0c bytes) 000320[1] 1788390905.214633 write 0x895f37 thru 0x896140 (0x20a bytes) 000322[1] 1788390905.259173 trunc from 0x896141 to 0x229c8b (0x66c4b5 bytes) 000323[1] 1788390905.277765 mapread 0x11edb7 thru 0x11f447 (0x691 bytes) 000324[1] 1788390905.287327 write 0x4dddf4 thru 0x4df4f1 (0x16fe bytes) 000327[1] 1788390905.313226 read 0x2d9a39 thru 0x2da2ad (0x875 bytes) 000329[1] 1788390905.336940 mapwrite 0x6bc2f2 thru 0x6bd290 (0xf9f bytes) 000331[0] 1788390905.449088 trunc from 0x6bd291 to 0x059730 (0x663b60 bytes) 000332[1] 1788390905.518264 mapread 0x04a777 thru 0x04b467 (0xcf1 bytes) 000333[1] 1788390905.554821 trunc from 0x059731 to 0x7de209 (0x784ad9 bytes) 000334[0] 1788390905.597984 write 0x2dbe05 thru 0x2dc1b1 (0x3ad bytes) 000335[1] 1788390905.649627 read 0x0b345b thru 0x0b4dbf (0x1965 bytes) 000337[0] 1788390905.696866 read 0x24b962 thru 0x24c357 (0x9f6 bytes) 000338[0] 1788390905.702528 read 0x0da829 thru 0x0db9d0 (0x11a8 bytes) 000344[1] 1788390905.826149 write 0x76aac9 thru 0x76bd0d (0x1245 bytes) 000345[1] 1788390905.845618 read 0x066c98 thru 0x068bd0 (0x1f39 bytes) 000347[1] 1788390905.876430 mapwrite 0x614978 thru 0x615c20 (0x12a9 bytes) 000348[1] 1788390905.934947 mapwrite 0x5f576a thru 0x5f5783 (0x1a bytes) 000349[1] 1788390905.995830 mapread 0x0e95ff thru 0x0e98bf (0x2c1 bytes) 000351[0] 1788390906.017880 write 0x6b97c7 thru 0x6b9e15 (0x64f bytes) 000352[1] 1788390906.117677 read 0x1f8b39 thru 0x1f99db (0xea3 bytes) 000353[1] 1788390906.134304 trunc from 0x7de20a to 0x5f70cf (0x1e713a bytes) 000354[1] 1788390906.213723 write 0x2a5e46 thru 0x2a5f28 (0xe3 bytes) 000356[0] 1788390906.244231 mapread 0x2681c9 thru 0x2683f4 (0x22c bytes) 000358[1] 1788390906.300808 punch from 0x16ebe2 to 0x17072e (0x1b4d bytes) 000360[1] 1788390906.339222 mapread 0x4df808 thru 0x4dfd98 (0x591 bytes) 000362[1] 1788390906.374952 read 0x1785b2 thru 0x17947e (0xecd bytes) 000364[1] 1788390906.410858 mapwrite 0x83961e thru 0x83b136 (0x1b19 bytes) 000365[0] 1788390906.492262 write 0x1d6596 thru 0x1d828d (0x1cf8 bytes) 000366[1] 1788390906.530512 read 0x08143b thru 0x0825a2 (0x1168 bytes) 000367[0] 1788390906.569584 punch from 0x7a21d4 to 0x7a408d (0x1eba bytes) 000369[1] 1788390906.586578 mapread 0x11c991 thru 0x11cc9b (0x30b bytes) 000373[0] 1788390906.620185 read 0x777223 thru 0x7791aa (0x1f88 bytes) 000375[0] 1788390906.655811 write 0x6b3223 thru 0x6b43fc (0x11da bytes) 000377[0] 1788390906.684381 punch from 0x6f2b3a to 0x6f4779 (0x1c40 bytes) 000379[0] 1788390906.712438 mapwrite 0x52ab7e thru 0x52af5a (0x3dd bytes) 000380[0] 1788390906.788982 punch from 0x52605b to 0x5264bd (0x463 bytes) 000383[0] 1788390906.813311 write 0x3c7c59 thru 0x3c87a9 (0xb51 bytes) 000384[0] 1788390906.872708 read 0x22d35b thru 0x22d741 (0x3e7 bytes) 000385[1] 1788390906.904805 mapread 0x3672a9 thru 0x368b1b (0x1873 bytes) 000386[1] 1788390906.948215 punch from 0x5dc584 to 0x5de442 (0x1ebf bytes) 000388[1] 1788390907.009139 punch from 0x700095 to 0x7004af (0x41b bytes) 000390[0] 1788390907.127526 read 0x117dbb thru 0x1184ca (0x710 bytes) 000392[1] 1788390907.196991 write 0x3f6d11 thru 0x3f6d8f (0x7f bytes) 000394[1] 1788390907.236739 write 0x1d5e90 thru 0x1d62af (0x420 bytes) 000395[0] 1788390907.257968 punch from 0x316899 to 0x31724b (0x9b3 bytes) 000397[1] 1788390907.384898 mapwrite 0x290a19 thru 0x29255a (0x1b42 bytes) 000398[1] 1788390907.570945 write 0x88ddfa thru 0x88ed1b (0xf22 bytes) 000400[1] 1788390907.632424 write 0x97de9a thru 0x97f062 (0x11c9 bytes) 000402[0] 1788390907.634293 read 0x121f9e thru 0x123693 (0x16f6 bytes) 000403[0] 1788390907.670587 punch from 0x87cbbb to 0x87e686 (0x1acc bytes) 000404[1] 1788390907.711880 mapwrite 0x7551cc thru 0x755d62 (0xb97 bytes) 000405[0] 1788390907.809051 mapread 0x73e9ca thru 0x73f471 (0xaa8 bytes) 000406[0] 1788390907.864406 trunc from 0x97f063 to 0x790be2 (0x1ee480 bytes) 000409[1] 1788390907.977302 punch from 0x525ef9 to 0x526a12 (0xb1a bytes) 000410[0] 1788390908.012740 mapread 0x035a90 thru 0x035b9e (0x10f bytes) 000411[0] 1788390908.046197 mapwrite 0x8bc743 thru 0x8bd7ec (0x10aa bytes) 000415[1] 1788390908.232999 read 0x77f8f7 thru 0x780165 (0x86f bytes) 000416[0] 1788390908.262143 punch from 0x24e92a to 0x24ed81 (0x458 bytes) 000417[0] 1788390908.281716 mapwrite 0x9ad0b7 thru 0x9ad10d (0x57 bytes) 000419[0] 1788390908.455440 punch from 0x8a47d8 to 0x8a4bd7 (0x400 bytes) 000420[0] 1788390908.487967 read 0x95c8cd thru 0x95e1a5 (0x18d9 bytes) 000421[1] 1788390908.520895 trunc from 0x9ad10e to 0x90f1bb (0x9df52 bytes) 000423[1] 1788390908.607307 trunc from 0x90f1bc to 0x7c28ed (0x14c8ce bytes) 000424[0] 1788390908.636167 punch from 0x4d7106 to 0x4d84d8 (0x13d3 bytes) 000426[1] 1788390908.697829 punch from 0x1d49b2 to 0x1d4ae7 (0x136 bytes) 000428[1] 1788390908.730936 mapwrite 0x209af5 thru 0x20af50 (0x145c bytes) 000429[0] 1788390908.819316 write 0x053199 thru 0x053944 (0x7ac bytes) 000431[0] 1788390908.935669 trunc from 0x7c28ee to 0x2be3ac (0x504541 bytes) 000432[1] 1788390908.972475 mapread 0x19272c thru 0x194173 (0x1a48 bytes) 000433[0] 1788390909.037246 write 0x04b60f thru 0x04c2c0 (0xcb2 bytes) 000434[1] 1788390909.066484 write 0x9412f8 thru 0x941aa2 (0x7ab bytes) 000435[0] 1788390909.106304 write 0x4c312e thru 0x4c36ba (0x58d bytes) 000437[1] 1788390909.136861 mapread 0x33aa59 thru 0x33bfbd (0x1565 bytes) 000439[0] 1788390909.206195 mapwrite 0x73aa98 thru 0x73c0fb (0x1664 bytes) 000440[1] 1788390909.366792 read 0x8ce460 thru 0x8d0272 (0x1e13 bytes) 000441[0] 1788390909.393318 mapread 0x132944 thru 0x132ca9 (0x366 bytes) 000442[0] 1788390909.455607 read 0x0c9221 thru 0x0ca6e6 (0x14c6 bytes) 000443[0] 1788390909.462390 write 0x8d51d5 thru 0x8d6f46 (0x1d72 bytes) 000444[1] 1788390909.491648 mapwrite 0x999548 thru 0x999f26 (0x9df bytes) 000447[1] 1788390909.769804 read 0x01a111 thru 0x01b9b2 (0x18a2 bytes) 000449[1] 1788390909.856258 trunc from 0x999f27 to 0x3f886f (0x5a16b7 bytes) 000450[1] 1788390909.928667 trunc from 0x3f8870 to 0x5d8874 (0x1e0005 bytes) 000451[0] 1788390910.033709 read 0x47b811 thru 0x47c1a3 (0x993 bytes) 000452[0] 1788390910.093972 write 0x10f95a thru 0x1115b5 (0x1c5c bytes) 000453[1] 1788390910.133326 punch from 0x541647 to 0x54268b (0x1045 bytes) 000454[1] 1788390910.206371 write 0x91253c thru 0x912e97 (0x95c bytes) 000458[1] 1788390910.299649 mapwrite 0x85aa44 thru 0x85ac40 (0x1fd bytes) 000459[1] 1788390910.354412 write 0x321e30 thru 0x3224bd (0x68e bytes) 000460[0] 1788390910.366511 write 0x169ac3 thru 0x16a774 (0xcb2 bytes) 000461[1] 1788390910.415881 mapwrite 0x52f91c thru 0x52feb8 (0x59d bytes) 000466[1] 1788390910.596363 mapread 0x716f6e thru 0x718e88 (0x1f1b bytes) 000468[0] 1788390910.703139 mapwrite 0x454db2 thru 0x454ef1 (0x140 bytes) 000469[1] 1788390910.837836 punch from 0x82e73e to 0x82e970 (0x233 bytes) 000471[1] 1788390910.899664 punch from 0x49ba9e to 0x49d7a1 (0x1d04 bytes) 000473[1] 1788390910.944872 read 0x0f1db8 thru 0x0f33ba (0x1603 bytes) 000474[1] 1788390910.980206 write 0x0d0f0e thru 0x0d1552 (0x645 bytes) 000475[0] 1788390910.991858 mapread 0x2c8f37 thru 0x2ca1a3 (0x126d bytes) 000476[0] 1788390911.043772 mapwrite 0x75a935 thru 0x75b730 (0xdfc bytes) 000478[1] 1788390911.137525 read 0x78e585 thru 0x78f671 (0x10ed bytes) 000480[0] 1788390911.167960 punch from 0x5415bf to 0x542d98 (0x17da bytes) 000481[0] 1788390911.216420 write 0x592a67 thru 0x59424d (0x17e7 bytes) 000482[1] 1788390911.233106 mapread 0x64d237 thru 0x64da16 (0x7e0 bytes) 000483[1] 1788390911.284330 punch from 0x4ca652 to 0x4ca73f (0xee bytes) 000484[1] 1788390911.305038 mapwrite 0x75bd9b thru 0x75c922 (0xb88 bytes) 000485[1] 1788390911.408953 mapwrite 0x4ecfd6 thru 0x4ed244 (0x26f bytes) 000487[1] 1788390911.510964 read 0x582ca0 thru 0x583ded (0x114e bytes) 000488[0] 1788390911.537770 write 0x980fea thru 0x982ef3 (0x1f0a bytes) 000490[1] 1788390911.597933 mapread 0x7c3d86 thru 0x7c4491 (0x70c bytes) 000494[0] 1788390911.702886 read 0x561c29 thru 0x563c26 (0x1ffe bytes) 000496[0] 1788390911.762323 trunc from 0x982ef4 to 0x40c9ef (0x576504 bytes) 000497[0] 1788390911.804331 write 0x741e78 thru 0x7437aa (0x1933 bytes) 000498[1] 1788390911.837907 write 0x1bd61c thru 0x1bf0e6 (0x1acb bytes) All operations completed A-OK! PASS 16b (26s) == sanityn test 16c: verify data consistency on ldiskfs with cache disabled (b=17397) ========================================================== 19:15:16 (1788390916) SKIP: sanityn test_16c dio on ldiskfs only SKIP 16c (4s) == sanityn test 16d: Verify DIO and buffer IO with two clients ========================================================== 19:15:20 (1788390920) 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.56787 s, 14.8 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.270796 s, 31.0 MB/s ldlm.namespaces.MGC192.168.201.153@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff96b686658000.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff96b688d5b800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff96b686658000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff96b688d5b800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff96b686658000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff96b688d5b800.lru_size=clear 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.1797 s, 10.3 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 3.14926 s, 33.3 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 2.92809 s, 35.8 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.50413 s, 19.1 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.48344 s, 12.4 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 3.64045 s, 28.8 MB/s PASS 16d (46s) == sanityn test 16e: Verify size consistency for O_DIRECT write ========================================================== 19:16:06 (1788390966) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.270428 s, 38.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133154 s, 7.9 MB/s PASS 16e (8s) == sanityn test 16f: rw sequential consistency vs drop_caches ========================================================== 19:16:14 (1788390974) TIMEOUT PASS 16f (27s) == sanityn test 16g: mmap rw sequential consistency vs drop_caches ========================================================== 19:16:41 (1788391001) TIMEOUT PASS 16g (26s) == sanityn test 16h: mmap read after truncate file ======= 19:17:07 (1788391027) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16h (5s) == sanityn test 16i: read after truncate file ============ 19:17:12 (1788391032) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16i (5s) == sanityn test 16j: race dio with buffered i/o ========== 19:17:18 (1788391038) 1+0 records in 1+0 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.77061 s, 14.8 MB/s bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.05899 s, 24.8 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.29667 s, 11.4 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.40038 s, 18.7 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.94348 s, 13.5 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.57398 s, 16.7 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.04156 s, 12.8 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.6655 s, 15.7 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.72212 s, 15.2 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.07824 s, 12.6 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.41322 s, 10.9 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.96026 s, 13.4 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.96979 s, 13.3 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.37593 s, 11.0 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.49871 s, 10.5 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.11483 s, 12.4 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.17828 s, 12.0 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.45668 s, 10.7 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.48904 s, 10.5 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK PASS 16j (31s) == sanityn test 16k: Parallel FSX and drop caches should not panic ========================================================== 19:17:48 (1788391068) + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.1 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.2 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.3 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.4 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.5 -l 67108864 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: skipping zero size read test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read skipping zero size read truncating to largest ever: 0x1a7062c truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x1a7062c truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab All operations completed A-OK! + fsx 1 done: rc=0 Wed Sep 2 19:18:22 EDT 2026 All operations completed A-OK! All operations completed A-OK! + fsx 2 done: rc=0 Wed Sep 2 19:18:23 EDT 2026 All operations completed A-OK! + fsx 3 done: rc=0 Wed Sep 2 19:18:23 EDT 2026 + fsx 4 done: rc=0 Wed Sep 2 19:18:23 EDT 2026 All operations completed A-OK! + fsx 5 done: rc=0 Wed Sep 2 19:18:23 EDT 2026 PASS 16k (43s) == sanityn test 17: resource creation/LVB creation race ========================================================================= 19:18:31 (1788391111) fail_loc=0x8000030a PASS 17 (8s) == sanityn test 18: mmap sanity check =========================================================================================== 19:18:39 (1788391119) mmap test1: basic mmap operation (PASS, 0.1106s) mmap test2: MAP_PRIVATE not write back (PASS, 0.053118s) mmap test3: concurrent mmap ops on two nodes (PASS, 9.569s) mmap test4: c1 write to f1 from mmapped f2, c2 write to f1 from mmapped f1 (PASS, 4.4421s) mmap test5: read/write file to/from the buffer which mmapped to just this file (PASS, 0.10046s) mmap test6: check mmap write/read content on two nodes (PASS, 0.18956s) mmap test7: file i/o with an unmapped buffer (SKIPPED, 0s) mmap test8: SIGBUS for beyond file size (SKIPPED, 0s) mmap test9: SIGBUS for negative file offset (SKIPPED, 0s) mmap test10: multi-thread mmap access (PASS, 0.97209s) mmap test11: multi-thread shared mmap (PASS, 1.0642s) mmap test10: mtime not change for readonly mmap access (PASS, 1.1079s) PASS 18 (24s) == sanityn test 19: test concurrent uncached read races ========================================================================= 19:19:03 (1788391143) oleg153-server: error: get_param: param_path 'osd-*/lustre-OST*/read_cache_enable': No such file or directory pdsh@oleg153-client: oleg153-server: ssh exited with exit code 2 SKIP: sanityn test_19 not cache-capable obdfilter SKIP 19 (3s) == sanityn test 20: test extra readahead page left in cache ============================================================== 19:19:06 (1788391146) PASS 20 (6s) == sanityn test 21: Try to remove mountpoint on another dir ============================================================== 19:19:12 (1788391152) rmdir: removing directory, '/mnt/lustre/d21.sanityn' rmdir: failed to remove '/mnt/lustre/d21.sanityn': Device or resource busy rmdir: removing directory, '/mnt/lustre2/d21.sanityn' Removed mounted directory from another mountpoint, needs to be fixed PASS 21 (5s) == sanityn test 23: others should see updated atime while another read============================================================== 19:19:17 (1788391157) now is 1788391159 starting reads multiop /mnt/lustre/f23.sanityn vor20_c TMPPIPE=/tmp/multiop_open_wait_pipe.7616 new atime is 1788391220 PASS 23 (68s) == sanityn test 24a: lfs df [-ih] [path] test =================================================================================== 19:20:25 (1788391225) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 5120 3764224 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 5120 3764224 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre2 UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 115.3K 403 114.9K 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 114.5K 400 114.1K 1% /mnt/lustre[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.3K 403 114.9K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 400 114.1K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre2 UUID bytes Used Available Use% Mounted on lustre-MDT0000_UUID 2.1G 2.1M 2.1G 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3.6G 5.0M 3.6G 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3.6G 32.0M 3.6G 1% /mnt/lustre[OST:1] filesystem_summary: 7.2G 37.0M 7.2G 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118099 403 117696 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117232 400 116832 1% /mnt/lustre2[OST:1] filesystem_summary: 234914 386 234528 1% /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 5120 3764224 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.3K 403 114.9K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 400 114.1K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 118099 403 117696 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 117232 400 116832 1% /mnt/lustre[OST:1] filesystem_summary: 234914 386 234528 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118099 403 117696 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117232 400 116832 1% /mnt/lustre2[OST:1] filesystem_summary: 234914 386 234528 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 5120 3764224 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 5120 3764224 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre2 PASS 24a (5s) == sanityn test 24b: lfs df should show both filesystems ========================================================================= 19:20:30 (1788391230) PASS 24b (5s) == sanityn test 25a: change ACL on one mountpoint be seen on another ============================================================= 19:20:35 (1788391235) running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] PASS 25a (6s) == sanityn test 25b: change ACL under remote dir on one mountpoint be seen on another ========================================================== 19:20:41 (1788391241) SKIP: sanityn test_25b needs >= 2 MDTs SKIP 25b (3s) == sanityn test 26a: allow mtime to get older ============ 19:20:44 (1788391244) utime: good mknod times 1788391245 <= 1788391246 <= 1788391246 for /mnt/lustre/f26a utime: good mknod times 1788391245 <= 1788391246 <= 1788391246 for /mnt/lustre2/f26a utime: good utime mtimes 100000, atime 200000 utime: updated times for /mnt/lustre2/f26a PASS 26a (6s) == sanityn test 26b: sync mtime between ost and mds ====== 19:20:50 (1788391250) PASS 26b (8s) == sanityn test 26c: set-in-past on open file is not lost on close ========================================================== 19:20:58 (1788391258) multiop /mnt/lustre/f26c.sanityn vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7616 File: /mnt/lustre/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502396 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-09-02 19:20:59.000000000 -0400 Birth: 2026-09-02 19:21:00.000000000 -0400 File: /mnt/lustre2/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502396 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-09-02 19:20:59.000000000 -0400 Birth: 2026-09-02 19:21:00.000000000 -0400 PASS 26c (7s) SKIP: sanityn test_27 skipping excluded test 27 == sanityn test 30: recreate file race =================== 19:21:07 (1788391267) 0 PASS 30 (8s) == sanityn test 31a: voluntary cancel / blocking ast race======================================================================== 19:21:15 (1788391275) fail_loc=0x314 PASS 31a (6s) == sanityn test 31b: voluntary OST cancel / blocking ast race======================================================================== 19:21:21 (1788391281) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg153-server mds-ost sync done. sleep 5 for ZFS MDS Waiting for MDT destroys to complete fail_loc=0x314 fail_loc=0x316 fail_loc=0 fail_loc=0 Filesystem 1K-blocks Used Available Use% Mounted on 192.168.201.153@tcp:/lustre 7542784 38912 7499776 1% /mnt/lustre2 PASS 31b (19s) == sanityn test 31r: open-rename(replace) race =========== 19:21:41 (1788391301) -rw-r--r-- 1 root root 0 Sep 2 19:21 /mnt/lustre/f31r.sanityn.target fail_loc=0x1419 fail_val=3 PASS 31r (7s) == sanityn test 31s: open should not revalidate invalid dentry ========================================================== 19:21:49 (1788391309) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 1205: 43142 Terminated tail -f $tfile PASS 31s (5s) == sanityn test 31t: getattr should not revalidate invalid dentry ========================================================== 19:21:54 (1788391314) PASS 31t (7s) == sanityn test 32b: lockless i/o ======================== 19:22:01 (1788391321) SKIP: sanityn test_32b max_nolock_bytes is removed >= 2.17.53 SKIP 32b (4s) == sanityn test 32c: contention detection ================ 19:22:05 (1788391325) ldlm.namespaces.filter-lustre-OST0000_UUID.contended_locks=2 ldlm.namespaces.filter-lustre-OST0001_UUID.contended_locks=2 ldlm.namespaces.filter-lustre-OST0000_UUID.contention_events=0 ldlm.namespaces.filter-lustre-OST0001_UUID.contention_events=0 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.07641 s, 48.7 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.10114 s, 47.6 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.6954 s, 30.9 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.01521 s, 26.0 MB/s contention_seconds 4, contention_events 17 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.42591 s, 36.8 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.60898 s, 32.6 MB/s ldlm.namespaces.filter-lustre-OST0000_UUID.contention_events=0 ldlm.namespaces.filter-lustre-OST0001_UUID.contention_events=0 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.93937 s, 55.8 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.951147 s, 55.1 MB/s PASS 32c (24s) SKIP: sanityn test_33a skipping SLOW test 33a == sanityn test 33b: COS: cross create/delete, 2 clients, benchmark under remote dir ========================================================== 19:22:30 (1788391350) SKIP: sanityn test_33b Need two or more clients, have 1 SKIP 33b (3s) == sanityn test 33c: Cancel cross-MDT lock should trigger Sync-on-Lock-Cancel ========================================================== 19:22:33 (1788391353) SKIP: sanityn test_33c needs >= 2 MDTs SKIP 33c (3s) == sanityn test 33d: dependent transactions should trigger COS ========================================================== 19:22:36 (1788391356) SKIP: sanityn test_33d needs >= 2 MDTs SKIP 33d (3s) == sanityn test 33e: independent transactions shouldn't trigger COS ========================================================== 19:22:39 (1788391359) SKIP: sanityn test_33e needs >= 2 MDTs SKIP 33e (3s) == sanityn test 34: no lock timeout under IO ============= 19:22:42 (1788391362) fail_loc=0x512 fail_loc=0x512 lock should not expire writing on client1 reading on client2 success fail_loc=0x511 fail_loc=0x511 lock should expire writing on client1 reading on client2 success oleg153-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff96b686658000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff96b686658000.ost_server_uuid in FULL state after 0 sec oleg153-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff96b686658000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff96b686658000.ost_server_uuid in IDLE state after 0 sec oleg153-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff96b686658000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff96b686658000.ost_server_uuid in FULL state after 0 sec oleg153-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff96b686658000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff96b686658000.ost_server_uuid in IDLE state after 0 sec oleg153-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff96b686658000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff96b686658000.ost_server_uuid in FULL state after 0 sec oleg153-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff96b686658000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff96b686658000.ost_server_uuid in IDLE state after 0 sec PASS 34 (123s) == sanityn test 35: -EINTR cp_ast vs. bl_ast race does not evict client ========================================================== 19:24:45 (1788391485) fail_loc=0x80000317 Race attempt 0 Wait for 50438 50482 for 60 sec... - open/close 933 (time 1788391501.03 total 10.00 last 93.28) - open/close 1854 (time 1788391511.04 total 20.02 last 91.98) - open/close 2738 (time 1788391521.04 total 30.02 last 88.39) - open/close 3747 (time 1788391531.06 total 40.03 last 100.76) total: 4000 open/close in 42.51 seconds: 94.09 ops/second fail_loc=0x0 Filesystem Size Used Avail Use% Mounted on 192.168.201.153@tcp:/lustre 7.2G 38M 7.2G 1% /mnt/lustre 192.168.201.153@tcp:/lustre 7.2G 38M 7.2G 1% /mnt/lustre2 PASS 35 (70s) == sanityn test 36: handle ESTALE/open-unlink correctly == 19:25:55 (1788391555) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg153-server mds-ost sync done. Waiting for MDT destroys to complete 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.0772 s, 48.7 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7616 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(0) *** before(7499776) after_dd(7443456) after(7494656) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.76661 s, 68.4 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7616 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(1) *** before(7494656) after_dd(7446528) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.19127 s, 44.0 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7616 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(2) *** before(7497728) after_dd(7442432) after(7493632) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.971746 s, 54.0 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7616 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(3) *** before(7493632) after_dd(7446528) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.12319 s, 46.7 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7616 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(4) *** before(7497728) after_dd(7446528) after(7496704) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.04351 s, 50.2 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7616 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(5) *** before(7496704) after_dd(7440384) after(7491584) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.944984 s, 55.5 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7616 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(6) *** before(7491584) after_dd(7438336) after(7489536) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.9246 s, 56.7 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7616 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(7) *** before(7489536) after_dd(7440384) after(7491584) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.858512 s, 61.1 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7616 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(8) *** before(7491584) after_dd(7443456) after(7494656) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.963407 s, 54.4 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7616 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(9) *** before(7494656) after_dd(7445504) after(7496704) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.911466 s, 57.5 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7616 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(10) *** before(7496704) after_dd(7441408) after(7492608) PASS 36 (291s) == sanityn test 37: check i_size is not updated for directory on close (bug 18695) ======================================================================== 19:30:46 (1788391846) multiop /mnt/lustre/d37.sanityn vD_c TMPPIPE=/tmp/multiop_open_wait_pipe.7616 - create 1909 (time 1788391859.25 total 10.00 last 190.87) - create 3718 (time 1788391869.25 total 20.01 last 180.80) - create 5353 (time 1788391879.26 total 30.01 last 163.50) - create 7136 (time 1788391889.26 total 40.01 last 178.22) - create 9034 (time 1788391899.26 total 50.02 last 189.74) total: 10000 create in 54.98 seconds: 181.87 ops/second PASS 37 (63s) == sanityn test 39a: file mtime does not change after rename ========================================================== 19:31:49 (1788391909) repeat after cancel_lru_locks PASS 39a (5s) == sanityn test 39b: file mtime the same on clients with/out lock ========================================================== 19:31:54 (1788391914) repeat after cancel_lru_locks PASS 39b (6s) == sanityn test 39c: check truncate mtime update ================================================================================ 19:32:00 (1788391920) repeat after cancel_lru_locks PASS 39c (6s) == sanityn test 39d: sync write should update mtime ====== 19:32:06 (1788391926) fail_loc=0x411 fail_loc=0 PASS 39d (4s) SKIP: sanityn test_40a skipping ALWAYS excluded test 40a == sanityn test 40b: pdirops: open|create and others ======================================================================== 19:32:12 (1788391932) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40b (13s) == sanityn test 40c: pdirops: link and others ======================================================================== 19:32:26 (1788391946) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40c (13s) == sanityn test 40d: pdirops: unlink and others ======================================================================== 19:32:39 (1788391959) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40d (13s) == sanityn test 40e: pdirops: rename and others ======================================================================== 19:32:52 (1788391972) No conflict No conflict No conflict No conflict No conflict No conflict PASS 40e (13s) == sanityn test 41a: pdirops: create vs mkdir ======================================================================== 19:33:05 (1788391985) mkdir: cannot create directory '/mnt/lustre2/f41a.sanityn': File exists Conflict PASS 41a (10s) == sanityn test 41b: pdirops: create vs create ======================================================================== 19:33:15 (1788391995) open: File exists Conflict PASS 41b (9s) == sanityn test 41c: pdirops: create vs link ======================================================================== 19:33:24 (1788392004) link: cannot create link '/mnt/lustre2/f41c.sanityn' to '/mnt/lustre2/f41c.sanityn-2': File exists Conflict PASS 41c (10s) == sanityn test 41d: pdirops: create vs unlink ======================================================================== 19:33:34 (1788392014) Conflict PASS 41d (10s) == sanityn test 41e: pdirops: create and rename (tgt) ======================================================================== 19:33:44 (1788392024) Conflict PASS 41e (11s) == sanityn test 41f: pdirops: create and rename (src) ======================================================================== 19:33:56 (1788392036) Conflict PASS 41f (9s) == sanityn test 41g: pdirops: create vs getattr ======================================================================== 19:34:05 (1788392045) Conflict PASS 41g (10s) == sanityn test 41h: pdirops: create vs readdir ======================================================================== 19:34:15 (1788392055) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 2427: lctl set_param -n fail_loc=0 2>/dev/null || true: No such file or directory Conflict PASS 41h (10s) == sanityn test 41i: reint_open: create vs create ======== 19:34:25 (1788392065) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 41i (857s) == sanityn test 42a: pdirops: mkdir vs mkdir ======================================================================== 19:48:42 (1788392922) mkdir: cannot create directory '/mnt/lustre2/f42a.sanityn': File exists Conflict PASS 42a (10s) == sanityn test 42b: pdirops: mkdir vs create ======================================================================== 19:48:52 (1788392932) open: File exists Conflict PASS 42b (10s) == sanityn test 42c: pdirops: mkdir vs link ======================================================================== 19:49:02 (1788392942) link: cannot create link '/mnt/lustre2/f42c.sanityn' to '/mnt/lustre2/f42c.sanityn-2': File exists Conflict PASS 42c (10s) == sanityn test 42d: pdirops: mkdir vs unlink ======================================================================== 19:49:12 (1788392952) Conflict PASS 42d (9s) == sanityn test 42e: pdirops: mkdir and rename (tgt) ======================================================================== 19:49:21 (1788392961) mv: cannot overwrite directory '/mnt/lustre2/f42e.sanityn' with non-directory Conflict PASS 42e (10s) == sanityn test 42f: pdirops: mkdir and rename (src) ======================================================================== 19:49:31 (1788392971) Conflict PASS 42f (10s) == sanityn test 42g: pdirops: mkdir vs getattr ======================================================================== 19:49:41 (1788392981) Conflict PASS 42g (11s) == sanityn test 42h: pdirops: mkdir vs readdir ======================================================================== 19:49:52 (1788392992) Conflict PASS 42h (10s) == sanityn test 43a: rmdir,mkdir doesn't return -EEXIST ======================================================================== 19:50:02 (1788393002) PASS 43a (64s) == sanityn test 43b: pdirops: unlink vs create ======================================================================== 19:51:07 (1788393067) Conflict PASS 43b (21s) == sanityn test 43c: pdirops: unlink vs link ======================================================================== 19:51:28 (1788393088) Conflict PASS 43c (20s) == sanityn test 43d: pdirops: unlink vs unlink ======================================================================== 19:51:48 (1788393108) rm: cannot remove '/mnt/lustre2/f43d.sanityn': No such file or directory Conflict PASS 43d (15s) == sanityn test 43e: pdirops: unlink and rename (tgt) ======================================================================== 19:52:03 (1788393123) Conflict PASS 43e (16s) == sanityn test 43f: pdirops: unlink and rename (src) ======================================================================== 19:52:20 (1788393140) mv: cannot stat '/mnt/lustre2/f43f.sanityn': No such file or directory Conflict PASS 43f (20s) == sanityn test 43g: pdirops: unlink vs getattr ======================================================================== 19:52:40 (1788393160) stat: cannot statx '/mnt/lustre2/f43g.sanityn': No such file or directory Conflict PASS 43g (14s) == sanityn test 43h: pdirops: unlink vs readdir ======================================================================== 19:52:54 (1788393174) Conflict PASS 43h (18s) == sanityn test 43i: pdirops: unlink vs remote mkdir ===== 19:53:12 (1788393192) SKIP: sanityn test_43i needs >= 2 MDTs SKIP 43i (4s) == sanityn test 43j: racy mkdir return EEXIST ======================================================================== 19:53:17 (1788393197) mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists PASS 43j (138s) == sanityn test 43k: unlink vs create ==================== 19:55:35 (1788393335) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 43k (1216s) == sanityn test 44a: pdirops: rename tgt vs mkdir ======================================================================== 20:15:50 (1788394550) mkdir: cannot create directory '/mnt/lustre2/f44a.sanityn': File exists Conflict Wed Sep 2 20:15:58 EDT 2026 PASS 44a (13s) == sanityn test 44b: pdirops: rename tgt vs create ======================================================================== 20:16:03 (1788394563) open: File exists Conflict PASS 44b (13s) == sanityn test 44c: pdirops: rename tgt vs link ======================================================================== 20:16:16 (1788394576) link: cannot create link '/mnt/lustre2/f44c.sanityn' to '/mnt/lustre2/f44c.sanityn-3': File exists Conflict PASS 44c (13s) == sanityn test 44d: pdirops: rename tgt vs unlink ======================================================================== 20:16:29 (1788394589) Conflict PASS 44d (14s) == sanityn test 44e: pdirops: rename tgt and rename (tgt) ======================================================================== 20:16:43 (1788394603) Conflict PASS 44e (14s) == sanityn test 44f: pdirops: rename tgt and rename (src) ======================================================================== 20:16:57 (1788394617) Conflict PASS 44f (12s) == sanityn test 44g: pdirops: rename tgt vs getattr ======================================================================== 20:17:09 (1788394629) Conflict PASS 44g (12s) == sanityn test 44h: pdirops: rename tgt vs readdir ======================================================================== 20:17:22 (1788394642) Conflict PASS 44h (10s) == sanityn test 44i: pdirops: rename tgt vs remote mkdir ========================================================== 20:17:33 (1788394653) SKIP: sanityn test_44i needs >= 2 MDTs SKIP 44i (2s) == sanityn test 45a: rename,mkdir doesn't return -EEXIST ======================================================================== 20:17:35 (1788394655) PASS 45a (107s) == sanityn test 45b: pdirops: rename src vs create ======================================================================== 20:19:22 (1788394762) Conflict PASS 45b (13s) == sanityn test 45c: pdirops: rename src vs link ======================================================================== 20:19:35 (1788394775) Conflict PASS 45c (12s) == sanityn test 45d: pdirops: rename src vs unlink ======================================================================== 20:19:48 (1788394788) rm: cannot remove '/mnt/lustre2/f45d.sanityn': No such file or directory Conflict PASS 45d (11s) == sanityn test 45e: pdirops: rename src and rename (tgt) ======================================================================== 20:19:59 (1788394799) Conflict PASS 45e (14s) == sanityn test 45f: pdirops: rename src and rename (src) ======================================================================== 20:20:13 (1788394813) mv: cannot stat '/mnt/lustre2/f45f.sanityn': No such file or directory Conflict PASS 45f (15s) == sanityn test 45g: pdirops: rename src vs getattr ======================================================================== 20:20:29 (1788394829) stat: cannot statx '/mnt/lustre2/f45g.sanityn': No such file or directory Conflict PASS 45g (11s) == sanityn test 45h: pdirops: unlink vs readdir ======================================================================== 20:20:41 (1788394841) Conflict PASS 45h (13s) == sanityn test 45i: pdirops: rename src vs remote mkdir ========================================================== 20:20:53 (1788394853) SKIP: sanityn test_45i needs >= 2 MDTs SKIP 45i (4s) == sanityn test 45j: read vs rename ====================== 20:20:57 (1788394857) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 45j (1267s) == sanityn test 46a: pdirops: link vs mkdir ======================================================================== 20:42:04 (1788396124) mkdir: cannot create directory '/mnt/lustre2/f46a.sanityn': File exists Conflict PASS 46a (13s) == sanityn test 46b: pdirops: link vs create ======================================================================== 20:42:17 (1788396137) open: File exists Conflict PASS 46b (13s) == sanityn test 46c: pdirops: link vs link ======================================================================== 20:42:30 (1788396150) link: cannot create link '/mnt/lustre2/f46c.sanityn' to '/mnt/lustre2/f46c.sanityn': File exists Conflict PASS 46c (13s) == sanityn test 46d: pdirops: link vs unlink ======================================================================== 20:42:44 (1788396164) Conflict PASS 46d (12s) == sanityn test 46e: pdirops: link and rename (tgt) ======================================================================== 20:42:56 (1788396176) Conflict PASS 46e (14s) == sanityn test 46f: pdirops: link and rename (src) ======================================================================== 20:43:10 (1788396190) Conflict PASS 46f (14s) == sanityn test 46g: pdirops: link vs getattr ======================================================================== 20:43:24 (1788396204) Conflict PASS 46g (16s) == sanityn test 46h: pdirops: link vs readdir ======================================================================== 20:43:41 (1788396221) Conflict PASS 46h (15s) == sanityn test 46i: pdirops: link vs remote mkdir ======= 20:43:55 (1788396235) SKIP: sanityn test_46i needs >= 2 MDTs SKIP 46i (3s) == sanityn test 47a: pdirops: remote mkdir vs mkdir ====== 20:43:58 (1788396238) SKIP: sanityn test_47a needs >= 2 MDTs SKIP 47a (4s) == sanityn test 47b: pdirops: remote mkdir vs create ===== 20:44:02 (1788396242) SKIP: sanityn test_47b needs >= 2 MDTs SKIP 47b (3s) == sanityn test 47c: pdirops: remote mkdir vs link ======= 20:44:05 (1788396245) SKIP: sanityn test_47c needs >= 2 MDTs SKIP 47c (4s) == sanityn test 47d: pdirops: remote mkdir vs unlink ===== 20:44:10 (1788396250) SKIP: sanityn test_47d needs >= 2 MDTs SKIP 47d (3s) == sanityn test 47e: pdirops: remote mkdir and rename (tgt) ========================================================== 20:44:13 (1788396253) SKIP: sanityn test_47e needs >= 2 MDTs SKIP 47e (4s) == sanityn test 47f: pdirops: remote mkdir and rename (src) ========================================================== 20:44:17 (1788396257) SKIP: sanityn test_47f needs >= 2 MDTs SKIP 47f (4s) == sanityn test 47g: pdirops: remote mkdir vs getattr ==== 20:44:21 (1788396261) SKIP: sanityn test_47g needs >= 2 MDTs SKIP 47g (5s) == sanityn test 50: osc lvb attrs: enqueue vs. CP AST ======================================================================== 20:44:26 (1788396266) 10+0 records in 10+0 records out 10240 bytes (10 kB, 10 KiB) copied, 0.207683 s, 49.3 kB/s fail_loc=0x410 fail_loc=0x0 PASS 50 (12s) == sanityn test 51a: layout lock: refresh layout should work ========================================================== 20:44:39 (1788396279) 0+1 records in 0+1 records out 158 bytes copied, 0.0322515 s, 4.9 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 3690: kill: (218995) - No such process PASS 51a (10s) == sanityn test 51b: layout lock: glimpse should be able to restart if layout changed ========================================================== 20:44:48 (1788396288) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0364104 s, 28.1 kB/s fail_loc=0x1404 fail_val=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0461412 s, 22.7 MB/s 3145728 PASS 51b (20s) == sanityn test 51c: layout lock: IT_LAYOUT blocked and correct layout can be returned ========================================================== 20:45:08 (1788396308) fail_loc=0x172 Setting layout to have 2 stripes ... 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0507868 s, 20.2 kB/s PASS 51c (12s) == sanityn test 51d: layout lock: losing layout lock should clean up memory map region ========================================================== 20:45:20 (1788396320) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0518007 s, 20.2 MB/s Before revoking layout lock: 1024 KB mapped PASS 51d (10s) == sanityn test 51e: lfs getstripe does not break leases, part 2 ========================================================== 20:45:30 (1788396330) /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 2450 0x992 0x280000400 write lease(2) has applied. /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 2450 0x992 0x280000400 read lease(1) has applied. PASS 51e (9s) == sanityn test 54: rename locking ======================= 20:45:39 (1788396339) llite.lustre-ffff96b686658000.xattr_cache=1 llite.lustre-ffff96b688d5b800.xattr_cache=1 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505013 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-02 20:45:41.000000000 -0400 Modify: 2026-09-02 20:45:41.000000000 -0400 Change: 2026-09-02 20:45:41.000000000 -0400 Birth: 2026-09-02 20:45:43.000000000 -0400 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000154 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505016 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-02 20:45:48.000000000 -0400 Modify: 2026-09-02 20:45:50.000000000 -0400 Change: 2026-09-02 20:45:50.000000000 -0400 Birth: 2026-09-02 20:45:50.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505019 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-02 20:45:55.000000000 -0400 Modify: 2026-09-02 20:45:55.000000000 -0400 Change: 2026-09-02 20:45:55.000000000 -0400 Birth: 2026-09-02 20:45:56.000000000 -0400 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505018 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-02 20:45:55.000000000 -0400 Modify: 2026-09-02 20:45:56.000000000 -0400 Change: 2026-09-02 20:45:56.000000000 -0400 Birth: 2026-09-02 20:45:56.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000154 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505022 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-02 20:46:01.000000000 -0400 Modify: 2026-09-02 20:46:02.000000000 -0400 Change: 2026-09-02 20:46:02.000000000 -0400 Birth: 2026-09-02 20:46:02.000000000 -0400 File: /mnt/lustre/d1 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505021 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-02 20:46:01.000000000 -0400 Modify: 2026-09-02 20:46:02.000000000 -0400 Change: 2026-09-02 20:46:02.000000000 -0400 Birth: 2026-09-02 20:46:02.000000000 -0400 PASS 54 (34s) == sanityn test 55a: rename vs unlink target dir ========= 20:46:13 (1788396373) fail_loc=0x80000156 mv: cannot move '/mnt/lustre/d55a.sanityn/d1/d2' to '/mnt/lustre/d55a.sanityn/d3/d2': No such file or directory PASS 55a (13s) == sanityn test 55b: rename vs unlink source dir ========= 20:46:26 (1788396386) fail_loc=0x80000156 mv: cannot stat '/mnt/lustre/d55b.sanityn/d1/d2': No such file or directory PASS 55b (13s) == sanityn test 55c: rename vs unlink orphan target dir == 20:46:39 (1788396399) fail_loc=0x156 mv: cannot move '/mnt/lustre/d55c.sanityn/d1/d2' to '/mnt/lustre/d55c.sanityn/d3/d2': No such file or directory PASS 55c (19s) == sanityn test 55d: rename file vs link ================= 20:46:58 (1788396418) fail_loc=0x155 mv: '/mnt/lustre/d55d.sanityn/f1' and '/mnt/lustre/d55d.sanityn/d55d.sanityn/f1' are the same file PASS 55d (16s) == sanityn test 55e: rename race AB/BA under the same parent dir ========================================================== 20:47:14 (1788396434) SKIP: sanityn test_55e needs >= 2 MDTs SKIP 55e (3s) == sanityn test 55f: rename: (P1/A -> P2/B) race with (P2/B -> P1/A) ========================================================== 20:47:17 (1788396437) fail_loc=0x155 We survived after P1A_P2B/P2B_P1A race test fail_loc=0x0 PASS 55f (21s) == sanityn test 55g: rename: race with trylock =========== 20:47:38 (1788396458) mdt.lustre-MDT0000.enable_rename_trylock=1 fail_loc=0x155 mv: cannot move '/mnt/lustre2/d55g.sanityn/b1/b2' to a subdirectory of itself, '/mnt/lustre2/d55g.sanityn/a1/a2/a3/a4/b2' We survived race with trylock test fail_loc=0x0 PASS 55g (25s) == sanityn test 56a: test llverdev with single large stripe ========================================================== 20:48:04 (1788396484) run_llverdev /mnt/lustre/f56a.sanityn -p -s 16000000 -c 4k llverdev: /mnt/lustre/f56a.sanityn is 16777216000000 bytes (15625 GB) in size timestamp: 1788396486 chunksize: 4096 size: 16777216000000 write offset: 16384000000kB 0.586914 MB/s read offset: 16384000000kB inf MB/s llverdev: data verified successfully PASS 56a (112s) == sanityn test 56b: test llverdev and partial verify of wide stripe file ========================================================== 20:49:57 (1788396597) run_llverdev /mnt/lustre/f56b.sanityn -p -s 32000000 -c 4k llverdev: /mnt/lustre/f56b.sanityn is 33554432000000 bytes (31250 GB) in size timestamp: 1788396602 chunksize: 4096 size: 33554432000000 write offset: 32768000000kB 0.865775 MB/s read offset: 32768000000kB inf MB/s llverdev: data verified successfully PASS 56b (154s) == sanityn test 60: Verify data_version behaviour ======== 20:52:32 (1788396752) PASS 60 (16s) == sanityn test 70a: cd directory && rm directory ======== 20:52:47 (1788396767) PASS 70a (10s) == sanityn test 70b: remove files after calling rm_entry ========================================================== 20:52:57 (1788396777) PASS 70b (10s) == sanityn test 71a: correct file map just after write operation is finished ========================================================== 20:53:07 (1788396787) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.00436498 s, 15.0 MB/s SKIP: sanityn test_71a ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71a (4s) == sanityn test 71b: check fiemap support for stripecount > 1 ========================================================== 20:53:11 (1788396791) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0331071 s, 2.0 MB/s SKIP: sanityn test_71b ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71b (5s) == sanityn test 71c: check FIEMAP_EXTENT_LAST flag with different extents number ========================================================== 20:53:16 (1788396796) SKIP: sanityn test_71c support only ldiskfs ost SKIP 71c (4s) == sanityn test 71d: fiemap corruption test with fm_extent_count=0 ========================================================== 20:53:20 (1788396800) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0587542 s, 1.1 MB/s SKIP: sanityn test_71d ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71d (4s) == sanityn test 72: getxattr/setxattr cache should be consistent between nodes ========================================================== 20:53:24 (1788396804) llite.lustre-ffff96b686658000.xattr_cache=1 llite.lustre-ffff96b688d5b800.xattr_cache=1 getfattr: Removing leading '/' from absolute path names user.attr1="value1" getfattr: Removing leading '/' from absolute path names user.attr1="value2" getfattr: Removing leading '/' from absolute path names getfattr: Removing leading '/' from absolute path names b362740ad371eae580e7f85452e8830b - 870e072da1ebfc20ea8c0cd3d6bdf3b6 - PASS 72 (11s) == sanityn test 73: getxattr should not cause xattr lock cancellation ========================================================== 20:53:35 (1788396815) llite.lustre-ffff96b686658000.xattr_cache=1 llite.lustre-ffff96b688d5b800.xattr_cache=1 getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" PASS 73 (9s) == sanityn test 74: flock deadlock: different mounts ======================================================================== 20:53:44 (1788396824) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0362538 s, 28.2 kB/s 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0121996 s, 83.9 kB/s 231824: get lock2 231825: get lock1 231824: done 231824: get lock1 231825: done 231825: get lock2 fcntl cmd 7 failed: Resource deadlock avoided 231825: cannot get lock2: Resource deadlock avoided 231825: exit rc=0 231824: done 231824: exit rc=0 PASS 74 (13s) == sanityn test 75: osc: upcall after unuse lock============================================================================= 20:53:57 (1788396837) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0864648 s, 24.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0363089 s, 28.9 MB/s fail_loc=0x31d /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 4373: 232406 Killed stat -c %s $DIR1/$tfile fail_loc=0 PASS 75 (13s) == sanityn test 76: Verify MDT open_files listing ======== 20:54:11 (1788396851) opening files: ........ closing files: ........ PASS 76 (77s) == sanityn test 77a: check FIFO NRS policy =============== 20:55:27 (1788396927) ost.OSS.ost.nrs_policies=fifo ost.OSS.ost_create.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_out.nrs_policies=fifo ost.OSS.ost_seq.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.426015 s, 39.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0424088 s, 24.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0610711 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0471928 s, 22.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133091 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104872 s, 10.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172929 s, 6.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135973 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189898 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.255156 s, 4.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff96b688d5b800.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.238046 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.267057 s, 3.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.284575 s, 3.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21205 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22223 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.25591 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.330035 s, 3.2 MB/s 1+0 records in 1+0 records out ldlm.namespaces.lustre-OST0001-osc-ffff96b688d5b800.lock_unused_count=2 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00679285 s, 154 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00688373 s, 152 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00193835 s, 541 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00195987 s, 535 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0959408 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0641723 s, 16.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0697752 s, 15.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109879 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111305 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0814658 s, 12.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0434905 s, 24.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117729 s, 8.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106495 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110436 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.056755 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0507594 s, 20.7 MB/s removed '/mnt/lustre/d77a.sanityn/nrs_r_oleg153-client.virtnet' removed '/mnt/lustre/d77a.sanityn/nrs_w_oleg153-client.virtnet' removed directory '/mnt/lustre/d77a.sanityn' PASS 77a (9s) == sanityn test 77b: check CRR-N NRS policy ============== 20:55:36 (1788396936) ost.OSS.ost.nrs_policies=crrn ost.OSS.ost_create.nrs_policies=crrn ost.OSS.ost_io.nrs_policies=crrn ost.OSS.ost_out.nrs_policies=crrn ost.OSS.ost_seq.nrs_policies=crrn ost.OSS.ost.nrs_crrn_quantum=1 ost.OSS.ost_create.nrs_crrn_quantum=1 ost.OSS.ost_io.nrs_crrn_quantum=1 ost.OSS.ost_out.nrs_crrn_quantum=1 ost.OSS.ost_seq.nrs_crrn_quantum=1 policy: crr-n, crrn_quantum 1 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.585305 s, 28.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0496732 s, 21.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0365282 s, 28.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0762322 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116147 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0351342 s, 29.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0930302 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160161 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123491 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100481 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.267027 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.25764 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.290964 s, 3.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff96b688d5b800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.365445 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.31224 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.261825 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.399417 s, 2.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.080949 s, 13.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0284286 s, 36.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0936559 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0708697 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162277 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147574 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178148 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176808 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134776 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127308 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153651 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.086421 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142228 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143486 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150952 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148768 s, 7.0 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg153-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg153-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost.nrs_crrn_quantum=64 ost.OSS.ost_create.nrs_crrn_quantum=64 ost.OSS.ost_io.nrs_crrn_quantum=64 ost.OSS.ost_out.nrs_crrn_quantum=64 ost.OSS.ost_seq.nrs_crrn_quantum=64 policy: crr-n, crrn_quantum 64 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.40836 s, 41.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0492663 s, 21.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128928 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0728868 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17749 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0996942 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120102 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193449 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103394 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126039 s, 8.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff96b688d5b800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201658 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179009 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.270856 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.378812 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.372307 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.313139 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.391745 s, 2.7 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff96b688d5b800.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0099889 s, 105 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0401255 s, 26.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0193284 s, 54.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0224102 s, 46.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0481018 s, 21.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0464736 s, 22.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0279255 s, 37.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0794863 s, 13.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0687536 s, 15.3 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0590665 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0525712 s, 19.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0614818 s, 17.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0535252 s, 19.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0532424 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0723772 s, 14.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0581651 s, 18.0 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg153-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg153-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77b (13s) == sanityn test 77c: check ORR NRS policy ================ 20:55:49 (1788396949) ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_orr_quantum=1 ost.OSS.ost_io.nrs_orr_offset_type=physical ost.OSS.ost_io.nrs_orr_supported=reads policy: orr, orr_quantum 1, orr_offset_type physical, orr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.505437 s, 33.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0650669 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.087726 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0766999 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0839212 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0765802 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145088 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134896 s, 7.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159202 s, 6.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0664245 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.33044 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.258946 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.348643 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.298794 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.326863 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.432924 s, 2.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff96b688d5b800.lock_unused_count=3 ldlm.namespaces.lustre-OST0001-osc-ffff96b688d5b800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.398186 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133976 s, 78.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164431 s, 63.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00335255 s, 313 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0120079 s, 87.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0027569 s, 380 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0396668 s, 26.4 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff96b688d5b800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0349345 s, 30.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00657468 s, 159 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.008727 s, 120 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00256626 s, 409 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129438 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0774707 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0093456 s, 112 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.042473 s, 24.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0202775 s, 51.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0080168 s, 131 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg153-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg153-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=writes ost.OSS.ost_io.nrs_orr_quantum=64 policy: orr, orr_quantum 64, orr_offset_type physical, orr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.386747 s, 43.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.053179 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0503558 s, 20.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196873 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151308 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.258505 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.273551 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242436 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.307458 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.350144 s, 3.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff96b688d5b800.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff96b688d5b800.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.370202 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.30721 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.348344 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.418738 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.329046 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.355614 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.412844 s, 2.5 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff96b688d5b800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0348516 s, 30.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0546167 s, 19.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00974117 s, 108 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00537998 s, 195 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0079549 s, 132 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00247426 s, 424 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.055161 s, 19.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156052 s, 67.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0176056 s, 59.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00438727 s, 239 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100528 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0655823 s, 16.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0445484 s, 23.5 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00320889 s, 327 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0629051 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126524 s, 82.9 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg153-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg153-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=reads_and_writes ost.OSS.ost_io.nrs_orr_offset_type=logical policy: orr, orr_quantum 64, orr_offset_type logical, orr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.669541 s, 25.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0455575 s, 23.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.054595 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15677 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129837 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168636 s, 6.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21723 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221821 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189851 s, 5.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196166 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.226875 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0491934 s, 21.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0778064 s, 13.5 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0614319 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.25437 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120898 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.278238 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.266962 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0970764 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.279111 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135247 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112093 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.304079 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155648 s, 6.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132359 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107242 s, 9.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0301064 s, 34.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0824212 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167503 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00616669 s, 170 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120515 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1824 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.317359 s, 3.3 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg153-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg153-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77c (17s) == sanityn test 77d: check TRR nrs policy ================ 20:56:07 (1788396967) ost.OSS.ost_io.nrs_policies=trr ost.OSS.ost_io.nrs_trr_quantum=1 ost.OSS.ost_io.nrs_trr_offset_type=physical ost.OSS.ost_io.nrs_trr_supported=reads policy: trr, trr_quantum 1, trr_offset_type physical, trr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.84165 s, 19.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102993 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154802 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223795 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227316 s, 4.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.258922 s, 4.0 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.254436 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.238328 s, 4.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.295982 s, 3.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.337441 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.269498 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.275471 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.390021 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.442041 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.488877 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.563091 s, 1.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff96b688d5b800.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff96b688d5b800.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.43252 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0369947 s, 28.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131201 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14585 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0815558 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161635 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0927453 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182705 s, 5.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155978 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105808 s, 9.9 MB/s 1+0 records in 1+0 records out ldlm.namespaces.lustre-OST0001-osc-ffff96b688d5b800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18692 s, 5.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1187 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178224 s, 5.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130286 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132571 s, 7.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195136 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144729 s, 7.2 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg153-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg153-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=writes ost.OSS.ost_io.nrs_trr_quantum=64 policy: trr, trr_quantum 64, trr_offset_type physical, trr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.737007 s, 22.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196569 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100866 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.275941 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.275235 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.238993 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.225748 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.288815 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.501609 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.626135 s, 1.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.320682 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130811 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13419 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109338 s, 9.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.472649 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0606034 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.415486 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.472399 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16051 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00733668 s, 143 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.639318 s, 1.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.794073 s, 1.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.842537 s, 1.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0754254 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0636701 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0523238 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0262359 s, 40.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163134 s, 64.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.068266 s, 15.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0612137 s, 17.1 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0826487 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0943172 s, 11.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110387 s, 9.5 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg153-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg153-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=reads_and_writes ost.OSS.ost_io.nrs_trr_offset_type=logical policy: trr, trr_quantum 64, trr_offset_type logical, trr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.415483 s, 40.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0753414 s, 13.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0841193 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177752 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128354 s, 8.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.219622 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104028 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0534168 s, 19.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0463812 s, 22.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110908 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222809 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.311739 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.303039 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.377872 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.37533 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.374102 s, 2.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff96b688d5b800.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff96b688d5b800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.441067 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0971137 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181985 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.243104 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126982 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.233181 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193675 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.247411 s, 4.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124796 s, 8.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115047 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122389 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.258225 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208482 s, 5.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142976 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.282543 s, 3.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.224662 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155574 s, 6.7 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg153-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg153-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77d (20s) == sanityn test 77e: check TBF NID nrs policy ============ 20:56:27 (1788396987) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_tbf_rule=start localhost nid={0@lo} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start clients nid={192.168.201.53@tcp} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start others nid={*.*.*.*@tcp} rate=50 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.265297 s, 63.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.038901 s, 27.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0680232 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124323 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190506 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.224145 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202001 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.226778 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192074 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.240222 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.258454 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.274266 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.261841 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.218767 s, 4.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff96b688d5b800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.245085 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.27269 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.364181 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0511595 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1899 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0447788 s, 23.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.081785 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0570165 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0403921 s, 26.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180706 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00265578 s, 395 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff96b688d5b800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00489606 s, 214 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.010086 s, 104 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168139 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124901 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142228 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133917 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00228445 s, 459 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.228617 s, 4.6 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg153-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg153-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change localhost rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change clients rate=101 ost.OSS.ost_io.nrs_tbf_rule=change others rate=51 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.679297 s, 24.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0440673 s, 23.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0612644 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0802316 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0413117 s, 25.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0731594 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211335 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.228054 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193848 s, 5.4 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff96b688d5b800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.313783 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.292908 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.303823 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.345988 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.393718 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.317987 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0638073 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194287 s, 5.4 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff96b688d5b800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0114595 s, 91.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141719 s, 74.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0172978 s, 60.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140453 s, 74.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0119095 s, 88.0 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0205778 s, 51.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0758336 s, 13.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0685062 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00933043 s, 112 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0802 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101766 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167231 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0878716 s, 11.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0679035 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00851656 s, 123 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0331142 s, 31.7 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg153-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg153-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop localhost ost.OSS.ost_io.nrs_tbf_rule=stop clients ost.OSS.ost_io.nrs_tbf_rule=stop others 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.741415 s, 22.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0761877 s, 13.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166882 s, 6.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121049 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17069 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0759567 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133647 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211704 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203541 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.327641 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.377887 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.27545 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.48644 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.511282 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.448056 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.423267 s, 2.5 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff96b688d5b800.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.700858 s, 1.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125863 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132732 s, 7.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0782113 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121313 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194153 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165053 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167115 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.31581 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208239 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227193 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116238 s, 9.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135722 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106306 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.224232 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23639 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116729 s, 9.0 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg153-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg153-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.775023 s, 21.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0438695 s, 23.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118465 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0764286 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173649 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.235749 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200463 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.241067 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.239472 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.265056 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.333113 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.507169 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.49158 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.416847 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.437859 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.570033 s, 1.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.547889 s, 1.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff96b688d5b800.lock_unused_count=3 ldlm.namespaces.lustre-OST0001-osc-ffff96b688d5b800.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0324563 s, 32.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136117 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0429474 s, 24.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0432539 s, 24.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0438768 s, 23.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0464879 s, 22.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0872976 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0535884 s, 19.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153534 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0638334 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0648109 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0751699 s, 13.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.076873 s, 13.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0794 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.086141 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0943457 s, 11.1 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg153-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg153-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' PASS 77e (29s) == sanityn test 77f: check TBF JobID nrs policy ========== 20:56:56 (1788397016) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start runas jobid={iozone.500 dd.500 tiotest.500} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start iozone_runas jobid={iozone.500} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg153-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.319235 s, 52.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.038387 s, 27.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0722929 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0888768 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105476 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0775698 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0703515 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115232 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0970744 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139355 s, 7.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff96b688d5b800.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222832 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190906 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23829 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198257 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.250693 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.27755 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.280218 s, 3.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0577797 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0666704 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0350381 s, 29.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0374709 s, 28.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0822338 s, 12.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0503084 s, 20.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0537199 s, 19.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0870384 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0648243 s, 16.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0838365 s, 12.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0845904 s, 12.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117586 s, 8.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0889935 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0971321 s, 10.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107742 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126189 s, 8.3 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg153-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg153-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change runas rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change iozone_runas rate=101 ost.OSS.ost_io.nrs_tbf_rule=change dd_runas rate=51 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg153-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.800555 s, 21.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18174 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145095 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151474 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0917483 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138818 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.239433 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158703 s, 6.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166206 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0902567 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.360655 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.403765 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.432878 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242992 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.384246 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.465347 s, 2.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff96b688d5b800.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffff96b688d5b800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.454456 s, 2.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0615692 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0511339 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0726422 s, 14.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0460899 s, 22.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124838 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0671871 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0567163 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0707349 s, 14.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0780547 s, 13.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0916524 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.072924 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122104 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10331 s, 10.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132207 s, 7.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138499 s, 7.6 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff96b688d5b800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146913 s, 7.1 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg153-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg153-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop runas ost.OSS.ost_io.nrs_tbf_rule=stop iozone_runas ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg153-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.43468 s, 38.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0884454 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0848141 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14899 s, 7.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213678 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.259815 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.286217 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.252375 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.306797 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.325557 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.33688 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.33856 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.414143 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.35443 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.373577 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.35118 s, 3.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff96b688d5b800.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffff96b688d5b800.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.478294 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166434 s, 63.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00413368 s, 254 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00242542 s, 432 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00238061 s, 440 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118374 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154477 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0976704 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118722 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198791 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173959 s, 60.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0957296 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.080717 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0697196 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0622371 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0513966 s, 20.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0570318 s, 18.4 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg153-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg153-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_policies=fifo running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg153-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.727076 s, 23.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0790776 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0871623 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0627597 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12502 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.214129 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206007 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242842 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.320776 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.238927 s, 4.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff96b688d5b800.lock_unused_count=5 ldlm.namespaces.lustre-OST0001-osc-ffff96b688d5b800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.317165 s, 3.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.32634 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.27449 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.404678 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.372654 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.472066 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.437304 s, 2.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff96b688d5b800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff96b688d5b800.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00514702 s, 204 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116712 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00243331 s, 431 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167501 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0864054 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016119 s, 65.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.075819 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0502453 s, 20.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155445 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0478863 s, 21.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0498507 s, 21.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0454141 s, 23.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.068704 s, 15.3 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0409812 s, 25.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0641224 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0397066 s, 26.4 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg153-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg153-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' PASS 77f (26s) == sanityn test 77g: Change TBF type directly ============ 20:57:22 (1788397042) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 ost.OSS.ost_io.nrs_policies=fifo PASS 77g (12s) == sanityn test 77h: Wrong policy name should report error, not LBUG ========================================================== 20:57:35 (1788397055) oleg153-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=abc: No such file or directory oleg153-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='abc': No such file or directory pdsh@oleg153-client: oleg153-server: ssh exited with exit code 2 oleg153-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc: Invalid argument oleg153-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc': Invalid argument pdsh@oleg153-client: oleg153-server: ssh exited with exit code 22 oleg153-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf reg abc: Invalid argument oleg153-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf reg abc': Invalid argument pdsh@oleg153-client: oleg153-server: ssh exited with exit code 22 oleg153-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc efg: Invalid argument oleg153-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc efg': Invalid argument pdsh@oleg153-client: oleg153-server: ssh exited with exit code 22 PASS 77h (13s) == sanityn test 77i: Change rank of TBF rule ============= 20:57:48 (1788397068) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start before jobid={jobid} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start after jobid={jobid} rate=1000 rank=default ost.OSS.ost_io.nrs_tbf_rule=start target jobid={jobid} rate=1000 rank=after Move before itself ost.OSS.ost_io.nrs_tbf_rule=change target rank=target Move to higher rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=before Move to lower rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=after Move before default ost.OSS.ost_io.nrs_tbf_rule=change target rank=default ost.OSS.ost_io.nrs_policies=fifo PASS 77i (26s) == sanityn test 77j: check TBF-OPCode NRS policy ========= 20:58:14 (1788397094) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf opcode ost.OSS.ost_io.nrs_tbf_rule=start ost_r opcode={ost_read} rate=5 ost.OSS.ost_io.nrs_tbf_rule=start ost_w opcode={ost_write} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.79506 s, 21.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0950214 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123044 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.241103 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163165 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144698 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146743 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.322231 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197891 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.269702 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.303927 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.341907 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.418487 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.258713 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.393327 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.392294 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.358427 s, 2.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff96b688d5b800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff96b688d5b800.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111459 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138942 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112255 s, 9.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130494 s, 8.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137767 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.082213 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0751814 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157353 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108224 s, 9.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152487 s, 6.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13533 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132032 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168937 s, 6.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127224 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145749 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118217 s, 8.9 MB/s removed '/mnt/lustre/d77j.sanityn/nrs_w_oleg153-client.virtnet' removed '/mnt/lustre/d77j.sanityn/nrs_r_oleg153-client.virtnet' removed directory '/mnt/lustre/d77j.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 5 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.14193 s, 12.9 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.6438 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77j.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_r ost.OSS.ost_io.nrs_tbf_rule=stop ost_w ost.OSS.ost_io.nrs_policies=fifo PASS 77j (69s) == sanityn test 77ja: check TBF-UID/GID NRS policy ======= 20:59:23 (1788397163) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg153-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg153-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg153-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg153-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg153-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.29488 s, 13.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0516283 s, 20.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142974 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155827 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159465 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199911 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181567 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19035 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209076 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.287299 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.303192 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.289858 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.219102 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.353842 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.326131 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242358 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.368489 s, 2.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0953359 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119849 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.068997 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137799 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131351 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0890178 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0925917 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109971 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163507 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14955 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109199 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134136 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172216 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120735 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130988 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.246077 s, 4.3 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg153-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg153-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.7493 s, 5.1 MB/s Write runtime is 22 s, speed is 4.545454 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8074 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg153-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg153-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg153-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg153-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg153-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.29419 s, 13.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0871955 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107953 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0897243 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121156 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147437 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166215 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205909 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199533 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.26255 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.278678 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185536 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.275417 s, 3.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.239175 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.321872 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.262236 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.390961 s, 2.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.050701 s, 20.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0244933 s, 42.8 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0226292 s, 46.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0287473 s, 36.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0645869 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0779471 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0439042 s, 23.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0395355 s, 26.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0385114 s, 27.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0413595 s, 25.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0517134 s, 20.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0537846 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0591267 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0703765 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0704728 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0894827 s, 11.7 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg153-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg153-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.0777 s, 5.2 MB/s Write runtime is 22 s, speed is 4.545454 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9497 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77ja (155s) == sanityn test 77jb: check TBF-UID/GID NRS policy on files that don't belong to us ========================================================== 21:01:58 (1788397318) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg153-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg153-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg153-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg153-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg153-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.30798 s, 12.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151406 s, 6.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161618 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138258 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205789 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.26581 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.262195 s, 4.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.300053 s, 3.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.303374 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.301091 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.278349 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.314801 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.349823 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.30432 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.429445 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.409624 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.492919 s, 2.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166877 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160408 s, 6.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160474 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179025 s, 5.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141046 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18503 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.204457 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147071 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207883 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185357 s, 5.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189743 s, 5.5 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21154 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18049 s, 5.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.229274 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.231667 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.228733 s, 4.6 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg153-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg153-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.3226 s, 5.2 MB/s Write runtime is 22 s, speed is 4.545454 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.0065 s, 5.2 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg153-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg153-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg153-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg153-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg153-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.29136 s, 13.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112323 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.090775 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112831 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174091 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172647 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187647 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.218139 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22436 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199155 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.263939 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.258186 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23539 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.292997 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.291981 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.288814 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.310889 s, 3.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0739576 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0807736 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.083125 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0748686 s, 14.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100246 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100421 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0545944 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0576483 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0940048 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0860867 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121592 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0901446 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0963633 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0705712 s, 14.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106041 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0844796 s, 12.4 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg153-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg153-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.4107 s, 5.1 MB/s Write runtime is 22 s, speed is 4.545454 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8167 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77jb (154s) == sanityn test 77k: check TBF policy with UID/GID/JobID/OPCode expression ========================================================== 21:04:32 (1788397472) ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg153-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.815159 s, 20.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112952 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0999856 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20207 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.244829 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.31716 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.351285 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.346301 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.300759 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.340345 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.3884 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.343386 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.375622 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.38035 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.39272 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.387749 s, 2.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied ldlm.namespaces.lustre-OST0000-osc-ffff96b688d5b800.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff96b688d5b800.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.479679 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147814 s, 70.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151361 s, 69.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123128 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171197 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115255 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162379 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199337 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139471 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0753321 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152373 s, 6.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.079835 s, 13.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0746991 s, 14.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00248205 s, 422 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0709385 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0647062 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0555645 s, 18.9 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg153-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg153-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.78522 s, 13.5 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.9693 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.201.53@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.201.53@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.422452 s, 39.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0790285 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.086094 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162451 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190795 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227422 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237707 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203106 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.271713 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.286849 s, 3.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.29149 s, 3.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.282532 s, 3.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.294475 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.254596 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.301539 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.277409 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.295857 s, 3.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0698768 s, 15.0 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109189 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0252288 s, 41.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116266 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105556 s, 9.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0951416 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0520891 s, 20.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.087107 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113581 s, 9.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0438929 s, 23.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110194 s, 9.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0723871 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0551664 s, 19.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0575823 s, 18.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111011 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.068482 s, 15.3 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg153-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg153-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.28343 s, 12.7 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.91686 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.201.53@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg153-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.581841 s, 28.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0719117 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0838796 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162765 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20267 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19669 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202608 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.254418 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.234516 s, 4.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.226054 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.235227 s, 4.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208374 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.28966 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.290113 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.266018 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.317814 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.28674 s, 3.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff96b688d5b800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff96b688d5b800.lock_unused_count=1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0861097 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0364097 s, 28.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112111 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0981614 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0757055 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.099519 s, 10.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0909069 s, 11.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0630242 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13176 s, 8.0 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137892 s, 7.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157594 s, 6.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136199 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118302 s, 8.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141415 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15089 s, 6.9 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg153-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg153-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.61533 s, 12.2 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.38016 s, 19.5 MB/s Read runtime is 7 s, speed is 14.285714 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg153-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.610005 s, 27.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0618347 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0782099 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137323 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206728 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222365 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16281 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0585417 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113197 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.234238 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.224865 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158674 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.285128 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201499 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0752442 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.29725 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.258606 s, 4.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0283487 s, 37.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0714084 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0826442 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107262 s, 9.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0491192 s, 21.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0533027 s, 19.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0534998 s, 19.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0397096 s, 26.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0637505 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109616 s, 9.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0730103 s, 14.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114027 s, 9.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0721067 s, 14.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0846537 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0919942 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112799 s, 9.3 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg153-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg153-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.3155 s, 9.3 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.90547 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.11058 s, 14.7 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.91803 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg153-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.06858 s, 15.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119509 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113507 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.273516 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.28502 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.343779 s, 3.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.245855 s, 4.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.350085 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.297019 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.259281 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.313375 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.289494 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.341013 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.412093 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.37332 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.369794 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.378832 s, 2.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.080874 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0293442 s, 35.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0449208 s, 23.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0341984 s, 30.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0979445 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0675819 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0499776 s, 21.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127809 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0520469 s, 20.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0739712 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0598322 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0634419 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0629517 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0733641 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0719967 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0778132 s, 13.5 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg153-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg153-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9413 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.7061 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_ug ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg153-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.472022 s, 35.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0372587 s, 28.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0342319 s, 30.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0893033 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173442 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18771 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22054 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186921 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200662 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.26078 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.278102 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.301083 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.279688 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.257615 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.255348 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.268869 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.373254 s, 2.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff96b688d5b800.lock_unused_count=3 ldlm.namespaces.lustre-OST0001-osc-ffff96b688d5b800.lock_unused_count=2 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0824925 s, 12.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0431344 s, 24.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0265215 s, 39.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0794239 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0291939 s, 35.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0327955 s, 32.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0899401 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0845259 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114207 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.061397 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11044 s, 9.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108781 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0608119 s, 17.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0620709 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.076296 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0864772 s, 12.1 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg153-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg153-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.70696 s, 13.6 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0196 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg153-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.880346 s, 19.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0473524 s, 22.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134501 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174123 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170703 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189122 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143431 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.327316 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.291915 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.244121 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.293151 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.284202 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.307616 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.428599 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.369949 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.405945 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.514032 s, 2.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013846 s, 75.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106584 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0586118 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164499 s, 63.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0450014 s, 23.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0122504 s, 85.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0842931 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0795502 s, 13.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0769562 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140641 s, 7.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112502 s, 9.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105225 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148201 s, 70.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0355972 s, 29.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161844 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00830406 s, 126 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg153-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg153-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.1736 s, 9.4 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.89739 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.6504 s, 9.0 MB/s Write runtime is 13 s, speed is 7.692307 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.90972 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_policies=fifo PASS 77k (446s) == sanityn test 77kb: Check different granular TBF type combination: jobid+opcode ========================================================== 21:11:58 (1788397918) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/nrs_r_oleg153-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.464638 s, 36.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0786126 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105075 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102971 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149884 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18784 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197688 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193964 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.243332 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.2835 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.323957 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.272382 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.323901 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.282515 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.327946 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.362252 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.336652 s, 3.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0658386 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0687546 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00585045 s, 179 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0630946 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0461567 s, 22.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0660228 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.034398 s, 30.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00257071 s, 408 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0776128 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0116966 s, 89.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.06152 s, 17.0 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119497 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0200444 s, 52.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.075483 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00892715 s, 117 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0348499 s, 30.1 MB/s removed '/mnt/lustre/d77kb.sanityn/nrs_w_oleg153-client.virtnet' removed '/mnt/lustre/d77kb.sanityn/nrs_r_oleg153-client.virtnet' removed directory '/mnt/lustre/d77kb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.19742 s, 11.4 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.97039 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kb (52s) == sanityn test 77kc: Check different granular TBF type combination: nid+opcode ========================================================== 21:12:51 (1788397971) ost.OSS.ost_io.nrs_policies=tbf nid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.201.53@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.201.53@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.537978 s, 31.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0651601 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101201 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199489 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207221 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138186 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.232462 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.226249 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.235809 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.269261 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.224544 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.294607 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.29448 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.299786 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.233126 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.272144 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.306138 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0692114 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00996643 s, 105 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0639739 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0112062 s, 93.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104681 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115221 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0954857 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136622 s, 76.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0228201 s, 45.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143677 s, 73.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0118446 s, 88.5 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00236783 s, 443 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00245678 s, 427 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10345 s, 10.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0250381 s, 41.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0297686 s, 35.2 MB/s removed '/mnt/lustre/d77kc.sanityn/nrs_r_oleg153-client.virtnet' removed '/mnt/lustre/d77kc.sanityn/nrs_w_oleg153-client.virtnet' removed directory '/mnt/lustre/d77kc.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.94488 s, 11.7 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.043 s, 10.4 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kc.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kc (55s) == sanityn test 77kd: Check different granular TBF type combination: nid+jobid ========================================================== 21:13:45 (1788398025) ost.OSS.ost_io.nrs_policies=tbf nid+jobid ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.201.53@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/nrs_r_oleg153-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.573349 s, 29.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119289 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103857 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173267 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.256534 s, 4.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff96b688d5b800.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.323265 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.315139 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.229809 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.339156 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.363682 s, 2.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.412416 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.321103 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.348394 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.368813 s, 2.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.386246 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.398925 s, 2.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.404518 s, 2.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff96b688d5b800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff96b688d5b800.lock_unused_count=4 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0770665 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0325711 s, 32.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0339987 s, 30.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0425913 s, 24.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0643597 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0505403 s, 20.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0522978 s, 20.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0740198 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0542058 s, 19.3 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0999004 s, 10.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0853246 s, 12.3 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0647 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0646818 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0724021 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0746789 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0756525 s, 13.9 MB/s removed '/mnt/lustre/d77kd.sanityn/nrs_r_oleg153-client.virtnet' removed '/mnt/lustre/d77kd.sanityn/nrs_w_oleg153-client.virtnet' removed directory '/mnt/lustre/d77kd.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.93953 s, 11.7 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kd.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.46077 s, 19.2 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kd.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kd (47s) == sanityn test 77ke: Check different granular TBF type combination: jobid+opcode ========================================================== 21:14:32 (1788398072) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/nrs_r_oleg153-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.86625 s, 19.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0642405 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107777 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.095703 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.2044 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180515 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.293182 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.276159 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.320407 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.307576 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.319198 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.312864 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.250025 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.328597 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.316623 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.374834 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.388601 s, 2.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00251058 s, 418 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0025827 s, 406 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0429727 s, 24.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00626848 s, 167 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00540409 s, 194 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.029007 s, 36.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00726326 s, 144 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0253441 s, 41.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0289178 s, 36.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0962149 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00754524 s, 139 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0463226 s, 22.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0048165 s, 218 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00849653 s, 123 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0712752 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137656 s, 76.2 MB/s removed '/mnt/lustre/d77ke.sanityn/nrs_w_oleg153-client.virtnet' removed '/mnt/lustre/d77ke.sanityn/nrs_r_oleg153-client.virtnet' removed directory '/mnt/lustre/d77ke.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.4064 s, 9.2 MB/s Write runtime is 13 s, speed is 7.692307 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ke.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.99943 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.9781 s, 13.1 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.89612 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77ke (97s) == sanityn test 77kf: Check different granular TBF type combination: uid+gid ========================================================== 21:16:09 (1788398169) ost.OSS.ost_io.nrs_policies=tbf uid+gid ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/nrs_r_oleg153-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.06983 s, 15.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0497807 s, 21.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111376 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112279 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0877273 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192439 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158996 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124919 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.248323 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210979 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23554 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170376 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.313952 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221568 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223065 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.297598 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.35309 s, 3.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0413761 s, 25.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117267 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0196803 s, 53.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0484139 s, 21.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0523456 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108573 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0527673 s, 19.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142383 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0702885 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0111096 s, 94.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138341 s, 75.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0213636 s, 49.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00497627 s, 211 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00880977 s, 119 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00869617 s, 121 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00316228 s, 332 MB/s removed '/mnt/lustre/d77kf.sanityn/nrs_r_oleg153-client.virtnet' removed '/mnt/lustre/d77kf.sanityn/nrs_w_oleg153-client.virtnet' removed directory '/mnt/lustre/d77kf.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.7793 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kf.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9825 s, 5.2 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77kf.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kf (77s) == sanityn test 77kg: check different granular TBF type combination: uid+opcode ========================================================== 21:17:26 (1788398246) ost.OSS.ost_io.nrs_policies=tbf uid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg153-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.563553 s, 29.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0615905 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.067956 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15585 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199947 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.229243 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242674 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184896 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.247278 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.226125 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.281963 s, 3.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.252882 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.273909 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.273766 s, 3.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.263539 s, 4.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.325495 s, 3.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff96b688d5b800.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffff96b688d5b800.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.374049 s, 2.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0689969 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.03204 s, 32.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0712355 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103701 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0623359 s, 16.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0531001 s, 19.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0568761 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0601287 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.060702 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0621016 s, 16.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0648654 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0683467 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0709489 s, 14.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0716074 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0719535 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0802364 s, 13.1 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg153-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg153-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.06657 s, 13.0 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.91109 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg153-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.348897 s, 48.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.040991 s, 25.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0757213 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100805 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144099 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181556 s, 5.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167808 s, 6.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199862 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.224888 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158381 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191408 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.287788 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.270288 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208776 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.230072 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.29161 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.277768 s, 3.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128875 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0332051 s, 31.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0585138 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143165 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142787 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158252 s, 6.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139561 s, 7.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153266 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151726 s, 6.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154165 s, 6.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191822 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168157 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106394 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16778 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11324 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213528 s, 4.9 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg153-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg153-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.8715 s, 9.6 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.95028 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.9604 s, 9.6 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.93672 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kg (139s) == sanityn test 77kh: Verify that Project ID support for NRS TBF rule ========================================================== 21:19:45 (1788398385) using SAVE_PROJECT_SUPPORTED=0 Stopping clients: oleg153-client.virtnet /mnt/lustre (opts:) Stopping client oleg153-client.virtnet /mnt/lustre opts: Stopping clients: oleg153-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg153-client.virtnet /mnt/lustre2 opts: Stopping /mnt/lustre-mds1 (opts:-f) on oleg153-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg153-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg153-server sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime) proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) devtmpfs on /dev type devtmpfs (rw,nosuid,mode=755) securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime) tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev) devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000) tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755) tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755) cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd) pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime) bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700) cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids) cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset) cgroup on /sys/fs/cgroup/misc type cgroup (rw,nosuid,nodev,noexec,relatime,misc) cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb) cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio) cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct) cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma) cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer) cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices) cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event) cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio) cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory) none on /sys/kernel/tracing type tracefs (rw,relatime) configfs on /sys/kernel/config type configfs (rw,relatime) /dev/nbd0 on / type ext4 (ro,relatime,stripe=32) rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) debugfs on /sys/kernel/debug type debugfs (rw,relatime) mqueue on /dev/mqueue type mqueue (rw,relatime) hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M) none on /mnt type ramfs (rw,relatime) /dev/vda on /home/green/git/lustre-release type squashfs (ro,relatime) none on /var/lib/stateless/writable type tmpfs (rw,relatime) none on /var/cache/man type tmpfs (rw,relatime) none on /var/log type tmpfs (rw,relatime) none on /var/lib/dbus type tmpfs (rw,relatime) none on /tmp type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) none on /var/tmp type tmpfs (rw,relatime) none on /var/lib/NetworkManager type tmpfs (rw,relatime) none on /var/lib/systemd/random-seed type tmpfs (rw,relatime) none on /var/spool type tmpfs (rw,relatime) none on /var/lib/nfs type tmpfs (rw,relatime) none on /var/lib/gssproxy type tmpfs (rw,relatime) none on /var/lib/logrotate type tmpfs (rw,relatime) none on /etc type tmpfs (rw,relatime) none on /var/lib/rsyslog type tmpfs (rw,relatime) none on /root/.ssh type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) 192.168.200.253:/exports/state/oleg153-client.virtnet on /var/lib/stateless/state type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.201.53,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg153-client.virtnet/boot on /boot type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.201.53,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg153-client.virtnet/etc/kdump.conf on /etc/kdump.conf type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.201.53,local_lock=none,addr=192.168.200.253) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=378180k,mode=700) 192.168.200.253:/exports/testreports/69399/testresults/sanityn-zfs-rocky8.10_x86_64-rocky8.10_x86_64-retry1 on /tmp/testlogs type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.201.53,local_lock=none,addr=192.168.200.253) /dev/vda on /usr/sbin/mount.lustre type squashfs (ro,relatime) Checking servers environments Checking clients oleg153-client.virtnet environments Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg153-server' oleg153-server: oleg153-server.virtnet: executing load_modules_local oleg153-server: kmemleak disabled oleg153-server: Loading modules from /home/green/git/lustre-release/lustre oleg153-server: detected 4 online CPUs by sysfs oleg153-server: MODOPTS_LIBCFS= oleg153-server: Force libcfs to create 2 CPU partitions oleg153-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Setup mgs, mdt, osts Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg153-server: oleg153-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg153-client: oleg153-server: ssh exited with exit code 1 Started lustre-MDT0000 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg153-server: oleg153-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg153-client: oleg153-server: ssh exited with exit code 1 Started lustre-OST0000 Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg153-server: oleg153-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg153-client: oleg153-server: ssh exited with exit code 1 Started lustre-OST0001 Starting client: oleg153-client.virtnet: -o user_xattr,flock 192.168.201.153@tcp:/lustre /mnt/lustre Starting client oleg153-client.virtnet: -o user_xattr,flock 192.168.201.153@tcp:/lustre /mnt/lustre Started clients oleg153-client.virtnet: 192.168.201.153@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Starting client: oleg153-client.virtnet: -o user_xattr,flock 192.168.201.153@tcp:/lustre /mnt/lustre2 Starting client oleg153-client.virtnet: -o user_xattr,flock 192.168.201.153@tcp:/lustre /mnt/lustre2 Started clients oleg153-client.virtnet: 192.168.201.153@tcp:/lustre on /mnt/lustre2 type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff96b686658800.idle_timeout=debug osc.lustre-OST0000-osc-ffff96b69ab2b000.idle_timeout=debug osc.lustre-OST0001-osc-ffff96b686658800.idle_timeout=debug osc.lustre-OST0001-osc-ffff96b69ab2b000.idle_timeout=debug disable quota as required ost.OSS.ost_io.nrs_policies=tbf projid ost.OSS.ost_io.nrs_tbf_rule=start ext_proj_rw projid={100} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.396538 s, 42.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15723 s, 6.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12251 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0891355 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137691 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142124 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.244516 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237025 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.291766 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.250631 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.25554 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.306403 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.293638 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.38935 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.277132 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.334539 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.283516 s, 3.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff96b69ab2b000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0233562 s, 44.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0595038 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0581241 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0309819 s, 33.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0345269 s, 30.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0345388 s, 30.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0367366 s, 28.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0441096 s, 23.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0804339 s, 13.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0762563 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0694487 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0527536 s, 19.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0784676 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0859486 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0659233 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0810306 s, 12.9 MB/s removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg153-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg153-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.09332 s, 14.8 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.23773 s, 20.0 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_proj_rw ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf projid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w projid={100}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r projid={100}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.750974 s, 22.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0547254 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107645 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106207 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176364 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185577 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.247255 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205886 s, 5.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.377472 s, 2.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.400225 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.357484 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.30867 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.337491 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.477921 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.458014 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.38816 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.377146 s, 2.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff96b69ab2b000.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffff96b69ab2b000.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190212 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1641 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132104 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175074 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13362 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136506 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140431 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129533 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205963 s, 5.1 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222568 s, 4.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149642 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150193 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145354 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156411 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158545 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187445 s, 5.6 MB/s removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg153-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg153-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.07707 s, 13.0 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.89149 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77kh (200s) == sanityn test 77ki: Add rule with unsupported TBF type should fail for generic TBF ========================================================== 21:23:05 (1788398585) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 oleg153-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_fail projid={100}&opcode={ost_write} rate=20: Invalid argument oleg153-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_fail projid={100}&opcode={ost_write} rate=20': Invalid argument pdsh@oleg153-client: oleg153-server: ssh exited with exit code 22 ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77ki (18s) == sanityn test 77kj: Verify nodemap support for NRS TBF rule ========================================================== 21:23:24 (1788398604) On MGS 192.168.201.153, active = nodemap.active=1 On MGS 192.168.201.153, TBF = ost.OSS.ost_io.nrs_policies=tbf nodemap ost.OSS.ost_io.nrs_tbf_rule=start ext_nm_rw nodemap={TBF} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.469091 s, 35.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0735004 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0751242 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147807 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171384 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172054 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206444 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.204621 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.263017 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.230757 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.229946 s, 4.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.236386 s, 4.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.272273 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.287752 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.274143 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.3013 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.281898 s, 3.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff96b69ab2b000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff96b69ab2b000.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0232454 s, 45.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0238893 s, 43.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0202888 s, 51.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0272299 s, 38.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.028717 s, 36.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0809719 s, 12.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0355241 s, 29.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0349366 s, 30.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0959793 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0358986 s, 29.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0419654 s, 25.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0463421 s, 22.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0515068 s, 20.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0564197 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0554439 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0585171 s, 17.9 MB/s removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg153-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg153-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.35797 s, 14.3 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.44004 s, 19.3 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete On MGS 192.168.201.153, TBF.id = On MGS 192.168.201.153, active = nodemap.active=0 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.632572 s, 26.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0804225 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.088941 s, 11.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167318 s, 6.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175156 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125164 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141033 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.268946 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155931 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242331 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139822 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197836 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.231617 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.269097 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.234507 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.286873 s, 3.7 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff96b69ab2b000.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128168 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0677011 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108869 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0986008 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111885 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0694379 s, 15.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0665101 s, 15.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135588 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.063031 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132078 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0921984 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10706 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0873564 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153844 s, 6.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149191 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188101 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130811 s, 8.0 MB/s removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg153-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg153-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 1 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.14291 s, 17.1 MB/s Write runtime is 7 s, speed is 14.285714 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 2.49577 s, 42.0 MB/s Read runtime is 4 s, speed is 25.000000 IOPS read rate (25.000000) should exceed limit (20 * 1) cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_nm_rw ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf nodemap+opcode oleg153-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_w nodemap={TBF}&opcode={ost_write} rate=20: Invalid argument oleg153-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_w nodemap={TBF}&opcode={ost_write} rate=20': Invalid argument oleg153-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_r nodemap={TBF}&opcode={ost_read} rate=10: Invalid argument oleg153-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_r nodemap={TBF}&opcode={ost_read} rate=10': Invalid argument pdsh@oleg153-client: oleg153-server: ssh exited with exit code 22 On MGS 192.168.201.153, active = nodemap.active=1 On MGS 192.168.201.153, TBF = ost.OSS.ost_io.nrs_tbf_rule=start ext_w nodemap={TBF}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nodemap={TBF}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.493236 s, 34.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0596499 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0752165 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0807613 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158892 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175398 s, 6.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212093 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201124 s, 5.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206278 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202998 s, 5.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227588 s, 4.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.234554 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.25137 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.235564 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.261689 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.278152 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.234001 s, 4.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff96b69ab2b000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0944122 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00850162 s, 123 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00921068 s, 114 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00520069 s, 202 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0516468 s, 20.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0107826 s, 97.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.010783 s, 97.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0276502 s, 37.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0284971 s, 36.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0195916 s, 53.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0225985 s, 46.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13305 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0825325 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0618254 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00852514 s, 123 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00770906 s, 136 MB/s removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg153-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg153-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.1772 s, 10.3 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0375 s, 10.4 MB/s Read runtime is 12 s, speed is 8.333333 IOPS cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo On MGS 192.168.201.153, TBF.id = On MGS 192.168.201.153, active = nodemap.active=0 PASS 77kj (179s) == sanityn test 77l: check the output of NRS policies for generic TBF ========================================================== 21:26:24 (1788398784) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_policies=fifo PASS 77l (9s) == sanityn test 77m: check NRS Delay slows write RPC processing ========================================================== 21:26:33 (1788398793) ost.OSS.ost_io.nrs_policies=delay ost.OSS.ost_io.nrs_delay_min=4 ost.OSS.ost_io.nrs_delay_max=4 ost.OSS.ost_io.nrs_delay_pct=100 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg153-client.virtnet] [bs=1M] [count=1] [oflag=direct] [conv=fdatasync] 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 3.30009 s, 318 kB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg153-client.virtnet] [bs=1M] [count=10] [oflag=direct] [conv=fdatasync] 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 40.1349 s, 261 kB/s ost.OSS.ost_io.nrs_policies=fifo PASS 77m (58s) == sanityn test 77n: check wildcard support for TBF JobID NRS policy ========================================================== 21:27:31 (1788398851) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={*.500} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.445239 s, 37.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0628968 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0840749 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109289 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199202 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17418 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128622 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.229388 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148336 s, 7.1 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207197 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184485 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190499 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.24119 s, 4.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff96b69ab2b000.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffff96b69ab2b000.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193352 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208513 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201704 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.367141 s, 2.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff96b69ab2b000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102275 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0239301 s, 43.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00308143 s, 340 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0570115 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0401207 s, 26.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0771783 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.078718 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101277 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100243 s, 10.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0786105 s, 13.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138219 s, 75.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109252 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0924847 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127459 s, 82.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165346 s, 63.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00600562 s, 175 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg153-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg153-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77n.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.46647 s, 14.0 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77n.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.2179 s, 20.1 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.*} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.587404 s, 28.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0713095 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0953028 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0723342 s, 14.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0313641 s, 33.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129062 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110979 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153634 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0341178 s, 30.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.06073 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0373207 s, 28.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23794 s, 4.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.244897 s, 4.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.266076 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169873 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183503 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.230258 s, 4.6 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff96b69ab2b000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0780274 s, 13.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127978 s, 8.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124921 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0713816 s, 14.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0732966 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0661729 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0627421 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0667964 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0971419 s, 10.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0879507 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105828 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17199 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161139 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107894 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192219 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128737 s, 8.1 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg153-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg153-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.18897 s, 14.6 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.16125 s, 20.3 MB/s Read runtime is 7 s, speed is 14.285714 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_policies=fifo PASS 77n (90s) == sanityn test 77o: Changing rank should not panic ====== 21:29:01 (1788398941) mds.MDS.mdt.nrs_policies=tbf nid mds.MDS.mdt.nrs_tbf_rule=start name nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=start name1 nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=change name1 rank=name mds.MDS.mdt.nrs_tbf_rule=stop name mds.MDS.mdt.nrs_policies=fifo PASS 77o (16s) == sanityn test 77q: Parallel TBF rule definitions should not panic ========================================================== 21:29:17 (1788398957) mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_policies=fifo PASS 77q (123s) == sanityn test 77p: Check validity of rule names for TBF policies ========================================================== 21:31:20 (1788399080) mds.MDS.mdt.nrs_policies=tbf oleg153-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_overflo uid={500} rate=500: File name too long oleg153-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_overflo uid={500} rate=500': File name too long pdsh@oleg153-client: oleg153-server: ssh exited with exit code 36 oleg153-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start uid={500} rate=500: Invalid argument oleg153-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start uid={500} rate=500': Invalid argument pdsh@oleg153-client: oleg153-server: ssh exited with exit code 22 oleg153-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_empty: Invalid argument oleg153-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_empty': Invalid argument pdsh@oleg153-client: oleg153-server: ssh exited with exit code 22 oleg153-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p@spec uid={500} rate=500: Invalid argument oleg153-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p@spec uid={500} rate=500': Invalid argument pdsh@oleg153-client: oleg153-server: ssh exited with exit code 22 oleg153-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p.spec uid={500} rate=500: Invalid argument oleg153-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p.spec uid={500} rate=500': Invalid argument pdsh@oleg153-client: oleg153-server: ssh exited with exit code 22 oleg153-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p~spec uid={500} rate=500: Invalid argument oleg153-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p~spec uid={500} rate=500': Invalid argument pdsh@oleg153-client: oleg153-server: ssh exited with exit code 22 oleg153-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p#spec uid={500} rate=500: Invalid argument oleg153-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p#spec uid={500} rate=500': Invalid argument pdsh@oleg153-client: oleg153-server: ssh exited with exit code 22 oleg153-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p/spec uid={500} rate=500: Invalid argument oleg153-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p/spec uid={500} rate=500': Invalid argument pdsh@oleg153-client: oleg153-server: ssh exited with exit code 22 oleg153-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p^spec uid={500} rate=500: Invalid argument oleg153-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p^spec uid={500} rate=500': Invalid argument pdsh@oleg153-client: oleg153-server: ssh exited with exit code 22 oleg153-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p%spec uid={500} rate=500: Invalid argument oleg153-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p%spec uid={500} rate=500': Invalid argument pdsh@oleg153-client: oleg153-server: ssh exited with exit code 22 oleg153-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p*spec uid={500} rate=500: Invalid argument oleg153-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p*spec uid={500} rate=500': Invalid argument pdsh@oleg153-client: oleg153-server: ssh exited with exit code 22 oleg153-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p;spec uid={500} rate=500: Invalid argument oleg153-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p;spec uid={500} rate=500': Invalid argument pdsh@oleg153-client: oleg153-server: ssh exited with exit code 22 oleg153-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p,spec uid={500} rate=500: Invalid argument oleg153-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p,spec uid={500} rate=500': Invalid argument pdsh@oleg153-client: oleg153-server: ssh exited with exit code 22 oleg153-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p?spec uid={500} rate=500: Invalid argument oleg153-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p?spec uid={500} rate=500': Invalid argument pdsh@oleg153-client: oleg153-server: ssh exited with exit code 22 oleg153-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77pspec uid={500} rate=500: Invalid argument oleg153-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p>spec uid={500} rate=500': Invalid argument pdsh@oleg153-client: oleg153-server: ssh exited with exit code 22 oleg153-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p:spec uid={500} rate=500: Invalid argument oleg153-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p:spec uid={500} rate=500': Invalid argument pdsh@oleg153-client: oleg153-server: ssh exited with exit code 22 oleg153-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p+spec uid={500} rate=500: Invalid argument oleg153-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p+spec uid={500} rate=500': Invalid argument pdsh@oleg153-client: oleg153-server: ssh exited with exit code 22 oleg153-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p=spec uid={500} rate=500: Invalid argument oleg153-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p=spec uid={500} rate=500': Invalid argument pdsh@oleg153-client: oleg153-server: ssh exited with exit code 22 oleg153-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p)spec uid={500} rate=500: Invalid argument oleg153-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p)spec uid={500} rate=500': Invalid argument pdsh@oleg153-client: oleg153-server: ssh exited with exit code 22 oleg153-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p(spec uid={500} rate=500: Invalid argument oleg153-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p(spec uid={500} rate=500': Invalid argument pdsh@oleg153-client: oleg153-server: ssh exited with exit code 22 oleg153-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p{spec uid={500} rate=500: Invalid argument oleg153-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p{spec uid={500} rate=500': Invalid argument pdsh@oleg153-client: oleg153-server: ssh exited with exit code 22 oleg153-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p}spec uid={500} rate=500: Invalid argument oleg153-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p}spec uid={500} rate=500': Invalid argument pdsh@oleg153-client: oleg153-server: ssh exited with exit code 22 oleg153-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p|spec uid={500} rate=500: Invalid argument oleg153-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p|spec uid={500} rate=500': Invalid argument pdsh@oleg153-client: oleg153-server: ssh exited with exit code 22 oleg153-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p[spec uid={500} rate=500: Invalid argument oleg153-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p[spec uid={500} rate=500': Invalid argument pdsh@oleg153-client: oleg153-server: ssh exited with exit code 22 oleg153-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p]spec uid={500} rate=500: Invalid argument oleg153-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p]spec uid={500} rate=500': Invalid argument pdsh@oleg153-client: oleg153-server: ssh exited with exit code 22 oleg153-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p!spec uid={500} rate=500: Invalid argument oleg153-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p!spec uid={500} rate=500': Invalid argument pdsh@oleg153-client: oleg153-server: ssh exited with exit code 22 oleg153-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p&spec uid={500} rate=500: Invalid argument oleg153-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p&spec uid={500} rate=500': Invalid argument pdsh@oleg153-client: oleg153-server: ssh exited with exit code 22 oleg153-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p$spec uid={500} rate=500: Invalid argument oleg153-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p$spec uid={500} rate=500': Invalid argument pdsh@oleg153-client: oleg153-server: ssh exited with exit code 22 oleg153-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p`spec uid={500} rate=500: Invalid argument oleg153-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p`spec uid={500} rate=500': Invalid argument pdsh@oleg153-client: oleg153-server: ssh exited with exit code 22 oleg153-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p\spec uid={500} rate=500: Invalid argument oleg153-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p\spec uid={500} rate=500': Invalid argument pdsh@oleg153-client: oleg153-server: ssh exited with exit code 22 mds.MDS.mdt.nrs_policies=fifo PASS 77p (37s) == sanityn test 77r: Change type of tbf policy at run time ========================================================== 21:31:57 (1788399117) mds.MDS.mdt.nrs_policies=tbf uid mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf gid mds.MDS.mdt.nrs_tbf_rule=start md_rule gid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start md_rule jobid={*.500} rate=1 mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=fifo check the number of requests in queue: fifo: 0 crrn: 0 tbf: 0 delay: 0 fifo: 0 crrn: 0 tbf: 0 delay: 0 cleanup 77r 273679 kill md_thread xargs PASS 77r (54s) == sanityn test 77s: Check TBF LRU shrinker ============== 21:32:51 (1788399171) jobid_name=%e.%p jobid_var=TEST_77s_JOBVAR mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start touch jobid={touch.*} rate=1000 mds.MDS.mdt.nrs_tbf_rule=start unlink jobid={unlink.*} rate=1000 touch jobid={touch.*} 1000, ref 9 touch jobid={touch.*} 1000, ref 0 unlink jobid={unlink.*} 1000, ref 9 unlink jobid={unlink.*} 1000, ref 0 mds.MDS.mdt.nrs_policies=fifo jobid_var=procname_uid jobid_name=%e.%u PASS 77s (62s) == sanityn test 78: Enable policy and specify tunings right away ========================================================== 21:33:53 (1788399233) ost.OSS.ost_io.nrs_orr_quantum=1 ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_policies=fifo PASS 78 (9s) == sanityn test 79: xattr: intent error ================== 21:34:02 (1788399242) File: /mnt/lustre/d79.sanityn Size: 16384 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115238826935627 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-02 21:34:04.000000000 -0400 Modify: 2026-09-02 21:34:04.000000000 -0400 Change: 2026-09-02 21:34:04.000000000 -0400 Birth: 2026-09-02 21:34:05.000000000 -0400 fail_val=0 fail_loc=0x80000160 fail_val=0 fail_loc=0x80000131 # file: mnt/lustre/d79.sanityn trusted.name1="value1" PASS 79 (18s) == sanityn test 80a: migrate directory when some children is being opened ========================================================== 21:34:20 (1788399260) SKIP: sanityn test_80a needs >= 2 MDTs SKIP 80a (5s) == sanityn test 80b: Accessing directory during migration ========================================================== 21:34:25 (1788399265) SKIP: sanityn test_80b needs >= 2 MDTs SKIP 80b (4s) == sanityn test 81a: rename and stat under striped directory ========================================================== 21:34:29 (1788399269) SKIP: sanityn test_81a needs >= 2 MDTs SKIP 81a (3s) == sanityn test 81b: rename under striped directory doesn't deadlock ========================================================== 21:34:33 (1788399273) SKIP: sanityn test_81b We need at least 2 MDTs for this test SKIP 81b (2s) == sanityn test 81c: rename revoke LOOKUP lock for remote object ========================================================== 21:34:35 (1788399275) SKIP: sanityn test_81c needs >= 4 MDTs SKIP 81c (4s) == sanityn test 81d: parallel rename file cross-dir on same MDT ========================================================== 21:34:39 (1788399279) - open/close 777 (time 1788399294.91 total 10.00 last 77.69) - open/close 1666 (time 1788399304.92 total 20.01 last 88.86) - open/close 2495 (time 1788399314.92 total 30.01 last 82.84) total: 2500 open/close in 30.05 seconds: 83.19 ops/second snapshot_time 1788399407.048580209 secs.nsecs start_time 1788399283.872152454 secs.nsecs elapsed_time 123.176427755 secs.nsecs open 2505 samples [usecs] 140 18092 3540042 8340971436 close 2505 samples [usecs] 193 9038 888990 638684088 mknod 2501 samples [usecs] 550 17998 3091192 6549181636 unlink 2500 samples [usecs] 665 91600 5335766 48901210180 mkdir 2 samples [usecs] 2319 6499 8818 47614762 rmdir 2 samples [usecs] 5890 8710 14600 110556200 rename 2500 samples [usecs] 502 80119 17216098 260979083970 getattr 3761 samples [usecs] 7 6967 269225 303444565 parallel_rename_file 2500 samples [usecs] 502 80119 17216098 260979083970 crossdir_rename 2500 samples [usecs] 502 80119 17216098 260979083970 readpage 17 samples [usecs] 6089 58400 358854 10903901572 PASS 81d (132s) == sanityn test 82: fsetxattr and fgetxattr on orphan files ========================================================== 21:36:52 (1788399412) multiop /mnt/lustre/f82.sanityn vO_ac TMPPIPE=/tmp/multiop_open_wait_pipe.7616 multiop /mnt/lustre2/f82.sanityn vo_Ac TMPPIPE=/tmp/multiop_open_wait_pipe.7616 PASS 82 (7s) == sanityn test 83: access striped directory while it is being created/unlinked ========================================================== 21:36:59 (1788399419) SKIP: sanityn test_83 needs >= 2 MDTs SKIP 83 (4s) == sanityn test 84: 0-nlink race in lu_object_find() ===== 21:37:03 (1788399423) fail_loc=0x8000060b PASS 84 (13s) == sanityn test 85: Lustre API root cache race =========== 21:37:17 (1788399437) Starting: llapi_root_test /mnt/lustre/f85.sanityn /mnt/lustre2/f85.sanityn Starting test test1 at 1788399438 Finishing test test1 at 1788399443 PASS 85 (12s) == sanityn test 90: open/create and unlink striped directory ========================================================== 21:37:28 (1788399448) SKIP: sanityn test_90 needs >= 2 MDTs SKIP 90 (3s) == sanityn test 91: chmod and unlink striped directory === 21:37:31 (1788399451) SKIP: sanityn test_91 needs >= 2 MDTs SKIP 91 (4s) == sanityn test 92: create remote directory under orphan directory ========================================================== 21:37:35 (1788399455) SKIP: sanityn test_92 needs >= 2 MDTs SKIP 92 (3s) == sanityn test 93: alloc_rr should not allocate on same ost ========================================================== 21:37:38 (1788399458) fail_loc=0x00000163 fail_loc=0x0 /mnt/lustre/f93.sanityn-1/file1 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5747 0x1673 0x240000400 1 5706 0x164a 0x280000400 /mnt/lustre2/f93.sanityn-2/file2 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 5707 0x164b 0x280000400 0 5748 0x1674 0x240000400 rc1=0 and rc2=0 PASS 93 (20s) == sanityn test 94: signal vs CP callback race =========== 21:37:58 (1788399478) osc.lustre-OST0000-osc-ffff96b686658800.idle_timeout=0 osc.lustre-OST0000-osc-ffff96b69ab2b000.idle_timeout=0 osc.lustre-OST0001-osc-ffff96b686658800.idle_timeout=0 osc.lustre-OST0001-osc-ffff96b69ab2b000.idle_timeout=0 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.436975 s, 18.7 kB/s fail_val=5 fail_loc=0x80000312 fail_val=6 fail_loc=0x80000329 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 3.05314 s, 1.3 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 6389: 290226 Alarm clock dd if=/dev/zero of=$DIR/$tfile conv=notrunc oflag=append bs=4k count=1 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0345878 s, 118 kB/s osc.lustre-OST0000-osc-ffff96b686658800.idle_timeout=debug osc.lustre-OST0000-osc-ffff96b69ab2b000.idle_timeout=debug osc.lustre-OST0001-osc-ffff96b686658800.idle_timeout=debug osc.lustre-OST0001-osc-ffff96b69ab2b000.idle_timeout=debug PASS 94 (16s) == sanityn test 95a: Check readpage() on a page that was removed from page cache ========================================================== 21:38:14 (1788399494) llite.lustre-ffff96b686658800.fast_read=0 llite.lustre-ffff96b69ab2b000.fast_read=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0687348 s, 30.5 MB/s fail_loc=0x80001421 fail_val=7 fail_loc=0x80001422 fail_val=10 llite.lustre-ffff96b686658800.fast_read=1 llite.lustre-ffff96b69ab2b000.fast_read=1 PASS 95a (22s) == sanityn test 95b: Check readpage() on a page that is no longer uptodate ========================================================== 21:38:36 (1788399516) llite.lustre-ffff96b686658800.fast_read=0 llite.lustre-ffff96b69ab2b000.fast_read=0 1+0 records in 1+0 records out 12288 bytes (12 kB, 12 KiB) copied, 0.00535354 s, 2.3 MB/s fail_loc=0x80001424 fail_val=5 fail_loc=0 llite.lustre-ffff96b686658800.fast_read=1 llite.lustre-ffff96b69ab2b000.fast_read=1 PASS 95b (13s) == sanityn test 100a: DoM: glimpse RPCs for stat without IO lock (DoM only file) ========================================================== 21:38:49 (1788399529) SKIP: sanityn test_100a Reserved for glimpse-ahead SKIP 100a (4s) == sanityn test 100b: DoM: no glimpse RPC for stat with IO lock (DoM only file) ========================================================== 21:38:53 (1788399533) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0053316 s, 768 kB/s /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK PASS 100b (8s) == sanityn test 100c: DoM: write vs stat without IO lock (combined file) ========================================================== 21:39:01 (1788399541) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.252234 s, 8.3 MB/s /mnt/lustre/d100c.sanityn/dom has type file OK /mnt/lustre/d100c.sanityn/dom has size 2097152 OK PASS 100c (8s) == sanityn test 100d: DoM: write+truncate vs stat without IO lock (combined file) ========================================================== 21:39:09 (1788399549) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.130016 s, 16.1 MB/s /mnt/lustre/d100d.sanityn/dom has type file OK /mnt/lustre/d100d.sanityn/dom has size 4096 OK PASS 100d (7s) == sanityn test 100e: DoM: read on open and file size ==== 21:39:17 (1788399557) 1+0 records in 1+0 records out 12000 bytes (12 kB, 12 KiB) copied, 0.0248612 s, 483 kB/s PASS 100e (10s) == sanityn test 101a: Discard DoM data on unlink ========= 21:39:26 (1788399566) /mnt/lustre/f101a.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00760067 s, 539 kB/s PASS 101a (8s) == sanityn test 101b: Discard DoM data on rename ========= 21:39:34 (1788399574) /mnt/lustre/f101b.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00839827 s, 488 kB/s PASS 101b (8s) == sanityn test 101c: Discard DoM data on close-unlink === 21:39:42 (1788399582) /mnt/lustre/f101c.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0194259 s, 211 kB/s PASS 101c (8s) == sanityn test 102: Test open by handle of unlinked file ========================================================== 21:39:50 (1788399590) file: /mnt/lustre/f102.sanityn fh_bytes: 32 fh_type: 151 fh_data: d1 0b 00 00 02 00 00 00 2c 0f 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 file's parent FID is [0:0x0:0x0] file FID is [0x200000bd1:0xf2c:0x0] access via mount point '/mnt/lustre2' - OK access after unlink - OK check_fhandle_syscalls test Passed! PASS 102 (8s) == sanityn test 103: Test size correctness with lockahead ========================================================== 21:39:58 (1788399598) fail_loc=0x415 fail_loc=0x415 fail_val=2 Incorrect size expected (no glimpse fix): Starting test test23 at 1788399601 lockahead_test: lockahead_test.c:1171: test23: assertion 'sb.st_size == sb2.st_size' failed: size on /mnt/lustre/d103.sanityn/f103.sanityn and /mnt/lustre2/d103.sanityn/f103.sanityn differs: 0 vs 1048576 fail_loc=0 fail_loc=0x214 fail_val=2 Starting test test23 at 1788399604 Finishing test test23 at 1788399604 PASS 103 (14s) == sanityn test 104: Verify that MDS stores atime/mtime/ctime during close ========================================================== 21:40:12 (1788399612) SKIP: sanityn test_104 ldiskfs only test SKIP 104 (4s) == sanityn test 105: Glimpse and lock cancel race ======== 21:40:17 (1788399617) fail_loc=0x416 PASS 105 (29s) == sanityn test 106a: Verify the btime via statx() ======= 21:40:45 (1788399645) SKIP: sanityn test_106a Test only for ldiskfs and statx() supported SKIP 106a (3s) == sanityn test 106b: Glimpse RPCs test for statx ======== 21:40:48 (1788399648) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00931401 s, 110 kB/s File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238826938163 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-02 21:40:50.000000000 -0400 Modify: 2026-09-02 21:40:50.000000000 -0400 Change: 2026-09-02 21:40:50.000000000 -0400 Birth: 2026-09-02 21:40:51.000000000 -0400 /mnt/lustre/f106b.sanityn 144115238826938163 -rw-r--r-- 1 0 0 1788399651 1788399650 1788399650 File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238826938163 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-02 21:40:50.000000000 -0400 Modify: 2026-09-02 21:40:50.000000000 -0400 Change: 2026-09-02 21:40:50.000000000 -0400 Birth: 2026-09-02 21:40:51.000000000 -0400 1788399650 1024 1 PASS 106b (7s) == sanityn test 106c: Verify statx attributes mask ======= 21:40:55 (1788399655) PASS 106c (7s) == sanityn test 107a: Basic grouplock conflict =========== 21:41:02 (1788399662) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.347266 s, 30.2 MB/s /mnt/lustre/f107a.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 5714 0x1652 0x280000400 multiop /mnt/lustre/f107a.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7616 multiop /mnt/lustre2/f107a.sanityn vO_G16022000r10g16022000c TMPPIPE=/tmp/multiop_open_wait_pipe.7616 First grouplock blocks second one PASS 107a (9s) == sanityn test 107b: Grouplock is added to the head of waiting list ========================================================== 21:41:12 (1788399672) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.296089 s, 35.4 MB/s /mnt/lustre/f107b.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5757 0x167d 0x240000400 multiop /mnt/lustre/f107b.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7616 Grouplock blocks IO First grouplock blocks second one Second grouplock blocks IO PASS 107b (14s) == sanityn test 108a: lseek: parallel updates ============ 21:41:25 (1788399685) 1+0 records in 1+0 records out 10000 bytes (10 kB, 9.8 KiB) copied, 0.00846526 s, 1.2 MB/s llite.lustre-ffff96b686658800.hybrid_io=0 llite.lustre-ffff96b69ab2b000.hybrid_io=0 fail_val=4 fail_loc=0x80000414 1+0 records in 1+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 4.63127 s, 1.8 MB/s PASS 108a (12s) == sanityn test 109: Race with several mount instances on 1 node ========================================================== 21:41:37 (1788399697) 192.168.201.153@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg153-client.virtnet /mnt/lustre (opts:) 192.168.201.153@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg153-client.virtnet /mnt/lustre2 (opts:) Starting race between client mounts (duration 30) Iteration 0 Starting client: oleg153-client.virtnet: -o user_xattr,flock 192.168.201.153@tcp:/lustre /mnt/lustre Starting client: oleg153-client.virtnet: -o user_xattr,flock 192.168.201.153@tcp:/lustre /mnt/lustre2 192.168.201.153@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.153@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg153-client.virtnet /mnt/lustre (opts:) Stopping client oleg153-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg153-server' oleg153-server: oleg153-server.virtnet: executing load_modules_local oleg153-server: kmemleak disabled oleg153-server: Loading modules from /home/green/git/lustre-release/lustre oleg153-server: detected 4 online CPUs by sysfs oleg153-server: MODOPTS_LIBCFS= oleg153-server: Force libcfs to create 2 CPU partitions oleg153-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Starting client: oleg153-client.virtnet: -o user_xattr,flock 192.168.201.153@tcp:/lustre /mnt/lustre Starting client: oleg153-client.virtnet: -o user_xattr,flock 192.168.201.153@tcp:/lustre /mnt/lustre2 PASS 109 (43s) == sanityn test 110: do not grant another lock on resend ========================================================== 21:42:20 (1788399740) fail_loc=0x525 fail_val=3 fail_loc=0x534 File: /mnt/lustre/f110.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238860488707 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-02 21:42:22.000000000 -0400 Modify: 2026-09-02 21:42:22.000000000 -0400 Change: 2026-09-02 21:42:22.000000000 -0400 Birth: 2026-09-02 21:42:23.000000000 -0400 fail_loc=0 fail_val=0 Connected clients: oleg153-client.virtnet oleg153-client.virtnet PASS 110 (171s) == sanityn test 111: A racy rename/link an open file should not cause fs corruption ========================================================== 21:45:12 (1788399912) SKIP: sanityn test_111 needs >= 2 MDTs SKIP 111 (3s) == sanityn test 112: update max-inherit in default LMV === 21:45:15 (1788399915) SKIP: sanityn test_112 We need at least 2 MDTs for this test SKIP 112 (4s) == sanityn test 113: check servers of specified fs ======= 21:45:19 (1788399919) lustre-OST0000-osc-ffff96b685d96000 active. lustre-OST0001-osc-ffff96b685d96000 active. lustre-OST0000-osc-ffff96b685d96000 active. lustre-OST0001-osc-ffff96b685d96000 active. MGC192.168.201.153@tcp active. PASS 113 (7s) == sanityn test 114: implicit default LMV inherit ======== 21:45:26 (1788399926) SKIP: sanityn test_114 We need at least 2 MDTs for this test SKIP 114 (4s) == sanityn test 115: ldiskfs doesn't check direntry for uniqueness ========================================================== 21:45:30 (1788399930) SKIP: sanityn test_115 ldiskfs only test SKIP 115 (4s) == sanityn test 116: DNE: Set default LMV layout from a remote client ========================================================== 21:45:34 (1788399934) SKIP: sanityn test_116 needs >= 2 MDTs SKIP 116 (4s) == sanityn test 121: trunc append race =================== 21:45:38 (1788399938) 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0435939 s, 470 kB/s fail_loc=0x80001436 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.068777 s, 298 kB/s 0000000 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0005000 0000 0000 0000 0000 0000 0000 0000 0000 * 000c800 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0011800 PASS 121 (9s) == sanityn test 122: directory size is consistent across mounts ========================================================== 21:45:47 (1788399947) PASS 122 (10s) == sanityn test 200: service remains healthy while able to process request ========================================================== 21:45:57 (1788399957) at_max=10 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=13 fail_loc=0x1e1 fail_val=20 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1256 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1554 s, 0.2 kB/s1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1087 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1358 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1276 s, 0.2 kB/s1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.2242 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1673 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1898 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.219 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.0508 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1659 s, 0.1 kB/s1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1118 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.2201 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1653 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1571 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.2944 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.2033 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.3612 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.1245 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.1541 s, 0.1 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.248 s, 0.1 kB/s4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2212 s, 0.1 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.1998 s, 0.1 kB/s 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2353 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.3604 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.4282 s, 0.1 kB/s at_max=600 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=116 oleg153-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff96b685d96000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff96b685d96000.ost_server_uuid in FULL state after 0 sec PASS 200 (82s) cleanup: ====================================================== == sanityn test complete, duration 9619 sec ============== 21:47:21 (1788400041) === sanityn: start cleanup 21:47:23 (1788400043) === Stopping clients: oleg153-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg153-client.virtnet /mnt/lustre2 opts: === sanityn: finish cleanup 21:51:23 (1788400283) ===