-----============= acceptance-small: sanityn ============----- Fri Jun 26 10:45:42 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg248-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanityn.*ex': No such file or directory excepting tests: 27 40a 102 skipping tests SLOW=no: 33a === sanityn: start setup 10:45:59 (1782485159) === Starting client oleg248-client.virtnet: -o user_xattr,flock 192.168.202.148@tcp:/lustre /mnt/lustre2 Mount client oleg248-client.virtnet: mount -t lustre -o user_xattr,flock 192.168.202.148@tcp:/lustre /mnt/lustre2 Started clients oleg248-client.virtnet: 192.168.202.148@tcp:/lustre on /mnt/lustre2 type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) oleg248-client.virtnet: executing check_config_client /mnt/lustre oleg248-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg248-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffffa00409991000.idle_timeout=debug osc.lustre-OST0000-osc-ffffa00411183000.idle_timeout=debug osc.lustre-OST0001-osc-ffffa00409991000.idle_timeout=debug osc.lustre-OST0001-osc-ffffa00411183000.idle_timeout=debug disable quota as required oleg248-server: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanityn: finish setup 10:46:28 (1782485188) === 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0120527 s, 87.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7587] == sanityn test 0: do_node_vp() and do_facet_vp() do the right thing ========================================================== 10:46:30 (1782485190) PASS 0 (6s) == sanityn test 1: Check attribute updates on 2 mount points ========================================================== 10:46:36 (1782485196) /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0777 OK /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0666 OK /mnt/lustre/f1.sanityn: absent OK PASS 1 (6s) == sanityn test 2a: check cached attribute updates on 2 mtpt's ================================================================== 10:46:42 (1782485202) -rw-r--r-- 1 root root 0 Jun 26 10:46 /mnt/lustre2/f2a /mnt/lustre/f2a has type file OK /mnt/lustre/f2a has perms 0777 OK PASS 2a (4s) == sanityn test 2b: check cached attribute updates on 2 mtpt's ================================================================== 10:46:47 (1782485207) -rw-r--r-- 1 root root 0 Jun 26 10:46 /mnt/lustre2/f2b /mnt/lustre2/f2b has type file OK /mnt/lustre2/f2b has perms 0777 OK PASS 2b (5s) == sanityn test 2c: check cached attribute updates on 2 mtpt's root ============================================================= 10:46:52 (1782485212) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0777 OK PASS 2c (5s) == sanityn test 2d: check cached attribute updates on 2 mtpt's root ============================================================= 10:46:56 (1782485216) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0755 OK PASS 2d (4s) == sanityn test 2e: check chmod on root is propagated to others ========================================================== 10:47:00 (1782485220) total 1 -rwxrwxrwx 1 root root 0 Jun 26 10:46 f2a -rwxrwxrwx 1 root root 0 Jun 26 10:46 f2b total 1 -rwxrwxrwx 1 root root 0 Jun 26 10:46 f2a -rwxrwxrwx 1 root root 0 Jun 26 10:46 f2b running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre2/f2e.sanityn] [count=1] 1+0 records in 1+0 records out 512 bytes copied, 0.010999 s, 46.5 kB/s PASS 2e (5s) == sanityn test 2f: check attr/owner updates on DNE with 2 mtpt's ========================================================== 10:47:05 (1782485225) SKIP: sanityn test_2f needs >= 2 MDTs SKIP 2f (3s) == sanityn test 2g: check blocks update on sync write ==== 10:47:08 (1782485228) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.283699 s, 7.4 MB/s /mnt/lustre/f2g.sanityn has 4103 blocks /mnt/lustre2/f2g.sanityn has 4103 blocks PASS 2g (5s) == sanityn test 3: symlink on one mtpt, readlink on another ===================================================================== 10:47:13 (1782485233) PASS 3 (5s) == sanityn test 4: fstat validation on multiple mount points ==================================================================== 10:47:18 (1782485238) PASS 4 (6s) == sanityn test 5: create a file on one mount, truncate it on the other ========================================================== 10:47:24 (1782485244) /mnt/lustre/f5 has type file OK /mnt/lustre/f5 has size 100 OK PASS 5 (6s) == sanityn test 6: remove of open file on other node ============================================================================ 10:47:30 (1782485250) opening writing unlinking /mnt/lustre2/f6.sanityn accessing (1) seeking (1) accessing (2) fstat... reading comparing data truncating seeking (2) writing again seeking (3) reading again comparing data again closing SUCCESS - goto beer PASS 6 (5s) == sanityn test 7: remove of open directory on other node ======================================================================= 10:47:35 (1782485255) creating directory /mnt/lustre/d7 opening directory unlinking /mnt/lustre/d7 Ok, everything goes well. PASS 7 (6s) == sanityn test 8: remove of open special file on other node ==================================================================== 10:47:42 (1782485262) creating special file /mnt/lustre/f8.sanityn opening file unlinking /mnt/lustre/f8.sanityn Ok, everything goes well. PASS 8 (6s) == sanityn test 9a: append of file with sub-page size on multiple mounts ========================================================== 10:47:48 (1782485268) PASS 9a (8s) == sanityn test 9b: append to striped sparse file ======== 10:47:56 (1782485276) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0718182 s, 14.6 MB/s 3+0 records in 3+0 records out 3 bytes copied, 0.0546915 s, 0.1 kB/s Data read (expecting 'foo'): foo PASS 9b (6s) == sanityn test 10a: write of file with sub-page size on multiple mounts ========================================================== 10:48:02 (1782485282) 1+0 records in 1+0 records out 1 byte copied, 0.015025 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00825072 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0128956 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0154784 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00883821 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0112015 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0102354 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0096651 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00938351 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0138801 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0097275 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0102626 s, 0.1 kB/s PASS 10a (8s) == sanityn test 10b: write of file with sub-page size on multiple mounts ========================================================== 10:48:10 (1782485290) 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.037742 s, 81.4 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0512043 s, 80.0 kB/s 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.000918125 s, 3.3 MB/s PASS 10b (6s) == sanityn test 11: execution of file opened for write should return error ============================================================== 10:48:17 (1782485297) multiop /mnt/lustre/d11/f vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7587 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 300: /mnt/lustre2/d11/f: Text file busy PASS 11 (6s) == sanityn test 12: test lock ordering (link, stat, unlink) ========================================================== 10:48:22 (1782485302) warning: '-runtime' deprecated, use '-t runtime' instead - link 383 (time 1782485314.74 total 10.04 last 38.13) - link 699 (time 1782485324.75 total 20.06 last 31.56) - link 998 (time 1782485334.76 total 30.07 last 29.87) - link 1332 (time 1782485344.79 total 40.09 last 33.31) - link 1655 (time 1782485354.80 total 50.11 last 32.25) - link 2030 (time 1782485364.81 total 60.12 last 37.47) - link 2412 (time 1782485374.86 total 70.16 last 38.03) - link 2802 (time 1782485384.86 total 80.17 last 38.97) - link 3157 (time 1782485394.90 total 90.20 last 35.38) total: 3503 link in 100.02 seconds: 35.02 ops/second using seed 3712265181 running for 100 seconds - stat 5815 (time 1782485316 ; total 11 ; last 11) - stat 10000 (time 1782485323 ; total 18 ; last 7) - stat 15373 (time 1782485334 ; total 29 ; last 11) - stat 20000 (time 1782485343 ; total 38 ; last 9) - stat 25231 (time 1782485354 ; total 49 ; last 11) - stat 28677 (time 1782485365 ; total 60 ; last 11) - stat 30000 (time 1782485369 ; total 64 ; last 4) - stat 33027 (time 1782485380 ; total 75 ; last 11) - stat 36023 (time 1782485391 ; total 86 ; last 11) - stat 38816 (time 1782485402 ; total 97 ; last 11) total: 39753 stats in 100 seconds: 397.529999 stats/second - unlinked 0 (time 1782485355 ; total 0 ; last 0) unlink(/mnt/lustre2/lockdir/lockfile3503) error: No such file or directory total: 3502 unlinks in 86 seconds: 40.720932 unlinks/second /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 77: kill: (21386) - No such process /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 78: kill: (21388) - No such process PASS 12 (144s) == sanityn test 13: test directory page revocation ======= 10:50:46 (1782485446) f13 f13 PASS 13 (7s) == sanityn test 14aa: execution of file open for write returns -ETXTBSY ========================================================== 10:50:53 (1782485453) multiop /mnt/lustre/d14aa.sanityn/f14aa.sanityn vOw_c TMPPIPE=/tmp/multiop_open_wait_pipe.7587 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 335: /mnt/lustre2/d14aa.sanityn/f14aa.sanityn: Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5019: 22571 Terminated $MULTIOP_PROG $FILE v$ARGS > $TMPPIPE (wd: ~) PASS 14aa (5s) == sanityn test 14ab: open(RDWR) of executing file returns -ETXTBSY ========================================================== 10:50:58 (1782485458) open(O_RDWR|O_CREAT): Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5019: 23150 Terminated $DIR1/$tdir/sleep 60 (wd: ~) PASS 14ab (5s) == sanityn test 14b: truncate of executing file returns -ETXTBSY ================================================================ 10:51:04 (1782485464) truncate: cannot truncate '/mnt/lustre2/d14b.sanityn/sleep' to length 60: Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 350: 23728 Terminated $DIR1/$tdir/sleep 60 PASS 14b (5s) == sanityn test 14c: open(O_TRUNC) of executing file return -ETXTBSY ============================================================ 10:51:09 (1782485469) cp: cannot create regular file '/mnt/lustre2/d14c.sanityn/sleep': Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 362: 24310 Terminated $DIR1/$tdir/sleep 60 PASS 14c (6s) == sanityn test 14d: chmod of executing file is still possible ================================================================== 10:51:15 (1782485475) chmod /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 373: 24891 Terminated $DIR1/$tdir/sleep 60 PASS 14d (7s) == sanityn test 15: test out-of-space with multiple writers ===================================================================== 10:51:22 (1782485482) PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin: MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky sleep 5 for ZFS MDS Waiting for MDT destroys to complete fallocate on zfs doesn't consume space STRIPECOUNT=2 ORIGFREE=7531520 MAXFREE=800000 SKIP: oos2 test_15 oos2.sh: 7531520KB free > 800000KB, increase MAXFREE (or reduce fs size) sleep 5 for ZFS MDS Waiting for MDT destroys to complete SKIP 15 (47s) == sanityn test 16a: 500 iterations of dual-mount fsx ==== 10:52:09 (1782485529) fallocate on zfs doesn't consume space fallocate not supported 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.325205 s, 25.8 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.221464 s, 37.9 MB/s Chance of close/open is 1 in 50 Seed set to 5532 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000001[0] 1782485533.090642 write 0x1a3301 thru 0x1b2501 (0xf201 bytes) 000002[1] 1782485533.139319 mapread 0x124619 thru 0x12481a (0x202 bytes) 000003[1] 1782485533.197399 read 0x04d762 thru 0x05a179 (0xca18 bytes) 000004[1] 1782485533.220673 write 0x51a944 thru 0x5231d9 (0x8896 bytes) truncating to largest ever: 0x963416 000008[0] 1782485533.259249 trunc from 0x5231da to 0x963415 (0x44023c bytes) 000009[1] 1782485533.364586 mapread 0x3ef479 thru 0x3fa286 (0xae0e bytes) 000015[0] 1782485533.515487 write 0x3ec58d thru 0x3f997b (0xd3ef bytes) 000016[1] 1782485533.561481 mapwrite 0x8f615e thru 0x8f940f (0x32b2 bytes) 000017[1] 1782485533.743932 read 0x4d3e0a thru 0x4e14ba (0xd6b1 bytes) 000019[0] 1782485533.823931 write 0x4d90e6 thru 0x4dc69a (0x35b5 bytes) 000020[1] 1782485533.888546 mapread 0x246064 thru 0x24b1d7 (0x5174 bytes) 000022[1] 1782485533.976154 mapread 0x4b15e0 thru 0x4baaeb (0x950c bytes) 000023[0] 1782485534.059389 write 0x8fbf22 thru 0x903f02 (0x7fe1 bytes) 000024[0] 1782485534.085739 write 0x907776 thru 0x90c4d3 (0x4d5e bytes) 000026[0] 1782485534.099246 mapread 0x8c7a08 thru 0x8cf5e5 (0x7bde bytes) 000028[0] 1782485534.135147 trunc from 0x963416 to 0x4bad2c (0x4a86e9 bytes) 000029[1] 1782485534.182429 mapwrite 0x7d4bbc thru 0x7da9aa (0x5def bytes) 000030[1] 1782485534.354565 write 0x6e37e9 thru 0x6e6a7c (0x3294 bytes) 000034[0] 1782485534.423599 mapwrite 0x8b053a thru 0x8b4207 (0x3cce bytes) 000037[1] 1782485534.577407 mapwrite 0x77a67c thru 0x786863 (0xc1e8 bytes) 000041[0] 1782485534.747771 write 0x10eefc thru 0x11a839 (0xb93e bytes) 000042[0] 1782485534.783245 mapwrite 0x3054ff thru 0x313001 (0xdb03 bytes) 000045[0] 1782485535.018576 trunc from 0x8b4208 to 0x90ae5e (0x56c57 bytes) 000049[0] 1782485535.237616 read 0x29d14d thru 0x2a60a7 (0x8f5b bytes) 000051[1] 1782485535.316174 mapread 0x20a1da thru 0x20f200 (0x5027 bytes) 000054[0] 1782485535.413177 trunc from 0x90ae5f to 0x41d255 (0x4edc09 bytes) 000055[1] 1782485535.513805 read 0x142512 thru 0x144690 (0x217f bytes) 000057[1] 1782485535.582920 mapread 0x1faf89 thru 0x1fb827 (0x89f bytes) 000058[0] 1782485535.668757 mapread 0x3c3333 thru 0x3c5154 (0x1e22 bytes) 000059[0] 1782485535.729733 mapread 0x0d0da7 thru 0x0d9632 (0x888c bytes) 000061[1] 1782485535.785669 mapread 0x3a51aa thru 0x3a9dd9 (0x4c30 bytes) 000062[0] 1782485535.849323 mapwrite 0x6cf829 thru 0x6d3c0e (0x43e6 bytes) 000063[1] 1782485536.019440 trunc from 0x6d3c0f to 0x1318cc (0x5a2342 bytes) 000065[1] 1782485536.100030 mapread 0x0fad1b thru 0x0fe763 (0x3a49 bytes) 000072[0] 1782485536.161647 read 0x045b46 thru 0x04d9c0 (0x7e7b bytes) 000074[1] 1782485536.232912 write 0x47877d thru 0x47b806 (0x308a bytes) 000076[1] 1782485536.256392 read 0x3c0256 thru 0x3c1f31 (0x1cdc bytes) 000079[1] 1782485536.275245 mapread 0x437ab2 thru 0x43c346 (0x4895 bytes) 000080[1] 1782485536.319075 trunc from 0x47b807 to 0x4b1c5b (0x36455 bytes) 000081[0] 1782485536.376705 read 0x281f31 thru 0x288285 (0x6355 bytes) 000083[0] 1782485536.459441 write 0x0e850e thru 0x0f3d78 (0xb86b bytes) 000087[1] 1782485536.592559 mapread 0x3e93b5 thru 0x3f2c43 (0x988f bytes) 000089[1] 1782485536.688807 write 0x8a6e51 thru 0x8b64f8 (0xf6a8 bytes) 000091[1] 1782485536.759657 read 0x86380e thru 0x86f022 (0xb815 bytes) 000092[1] 1782485536.784260 read 0x009e1b thru 0x00a157 (0x33d bytes) 000093[0] 1782485536.792273 read 0x3d711a thru 0x3da59d (0x3484 bytes) 000094[1] 1782485536.846169 mapread 0x1e7dc0 thru 0x1f0e14 (0x9055 bytes) 000095[0] 1782485536.887976 trunc from 0x8b64f9 to 0x338649 (0x57deaf bytes) 000096[0] 1782485536.936583 read 0x031f97 thru 0x03febc (0xdf26 bytes) 000097[1] 1782485537.016630 mapwrite 0x04631d thru 0x04e156 (0x7e3a bytes) 000098[0] 1782485537.137288 mapwrite 0x916b50 thru 0x922509 (0xb9ba bytes) 000104[1] 1782485537.385902 read 0x81038d thru 0x8163a3 (0x6017 bytes) 000106[1] 1782485537.451707 write 0x378714 thru 0x3791b7 (0xaa4 bytes) 000108[0] 1782485537.538249 read 0x1d2d58 thru 0x1dba92 (0x8d3b bytes) 000109[1] 1782485537.564687 read 0x1b3967 thru 0x1b433a (0x9d4 bytes) 000110[1] 1782485537.577770 read 0x5a7b8b thru 0x5af7fb (0x7c71 bytes) 000113[1] 1782485537.634744 trunc from 0x92250a to 0x4aa242 (0x4782c7 bytes) 000115[1] 1782485537.732799 read 0x0c2313 thru 0x0cf5a8 (0xd296 bytes) 000117[1] 1782485537.820967 write 0x0d39ea thru 0x0e22dc (0xe8f3 bytes) 000118[0] 1782485537.857085 write 0x4c0a04 thru 0x4c7cc2 (0x72bf bytes) 000119[0] 1782485537.911407 read 0x186e8d thru 0x18bde9 (0x4f5d bytes) 000120[1] 1782485538.041254 trunc from 0x4c7cc3 to 0x02e43d (0x499885 bytes) 000121[1] 1782485538.152339 trunc from 0x02e43e to 0x2dd3d4 (0x2aef97 bytes) 000123[1] 1782485538.223632 mapread 0x231f4b thru 0x239c82 (0x7d38 bytes) 000126[1] 1782485538.276397 read 0x10e7b9 thru 0x11ae87 (0xc6cf bytes) 000127[0] 1782485538.331388 mapread 0x0bd2e7 thru 0x0c5dfe (0x8b18 bytes) truncating to largest ever: 0x9faf62 000128[0] 1782485538.392469 trunc from 0x2dd3d5 to 0x9faf61 (0x71db8d bytes) 000129[0] 1782485538.512635 mapwrite 0x7b3ad6 thru 0x7b689b (0x2dc6 bytes) 000131[0] 1782485538.682655 write 0x0d51f7 thru 0x0d7d67 (0x2b71 bytes) 000132[0] 1782485538.693363 mapwrite 0x02344f thru 0x02f295 (0xbe47 bytes) 000133[0] 1782485538.796957 trunc from 0x9faf62 to 0x825d19 (0x1d5248 bytes) 000134[1] 1782485538.830586 trunc from 0x825d1a to 0x7ff01c (0x26cfd bytes) 000136[1] 1782485538.904576 trunc from 0x7ff01d to 0x0a2f72 (0x75c0aa bytes) 000139[1] 1782485538.953618 mapread 0x08b2ce thru 0x094bc3 (0x98f6 bytes) 000140[0] 1782485539.053716 mapread 0x044d75 thru 0x046ac8 (0x1d54 bytes) 000142[1] 1782485539.153599 mapwrite 0x0b2e33 thru 0x0bb980 (0x8b4e bytes) 000147[0] 1782485539.488853 read 0x0625bc thru 0x063964 (0x13a9 bytes) 000148[0] 1782485539.535377 write 0x374d27 thru 0x376873 (0x1b4d bytes) 000149[0] 1782485539.561641 write 0x2f94f0 thru 0x306062 (0xcb73 bytes) 000151[0] 1782485539.616993 trunc from 0x376874 to 0x382b60 (0xc2ed bytes) 000154[0] 1782485539.731424 mapwrite 0x458639 thru 0x461da0 (0x9768 bytes) 000159[1] 1782485539.850309 write 0x6c15a6 thru 0x6cec95 (0xd6f0 bytes) 000160[1] 1782485539.886540 trunc from 0x6cec96 to 0x485f22 (0x248d73 bytes) 000162[0] 1782485539.972798 mapwrite 0x6f14e8 thru 0x6f2a93 (0x15ac bytes) 000163[1] 1782485540.185260 mapread 0x281f6e thru 0x286daa (0x4e3d bytes) 000167[0] 1782485540.321270 mapwrite 0x1b48c2 thru 0x1bd1a2 (0x88e1 bytes) 000170[0] 1782485540.507474 mapwrite 0x0cdfb4 thru 0x0d0449 (0x2496 bytes) 000171[0] 1782485540.590942 write 0x14392c thru 0x14b8b8 (0x7f8d bytes) 000172[0] 1782485540.625359 mapwrite 0x015c5f thru 0x0202f2 (0xa694 bytes) 000178[1] 1782485540.858800 write 0x56b8a6 thru 0x57b523 (0xfc7e bytes) 000181[0] 1782485540.985682 mapread 0x317f3a thru 0x31852f (0x5f6 bytes) 000183[0] 1782485541.028676 mapread 0x2c4222 thru 0x2d1625 (0xd404 bytes) 000184[0] 1782485541.139190 mapwrite 0x97e895 thru 0x986fb9 (0x8725 bytes) 000185[0] 1782485541.300556 read 0x333f22 thru 0x33ecde (0xadbd bytes) 000188[1] 1782485541.380657 trunc from 0x986fba to 0x4494de (0x53dadb bytes) 000190[0] 1782485541.474754 mapwrite 0x259ea5 thru 0x267b3e (0xdc9a bytes) 000191[0] 1782485541.618181 mapread 0x042cee thru 0x04a77b (0x7a8e bytes) 000193[0] 1782485541.696725 read 0x3f0608 thru 0x3f6fc9 (0x69c2 bytes) 000194[0] 1782485541.735801 mapwrite 0x548c95 thru 0x557818 (0xeb84 bytes) 000196[0] 1782485542.073403 trunc from 0x557819 to 0x138571 (0x41f2a7 bytes) 000203[0] 1782485542.255225 mapread 0x058056 thru 0x06715d (0xf108 bytes) 000205[0] 1782485542.310113 mapread 0x0bf11c thru 0x0c8690 (0x9575 bytes) 000206[0] 1782485542.330504 read 0x0ea98b thru 0x0f36c7 (0x8d3d bytes) 000210[0] 1782485542.460456 mapread 0x0c2acf thru 0x0cd5bc (0xaaee bytes) 000212[0] 1782485542.491781 read 0x1042a3 thru 0x10b76f (0x74cd bytes) 000213[0] 1782485542.494707 mapread 0x08b16d thru 0x094d88 (0x9c1c bytes) 000217[0] 1782485542.531550 trunc from 0x138572 to 0x59be99 (0x463928 bytes) 000221[1] 1782485542.616330 trunc from 0x59be9a to 0x715c3e (0x179da5 bytes) 000222[0] 1782485542.682316 read 0x59efd1 thru 0x5ab049 (0xc079 bytes) 000223[1] 1782485542.778783 write 0x267899 thru 0x2742fb (0xca63 bytes) 000225[0] 1782485542.812631 read 0x3141e7 thru 0x321896 (0xd6b0 bytes) 000227[0] 1782485542.848779 read 0x6538e5 thru 0x65db3f (0xa25b bytes) 000229[1] 1782485542.857897 mapwrite 0x08c9b1 thru 0x096928 (0x9f78 bytes) 000231[1] 1782485543.010204 read 0x373f09 thru 0x37ea63 (0xab5b bytes) 000232[1] 1782485543.057184 mapwrite 0x7656ce thru 0x768fa0 (0x38d3 bytes) 000238[0] 1782485543.344225 write 0x74ee29 thru 0x7560cc (0x72a4 bytes) 000240[0] 1782485543.431073 mapwrite 0x6aa40f thru 0x6b6a05 (0xc5f7 bytes) 000242[1] 1782485543.658940 trunc from 0x768fa1 to 0x934970 (0x1cb9d0 bytes) 000244[1] 1782485543.741558 read 0x294062 thru 0x29af85 (0x6f24 bytes) 000246[1] 1782485543.754251 mapwrite 0x627330 thru 0x62ebb6 (0x7887 bytes) 000247[0] 1782485543.846032 read 0x262283 thru 0x264f22 (0x2ca0 bytes) 000248[0] 1782485543.876058 trunc from 0x934971 to 0x2575a1 (0x6dd3cf bytes) 000249[1] 1782485543.915403 read 0x1a1a95 thru 0x1afa1a (0xdf86 bytes) 000250[1] 1782485543.940571 mapread 0x06bf18 thru 0x07024c (0x4335 bytes) 000251[1] 1782485544.040514 mapread 0x0ec76d thru 0x0f768a (0xaf1e bytes) 000252[1] 1782485544.085712 mapread 0x1fa97a thru 0x1fd24d (0x28d4 bytes) 000254[1] 1782485544.134426 mapread 0x1b593e thru 0x1c056e (0xac31 bytes) 000258[0] 1782485544.189726 write 0x3a1930 thru 0x3a4a11 (0x30e2 bytes) 000259[1] 1782485544.253687 read 0x104644 thru 0x10de9f (0x985c bytes) 000260[1] 1782485544.370223 write 0x73c578 thru 0x73eced (0x2776 bytes) 000261[1] 1782485544.387215 trunc from 0x73ecee to 0x156814 (0x5e84d9 bytes) 000263[0] 1782485544.431309 trunc from 0x156815 to 0x337eab (0x1e1697 bytes) 000265[0] 1782485544.520655 read 0x01b2db thru 0x0291de (0xdf04 bytes) 000268[0] 1782485544.614241 mapread 0x238685 thru 0x243a69 (0xb3e5 bytes) 000269[0] 1782485544.615753 mapwrite 0x310955 thru 0x313102 (0x27ae bytes) 000272[1] 1782485544.719078 mapwrite 0x04a562 thru 0x059512 (0xefb1 bytes) 000274[0] 1782485544.923330 read 0x2234dd thru 0x230037 (0xcb5b bytes) 000277[0] 1782485545.028722 read 0x138d09 thru 0x13a9a7 (0x1c9f bytes) 000278[0] 1782485545.029434 write 0x047f5d thru 0x04be4f (0x3ef3 bytes) 000280[1] 1782485545.141375 mapwrite 0x549521 thru 0x55454d (0xb02d bytes) 000281[0] 1782485545.304426 trunc from 0x55454e to 0x3588ae (0x1fbc9f bytes) 000282[0] 1782485545.384376 trunc from 0x3588af to 0x742821 (0x3e9f73 bytes) 000283[0] 1782485545.438355 read 0x123ba7 thru 0x12ad49 (0x71a3 bytes) 000284[0] 1782485545.468964 trunc from 0x742822 to 0x9bcf4e (0x27a72d bytes) 000285[0] 1782485545.490374 mapread 0x5139c5 thru 0x5171bb (0x37f7 bytes) 000286[0] 1782485545.537339 mapwrite 0x761b50 thru 0x769a28 (0x7ed9 bytes) 000289[0] 1782485545.662716 write 0x6bcb5f thru 0x6c33e2 (0x6884 bytes) 000290[0] 1782485545.693743 read 0x5ce982 thru 0x5da979 (0xbff8 bytes) 000295[0] 1782485545.707635 read 0x6a3cde thru 0x6af953 (0xbc76 bytes) 000296[1] 1782485545.742127 trunc from 0x9bcf4f to 0x9a63dc (0x16b72 bytes) 000297[0] 1782485545.825103 write 0x4b8940 thru 0x4c0ba9 (0x826a bytes) 000298[0] 1782485545.866370 write 0x9032fb thru 0x90f1b2 (0xbeb8 bytes) 000301[1] 1782485545.935570 write 0x8662bf thru 0x86d351 (0x7093 bytes) 000303[1] 1782485545.992455 mapwrite 0x69587f thru 0x6a38c1 (0xe043 bytes) 000304[1] 1782485546.139722 trunc from 0x9a63dd to 0x034954 (0x971a88 bytes) 000305[1] 1782485546.156129 read 0x00c258 thru 0x016d41 (0xaaea bytes) 000306[1] 1782485546.162992 write 0x6ee13b thru 0x6f1fd2 (0x3e98 bytes) 000310[1] 1782485546.204587 trunc from 0x6f1fd3 to 0x57af4e (0x177084 bytes) 000313[0] 1782485546.255195 mapread 0x51c44f thru 0x5263b7 (0x9f69 bytes) 000316[0] 1782485546.346581 trunc from 0x57af4f to 0x12c37d (0x44ebd1 bytes) 000324[0] 1782485546.527133 write 0x93c0ca thru 0x943256 (0x718d bytes) 000325[1] 1782485546.531775 write 0x310216 thru 0x310eb0 (0xc9b bytes) 000327[1] 1782485546.665200 mapread 0x87338c thru 0x87fddb (0xca50 bytes) 000329[0] 1782485546.838778 mapread 0x06eefd thru 0x07eccf (0xfdd3 bytes) 000333[1] 1782485546.909449 mapwrite 0x8f647f thru 0x903ea6 (0xda28 bytes) 000335[0] 1782485547.081435 mapread 0x1d2d16 thru 0x1dd0c9 (0xa3b4 bytes) 000337[1] 1782485547.136733 trunc from 0x943257 to 0x118f7e (0x82a2d8 bytes) 000338[1] 1782485547.214274 write 0x42602a thru 0x432480 (0xc457 bytes) 000342[0] 1782485547.291717 mapwrite 0x279899 thru 0x288d62 (0xf4ca bytes) 000343[0] 1782485547.834450 write 0x616363 thru 0x61e5d7 (0x8275 bytes) 000344[1] 1782485547.894697 read 0x328144 thru 0x330599 (0x8456 bytes) 000347[0] 1782485547.970453 write 0x9baa3f thru 0x9c68af (0xbe71 bytes) 000348[1] 1782485547.987477 read 0x6d1205 thru 0x6dd5f1 (0xc3ed bytes) 000351[1] 1782485548.117630 write 0x673f4b thru 0x67afdc (0x7092 bytes) 000352[0] 1782485548.147666 mapwrite 0x132bed thru 0x1417bd (0xebd1 bytes) 000353[0] 1782485548.260732 read 0x763b4d thru 0x76cd26 (0x91da bytes) 000356[0] 1782485548.353745 mapread 0x389c6c thru 0x39700c (0xd3a1 bytes) 000362[1] 1782485548.537502 mapwrite 0x721cf7 thru 0x7298af (0x7bb9 bytes) 000364[0] 1782485549.021971 read 0x55bb4d thru 0x56a100 (0xe5b4 bytes) 000369[0] 1782485549.129497 trunc from 0x9c68b0 to 0x610a25 (0x3b5e8a bytes) 000370[0] 1782485549.201298 trunc from 0x610a26 to 0x928b4c (0x318127 bytes) 000371[1] 1782485549.229803 mapwrite 0x94b49a thru 0x958a4a (0xd5b1 bytes) 000372[0] 1782485549.391075 write 0x3605f0 thru 0x3606e3 (0xf4 bytes) 000373[0] 1782485549.496115 mapwrite 0x72c726 thru 0x72cbcd (0x4a8 bytes) 000377[1] 1782485549.744678 read 0x1b8f27 thru 0x1b96e4 (0x7be bytes) 000378[0] 1782485549.793534 read 0x76624d thru 0x76ad2a (0x4ade bytes) 000381[0] 1782485549.826712 mapread 0x60e450 thru 0x60ef07 (0xab8 bytes) 000383[0] 1782485549.851402 mapread 0x892a50 thru 0x897a67 (0x5018 bytes) 000385[1] 1782485549.890041 read 0x8e5c23 thru 0x8f3df3 (0xe1d1 bytes) 000386[0] 1782485549.898260 mapread 0x5131ee thru 0x521409 (0xe21c bytes) 000388[0] 1782485549.932404 read 0x809f83 thru 0x81492e (0xa9ac bytes) 000391[0] 1782485550.001507 read 0x88bc2a thru 0x898906 (0xccdd bytes) 000392[1] 1782485550.031608 write 0x0a7ed0 thru 0x0b3293 (0xb3c4 bytes) 000393[0] 1782485550.080546 mapwrite 0x265261 thru 0x26a6e4 (0x5484 bytes) 000394[0] 1782485550.186631 write 0x8b4eb1 thru 0x8b9919 (0x4a69 bytes) 000396[1] 1782485550.223161 write 0x028d80 thru 0x02cf33 (0x41b4 bytes) 000397[0] 1782485550.266724 mapwrite 0x934101 thru 0x943510 (0xf410 bytes) 000398[0] 1782485550.467104 mapwrite 0x05b7e8 thru 0x06a645 (0xee5e bytes) 000401[0] 1782485550.561532 mapread 0x93b934 thru 0x942eaf (0x757c bytes) 000404[0] 1782485550.590548 mapwrite 0x188f6a thru 0x196319 (0xd3b0 bytes) 000405[0] 1782485550.719683 write 0x5b4a1f thru 0x5bf69f (0xac81 bytes) 000408[1] 1782485550.779749 trunc from 0x958a4b to 0x791d2f (0x1c6d1b bytes) 000414[1] 1782485551.075305 mapread 0x0881bf thru 0x095ff2 (0xde34 bytes) 000415[1] 1782485551.159358 trunc from 0x791d30 to 0x8b7d58 (0x126029 bytes) 000416[0] 1782485551.230133 write 0x300bf6 thru 0x30a046 (0x9451 bytes) 000419[0] 1782485551.389104 mapread 0x7ff2f5 thru 0x8058b3 (0x65bf bytes) 000420[0] 1782485551.452076 mapwrite 0x210e82 thru 0x21d98e (0xcb0d bytes) 000422[0] 1782485551.744573 mapread 0x325a3c thru 0x3309f2 (0xafb7 bytes) 000423[1] 1782485551.829648 mapread 0x01196f thru 0x01d173 (0xb805 bytes) 000426[1] 1782485551.883400 read 0x791b8a thru 0x79eba7 (0xd01e bytes) 000428[0] 1782485551.895502 mapread 0x7a256b thru 0x7b0067 (0xdafd bytes) 000429[1] 1782485551.919384 read 0x0b5ab4 thru 0x0bb2bf (0x580c bytes) 000434[0] 1782485551.928550 mapwrite 0x6220d9 thru 0x624090 (0x1fb8 bytes) 000435[1] 1782485552.000426 write 0x9581b3 thru 0x966a17 (0xe865 bytes) 000436[1] 1782485552.039675 mapread 0x50e07e thru 0x512119 (0x409c bytes) 000438[0] 1782485552.116323 write 0x50073b thru 0x50de36 (0xd6fc bytes) 000440[1] 1782485552.172497 mapwrite 0x7a12bc thru 0x7acefb (0xbc40 bytes) 000441[0] 1782485552.304512 mapread 0x0aa4ec thru 0x0b2983 (0x8498 bytes) 000442[0] 1782485552.370360 trunc from 0x966a18 to 0x21c313 (0x74a704 bytes) 000443[1] 1782485552.419554 trunc from 0x21c314 to 0x508c09 (0x2ec8f6 bytes) 000444[0] 1782485552.518908 trunc from 0x508c0a to 0x41e837 (0xea3d2 bytes) 000445[0] 1782485552.591334 mapwrite 0x04fd10 thru 0x05d7f9 (0xdaea bytes) 000446[1] 1782485552.706569 write 0x9c80e3 thru 0x9d01f9 (0x8117 bytes) 000447[1] 1782485552.752293 mapwrite 0x546ff5 thru 0x550f3d (0x9f49 bytes) 000449[1] 1782485552.891218 mapread 0x597a3e thru 0x5a759b (0xfb5e bytes) 000451[1] 1782485552.972501 mapwrite 0x951273 thru 0x95572f (0x44bd bytes) 000452[0] 1782485553.064524 mapread 0x2fa78a thru 0x307a9e (0xd315 bytes) 000453[0] 1782485553.149957 read 0x1606de thru 0x16fd00 (0xf623 bytes) 000454[1] 1782485553.171379 read 0x74712b thru 0x749f25 (0x2dfb bytes) 000455[1] 1782485553.199166 trunc from 0x9d01fa to 0x729468 (0x2a6d91 bytes) 000456[0] 1782485553.270643 mapwrite 0x1b6268 thru 0x1c392c (0xd6c5 bytes) 000457[1] 1782485553.399734 mapread 0x6704a1 thru 0x67914c (0x8cac bytes) 000460[1] 1782485553.482946 mapwrite 0x723a90 thru 0x72840e (0x497f bytes) 000461[0] 1782485553.571318 mapread 0x23d972 thru 0x24104b (0x36da bytes) 000462[1] 1782485553.604762 mapwrite 0x15e631 thru 0x163a89 (0x5459 bytes) 000467[0] 1782485553.769208 write 0x32f867 thru 0x33b0eb (0xb885 bytes) 000468[0] 1782485553.804344 trunc from 0x729469 to 0x515ec6 (0x2135a2 bytes) 000470[1] 1782485553.895865 mapwrite 0x3c9321 thru 0x3cb49f (0x217f bytes) 000471[0] 1782485554.067505 write 0x9b31d2 thru 0x9c17ac (0xe5db bytes) 000474[0] 1782485554.128120 mapread 0x4dcd00 thru 0x4e4ce2 (0x7fe3 bytes) 000476[0] 1782485554.164534 mapwrite 0x336d95 thru 0x34042f (0x969b bytes) 000477[1] 1782485554.233105 trunc from 0x9c17ad to 0x0d5b07 (0x8ebca5 bytes) 000478[1] 1782485554.318219 trunc from 0x0d5b08 to 0x7fe612 (0x728b0b bytes) 000479[1] 1782485554.367581 write 0x30925c thru 0x3173fd (0xe1a2 bytes) 000480[0] 1782485554.395582 mapread 0x30ff6e thru 0x31d1bb (0xd24e bytes) 000481[0] 1782485554.451281 mapwrite 0x05991b thru 0x064646 (0xad2c bytes) 000484[1] 1782485554.616515 read 0x5568c2 thru 0x56396f (0xd0ae bytes) 000485[0] 1782485554.647265 trunc from 0x7fe613 to 0x0c8af7 (0x735b1b bytes) 000487[0] 1782485554.714269 mapwrite 0x3c7736 thru 0x3d055f (0x8e2a bytes) 000488[1] 1782485554.810502 mapwrite 0x094a40 thru 0x09d0b6 (0x8677 bytes) 000490[0] 1782485554.934676 mapwrite 0x05b060 thru 0x064095 (0x9036 bytes) 000494[1] 1782485555.068261 read 0x2e4e3f thru 0x2eb1fb (0x63bd bytes) 000497[1] 1782485555.168608 trunc from 0x3d0560 to 0x702de5 (0x332886 bytes) 000498[1] 1782485555.264769 trunc from 0x702de6 to 0x12b7af (0x5d7636 bytes) 000499[1] 1782485555.327428 trunc from 0x12b7b0 to 0x305986 (0x1da1d7 bytes) 000500[1] 1782485555.387996 read 0x11c5c4 thru 0x12baeb (0xf528 bytes) All operations completed A-OK! Chance of close/open is 1 in 50 Seed set to 5555 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read skipping zero size read skipping zero size read 000006[1] 1782485556.301613 write 0x1d3000 thru 0x1d8ea7 (0x5ea8 bytes) 000009[0] 1782485556.323139 read_OD 0x06d000 thru 0x075fff (0x9000 bytes) truncating to largest ever: 0x22066a 000011[0] 1782485556.393128 trunc from 0x1d8ea8 to 0x220669 (0x477c2 bytes) 000014[1] 1782485556.483432 read 0x066000 thru 0x06b6fc (0x56fd bytes) 000016[1] 1782485556.517735 mapwrite 0x6a3000 thru 0x6a3453 (0x454 bytes) 000018[1] 1782485556.648619 mapread 0x185000 thru 0x18a534 (0x5535 bytes) 000020[0] 1782485556.700809 mapwrite 0x474000 thru 0x47d4cb (0x94cc bytes) 000021[1] 1782485556.820516 mapread 0x663000 thru 0x664f07 (0x1f08 bytes) truncating to largest ever: 0x735725 000022[1] 1782485556.889727 trunc from 0x6a3454 to 0x735724 (0x922d1 bytes) 000026[1] 1782485556.980245 mapread 0x719000 thru 0x7208c5 (0x78c6 bytes) 000029[0] 1782485557.048519 mapread 0x489000 thru 0x48a58b (0x158c bytes) 000030[1] 1782485557.093120 read_OD 0x040000 thru 0x042fff (0x3000 bytes) 000031[1] 1782485557.098160 mapread 0x464000 thru 0x46e165 (0xa166 bytes) 000033[1] 1782485557.125550 trunc from 0x735725 to 0x42d53e (0x3081e6 bytes) 000034[0] 1782485557.169081 write_OD 0x670000 thru 0x670fff (0x1000 bytes) 000035[1] 1782485557.224966 mapwrite 0x49a000 thru 0x4a17b0 (0x77b1 bytes) 000036[1] 1782485557.332667 write_OD 0x093000 thru 0x09bfff (0x9000 bytes) 000040[0] 1782485557.392805 mapwrite 0x5af000 thru 0x5bbabb (0xcabc bytes) 000044[0] 1782485557.542506 read_OD 0x50e000 thru 0x519fff (0xc000 bytes) truncating to largest ever: 0x879824 000045[1] 1782485557.583897 trunc from 0x671000 to 0x879823 (0x208824 bytes) 000048[0] 1782485557.635593 read_OD 0x285000 thru 0x28ffff (0xb000 bytes) 000051[1] 1782485557.673304 mapwrite 0x009000 thru 0x00dae8 (0x4ae9 bytes) 000055[1] 1782485557.841273 read_OD 0x10a000 thru 0x114fff (0xb000 bytes) 000056[0] 1782485557.865662 mapwrite 0x2bf000 thru 0x2c313f (0x4140 bytes) 000057[1] 1782485557.959808 write_OD 0x03e000 thru 0x044fff (0x7000 bytes) 000059[1] 1782485558.029476 write_OD 0x397000 thru 0x39afff (0x4000 bytes) 000060[1] 1782485558.075700 write_OD 0x90f000 thru 0x91dfff (0xf000 bytes) 000061[0] 1782485558.124586 trunc from 0x91e000 to 0x4dcdbe (0x441241 bytes) 000062[1] 1782485558.170725 mapwrite 0x134000 thru 0x13e8dd (0xa8de bytes) 000063[0] 1782485558.290377 write_OD 0x3c9000 thru 0x3d6fff (0xe000 bytes) truncating to largest ever: 0x8a969b 000064[0] 1782485558.361011 trunc from 0x4dcdbf to 0x8a969a (0x3cc8dc bytes) 000068[1] 1782485558.457541 read_OD 0x80b000 thru 0x814fff (0xa000 bytes) 000069[0] 1782485558.487697 write_OD 0x025000 thru 0x02efff (0xa000 bytes) 000070[1] 1782485558.551271 write_OD 0x215000 thru 0x21efff (0xa000 bytes) 000071[1] 1782485558.641299 write_OD 0x080000 thru 0x086fff (0x7000 bytes) 000072[0] 1782485558.698641 write_OD 0x62f000 thru 0x62ffff (0x1000 bytes) 000073[0] 1782485558.753467 write_OD 0x475000 thru 0x47ffff (0xb000 bytes) 000076[0] 1782485558.819383 write_OD 0x499000 thru 0x4a5fff (0xd000 bytes) 000077[0] 1782485558.896602 mapread 0x558000 thru 0x55b5e0 (0x35e1 bytes) 000079[1] 1782485558.918536 mapread 0x56a000 thru 0x579b1b (0xfb1c bytes) 000080[0] 1782485558.956295 mapread 0x40c000 thru 0x40c316 (0x317 bytes) 000081[0] 1782485558.962530 mapwrite 0x837000 thru 0x84233c (0xb33d bytes) 000084[0] 1782485559.116601 trunc from 0x8a969b to 0x5623aa (0x3472f0 bytes) 000085[0] 1782485559.151495 trunc from 0x5623ab to 0x13ab26 (0x427884 bytes) 000086[0] 1782485559.171931 mapread 0x0c6000 thru 0x0cc52b (0x652c bytes) 000087[1] 1782485559.197476 trunc from 0x13ab27 to 0x13ce15 (0x22ef bytes) 000089[0] 1782485559.258141 mapwrite 0x7ad000 thru 0x7b9943 (0xc944 bytes) 000091[1] 1782485559.443879 mapwrite 0x669000 thru 0x6739f9 (0xa9fa bytes) 000092[0] 1782485559.578277 mapwrite 0x022000 thru 0x029bab (0x7bac bytes) 000094[1] 1782485559.665157 mapwrite 0x03c000 thru 0x044433 (0x8434 bytes) 000096[1] 1782485559.767785 read_OD 0x496000 thru 0x49bfff (0x6000 bytes) 000098[0] 1782485559.787515 read_OD 0x7b5000 thru 0x7b8fff (0x4000 bytes) 000101[1] 1782485559.797754 write_OD 0x45c000 thru 0x462fff (0x7000 bytes) 000102[1] 1782485559.852311 trunc from 0x7b9944 to 0x024ac4 (0x794e7f bytes) truncating to largest ever: 0x944976 000103[0] 1782485559.897695 trunc from 0x024ac5 to 0x944975 (0x91feb1 bytes) 000104[0] 1782485559.946266 mapread 0x295000 thru 0x297def (0x2df0 bytes) 000107[0] 1782485560.013869 mapread 0x1aa000 thru 0x1ae6f0 (0x46f1 bytes) 000108[1] 1782485560.029127 write_OD 0x03e000 thru 0x044fff (0x7000 bytes) 000109[1] 1782485560.068050 trunc from 0x944976 to 0x1d0f77 (0x7739fe bytes) 000110[0] 1782485560.099176 read_OD 0x0ef000 thru 0x0effff (0x1000 bytes) 000112[0] 1782485560.128683 mapread 0x101000 thru 0x10d205 (0xc206 bytes) 000113[0] 1782485560.161208 read_OD 0x00f000 thru 0x017fff (0x9000 bytes) 000115[0] 1782485560.172172 read_OD 0x078000 thru 0x083fff (0xc000 bytes) 000116[0] 1782485560.184331 mapread 0x19c000 thru 0x1a4b34 (0x8b35 bytes) 000118[1] 1782485560.185520 write_OD 0x404000 thru 0x410fff (0xd000 bytes) 000119[1] 1782485560.257003 write_OD 0x914000 thru 0x922fff (0xf000 bytes) 000120[1] 1782485560.318652 mapwrite 0x719000 thru 0x7245c8 (0xb5c9 bytes) 000121[1] 1782485560.483321 mapread 0x85b000 thru 0x863c0e (0x8c0f bytes) 000125[0] 1782485560.534524 mapwrite 0x618000 thru 0x61a9be (0x29bf bytes) 000135[0] 1782485560.885653 read_OD 0x0af000 thru 0x0bdfff (0xf000 bytes) 000141[0] 1782485560.939766 trunc from 0x923000 to 0x0b4901 (0x86e6fe bytes) 000142[1] 1782485560.975716 mapread 0x024000 thru 0x02da60 (0x9a61 bytes) 000143[0] 1782485561.032113 write_OD 0x1e5000 thru 0x1eafff (0x6000 bytes) 000144[1] 1782485561.096048 write_OD 0x0dc000 thru 0x0e1fff (0x6000 bytes) 000146[1] 1782485561.196383 write_OD 0x604000 thru 0x609fff (0x6000 bytes) 000149[1] 1782485561.281046 mapwrite 0x32d000 thru 0x32df11 (0xf12 bytes) 000150[0] 1782485561.377939 mapread 0x233000 thru 0x239ed7 (0x6ed8 bytes) 000152[1] 1782485561.451444 mapread 0x043000 thru 0x0479c2 (0x49c3 bytes) 000153[0] 1782485561.469974 mapread 0x04d000 thru 0x056df0 (0x9df1 bytes) 000154[1] 1782485561.491613 trunc from 0x60a000 to 0x28a5fb (0x37fa04 bytes) 000155[1] 1782485561.534232 write_OD 0x1c9000 thru 0x1d7fff (0xf000 bytes) 000156[1] 1782485561.581191 read_OD 0x0bd000 thru 0x0c9fff (0xd000 bytes) 000157[0] 1782485561.602057 mapread 0x21e000 thru 0x226cba (0x8cbb bytes) 000159[0] 1782485561.634132 mapread 0x022000 thru 0x022680 (0x681 bytes) 000161[1] 1782485561.639190 mapread 0x151000 thru 0x15d120 (0xc121 bytes) 000162[1] 1782485561.678562 mapwrite 0x851000 thru 0x8513c0 (0x3c1 bytes) 000163[1] 1782485561.834128 write_OD 0x87a000 thru 0x888fff (0xf000 bytes) 000164[1] 1782485561.917529 read_OD 0x46e000 thru 0x479fff (0xc000 bytes) 000165[0] 1782485561.932261 write_OD 0x5cf000 thru 0x5d0fff (0x2000 bytes) 000168[0] 1782485561.992689 mapwrite 0x25b000 thru 0x2604b7 (0x54b8 bytes) 000169[0] 1782485562.070494 mapwrite 0x869000 thru 0x869fcc (0xfcd bytes) 000170[1] 1782485562.124070 trunc from 0x889000 to 0x016f35 (0x8720ca bytes) 000172[1] 1782485562.181454 mapread 0x00b000 thru 0x010632 (0x5633 bytes) 000174[0] 1782485562.219251 read_OD 0x007000 thru 0x011fff (0xb000 bytes) 000175[0] 1782485562.250087 write_OD 0x5c8000 thru 0x5cdfff (0x6000 bytes) 000177[1] 1782485562.330185 mapread 0x4ff000 thru 0x50bb7b (0xcb7c bytes) 000179[1] 1782485562.358675 trunc from 0x5ce000 to 0x5cb7cf (0x2830 bytes) 000180[0] 1782485562.408997 write_OD 0x26b000 thru 0x272fff (0x8000 bytes) 000183[1] 1782485562.477513 mapread 0x214000 thru 0x217966 (0x3967 bytes) 000189[1] 1782485562.513634 trunc from 0x5cb7d0 to 0x54a604 (0x811cb bytes) 000190[1] 1782485562.550035 mapread 0x191000 thru 0x1986ce (0x76cf bytes) 000193[0] 1782485562.603581 trunc from 0x54a605 to 0x62a8e1 (0xe02dd bytes) 000194[1] 1782485562.654886 write_OD 0x26f000 thru 0x279fff (0xb000 bytes) 000199[1] 1782485562.735496 write_OD 0x12e000 thru 0x134fff (0x7000 bytes) 000200[0] 1782485562.790339 write_OD 0x6a7000 thru 0x6b3fff (0xd000 bytes) 000201[0] 1782485562.853532 read_OD 0x2c0000 thru 0x2c7fff (0x8000 bytes) truncating to largest ever: 0x979a52 000202[1] 1782485562.862220 trunc from 0x6b4000 to 0x979a51 (0x2c5a52 bytes) 000203[0] 1782485562.900514 mapwrite 0x99f000 thru 0x9a0da1 (0x1da2 bytes) 000204[0] 1782485562.973811 read_OD 0x959000 thru 0x95efff (0x6000 bytes) 000205[0] 1782485562.991618 trunc from 0x9a0da2 to 0x6f63a3 (0x2aa9fe bytes) 000207[1] 1782485563.025295 mapread 0x400000 thru 0x400fbb (0xfbc bytes) 000209[1] 1782485563.080001 read_OD 0x0dc000 thru 0x0e6fff (0xb000 bytes) 000212[1] 1782485563.098563 mapwrite 0x072000 thru 0x07f7ca (0xd7cb bytes) 000217[1] 1782485563.245507 write_OD 0x009000 thru 0x014fff (0xc000 bytes) 000218[0] 1782485563.295333 mapwrite 0x755000 thru 0x75656f (0x1570 bytes) 000223[1] 1782485563.441350 read_OD 0x04b000 thru 0x04bfff (0x1000 bytes) 000225[0] 1782485563.472324 read_OD 0x0a2000 thru 0x0b0fff (0xf000 bytes) 000226[0] 1782485563.482391 write_OD 0x190000 thru 0x19bfff (0xc000 bytes) 000227[0] 1782485563.546048 read_OD 0x248000 thru 0x256fff (0xf000 bytes) 000228[1] 1782485563.561732 read_OD 0x064000 thru 0x065fff (0x2000 bytes) 000229[0] 1782485563.568140 read_OD 0x2ea000 thru 0x2ebfff (0x2000 bytes) 000231[1] 1782485563.575366 mapwrite 0x4eb000 thru 0x4fa2d1 (0xf2d2 bytes) 000235[1] 1782485563.753606 mapwrite 0x418000 thru 0x4238ef (0xb8f0 bytes) 000236[1] 1782485563.927066 read_OD 0x752000 thru 0x754fff (0x3000 bytes) 000238[0] 1782485563.949314 write_OD 0x242000 thru 0x24afff (0x9000 bytes) 000239[0] 1782485563.995624 mapwrite 0x154000 thru 0x15fc21 (0xbc22 bytes) 000242[0] 1782485564.152564 mapread 0x629000 thru 0x6330c1 (0xa0c2 bytes) 000244[0] 1782485564.203909 read_OD 0x31b000 thru 0x31ffff (0x5000 bytes) 000246[1] 1782485564.232127 trunc from 0x756570 to 0x6fb20f (0x5b360 bytes) 000248[0] 1782485564.288758 write_OD 0x01f000 thru 0x02dfff (0xf000 bytes) 000249[0] 1782485564.361387 mapread 0x3e9000 thru 0x3ee339 (0x533a bytes) 000251[1] 1782485564.382452 read_OD 0x4d5000 thru 0x4e3fff (0xf000 bytes) 000255[1] 1782485564.389739 mapwrite 0x266000 thru 0x2722d8 (0xc2d9 bytes) 000261[0] 1782485564.525707 trunc from 0x6fb210 to 0x344563 (0x3b6cac bytes) 000262[1] 1782485564.573939 read_OD 0x301000 thru 0x30afff (0xa000 bytes) 000263[0] 1782485564.606544 read_OD 0x129000 thru 0x12ffff (0x7000 bytes) 000264[0] 1782485564.623508 mapread 0x19c000 thru 0x1a5423 (0x9424 bytes) 000268[0] 1782485564.687355 read_OD 0x06a000 thru 0x06afff (0x1000 bytes) 000271[1] 1782485564.693497 read_OD 0x04e000 thru 0x04ffff (0x2000 bytes) 000275[1] 1782485564.699891 read_OD 0x10e000 thru 0x11afff (0xd000 bytes) 000278[1] 1782485564.707898 mapread 0x2eb000 thru 0x2ed7ca (0x27cb bytes) 000279[1] 1782485564.755133 mapread 0x134000 thru 0x13c1c4 (0x81c5 bytes) 000280[0] 1782485564.759301 write_OD 0x0ca000 thru 0x0d2fff (0x9000 bytes) 000282[1] 1782485564.814360 write_OD 0x28d000 thru 0x29afff (0xe000 bytes) 000284[1] 1782485564.862963 mapwrite 0x49a000 thru 0x4a0811 (0x6812 bytes) 000286[0] 1782485564.954948 mapwrite 0x4d6000 thru 0x4db840 (0x5841 bytes) 000288[0] 1782485565.088070 trunc from 0x4db841 to 0x0fcbc3 (0x3dec7d bytes) 000290[1] 1782485565.129803 read_OD 0x0e3000 thru 0x0e3fff (0x1000 bytes) 000292[1] 1782485565.171265 read_OD 0x096000 thru 0x09cfff (0x7000 bytes) 000293[0] 1782485565.182060 mapwrite 0x466000 thru 0x46ad57 (0x4d58 bytes) 000294[0] 1782485565.301707 read_OD 0x2a4000 thru 0x2a7fff (0x4000 bytes) 000295[0] 1782485565.335518 mapread 0x21e000 thru 0x2206a0 (0x26a1 bytes) 000298[0] 1782485565.369316 read_OD 0x3f0000 thru 0x3fafff (0xb000 bytes) 000300[0] 1782485565.379723 write_OD 0x7c4000 thru 0x7c4fff (0x1000 bytes) 000303[1] 1782485565.467982 mapwrite 0x7fd000 thru 0x7fd940 (0x941 bytes) 000305[0] 1782485565.629371 read_OD 0x124000 thru 0x128fff (0x5000 bytes) 000309[0] 1782485565.671167 mapread 0x492000 thru 0x497a15 (0x5a16 bytes) 000311[0] 1782485565.695449 mapwrite 0x0b7000 thru 0x0c6c52 (0xfc53 bytes) 000313[0] 1782485565.815527 mapwrite 0x123000 thru 0x124e12 (0x1e13 bytes) 000314[0] 1782485565.903316 mapwrite 0x751000 thru 0x7512b7 (0x2b8 bytes) 000316[0] 1782485565.945496 trunc from 0x7fd941 to 0x197242 (0x6666fe bytes) 000320[0] 1782485565.970205 read_OD 0x152000 thru 0x15efff (0xd000 bytes) 000321[1] 1782485565.990210 mapwrite 0x089000 thru 0x0966b0 (0xd6b1 bytes) 000324[0] 1782485566.131479 mapread 0x188000 thru 0x1950d1 (0xd0d2 bytes) 000325[1] 1782485566.208600 read_OD 0x03e000 thru 0x048fff (0xb000 bytes) 000331[1] 1782485566.229285 mapread 0x062000 thru 0x0703e8 (0xe3e9 bytes) 000332[1] 1782485566.283687 mapread 0x16f000 thru 0x171116 (0x2117 bytes) 000333[0] 1782485566.289889 mapwrite 0x5c8000 thru 0x5d4a01 (0xca02 bytes) 000335[1] 1782485566.496668 mapread 0x0f1000 thru 0x0f49cc (0x39cd bytes) 000337[1] 1782485566.623304 mapwrite 0x5cc000 thru 0x5d9b7f (0xdb80 bytes) 000339[0] 1782485566.774666 write_OD 0x00c000 thru 0x012fff (0x7000 bytes) 000340[0] 1782485566.832683 read_OD 0x48a000 thru 0x492fff (0x9000 bytes) 000341[0] 1782485566.853549 mapread 0x4e4000 thru 0x4e6388 (0x2389 bytes) 000342[1] 1782485566.873577 trunc from 0x5d9b80 to 0x3b0699 (0x2294e6 bytes) 000349[0] 1782485566.965634 mapread 0x2dc000 thru 0x2e0148 (0x4149 bytes) 000350[0] 1782485567.006579 read_OD 0x38b000 thru 0x391fff (0x7000 bytes) 000353[1] 1782485567.014135 read_OD 0x14a000 thru 0x14efff (0x5000 bytes) 000354[0] 1782485567.027301 mapread 0x25b000 thru 0x25fc8d (0x4c8e bytes) 000358[1] 1782485567.082422 mapread 0x281000 thru 0x2828dd (0x18de bytes) 000359[0] 1782485567.149510 write_OD 0x24a000 thru 0x24cfff (0x3000 bytes) 000361[0] 1782485567.231101 read_OD 0x023000 thru 0x025fff (0x3000 bytes) 000362[1] 1782485567.235534 mapread 0x1a7000 thru 0x1b4169 (0xd16a bytes) 000364[0] 1782485567.310591 write_OD 0x112000 thru 0x118fff (0x7000 bytes) 000365[0] 1782485567.349975 mapread 0x2dc000 thru 0x2e9a62 (0xda63 bytes) 000366[1] 1782485567.406943 trunc from 0x3b069a to 0x353e36 (0x5c863 bytes) 000367[1] 1782485567.455646 mapwrite 0x008000 thru 0x00b746 (0x3747 bytes) 000368[0] 1782485567.535293 write_OD 0x71e000 thru 0x724fff (0x7000 bytes) 000369[1] 1782485567.587486 read_OD 0x6f2000 thru 0x6fcfff (0xb000 bytes) 000370[1] 1782485567.608152 write_OD 0x39d000 thru 0x39ffff (0x3000 bytes) 000371[1] 1782485567.709999 mapread 0x4d0000 thru 0x4df612 (0xf613 bytes) 000372[1] 1782485567.829102 trunc from 0x725000 to 0x58ac4f (0x19a3b0 bytes) 000374[1] 1782485567.912488 write_OD 0x0c4000 thru 0x0cdfff (0xa000 bytes) 000375[0] 1782485567.972586 mapwrite 0x062000 thru 0x06c5d8 (0xa5d9 bytes) 000380[0] 1782485568.123267 mapread 0x52e000 thru 0x537f52 (0x9f53 bytes) 000384[0] 1782485568.220892 mapwrite 0x96d000 thru 0x979695 (0xc696 bytes) 000386[0] 1782485568.379304 mapread 0x87e000 thru 0x8855a7 (0x75a8 bytes) 000388[1] 1782485568.386545 read_OD 0x4d9000 thru 0x4ddfff (0x5000 bytes) 000392[1] 1782485568.434285 read_OD 0x1d5000 thru 0x1dafff (0x6000 bytes) 000393[0] 1782485568.446094 trunc from 0x979696 to 0x49e007 (0x4db68e bytes) 000394[1] 1782485568.513438 mapwrite 0x43a000 thru 0x443857 (0x9858 bytes) 000395[0] 1782485568.686797 read_OD 0x375000 thru 0x37ffff (0xb000 bytes) 000396[1] 1782485568.724383 mapread 0x40c000 thru 0x41774a (0xb74b bytes) 000402[1] 1782485568.747529 mapwrite 0x97f000 thru 0x983fa2 (0x4fa3 bytes) 000403[1] 1782485568.855136 trunc from 0x983fa3 to 0x3c645b (0x5bdb47 bytes) 000404[1] 1782485568.876344 write_OD 0x144000 thru 0x144fff (0x1000 bytes) truncating to largest ever: 0x9ea85a 000405[1] 1782485568.928567 trunc from 0x3c645c to 0x9ea859 (0x6243fe bytes) 000407[0] 1782485568.979623 write_OD 0x6a1000 thru 0x6affff (0xf000 bytes) 000409[0] 1782485569.030294 mapwrite 0x558000 thru 0x5644f2 (0xc4f3 bytes) 000410[1] 1782485569.144503 mapwrite 0x4ba000 thru 0x4c06c8 (0x66c9 bytes) 000411[1] 1782485569.278125 mapread 0x5a0000 thru 0x5a427d (0x427e bytes) 000415[1] 1782485569.339535 write_OD 0x7d1000 thru 0x7dbfff (0xb000 bytes) 000418[1] 1782485569.392494 trunc from 0x9ea85a to 0x530b5e (0x4b9cfb bytes) 000420[0] 1782485569.450797 write_OD 0x8cd000 thru 0x8d9fff (0xd000 bytes) 000424[0] 1782485569.526213 write_OD 0x9ce000 thru 0x9d2fff (0x5000 bytes) 000427[0] 1782485569.620527 mapwrite 0x760000 thru 0x76a114 (0xa115 bytes) 000428[1] 1782485569.759308 write_OD 0x7f5000 thru 0x7fdfff (0x9000 bytes) 000429[1] 1782485569.831559 mapread 0x584000 thru 0x58a5f5 (0x65f6 bytes) 000435[0] 1782485569.851585 mapwrite 0x61e000 thru 0x621be4 (0x3be5 bytes) 000438[0] 1782485569.939295 mapread 0x7e9000 thru 0x7f8ef2 (0xfef3 bytes) 000439[1] 1782485569.988658 mapread 0x698000 thru 0x6a71dc (0xf1dd bytes) 000440[1] 1782485570.045259 write_OD 0x506000 thru 0x510fff (0xb000 bytes) 000441[0] 1782485570.130677 mapwrite 0x60a000 thru 0x60eb98 (0x4b99 bytes) 000442[1] 1782485570.219769 write_OD 0x54e000 thru 0x556fff (0x9000 bytes) 000443[1] 1782485570.288665 mapwrite 0x9fc000 thru 0x9ff987 (0x3988 bytes) 000444[1] 1782485570.377944 mapread 0x3a8000 thru 0x3b44cd (0xc4ce bytes) 000445[0] 1782485570.425750 mapread 0x1f6000 thru 0x205b8f (0xfb90 bytes) 000446[1] 1782485570.488634 read_OD 0x57e000 thru 0x582fff (0x5000 bytes) 000447[0] 1782485570.505021 trunc from 0x9ff988 to 0x424ebd (0x5daaca bytes) 000448[1] 1782485570.543709 write_OD 0x771000 thru 0x773fff (0x3000 bytes) 000449[0] 1782485570.600568 read_OD 0x76b000 thru 0x76efff (0x4000 bytes) 000450[0] 1782485570.607366 mapwrite 0x46b000 thru 0x47213f (0x7140 bytes) 000452[0] 1782485570.763030 mapwrite 0x417000 thru 0x41d8b8 (0x68b9 bytes) 000453[1] 1782485570.902883 mapread 0x3d5000 thru 0x3db5b3 (0x65b4 bytes) 000457[0] 1782485570.968159 mapwrite 0x803000 thru 0x811cce (0xeccf bytes) 000462[1] 1782485571.229942 trunc from 0x811ccf to 0x1b2e59 (0x65ee75 bytes) 000463[1] 1782485571.282217 mapread 0x03d000 thru 0x046b79 (0x9b7a bytes) 000464[0] 1782485571.307591 trunc from 0x1b2e5a to 0x8b3c81 (0x700e28 bytes) 000465[0] 1782485571.351371 trunc from 0x8b3c82 to 0x673bc5 (0x2400bc bytes) 000467[1] 1782485571.392208 trunc from 0x673bc6 to 0x835f30 (0x1c236b bytes) 000468[1] 1782485571.441480 trunc from 0x835f31 to 0x491446 (0x3a4aea bytes) 000469[1] 1782485571.457718 mapwrite 0x169000 thru 0x16bc6d (0x2c6e bytes) 000470[0] 1782485571.552417 read_OD 0x076000 thru 0x07efff (0x9000 bytes) 000471[1] 1782485571.573527 mapwrite 0x6f4000 thru 0x6fae89 (0x6e8a bytes) 000476[1] 1782485571.774620 read_OD 0x41a000 thru 0x41afff (0x1000 bytes) 000477[1] 1782485571.795931 read_OD 0x276000 thru 0x278fff (0x3000 bytes) 000481[1] 1782485571.814941 read_OD 0x0fe000 thru 0x100fff (0x3000 bytes) 000482[0] 1782485571.821346 trunc from 0x6fae8a to 0x84d391 (0x152508 bytes) 000483[1] 1782485571.889504 mapwrite 0x019000 thru 0x028e40 (0xfe41 bytes) 000487[1] 1782485572.040547 read_OD 0x134000 thru 0x137fff (0x4000 bytes) 000489[0] 1782485572.062657 mapwrite 0x667000 thru 0x6682b9 (0x12ba bytes) 000490[0] 1782485572.130945 write_OD 0x741000 thru 0x74dfff (0xd000 bytes) 000492[1] 1782485572.194497 read_OD 0x77b000 thru 0x783fff (0x9000 bytes) 000495[1] 1782485572.207388 read_OD 0x1da000 thru 0x1dafff (0x1000 bytes) 000496[1] 1782485572.212486 mapread 0x7c0000 thru 0x7cc9d0 (0xc9d1 bytes) 000497[0] 1782485572.239415 read_OD 0x0c6000 thru 0x0c9fff (0x4000 bytes) 000500[1] 1782485572.246865 read_OD 0x441000 thru 0x447fff (0x7000 bytes) All operations completed A-OK! PASS 16a (47s) == sanityn test 16b: 500 iterations of dual-mount fsx at small size ========================================================== 10:52:56 (1782485576) fallocate on zfs doesn't consume space fallocate not supported 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.284517 s, 29.5 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.285042 s, 29.4 MB/s Chance of close/open is 1 in 50 Seed set to 5579 fd 0: /mnt/lustre/f16b.sanityn fd 1: /mnt/lustre2/f16b.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000001[1] 1782485579.880789 write 0x9a447f thru 0x9a4ae2 (0x664 bytes) 000002[1] 1782485579.899500 mapread 0x7dcb53 thru 0x7dce24 (0x2d2 bytes) 000005[1] 1782485579.940628 write 0x3fa50e thru 0x3fb146 (0xc39 bytes) 000007[0] 1782485579.952559 read 0x444c58 thru 0x445ea4 (0x124d bytes) 000008[1] 1782485580.040154 mapread 0x941971 thru 0x9435a8 (0x1c38 bytes) truncating to largest ever: 0x10f89d 000009[0] 1782485580.094251 trunc from 0x9a4ae3 to 0x10f89c (0x895246 bytes) 000010[1] 1782485580.168086 read 0x0939e0 thru 0x094a00 (0x1021 bytes) 000016[0] 1782485580.228577 trunc from 0x10f89d to 0x045001 (0xca89b bytes) 000018[1] 1782485580.296708 mapwrite 0x73c613 thru 0x73d072 (0xa60 bytes) 000023[0] 1782485580.543213 read 0x23b4c5 thru 0x23d1f7 (0x1d33 bytes) 000026[1] 1782485580.611863 write 0x4fc08a thru 0x4fc574 (0x4eb bytes) 000029[1] 1782485580.645730 mapread 0x6fcb60 thru 0x6fe3d6 (0x1877 bytes) truncating to largest ever: 0x1a475e 000030[0] 1782485580.675576 trunc from 0x73d073 to 0x1a475d (0x598915 bytes) 000033[1] 1782485580.779492 mapwrite 0x2b6d5f thru 0x2b8505 (0x17a7 bytes) 000035[0] 1782485580.966916 read 0x1d1d46 thru 0x1d2c15 (0xed0 bytes) truncating to largest ever: 0x680745 000036[0] 1782485581.058316 trunc from 0x2b8506 to 0x680744 (0x3c823f bytes) truncating to largest ever: 0x98bc30 000037[0] 1782485581.160873 trunc from 0x680745 to 0x98bc2f (0x30b4eb bytes) 000038[0] 1782485581.209097 trunc from 0x98bc30 to 0x1879ca (0x804265 bytes) 000041[1] 1782485581.285481 mapwrite 0x7ab67e thru 0x7abd5c (0x6df bytes) 000043[0] 1782485581.390756 read 0x14482c thru 0x145ce9 (0x14be bytes) 000044[1] 1782485581.524613 trunc from 0x7abd5d to 0x62eb17 (0x17d245 bytes) 000045[1] 1782485581.594720 read 0x43025a thru 0x430815 (0x5bc bytes) 000046[0] 1782485581.607143 mapwrite 0x6bc8ca thru 0x6be216 (0x194d bytes) 000048[1] 1782485581.769104 mapwrite 0x4c6ced thru 0x4c88a3 (0x1bb7 bytes) 000050[1] 1782485581.914865 mapread 0x05e929 thru 0x05f8e4 (0xfbc bytes) 000052[1] 1782485581.978505 mapwrite 0x4758d3 thru 0x4777b5 (0x1ee3 bytes) 000053[1] 1782485582.088457 read 0x545536 thru 0x5468f3 (0x13be bytes) 000054[1] 1782485582.123955 trunc from 0x6be217 to 0x378031 (0x3461e5 bytes) 000055[1] 1782485582.212176 read 0x085236 thru 0x086ce1 (0x1aac bytes) 000057[1] 1782485582.341571 write 0x696d13 thru 0x6985de (0x18cc bytes) 000058[0] 1782485582.376979 mapread 0x482382 thru 0x48430c (0x1f8b bytes) 000059[0] 1782485582.467546 write 0x9ee958 thru 0x9f0929 (0x1fd2 bytes) 000060[0] 1782485582.532528 read 0x4e71fb thru 0x4e7a43 (0x849 bytes) 000061[1] 1782485582.564605 mapread 0x586b5e thru 0x587629 (0xacc bytes) 000062[1] 1782485582.581987 mapwrite 0x76dafc thru 0x76e7d9 (0xcde bytes) 000063[1] 1782485582.724696 mapwrite 0x22792d thru 0x228461 (0xb35 bytes) 000064[0] 1782485582.868463 read 0x6f62a1 thru 0x6f7ab3 (0x1813 bytes) 000066[0] 1782485582.920729 trunc from 0x9f092a to 0x1891f1 (0x867738 bytes) 000068[0] 1782485583.017738 trunc from 0x1891f2 to 0x4fc823 (0x373632 bytes) 000070[0] 1782485583.076199 mapread 0x053a5f thru 0x0547dc (0xd7e bytes) 000071[1] 1782485583.089008 mapread 0x175739 thru 0x175e55 (0x71d bytes) 000074[0] 1782485583.136117 mapwrite 0x585c5f thru 0x586e10 (0x11b2 bytes) 000080[1] 1782485583.388445 mapread 0x55d8d9 thru 0x55e67e (0xda6 bytes) 000082[0] 1782485583.569585 write 0x5f2c04 thru 0x5f3afb (0xef8 bytes) 000084[1] 1782485583.624568 mapwrite 0x9e096e thru 0x9e1f02 (0x1595 bytes) 000091[0] 1782485583.864388 mapwrite 0x8772c2 thru 0x87833e (0x107d bytes) 000096[0] 1782485584.051075 trunc from 0x9e1f03 to 0x5126c5 (0x4cf83d bytes) 000099[0] 1782485584.094648 read 0x2ed2bf thru 0x2edd74 (0xab6 bytes) 000100[0] 1782485584.142130 write 0x3e4429 thru 0x3e5a54 (0x162c bytes) 000103[0] 1782485584.156269 write 0x828df6 thru 0x82ab7c (0x1d87 bytes) 000104[0] 1782485584.161489 read 0x821b32 thru 0x821c40 (0x10f bytes) 000108[1] 1782485584.229277 read 0x756497 thru 0x757a4e (0x15b8 bytes) 000109[0] 1782485584.278710 mapread 0x265d13 thru 0x267b80 (0x1e6e bytes) 000111[1] 1782485584.339289 mapwrite 0x2df1ea thru 0x2e0d23 (0x1b3a bytes) 000115[1] 1782485584.431563 mapwrite 0x56a082 thru 0x56a4ad (0x42c bytes) 000116[0] 1782485584.498370 write 0x2f1753 thru 0x2f2063 (0x911 bytes) 000118[1] 1782485584.554003 read 0x47f328 thru 0x47f3cd (0xa6 bytes) 000120[1] 1782485584.593400 mapread 0x6f0054 thru 0x6f0e8b (0xe38 bytes) 000122[0] 1782485584.659630 mapwrite 0x0fb850 thru 0x0fccce (0x147f bytes) 000124[1] 1782485584.855431 mapwrite 0x0fbba9 thru 0x0fc161 (0x5b9 bytes) 000126[1] 1782485584.953308 mapread 0x6248da thru 0x625861 (0xf88 bytes) 000127[0] 1782485584.998722 read 0x2b9001 thru 0x2b996f (0x96f bytes) 000131[0] 1782485585.021055 trunc from 0x82ab7d to 0x3bebc2 (0x46bfba bytes) 000133[1] 1782485585.071549 mapwrite 0x7768bd thru 0x7787d4 (0x1f18 bytes) 000136[1] 1782485585.288087 trunc from 0x7787d5 to 0x421546 (0x35728e bytes) 000139[1] 1782485585.336434 read 0x356dda thru 0x358cc0 (0x1ee7 bytes) 000140[1] 1782485585.344158 mapread 0x0b317c thru 0x0b3683 (0x508 bytes) 000142[0] 1782485585.382545 write 0x177070 thru 0x1788dd (0x186e bytes) 000144[1] 1782485585.434237 read 0x2add12 thru 0x2aedbd (0x10ac bytes) 000145[0] 1782485585.495652 mapread 0x190aa3 thru 0x1923ce (0x192c bytes) 000146[1] 1782485585.551295 trunc from 0x421547 to 0x575b9f (0x154659 bytes) 000147[1] 1782485585.598130 read 0x0bbc32 thru 0x0bd067 (0x1436 bytes) 000149[1] 1782485585.637944 write 0x533656 thru 0x534a2d (0x13d8 bytes) 000151[1] 1782485585.720206 write 0x1c0b05 thru 0x1c0b07 (0x3 bytes) 000153[0] 1782485585.725938 trunc from 0x575ba0 to 0x1ad69e (0x3c8501 bytes) 000154[0] 1782485585.815540 mapread 0x0649b4 thru 0x065257 (0x8a4 bytes) 000156[1] 1782485585.864407 mapwrite 0x3f5a92 thru 0x3f70f9 (0x1668 bytes) 000158[0] 1782485585.994389 mapread 0x24d746 thru 0x24e2b0 (0xb6b bytes) 000159[0] 1782485586.048306 write 0x0ffc6a thru 0x1018cd (0x1c64 bytes) 000161[0] 1782485586.107442 write 0x48564e thru 0x486790 (0x1143 bytes) 000163[1] 1782485586.162587 mapread 0x16217f thru 0x162df4 (0xc76 bytes) 000165[0] 1782485586.220413 mapwrite 0x911874 thru 0x912cda (0x1467 bytes) 000166[1] 1782485586.343390 mapwrite 0x760f37 thru 0x76149f (0x569 bytes) 000168[1] 1782485586.487880 read 0x23c539 thru 0x23cb8c (0x654 bytes) 000170[1] 1782485586.524194 mapread 0x1e9dd6 thru 0x1eafff (0x122a bytes) 000171[1] 1782485586.536777 write 0x6ee7f7 thru 0x6ef6cb (0xed5 bytes) 000174[0] 1782485586.565197 read 0x0951e6 thru 0x096dd3 (0x1bee bytes) 000180[0] 1782485586.630363 read 0x20d399 thru 0x20d498 (0x100 bytes) 000185[1] 1782485586.671355 read 0x1dc150 thru 0x1dd1c9 (0x107a bytes) 000187[0] 1782485586.710740 mapread 0x79d13c thru 0x79e115 (0xfda bytes) 000191[1] 1782485586.758543 read 0x07c3fe thru 0x07ca5e (0x661 bytes) 000192[0] 1782485586.764921 write 0x6f63f4 thru 0x6f7cf8 (0x1905 bytes) 000193[1] 1782485586.792499 read 0x5e0b22 thru 0x5e0c8b (0x16a bytes) 000196[0] 1782485586.826527 read 0x2ba944 thru 0x2bb7b7 (0xe74 bytes) 000198[0] 1782485586.833810 write 0x41d2bc thru 0x41ee29 (0x1b6e bytes) 000199[0] 1782485586.878564 mapwrite 0x3e7341 thru 0x3e7914 (0x5d4 bytes) 000200[0] 1782485586.945565 write 0x34ebb6 thru 0x350a37 (0x1e82 bytes) 000205[1] 1782485587.038149 mapread 0x8c8d3f thru 0x8cab30 (0x1df2 bytes) 000209[0] 1782485587.115799 read 0x3c8c89 thru 0x3c91a3 (0x51b bytes) 000211[1] 1782485587.143379 read 0x76b493 thru 0x76d207 (0x1d75 bytes) 000212[1] 1782485587.175529 read 0x41708a thru 0x417bdf (0xb56 bytes) 000213[0] 1782485587.182862 mapread 0x2bee70 thru 0x2c0a6e (0x1bff bytes) 000214[1] 1782485587.209144 trunc from 0x912cdb to 0x427c00 (0x4eb0da bytes) 000216[1] 1782485587.331241 write 0x913d3d thru 0x915ba2 (0x1e66 bytes) 000217[1] 1782485587.361764 write 0x7536fa thru 0x753c88 (0x58f bytes) 000221[1] 1782485587.450428 trunc from 0x915ba3 to 0x8b9643 (0x5c55f bytes) 000222[1] 1782485587.493097 trunc from 0x8b9644 to 0x2c58c8 (0x5f3d7b bytes) 000223[1] 1782485587.552915 read 0x0cbc66 thru 0x0cc1c0 (0x55b bytes) 000224[1] 1782485587.641318 trunc from 0x2c58c9 to 0x83a12f (0x574867 bytes) 000227[1] 1782485587.697337 write 0x017f1f thru 0x0184b9 (0x59b bytes) 000228[0] 1782485587.722186 mapread 0x2f029e thru 0x2f1569 (0x12cc bytes) 000230[0] 1782485587.790286 mapread 0x7f247f thru 0x7f42b4 (0x1e36 bytes) 000231[1] 1782485587.832685 trunc from 0x83a130 to 0x3c097e (0x4797b1 bytes) 000232[0] 1782485587.899431 write 0x66158b thru 0x661e3f (0x8b5 bytes) 000233[0] 1782485587.937763 write 0x2b09f5 thru 0x2b16dc (0xce8 bytes) 000234[1] 1782485587.989968 mapwrite 0x2dcdd1 thru 0x2dd8cd (0xafd bytes) 000236[0] 1782485588.123857 trunc from 0x661e40 to 0x82c9c7 (0x1cab88 bytes) 000237[0] 1782485588.168918 read 0x75bdd6 thru 0x75dd98 (0x1fc3 bytes) 000240[1] 1782485588.293884 write 0x3cd5c2 thru 0x3ceb50 (0x158f bytes) 000241[1] 1782485588.377425 mapread 0x745a08 thru 0x746407 (0xa00 bytes) 000243[0] 1782485588.497026 mapwrite 0x9090c8 thru 0x90a54b (0x1484 bytes) 000245[0] 1782485588.661726 write 0x6abfab thru 0x6ac8c1 (0x917 bytes) 000246[1] 1782485588.674784 mapread 0x8110be thru 0x811298 (0x1db bytes) 000255[0] 1782485588.825397 write 0x7b2c9c thru 0x7b3aed (0xe52 bytes) 000257[0] 1782485588.850235 read 0x7f39e4 thru 0x7f460f (0xc2c bytes) 000260[0] 1782485588.884210 mapwrite 0x72a8bc thru 0x72ba66 (0x11ab bytes) 000261[0] 1782485588.944312 mapwrite 0x4a0ad3 thru 0x4a12c1 (0x7ef bytes) 000263[1] 1782485589.012442 write 0x35d24c thru 0x35e4b1 (0x1266 bytes) 000269[1] 1782485589.170194 mapwrite 0x5cf523 thru 0x5d0125 (0xc03 bytes) 000270[0] 1782485589.307324 write 0x230153 thru 0x230643 (0x4f1 bytes) 000271[0] 1782485589.372356 trunc from 0x90a54c to 0x4ec277 (0x41e2d4 bytes) 000272[1] 1782485589.442128 write 0x5d730a thru 0x5d7d2f (0xa26 bytes) 000274[0] 1782485589.552801 mapwrite 0x0627a1 thru 0x062cc4 (0x524 bytes) 000275[0] 1782485589.646197 mapread 0x3268c6 thru 0x326e00 (0x53b bytes) 000276[1] 1782485589.679446 mapread 0x034311 thru 0x036002 (0x1cf2 bytes) 000277[1] 1782485589.712244 write 0x3e4345 thru 0x3e6292 (0x1f4e bytes) 000278[1] 1782485589.735223 mapwrite 0x0a4fc4 thru 0x0a51ff (0x23c bytes) 000279[0] 1782485589.826507 read 0x2f1fc0 thru 0x2f35ed (0x162e bytes) 000280[1] 1782485589.843639 mapwrite 0x1c0e51 thru 0x1c1c0b (0xdbb bytes) 000282[0] 1782485589.912992 trunc from 0x5d7d30 to 0x345b30 (0x2921ff bytes) 000283[0] 1782485589.991508 write 0x24f653 thru 0x24fffa (0x9a8 bytes) 000289[0] 1782485590.130798 trunc from 0x345b31 to 0x683976 (0x33de46 bytes) 000290[1] 1782485590.165906 write 0x1be57e thru 0x1bec69 (0x6ec bytes) 000293[0] 1782485590.248087 mapread 0x06b265 thru 0x06b2f1 (0x8d bytes) 000294[1] 1782485590.294944 read 0x614f1c thru 0x615c3c (0xd21 bytes) 000295[0] 1782485590.319080 read 0x014216 thru 0x0143f1 (0x1dc bytes) 000297[1] 1782485590.326402 trunc from 0x683977 to 0x1acd44 (0x4d6c32 bytes) 000298[1] 1782485590.380029 read 0x187188 thru 0x188459 (0x12d2 bytes) 000299[1] 1782485590.428180 write 0x6fc11c thru 0x6fdb07 (0x19ec bytes) 000301[1] 1782485590.511936 mapwrite 0x15e0b4 thru 0x15f889 (0x17d6 bytes) 000303[1] 1782485590.596743 trunc from 0x6fdb08 to 0x132882 (0x5cb285 bytes) 000304[1] 1782485590.644637 read 0x118768 thru 0x119f86 (0x181f bytes) 000305[0] 1782485590.650319 read 0x0d2688 thru 0x0d2cb7 (0x630 bytes) 000307[1] 1782485590.719710 mapwrite 0x2228bf thru 0x222f84 (0x6c6 bytes) 000310[0] 1782485590.853289 mapread 0x06f445 thru 0x06fcd2 (0x88e bytes) 000311[0] 1782485590.912237 mapread 0x06d534 thru 0x06ec29 (0x16f6 bytes) 000313[0] 1782485590.971496 read 0x0a3254 thru 0x0a4eab (0x1c58 bytes) 000315[1] 1782485591.004421 mapwrite 0x3d0293 thru 0x3d1347 (0x10b5 bytes) 000316[0] 1782485591.125980 mapread 0x1c7e54 thru 0x1c8788 (0x935 bytes) 000317[0] 1782485591.213886 write 0x3a787e thru 0x3a8e37 (0x15ba bytes) 000319[1] 1782485591.274388 read 0x33c80a thru 0x33e746 (0x1f3d bytes) 000320[1] 1782485591.386410 trunc from 0x3d1348 to 0x01bccf (0x3b5678 bytes) 000321[0] 1782485591.457430 write 0x99c7ed thru 0x99d0e3 (0x8f7 bytes) 000324[0] 1782485591.548557 mapread 0x1075c9 thru 0x108b9b (0x15d3 bytes) 000325[1] 1782485591.583727 trunc from 0x99d0e4 to 0x222040 (0x77b0a3 bytes) 000326[1] 1782485591.622424 write 0x238f41 thru 0x2398c8 (0x988 bytes) 000327[0] 1782485591.652742 read 0x1b91b9 thru 0x1badab (0x1bf3 bytes) 000329[1] 1782485591.734734 mapread 0x21359a thru 0x214334 (0xd9b bytes) 000331[1] 1782485591.760268 write 0x45bbfd thru 0x45bc0b (0xf bytes) 000335[1] 1782485591.806107 mapwrite 0x12eae4 thru 0x12f445 (0x962 bytes) 000340[1] 1782485591.920413 mapwrite 0x2f83ba thru 0x2f9589 (0x11d0 bytes) 000342[0] 1782485591.966656 mapread 0x391bc0 thru 0x39298b (0xdcc bytes) 000343[0] 1782485592.038343 trunc from 0x45bc0c to 0x2972c5 (0x1c4946 bytes) 000344[1] 1782485592.114261 write 0x2ccb31 thru 0x2cdad9 (0xfa9 bytes) 000345[0] 1782485592.143671 write 0x388134 thru 0x388fc7 (0xe94 bytes) 000347[0] 1782485592.186301 mapread 0x1e7304 thru 0x1e7614 (0x311 bytes) 000348[1] 1782485592.200765 mapread 0x0aee2d thru 0x0b01b4 (0x1388 bytes) 000351[1] 1782485592.291505 mapread 0x1244ab thru 0x125ba4 (0x16fa bytes) 000352[1] 1782485592.332776 mapread 0x0fd00c thru 0x0fe46c (0x1461 bytes) 000353[1] 1782485592.351034 mapwrite 0x8f2af6 thru 0x8f4ace (0x1fd9 bytes) 000354[0] 1782485592.548769 mapwrite 0x858db4 thru 0x85909d (0x2ea bytes) 000356[1] 1782485592.666255 write 0x8767d1 thru 0x87682e (0x5e bytes) 000357[0] 1782485592.709638 read 0x6224f4 thru 0x622aed (0x5fa bytes) 000358[1] 1782485592.736648 read 0x66fdea thru 0x670790 (0x9a7 bytes) 000359[0] 1782485592.760330 trunc from 0x8f4acf to 0x1d39a1 (0x72112d bytes) 000362[0] 1782485592.881344 write 0x3b5fe8 thru 0x3b607e (0x97 bytes) 000363[0] 1782485592.894484 trunc from 0x3b607f to 0x4df9ce (0x129950 bytes) 000364[1] 1782485592.942622 read 0x245b9a thru 0x247b47 (0x1fae bytes) 000365[1] 1782485592.979993 read 0x4c9ce9 thru 0x4ca046 (0x35e bytes) 000366[0] 1782485593.004769 trunc from 0x4df9cf to 0x2b1be5 (0x22dde9 bytes) 000372[1] 1782485593.103807 mapwrite 0x32f3d7 thru 0x32ff06 (0xb30 bytes) 000378[1] 1782485593.244146 write 0x27a794 thru 0x27b619 (0xe86 bytes) 000380[1] 1782485593.280299 trunc from 0x32ff07 to 0x94e8db (0x61e9d5 bytes) 000382[1] 1782485593.358578 read 0x55676c thru 0x556b48 (0x3dd bytes) 000385[1] 1782485593.400130 read 0x707775 thru 0x7083d6 (0xc62 bytes) 000386[0] 1782485593.408717 write 0x348e20 thru 0x3496f8 (0x8d9 bytes) 000388[0] 1782485593.581154 trunc from 0x94e8dc to 0x24a987 (0x703f54 bytes) 000390[0] 1782485593.670174 trunc from 0x24a988 to 0x504a05 (0x2ba07e bytes) 000393[1] 1782485593.758537 mapwrite 0x1f0b49 thru 0x1f1ee6 (0x139e bytes) 000396[1] 1782485593.950069 read 0x0abf61 thru 0x0ac8fe (0x99e bytes) 000400[0] 1782485594.017806 mapwrite 0x398c75 thru 0x3994ec (0x878 bytes) 000401[1] 1782485594.101598 mapwrite 0x854a68 thru 0x85562b (0xbc4 bytes) 000402[0] 1782485594.192390 mapwrite 0x46ab2a thru 0x46c198 (0x166f bytes) 000403[1] 1782485594.276287 trunc from 0x85562c to 0x250f2d (0x6046fe bytes) 000405[0] 1782485594.373322 mapwrite 0x29ce16 thru 0x29d0a4 (0x28f bytes) 000406[0] 1782485594.479109 mapread 0x03c88b thru 0x03cba7 (0x31d bytes) 000411[0] 1782485594.534326 read 0x001ce6 thru 0x00375e (0x1a79 bytes) 000419[1] 1782485594.778307 mapread 0x0b66b9 thru 0x0b8113 (0x1a5b bytes) 000420[0] 1782485594.830158 write 0x09694b thru 0x097d1c (0x13d2 bytes) 000421[1] 1782485594.875120 write 0x9452f8 thru 0x9456bc (0x3c5 bytes) 000423[1] 1782485594.921120 write 0x401548 thru 0x40349c (0x1f55 bytes) 000424[1] 1782485594.959772 trunc from 0x9456bd to 0x4828c7 (0x4c2df5 bytes) 000426[0] 1782485595.016107 trunc from 0x4828c8 to 0x023091 (0x45f836 bytes) 000428[1] 1782485595.096440 trunc from 0x023092 to 0x400138 (0x3dd0a7 bytes) 000430[1] 1782485595.162118 mapread 0x31ca06 thru 0x31d005 (0x600 bytes) 000432[0] 1782485595.197303 write 0x465029 thru 0x46666f (0x1647 bytes) 000434[1] 1782485595.238060 write 0x18a8be thru 0x18ba1a (0x115d bytes) 000439[1] 1782485595.322223 read 0x076ccc thru 0x0770b7 (0x3ec bytes) 000441[0] 1782485595.362074 trunc from 0x466670 to 0x783807 (0x31d198 bytes) 000442[1] 1782485595.406119 mapread 0x00b3a7 thru 0x00d0c8 (0x1d22 bytes) 000443[0] 1782485595.460114 write 0x51d03d thru 0x51ec5b (0x1c1f bytes) 000446[0] 1782485595.488217 mapread 0x4c77e4 thru 0x4c7d71 (0x58e bytes) 000448[0] 1782485595.579055 mapwrite 0x0ce23c thru 0x0cfc6c (0x1a31 bytes) 000449[1] 1782485595.661475 trunc from 0x783808 to 0x1a7d82 (0x5dba85 bytes) 000451[1] 1782485595.739609 mapwrite 0x3f61b2 thru 0x3f753e (0x138d bytes) 000452[1] 1782485595.809900 read 0x095c76 thru 0x096cb9 (0x1044 bytes) 000455[1] 1782485595.922140 mapwrite 0x007f0d thru 0x0091ff (0x12f3 bytes) 000457[1] 1782485596.017532 trunc from 0x3f753f to 0x434614 (0x3d0d6 bytes) 000459[1] 1782485596.078321 write 0x8333b6 thru 0x8337b5 (0x400 bytes) 000460[1] 1782485596.082924 trunc from 0x8337b6 to 0x8c1d99 (0x8e5e4 bytes) 000461[0] 1782485596.122538 write 0x54f40d thru 0x54f89f (0x493 bytes) 000462[1] 1782485596.168177 read 0x019e83 thru 0x01a3d0 (0x54e bytes) 000463[1] 1782485596.206129 trunc from 0x8c1d9a to 0x004494 (0x8bd905 bytes) 000465[1] 1782485596.316808 write 0x6e05a3 thru 0x6e0f22 (0x980 bytes) 000467[1] 1782485596.346195 trunc from 0x6e0f23 to 0x64417d (0x9cda5 bytes) 000468[1] 1782485596.373279 read 0x4b041c thru 0x4b173e (0x1323 bytes) 000469[0] 1782485596.381554 trunc from 0x64417e to 0x3db3e2 (0x268d9b bytes) 000472[0] 1782485596.449284 write 0x0bd27a thru 0x0be57e (0x1305 bytes) 000473[0] 1782485596.486621 trunc from 0x3db3e3 to 0x482334 (0xa6f52 bytes) 000475[1] 1782485596.546878 mapread 0x26bcb6 thru 0x26c40e (0x759 bytes) 000476[1] 1782485596.604895 mapwrite 0x437642 thru 0x437a05 (0x3c4 bytes) 000477[0] 1782485596.691739 read 0x2e1279 thru 0x2e12c1 (0x49 bytes) 000478[0] 1782485596.699214 read 0x1abbc9 thru 0x1ac3ac (0x7e4 bytes) 000479[1] 1782485596.789622 mapread 0x132f8f thru 0x134c0f (0x1c81 bytes) 000481[1] 1782485596.833385 mapread 0x168314 thru 0x168aa6 (0x793 bytes) 000483[0] 1782485596.882188 mapread 0x3afd41 thru 0x3b1c06 (0x1ec6 bytes) 000484[1] 1782485596.891882 write 0x354adc thru 0x35557f (0xaa4 bytes) 000487[1] 1782485597.001587 mapread 0x1a2d83 thru 0x1a31b8 (0x436 bytes) 000491[0] 1782485597.093664 read 0x088c0e thru 0x089cdf (0x10d2 bytes) 000493[1] 1782485597.138800 read 0x14233f thru 0x1426f8 (0x3ba bytes) 000495[1] 1782485597.157119 write 0x6f5423 thru 0x6f672c (0x130a bytes) 000499[1] 1782485597.177711 read 0x6313e1 thru 0x633211 (0x1e31 bytes) All operations completed A-OK! PASS 16b (25s) == sanityn test 16c: verify data consistency on ldiskfs with cache disabled (b=17397) ========================================================== 10:53:22 (1782485602) SKIP: sanityn test_16c dio on ldiskfs only SKIP 16c (4s) == sanityn test 16d: Verify DIO and buffer IO with two clients ========================================================== 10:53:25 (1782485605) 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.419711 s, 20.0 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.311797 s, 26.9 MB/s ldlm.namespaces.MGC192.168.202.148@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffffa00409991000.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffffa00411183000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffffa00409991000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffffa00411183000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffffa00409991000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffffa00411183000.lru_size=clear 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.81937 s, 11.9 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 2.94345 s, 35.6 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 2.36239 s, 44.4 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.12597 s, 20.5 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.96199 s, 15.1 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 3.47999 s, 30.1 MB/s PASS 16d (42s) == sanityn test 16e: Verify size consistency for O_DIRECT write ========================================================== 10:54:07 (1782485647) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.263669 s, 39.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100504 s, 10.4 MB/s PASS 16e (6s) == sanityn test 16f: rw sequential consistency vs drop_caches ========================================================== 10:54:13 (1782485653) TIMEOUT PASS 16f (27s) == sanityn test 16g: mmap rw sequential consistency vs drop_caches ========================================================== 10:54:40 (1782485680) TIMEOUT PASS 16g (31s) == sanityn test 16h: mmap read after truncate file ======= 10:55:11 (1782485711) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16h (14s) == sanityn test 16i: read after truncate file ============ 10:55:25 (1782485725) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16i (12s) == sanityn test 16j: race dio with buffered i/o ========== 10:55:37 (1782485737) 1+0 records in 1+0 records out 26214400 bytes (26 MB, 25 MiB) copied, 4.62643 s, 5.7 MB/s bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.1633 s, 12.1 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 5.30256 s, 4.9 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.04411 s, 8.6 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 4.14744 s, 6.3 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.19694 s, 8.2 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.87509 s, 6.8 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.5483 s, 16.9 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.22144 s, 11.8 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 4.10976 s, 6.4 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 4.20384 s, 6.2 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.76555 s, 9.5 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.21838 s, 8.1 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 4.01123 s, 6.5 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 4.16681 s, 6.3 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.03465 s, 8.6 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.48215 s, 7.5 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.85338 s, 6.8 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 4.08813 s, 6.4 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK PASS 16j (60s) == sanityn test 16k: Parallel FSX and drop caches should not panic ========================================================== 10:56:37 (1782485797) + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.1 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.2 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.3 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.4 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.5 -l 67108864 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x1a7062c truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab All operations completed A-OK! + fsx 1 done: rc=0 Fri Jun 26 10:57:35 EDT 2026 All operations completed A-OK! + fsx 2 done: rc=0 Fri Jun 26 10:57:35 EDT 2026 All operations completed A-OK! All operations completed A-OK! + fsx 3 done: rc=0 Fri Jun 26 10:57:36 EDT 2026 All operations completed A-OK! + fsx 4 done: rc=0 Fri Jun 26 10:57:36 EDT 2026 + fsx 5 done: rc=0 Fri Jun 26 10:57:36 EDT 2026 PASS 16k (70s) == sanityn test 17: resource creation/LVB creation race ========================================================================= 10:57:48 (1782485868) fail_loc=0x8000030a PASS 17 (12s) == sanityn test 18: mmap sanity check =========================================================================================== 10:58:00 (1782485880) mmap test1: basic mmap operation (PASS, 0.20499s) mmap test2: MAP_PRIVATE not write back (PASS, 0.19647s) mmap test3: concurrent mmap ops on two nodes (PASS, 9.5777s) mmap test4: c1 write to f1 from mmapped f2, c2 write to f1 from mmapped f1 (PASS, 4.8693s) mmap test5: read/write file to/from the buffer which mmapped to just this file (PASS, 0.39307s) mmap test6: check mmap write/read content on two nodes (PASS, 0.5871s) mmap test7: file i/o with an unmapped buffer (SKIPPED, 0s) mmap test8: SIGBUS for beyond file size (SKIPPED, 0s) mmap test9: SIGBUS for negative file offset (SKIPPED, 0s) mmap test10: multi-thread mmap access (PASS, 2.0264s) mmap test11: multi-thread shared mmap (PASS, 1.5367s) mmap test10: mtime not change for readonly mmap access (PASS, 1.1313s) PASS 18 (30s) == sanityn test 19: test concurrent uncached read races ========================================================================= 10:58:30 (1782485910) oleg248-server: error: get_param: param_path 'osd-*/lustre-OST*/read_cache_enable': No such file or directory pdsh@oleg248-client: oleg248-server: ssh exited with exit code 2 SKIP: sanityn test_19 not cache-capable obdfilter SKIP 19 (5s) == sanityn test 20: test extra readahead page left in cache ============================================================== 10:58:35 (1782485915) PASS 20 (8s) == sanityn test 21: Try to remove mountpoint on another dir ============================================================== 10:58:43 (1782485923) rmdir: removing directory, '/mnt/lustre/d21.sanityn' rmdir: failed to remove '/mnt/lustre/d21.sanityn': Device or resource busy rmdir: removing directory, '/mnt/lustre2/d21.sanityn' Removed mounted directory from another mountpoint, needs to be fixed PASS 21 (7s) == sanityn test 23: others should see updated atime while another read============================================================== 10:58:50 (1782485930) now is 1782485933 starting reads multiop /mnt/lustre/f23.sanityn vor20_c TMPPIPE=/tmp/multiop_open_wait_pipe.7587 new atime is 1782485994 PASS 23 (70s) == sanityn test 24a: lfs df [-ih] [path] test =================================================================================== 11:00:01 (1782486001) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 30720 3738624 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 30720 3738624 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre2 UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 115.2K 400 114.8K 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 114.5K 402 114.2K 1% /mnt/lustre[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.2K 400 114.8K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 402 114.2K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre2 UUID bytes Used Available Use% Mounted on lustre-MDT0000_UUID 2.1G 2.1M 2.1G 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3.6G 8.0M 3.6G 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3.6G 30.0M 3.6G 1% /mnt/lustre[OST:1] filesystem_summary: 7.2G 38.0M 7.2G 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118000 400 117600 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117298 402 116896 1% /mnt/lustre2[OST:1] filesystem_summary: 234882 386 234496 1% /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 30720 3738624 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.2K 400 114.8K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 402 114.2K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 118000 400 117600 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 117298 402 116896 1% /mnt/lustre[OST:1] filesystem_summary: 234882 386 234496 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118000 400 117600 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117298 402 116896 1% /mnt/lustre2[OST:1] filesystem_summary: 234882 386 234496 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 30720 3738624 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 30720 3738624 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre2 PASS 24a (9s) == sanityn test 24b: lfs df should show both filesystems ========================================================================= 11:00:09 (1782486009) PASS 24b (9s) == sanityn test 25a: change ACL on one mountpoint be seen on another ============================================================= 11:00:18 (1782486018) running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] PASS 25a (9s) == sanityn test 25b: change ACL under remote dir on one mountpoint be seen on another ========================================================== 11:00:27 (1782486027) SKIP: sanityn test_25b needs >= 2 MDTs SKIP 25b (3s) == sanityn test 26a: allow mtime to get older ============ 11:00:31 (1782486031) utime: good mknod times 1782486031 <= 1782486032 <= 1782486033 for /mnt/lustre/f26a utime: good mknod times 1782486031 <= 1782486032 <= 1782486033 for /mnt/lustre2/f26a utime: good utime mtimes 100000, atime 200000 utime: updated times for /mnt/lustre2/f26a PASS 26a (9s) == sanityn test 26b: sync mtime between ost and mds ====== 11:00:40 (1782486040) PASS 26b (12s) == sanityn test 26c: set-in-past on open file is not lost on close ========================================================== 11:00:52 (1782486052) multiop /mnt/lustre/f26c.sanityn vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7587 File: /mnt/lustre/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502398 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-06-26 11:00:55.000000000 -0400 Birth: 2026-06-26 11:00:54.000000000 -0400 File: /mnt/lustre2/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502398 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-06-26 11:00:55.000000000 -0400 Birth: 2026-06-26 11:00:54.000000000 -0400 PASS 26c (12s) SKIP: sanityn test_27 skipping excluded test 27 == sanityn test 30: recreate file race =================== 11:01:06 (1782486066) 0 PASS 30 (11s) == sanityn test 31a: voluntary cancel / blocking ast race======================================================================== 11:01:17 (1782486077) fail_loc=0x314 PASS 31a (9s) == sanityn test 31b: voluntary OST cancel / blocking ast race======================================================================== 11:01:26 (1782486086) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg248-server mds-ost sync done. sleep 5 for ZFS MDS Waiting for MDT destroys to complete fail_loc=0x314 fail_loc=0x316 fail_loc=0 fail_loc=0 Filesystem 1K-blocks Used Available Use% Mounted on 192.168.202.148@tcp:/lustre 7542784 39936 7496704 1% /mnt/lustre2 PASS 31b (25s) == sanityn test 31r: open-rename(replace) race =========== 11:01:51 (1782486111) -rw-r--r-- 1 root root 0 Jun 26 11:01 /mnt/lustre/f31r.sanityn.target fail_loc=0x1419 fail_val=3 PASS 31r (10s) == sanityn test 31s: open should not revalidate invalid dentry ========================================================== 11:02:01 (1782486121) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 1207: 43105 Terminated tail -f $tfile PASS 31s (8s) == sanityn test 31t: getattr should not revalidate invalid dentry ========================================================== 11:02:10 (1782486130) PASS 31t (8s) SKIP: sanityn test_33a skipping SLOW test 33a == sanityn test 33b: COS: cross create/delete, 2 clients, benchmark under remote dir ========================================================== 11:02:19 (1782486139) SKIP: sanityn test_33b Need two or more clients, have 1 SKIP 33b (4s) == sanityn test 33c: Cancel cross-MDT lock should trigger Sync-on-Lock-Cancel ========================================================== 11:02:23 (1782486143) SKIP: sanityn test_33c needs >= 2 MDTs SKIP 33c (3s) == sanityn test 33d: dependent transactions should trigger COS ========================================================== 11:02:26 (1782486146) SKIP: sanityn test_33d needs >= 2 MDTs SKIP 33d (3s) == sanityn test 33e: independent transactions shouldn't trigger COS ========================================================== 11:02:30 (1782486150) SKIP: sanityn test_33e needs >= 2 MDTs SKIP 33e (3s) == sanityn test 34: no lock timeout under IO ============= 11:02:33 (1782486153) fail_loc=0x512 fail_loc=0x512 lock should not expire writing on client1 reading on client2 success fail_loc=0x511 fail_loc=0x511 lock should expire writing on client1 reading on client2 success oleg248-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffffa00409991000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffffa00409991000.ost_server_uuid in IDLE state after 0 sec oleg248-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffffa00409991000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffffa00409991000.ost_server_uuid in FULL state after 0 sec oleg248-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffffa00409991000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffffa00409991000.ost_server_uuid in IDLE state after 0 sec oleg248-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffffa00409991000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffffa00409991000.ost_server_uuid in FULL state after 0 sec oleg248-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffffa00409991000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffffa00409991000.ost_server_uuid in IDLE state after 0 sec oleg248-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffffa00409991000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffffa00409991000.ost_server_uuid in FULL state after 0 sec PASS 34 (132s) == sanityn test 35: -EINTR cp_ast vs. bl_ast race does not evict client ========================================================== 11:04:45 (1782486285) fail_loc=0x80000317 Race attempt 0 Wait for 49289 49356 for 60 sec... - open/close 732 (time 1782486300.96 total 10.00 last 73.18) - open/close 1472 (time 1782486310.97 total 20.01 last 73.92) - open/close 2240 (time 1782486320.97 total 30.02 last 76.77) - open/close 2962 (time 1782486330.98 total 40.02 last 72.16) - open/close 3698 (time 1782486340.99 total 50.03 last 73.53) total: 4000 open/close in 53.64 seconds: 74.57 ops/second fail_loc=0x0 Filesystem Size Used Avail Use% Mounted on 192.168.202.148@tcp:/lustre 7.2G 41M 7.2G 1% /mnt/lustre 192.168.202.148@tcp:/lustre 7.2G 41M 7.2G 1% /mnt/lustre2 PASS 35 (71s) == sanityn test 36: handle ESTALE/open-unlink correctly == 11:05:56 (1782486356) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg248-server mds-ost sync done. Waiting for MDT destroys to complete 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.22308 s, 42.9 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7587 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(0) *** before(7496704) after_dd(7441408) after(7492608) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.873207 s, 60.0 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7587 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(1) *** before(7492608) after_dd(7443456) after(7494656) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.05129 s, 49.9 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7587 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(2) *** before(7494656) after_dd(7439360) after(7490560) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.03514 s, 50.6 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7587 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(3) *** before(7490560) after_dd(7441408) after(7492608) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.16917 s, 44.8 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7587 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(4) *** before(7492608) after_dd(7441408) after(7492608) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.05038 s, 49.9 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7587 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(5) *** before(7492608) after_dd(7443456) after(7494656) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.914017 s, 57.4 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7587 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(6) *** before(7494656) after_dd(7439360) after(7490560) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.15663 s, 45.3 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7587 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(7) *** before(7490560) after_dd(7436288) after(7487488) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.08647 s, 48.3 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7587 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(8) *** before(7487488) after_dd(7443456) after(7494656) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.04964 s, 49.9 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7587 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(9) *** before(7494656) after_dd(7443456) after(7494656) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.796615 s, 65.8 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7587 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(10) *** before(7494656) after_dd(7439360) after(7490560) PASS 36 (296s) == sanityn test 37: check i_size is not updated for directory on close (bug 18695) ======================================================================== 11:10:52 (1782486652) multiop /mnt/lustre/d37.sanityn vD_c TMPPIPE=/tmp/multiop_open_wait_pipe.7587 - create 927 (time 1782486668.44 total 10.00 last 92.70) - create 1998 (time 1782486678.44 total 20.00 last 107.10) - create 3305 (time 1782486688.44 total 30.00 last 130.67) - create 4214 (time 1782486698.45 total 40.01 last 90.82) - create 5228 (time 1782486708.45 total 50.01 last 101.37) - create 6174 (time 1782486718.46 total 60.01 last 94.60) - create 7411 (time 1782486728.46 total 70.02 last 123.64) - create 8613 (time 1782486738.46 total 80.02 last 120.16) - create 9846 (time 1782486748.47 total 90.03 last 123.16) total: 10000 create in 91.54 seconds: 109.25 ops/second PASS 37 (104s) == sanityn test 39a: file mtime does not change after rename ========================================================== 11:12:36 (1782486756) repeat after cancel_lru_locks PASS 39a (8s) == sanityn test 39b: file mtime the same on clients with/out lock ========================================================== 11:12:44 (1782486764) repeat after cancel_lru_locks PASS 39b (8s) == sanityn test 39c: check truncate mtime update ================================================================================ 11:12:52 (1782486772) repeat after cancel_lru_locks PASS 39c (7s) == sanityn test 39d: sync write should update mtime ====== 11:13:00 (1782486780) fail_loc=0x411 fail_loc=0 PASS 39d (6s) SKIP: sanityn test_40a skipping ALWAYS excluded test 40a == sanityn test 40b: pdirops: open|create and others ======================================================================== 11:13:08 (1782486788) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40b (16s) == sanityn test 40c: pdirops: link and others ======================================================================== 11:13:24 (1782486804) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40c (16s) == sanityn test 40d: pdirops: unlink and others ======================================================================== 11:13:40 (1782486820) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40d (16s) == sanityn test 40e: pdirops: rename and others ======================================================================== 11:13:56 (1782486836) No conflict No conflict No conflict No conflict No conflict No conflict PASS 40e (15s) == sanityn test 41a: pdirops: create vs mkdir ======================================================================== 11:14:11 (1782486851) mkdir: cannot create directory '/mnt/lustre2/f41a.sanityn': File exists Conflict PASS 41a (12s) == sanityn test 41b: pdirops: create vs create ======================================================================== 11:14:23 (1782486863) open: File exists Conflict PASS 41b (11s) == sanityn test 41c: pdirops: create vs link ======================================================================== 11:14:34 (1782486874) link: cannot create link '/mnt/lustre2/f41c.sanityn' to '/mnt/lustre2/f41c.sanityn-2': File exists Conflict PASS 41c (12s) == sanityn test 41d: pdirops: create vs unlink ======================================================================== 11:14:47 (1782486887) Conflict PASS 41d (13s) == sanityn test 41e: pdirops: create and rename (tgt) ======================================================================== 11:15:00 (1782486900) Conflict PASS 41e (11s) == sanityn test 41f: pdirops: create and rename (src) ======================================================================== 11:15:11 (1782486911) Conflict PASS 41f (12s) == sanityn test 41g: pdirops: create vs getattr ======================================================================== 11:15:23 (1782486923) Conflict PASS 41g (11s) == sanityn test 41h: pdirops: create vs readdir ======================================================================== 11:15:34 (1782486934) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 2318: lctl set_param -n fail_loc=0 2>/dev/null || true: No such file or directory Conflict PASS 41h (12s) == sanityn test 41i: reint_open: create vs create ======== 11:15:46 (1782486946) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 41i (914s) == sanityn test 42a: pdirops: mkdir vs mkdir ======================================================================== 11:31:01 (1782487861) mkdir: cannot create directory '/mnt/lustre2/f42a.sanityn': File exists Conflict PASS 42a (16s) == sanityn test 42b: pdirops: mkdir vs create ======================================================================== 11:31:17 (1782487877) open: File exists Conflict PASS 42b (16s) == sanityn test 42c: pdirops: mkdir vs link ======================================================================== 11:31:33 (1782487893) link: cannot create link '/mnt/lustre2/f42c.sanityn' to '/mnt/lustre2/f42c.sanityn-2': File exists Conflict PASS 42c (16s) == sanityn test 42d: pdirops: mkdir vs unlink ======================================================================== 11:31:49 (1782487909) Conflict PASS 42d (13s) == sanityn test 42e: pdirops: mkdir and rename (tgt) ======================================================================== 11:32:02 (1782487922) mv: cannot overwrite directory '/mnt/lustre2/f42e.sanityn' with non-directory Conflict PASS 42e (16s) == sanityn test 42f: pdirops: mkdir and rename (src) ======================================================================== 11:32:18 (1782487938) Conflict PASS 42f (18s) == sanityn test 42g: pdirops: mkdir vs getattr ======================================================================== 11:32:37 (1782487957) Conflict PASS 42g (17s) == sanityn test 42h: pdirops: mkdir vs readdir ======================================================================== 11:32:53 (1782487973) Conflict PASS 42h (17s) == sanityn test 43a: rmdir,mkdir doesn't return -EEXIST ======================================================================== 11:33:10 (1782487990) PASS 43a (74s) == sanityn test 43b: pdirops: unlink vs create ======================================================================== 11:34:24 (1782488064) Conflict PASS 43b (22s) == sanityn test 43c: pdirops: unlink vs link ======================================================================== 11:34:46 (1782488086) Conflict PASS 43c (18s) == sanityn test 43d: pdirops: unlink vs unlink ======================================================================== 11:35:04 (1782488104) rm: cannot remove '/mnt/lustre2/f43d.sanityn': No such file or directory Conflict PASS 43d (15s) == sanityn test 43e: pdirops: unlink and rename (tgt) ======================================================================== 11:35:19 (1782488119) Conflict PASS 43e (14s) == sanityn test 43f: pdirops: unlink and rename (src) ======================================================================== 11:35:33 (1782488133) mv: cannot stat '/mnt/lustre2/f43f.sanityn': No such file or directory Conflict PASS 43f (15s) == sanityn test 43g: pdirops: unlink vs getattr ======================================================================== 11:35:48 (1782488148) stat: cannot statx '/mnt/lustre2/f43g.sanityn': No such file or directory Conflict PASS 43g (13s) == sanityn test 43h: pdirops: unlink vs readdir ======================================================================== 11:36:01 (1782488161) Conflict PASS 43h (15s) == sanityn test 43i: pdirops: unlink vs remote mkdir ===== 11:36:16 (1782488176) SKIP: sanityn test_43i needs >= 2 MDTs SKIP 43i (4s) == sanityn test 43j: racy mkdir return EEXIST ======================================================================== 11:36:21 (1782488181) mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists PASS 43j (142s) == sanityn test 43k: unlink vs create ==================== 11:38:44 (1782488324) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 43k (838s) == sanityn test 44a: pdirops: rename tgt vs mkdir ======================================================================== 11:52:42 (1782489162) mkdir: cannot create directory '/mnt/lustre2/f44a.sanityn': File exists Conflict Fri Jun 26 11:52:51 EDT 2026 PASS 44a (15s) == sanityn test 44b: pdirops: rename tgt vs create ======================================================================== 11:52:57 (1782489177) open: File exists Conflict PASS 44b (15s) == sanityn test 44c: pdirops: rename tgt vs link ======================================================================== 11:53:13 (1782489193) link: cannot create link '/mnt/lustre2/f44c.sanityn' to '/mnt/lustre2/f44c.sanityn-3': File exists Conflict PASS 44c (16s) == sanityn test 44d: pdirops: rename tgt vs unlink ======================================================================== 11:53:30 (1782489210) Conflict PASS 44d (17s) == sanityn test 44e: pdirops: rename tgt and rename (tgt) ======================================================================== 11:53:47 (1782489227) Conflict PASS 44e (16s) == sanityn test 44f: pdirops: rename tgt and rename (src) ======================================================================== 11:54:03 (1782489243) Conflict PASS 44f (17s) == sanityn test 44g: pdirops: rename tgt vs getattr ======================================================================== 11:54:20 (1782489260) Conflict PASS 44g (19s) == sanityn test 44h: pdirops: rename tgt vs readdir ======================================================================== 11:54:40 (1782489280) Conflict PASS 44h (18s) == sanityn test 44i: pdirops: rename tgt vs remote mkdir ========================================================== 11:54:57 (1782489297) SKIP: sanityn test_44i needs >= 2 MDTs SKIP 44i (5s) == sanityn test 45a: rename,mkdir doesn't return -EEXIST ======================================================================== 11:55:02 (1782489302) PASS 45a (133s) == sanityn test 45b: pdirops: rename src vs create ======================================================================== 11:57:15 (1782489435) Conflict PASS 45b (14s) == sanityn test 45c: pdirops: rename src vs link ======================================================================== 11:57:30 (1782489450) Conflict PASS 45c (13s) == sanityn test 45d: pdirops: rename src vs unlink ======================================================================== 11:57:43 (1782489463) rm: cannot remove '/mnt/lustre2/f45d.sanityn': No such file or directory Conflict PASS 45d (16s) == sanityn test 45e: pdirops: rename src and rename (tgt) ======================================================================== 11:57:59 (1782489479) Conflict PASS 45e (15s) == sanityn test 45f: pdirops: rename src and rename (src) ======================================================================== 11:58:14 (1782489494) mv: cannot stat '/mnt/lustre2/f45f.sanityn': No such file or directory Conflict PASS 45f (14s) == sanityn test 45g: pdirops: rename src vs getattr ======================================================================== 11:58:28 (1782489508) stat: cannot statx '/mnt/lustre2/f45g.sanityn': No such file or directory Conflict PASS 45g (15s) == sanityn test 45h: pdirops: unlink vs readdir ======================================================================== 11:58:43 (1782489523) Conflict PASS 45h (16s) == sanityn test 45i: pdirops: rename src vs remote mkdir ========================================================== 11:58:59 (1782489539) SKIP: sanityn test_45i needs >= 2 MDTs SKIP 45i (4s) == sanityn test 45j: read vs rename ====================== 11:59:03 (1782489543) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 45j (1252s) == sanityn test 46a: pdirops: link vs mkdir ======================================================================== 12:19:55 (1782490795) mkdir: cannot create directory '/mnt/lustre2/f46a.sanityn': File exists Conflict PASS 46a (14s) == sanityn test 46b: pdirops: link vs create ======================================================================== 12:20:10 (1782490810) open: File exists Conflict PASS 46b (15s) == sanityn test 46c: pdirops: link vs link ======================================================================== 12:20:25 (1782490825) link: cannot create link '/mnt/lustre2/f46c.sanityn' to '/mnt/lustre2/f46c.sanityn': File exists Conflict PASS 46c (16s) == sanityn test 46d: pdirops: link vs unlink ======================================================================== 12:20:42 (1782490842) Conflict PASS 46d (19s) == sanityn test 46e: pdirops: link and rename (tgt) ======================================================================== 12:21:01 (1782490861) Conflict PASS 46e (15s) == sanityn test 46f: pdirops: link and rename (src) ======================================================================== 12:21:16 (1782490876) Conflict PASS 46f (18s) == sanityn test 46g: pdirops: link vs getattr ======================================================================== 12:21:35 (1782490895) Conflict PASS 46g (17s) == sanityn test 46h: pdirops: link vs readdir ======================================================================== 12:21:51 (1782490911) Conflict PASS 46h (16s) == sanityn test 46i: pdirops: link vs remote mkdir ======= 12:22:08 (1782490928) SKIP: sanityn test_46i needs >= 2 MDTs SKIP 46i (5s) == sanityn test 47a: pdirops: remote mkdir vs mkdir ====== 12:22:12 (1782490932) SKIP: sanityn test_47a needs >= 2 MDTs SKIP 47a (4s) == sanityn test 47b: pdirops: remote mkdir vs create ===== 12:22:16 (1782490936) SKIP: sanityn test_47b needs >= 2 MDTs SKIP 47b (3s) == sanityn test 47c: pdirops: remote mkdir vs link ======= 12:22:20 (1782490940) SKIP: sanityn test_47c needs >= 2 MDTs SKIP 47c (3s) == sanityn test 47d: pdirops: remote mkdir vs unlink ===== 12:22:23 (1782490943) SKIP: sanityn test_47d needs >= 2 MDTs SKIP 47d (3s) == sanityn test 47e: pdirops: remote mkdir and rename (tgt) ========================================================== 12:22:26 (1782490946) SKIP: sanityn test_47e needs >= 2 MDTs SKIP 47e (4s) == sanityn test 47f: pdirops: remote mkdir and rename (src) ========================================================== 12:22:30 (1782490950) SKIP: sanityn test_47f needs >= 2 MDTs SKIP 47f (4s) == sanityn test 47g: pdirops: remote mkdir vs getattr ==== 12:22:34 (1782490954) SKIP: sanityn test_47g needs >= 2 MDTs SKIP 47g (3s) == sanityn test 50: osc lvb attrs: enqueue vs. CP AST ======================================================================== 12:22:37 (1782490957) 10+0 records in 10+0 records out 10240 bytes (10 kB, 10 KiB) copied, 0.134025 s, 76.4 kB/s fail_loc=0x410 fail_loc=0x0 PASS 50 (16s) == sanityn test 51a: layout lock: refresh layout should work ========================================================== 12:22:54 (1782490974) 0+1 records in 0+1 records out 158 bytes copied, 0.0540904 s, 2.9 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 3581: kill: (217726) - No such process PASS 51a (14s) == sanityn test 51b: layout lock: glimpse should be able to restart if layout changed ========================================================== 12:23:07 (1782490987) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0261317 s, 39.2 kB/s fail_loc=0x1404 fail_val=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187858 s, 5.6 MB/s 3145728 PASS 51b (23s) == sanityn test 51c: layout lock: IT_LAYOUT blocked and correct layout can be returned ========================================================== 12:23:31 (1782491011) fail_loc=0x172 Setting layout to have 2 stripes ... 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0363777 s, 28.1 kB/s PASS 51c (13s) == sanityn test 51d: layout lock: losing layout lock should clean up memory map region ========================================================== 12:23:44 (1782491024) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0490064 s, 21.4 MB/s Before revoking layout lock: 1024 KB mapped PASS 51d (9s) == sanityn test 51e: lfs getstripe does not break leases, part 2 ========================================================== 12:23:54 (1782491034) /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 2452 0x994 0x240000400 write lease(2) has applied. /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 2452 0x994 0x240000400 read lease(1) has applied. PASS 51e (10s) == sanityn test 54: rename locking ======================= 12:24:04 (1782491044) llite.lustre-ffffa00409991000.xattr_cache=1 llite.lustre-ffffa00411183000.xattr_cache=1 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 7680 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505011 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-06-26 12:24:06.000000000 -0400 Modify: 2026-06-26 12:24:06.000000000 -0400 Change: 2026-06-26 12:24:06.000000000 -0400 Birth: 2026-06-26 12:24:06.000000000 -0400 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000154 File: /mnt/lustre/d1/d2 Size: 7680 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505014 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-06-26 12:24:13.000000000 -0400 Modify: 2026-06-26 12:24:14.000000000 -0400 Change: 2026-06-26 12:24:14.000000000 -0400 Birth: 2026-06-26 12:24:12.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 512 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505017 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-06-26 12:24:19.000000000 -0400 Modify: 2026-06-26 12:24:19.000000000 -0400 Change: 2026-06-26 12:24:19.000000000 -0400 Birth: 2026-06-26 12:24:19.000000000 -0400 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1 Size: 7680 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505016 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-06-26 12:24:19.000000000 -0400 Modify: 2026-06-26 12:24:20.000000000 -0400 Change: 2026-06-26 12:24:20.000000000 -0400 Birth: 2026-06-26 12:24:19.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000154 File: /mnt/lustre/d1 Size: 7680 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505019 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-06-26 12:24:25.000000000 -0400 Modify: 2026-06-26 12:24:27.000000000 -0400 Change: 2026-06-26 12:24:27.000000000 -0400 Birth: 2026-06-26 12:24:25.000000000 -0400 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1/d2 Size: 7680 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505020 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-06-26 12:24:26.000000000 -0400 Modify: 2026-06-26 12:24:27.000000000 -0400 Change: 2026-06-26 12:24:27.000000000 -0400 Birth: 2026-06-26 12:24:25.000000000 -0400 PASS 54 (34s) == sanityn test 55a: rename vs unlink target dir ========= 12:24:38 (1782491078) fail_loc=0x80000156 mv: cannot move '/mnt/lustre/d55a.sanityn/d1/d2' to '/mnt/lustre/d55a.sanityn/d3/d2': No such file or directory PASS 55a (15s) == sanityn test 55b: rename vs unlink source dir ========= 12:24:53 (1782491093) fail_loc=0x80000156 mv: cannot stat '/mnt/lustre/d55b.sanityn/d1/d2': No such file or directory PASS 55b (14s) == sanityn test 55c: rename vs unlink orphan target dir == 12:25:07 (1782491107) fail_loc=0x156 mv: cannot move '/mnt/lustre/d55c.sanityn/d1/d2' to '/mnt/lustre/d55c.sanityn/d3/d2': No such file or directory PASS 55c (19s) == sanityn test 55d: rename file vs link ================= 12:25:26 (1782491126) fail_loc=0x155 mv: '/mnt/lustre/d55d.sanityn/f1' and '/mnt/lustre/d55d.sanityn/d55d.sanityn/f1' are the same file PASS 55d (16s) == sanityn test 55e: rename race AB/BA under the same parent dir ========================================================== 12:25:42 (1782491142) SKIP: sanityn test_55e needs >= 2 MDTs SKIP 55e (3s) == sanityn test 55f: rename: (P1/A -> P2/B) race with (P2/B -> P1/A) ========================================================== 12:25:46 (1782491146) fail_loc=0x155 We survived after P1A_P2B/P2B_P1A race test fail_loc=0x0 PASS 55f (21s) == sanityn test 55g: rename: race with trylock =========== 12:26:07 (1782491167) mdt.lustre-MDT0000.enable_rename_trylock=1 fail_loc=0x155 mv: cannot move '/mnt/lustre/d55g.sanityn/a1/a2' to a subdirectory of itself, '/mnt/lustre/d55g.sanityn/b1/b2/b3/b4/a2' We survived race with trylock test fail_loc=0x0 PASS 55g (24s) == sanityn test 56a: test llverdev with single large stripe ========================================================== 12:26:31 (1782491191) run_llverdev /mnt/lustre/f56a.sanityn -p -s 16000000 -c 4k llverdev: /mnt/lustre/f56a.sanityn is 16777216000000 bytes (15625 GB) in size timestamp: 1782491193 chunksize: 4096 size: 16777216000000 write offset: 16384000000kB 0.53543 MB/s read offset: 16384000000kB inf MB/s llverdev: data verified successfully PASS 56a (122s) == sanityn test 56b: test llverdev and partial verify of wide stripe file ========================================================== 12:28:34 (1782491314) run_llverdev /mnt/lustre/f56b.sanityn -p -s 32000000 -c 4k llverdev: /mnt/lustre/f56b.sanityn is 33554432000000 bytes (31250 GB) in size timestamp: 1782491320 chunksize: 4096 size: 33554432000000 write offset: 32768000000kB 0.968843 MB/s read offset: 32768000000kB inf MB/s llverdev: data verified successfully PASS 56b (141s) == sanityn test 60: Verify data_version behaviour ======== 12:30:55 (1782491455) PASS 60 (7s) == sanityn test 70a: cd directory && rm directory ======== 12:31:02 (1782491462) PASS 70a (9s) == sanityn test 70b: remove files after calling rm_entry ========================================================== 12:31:11 (1782491471) PASS 70b (8s) == sanityn test 71a: correct file map just after write operation is finished ========================================================== 12:31:19 (1782491479) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.00738664 s, 8.9 MB/s SKIP: sanityn test_71a ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71a (4s) == sanityn test 71b: check fiemap support for stripecount > 1 ========================================================== 12:31:23 (1782491483) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0198989 s, 3.3 MB/s SKIP: sanityn test_71b ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71b (4s) == sanityn test 71c: check FIEMAP_EXTENT_LAST flag with different extents number ========================================================== 12:31:27 (1782491487) SKIP: sanityn test_71c support only ldiskfs ost SKIP 71c (3s) == sanityn test 71d: fiemap corruption test with fm_extent_count=0 ========================================================== 12:31:30 (1782491490) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0231347 s, 2.8 MB/s SKIP: sanityn test_71d ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71d (4s) == sanityn test 72: getxattr/setxattr cache should be consistent between nodes ========================================================== 12:31:34 (1782491494) llite.lustre-ffffa00409991000.xattr_cache=1 llite.lustre-ffffa00411183000.xattr_cache=1 getfattr: Removing leading '/' from absolute path names user.attr1="value1" getfattr: Removing leading '/' from absolute path names user.attr1="value2" getfattr: Removing leading '/' from absolute path names getfattr: Removing leading '/' from absolute path names b362740ad371eae580e7f85452e8830b - 870e072da1ebfc20ea8c0cd3d6bdf3b6 - PASS 72 (7s) == sanityn test 73: getxattr should not cause xattr lock cancellation ========================================================== 12:31:41 (1782491501) llite.lustre-ffffa00409991000.xattr_cache=1 llite.lustre-ffffa00411183000.xattr_cache=1 getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" PASS 73 (7s) == sanityn test 74: flock deadlock: different mounts ======================================================================== 12:31:48 (1782491508) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0193999 s, 52.8 kB/s 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0124843 s, 82.0 kB/s 230541: get lock2 230542: get lock1 230542: done 230542: get lock2 230541: done 230541: get lock1 fcntl cmd 7 failed: Resource deadlock avoided 230541: cannot get lock1: Resource deadlock avoided 230542: done 230542: exit rc=0 230541: done 230541: exit rc=0 PASS 74 (9s) == sanityn test 75: osc: upcall after unuse lock============================================================================= 12:31:57 (1782491517) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0626573 s, 33.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.040873 s, 25.7 MB/s fail_loc=0x31d /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 4264: 231122 Killed stat -c %s $DIR1/$tfile fail_loc=0 PASS 75 (12s) == sanityn test 76: Verify MDT open_files listing ======== 12:32:09 (1782491529) opening files: ........ closing files: ........ PASS 76 (75s) == sanityn test 77a: check FIFO NRS policy =============== 12:33:24 (1782491604) ost.OSS.ost.nrs_policies=fifo ost.OSS.ost_create.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_out.nrs_policies=fifo ost.OSS.ost_seq.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.574765 s, 29.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0512893 s, 20.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0391355 s, 26.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0465887 s, 22.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120317 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19324 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160873 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161082 s, 6.5 MB/s ldlm.namespaces.lustre-OST0001-osc-ffffa00411183000.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188566 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.270352 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216944 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223321 s, 4.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.332546 s, 3.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.264575 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.320939 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.257839 s, 4.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffffa00411183000.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffffa00411183000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.275408 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0901135 s, 11.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0109078 s, 96.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0108347 s, 96.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0919032 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0468556 s, 22.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0451342 s, 23.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0779118 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11662 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00385618 s, 272 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0543852 s, 19.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0830708 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0347825 s, 30.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.042449 s, 24.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0244642 s, 42.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0323465 s, 32.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0236469 s, 44.3 MB/s removed '/mnt/lustre/d77a.sanityn/nrs_r_oleg248-client.virtnet' removed '/mnt/lustre/d77a.sanityn/nrs_w_oleg248-client.virtnet' removed directory '/mnt/lustre/d77a.sanityn' PASS 77a (9s) == sanityn test 77b: check CRR-N NRS policy ============== 12:33:33 (1782491613) ost.OSS.ost.nrs_policies=crrn ost.OSS.ost_create.nrs_policies=crrn ost.OSS.ost_io.nrs_policies=crrn ost.OSS.ost_out.nrs_policies=crrn ost.OSS.ost_seq.nrs_policies=crrn ost.OSS.ost.nrs_crrn_quantum=1 ost.OSS.ost_create.nrs_crrn_quantum=1 ost.OSS.ost_io.nrs_crrn_quantum=1 ost.OSS.ost_out.nrs_crrn_quantum=1 ost.OSS.ost_seq.nrs_crrn_quantum=1 policy: crr-n, crrn_quantum 1 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.435336 s, 38.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0684795 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0501348 s, 20.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0790937 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0906858 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.251054 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.228205 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.226191 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.226794 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221596 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.308009 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.258559 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.342921 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.307676 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.279753 s, 3.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffffa00411183000.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffffa00411183000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.282603 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.366162 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00852791 s, 123 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00544587 s, 193 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00639131 s, 164 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0039383 s, 266 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144247 s, 72.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0062903 s, 167 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00515729 s, 203 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00310725 s, 337 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0581576 s, 18.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105945 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0443801 s, 23.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0774109 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0332306 s, 31.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0552327 s, 19.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0184049 s, 57.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0393347 s, 26.7 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg248-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg248-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost.nrs_crrn_quantum=64 ost.OSS.ost_create.nrs_crrn_quantum=64 ost.OSS.ost_io.nrs_crrn_quantum=64 ost.OSS.ost_out.nrs_crrn_quantum=64 ost.OSS.ost_seq.nrs_crrn_quantum=64 policy: crr-n, crrn_quantum 64 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.354949 s, 47.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.069874 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0802313 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104039 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149895 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121749 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10531 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141611 s, 7.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.26034 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.266695 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.255068 s, 4.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.217562 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.303369 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.283951 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.312684 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.333125 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.363978 s, 2.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118012 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136443 s, 7.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0768511 s, 13.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138505 s, 7.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0597062 s, 17.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140885 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0514569 s, 20.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143467 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0542284 s, 19.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.055625 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0495608 s, 21.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.055786 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0569294 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160308 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0599839 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0644455 s, 16.3 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg248-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg248-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77b (11s) == sanityn test 77c: check ORR NRS policy ================ 12:33:45 (1782491625) ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_orr_quantum=1 ost.OSS.ost_io.nrs_orr_offset_type=physical ost.OSS.ost_io.nrs_orr_supported=reads policy: orr, orr_quantum 1, orr_offset_type physical, orr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.290957 s, 57.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.029912 s, 35.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0546034 s, 19.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0613195 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0726945 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104461 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.261928 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172272 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222812 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196722 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206412 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190989 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.261313 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.240851 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.323088 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.279781 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.419425 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0601358 s, 17.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0858978 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0643047 s, 16.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113825 s, 9.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0677544 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0757704 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0775821 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0824456 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120587 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0881786 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0897278 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0862697 s, 12.2 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104138 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100807 s, 10.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140274 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115069 s, 9.1 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg248-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg248-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=writes ost.OSS.ost_io.nrs_orr_quantum=64 policy: orr, orr_quantum 64, orr_offset_type physical, orr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.474109 s, 35.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0800502 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0781095 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148294 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126939 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198992 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.252262 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160434 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242327 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223016 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.333116 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.226833 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.315054 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.374349 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.260971 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.282286 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.392119 s, 2.7 MB/s ldlm.namespaces.lustre-OST0001-osc-ffffa00411183000.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121316 s, 8.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0042192 s, 249 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10879 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151602 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0050467 s, 208 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0959192 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0730819 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0687007 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0577821 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0528994 s, 19.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.047198 s, 22.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0478574 s, 21.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0445993 s, 23.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0306541 s, 34.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0371485 s, 28.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0400929 s, 26.2 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg248-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg248-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=reads_and_writes ost.OSS.ost_io.nrs_orr_offset_type=logical policy: orr, orr_quantum 64, orr_offset_type logical, orr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.465856 s, 36.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0542 s, 19.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0769776 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117125 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172685 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168713 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166365 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184921 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165699 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182382 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178837 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206117 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207078 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186889 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.228333 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.218276 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.251002 s, 4.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffffa00411183000.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffffa00411183000.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00336802 s, 311 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00252571 s, 415 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0714421 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0553774 s, 18.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0039921 s, 263 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109064 s, 9.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0859667 s, 12.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0558225 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0452665 s, 23.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.041565 s, 25.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0400485 s, 26.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0284395 s, 36.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0192684 s, 54.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0267621 s, 39.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0230352 s, 45.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152297 s, 68.9 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg248-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg248-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77c (13s) == sanityn test 77d: check TRR nrs policy ================ 12:33:58 (1782491638) ost.OSS.ost_io.nrs_policies=trr ost.OSS.ost_io.nrs_trr_quantum=1 ost.OSS.ost_io.nrs_trr_offset_type=physical ost.OSS.ost_io.nrs_trr_supported=reads policy: trr, trr_quantum 1, trr_offset_type physical, trr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.463799 s, 36.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0414721 s, 25.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0739283 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0972048 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147281 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198044 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152572 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20015 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.231494 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.261338 s, 4.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.256832 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.310528 s, 3.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffffa00411183000.lock_unused_count=5 ldlm.namespaces.lustre-OST0001-osc-ffffa00411183000.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.307225 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.332506 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.36199 s, 2.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.317259 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.335887 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136861 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00422852 s, 248 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.062503 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0843165 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140966 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140248 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149219 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0490741 s, 21.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144997 s, 72.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0666114 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0330341 s, 31.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0451718 s, 23.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0523745 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0415277 s, 25.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0383358 s, 27.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0294434 s, 35.6 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg248-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg248-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=writes ost.OSS.ost_io.nrs_trr_quantum=64 policy: trr, trr_quantum 64, trr_offset_type physical, trr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.386926 s, 43.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.040322 s, 26.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0597914 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0944206 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138005 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170863 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160667 s, 6.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffffa00411183000.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209703 s, 5.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175473 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.26297 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188234 s, 5.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.249232 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.272077 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.313227 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.308797 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.340473 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.404017 s, 2.6 MB/s ldlm.namespaces.lustre-OST0001-osc-ffffa00411183000.lock_unused_count=3 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0245678 s, 42.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0557501 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0267271 s, 39.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0308571 s, 34.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.070078 s, 15.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0339697 s, 30.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0420247 s, 25.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0361108 s, 29.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0402425 s, 26.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0480602 s, 21.8 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0477276 s, 22.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101245 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0483529 s, 21.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0602124 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0971793 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0748777 s, 14.0 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg248-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg248-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=reads_and_writes ost.OSS.ost_io.nrs_trr_offset_type=logical policy: trr, trr_quantum 64, trr_offset_type logical, trr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.457935 s, 36.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0419312 s, 25.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.065687 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.021974 s, 47.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0747066 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0314128 s, 33.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0799239 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0573028 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.241885 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196265 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173979 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0967939 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185222 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.280144 s, 3.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.244891 s, 4.3 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152785 s, 6.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffffa00411183000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffffa00411183000.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237706 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0295743 s, 35.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00872381 s, 120 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0684447 s, 15.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.031301 s, 33.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffffa00411183000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00809557 s, 130 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00240901 s, 435 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00918013 s, 114 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00338476 s, 310 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00579175 s, 181 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00251026 s, 418 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0830667 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0442849 s, 23.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0276286 s, 38.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0229761 s, 45.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.020193 s, 51.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00609895 s, 172 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg248-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg248-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77d (13s) == sanityn test 77e: check TBF NID nrs policy ============ 12:34:11 (1782491651) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_tbf_rule=start localhost nid={0@lo} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start clients nid={192.168.202.48@tcp} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start others nid={*.*.*.*@tcp} rate=50 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.44967 s, 37.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0611973 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0838933 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126356 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16219 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180032 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19435 s, 5.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffffa00411183000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffffa00411183000.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203407 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.228026 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.235059 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227807 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221254 s, 4.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20717 s, 5.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195056 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.229281 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.259768 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201816 s, 5.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffffa00411183000.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffffa00411183000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168176 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124154 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137118 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162926 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149508 s, 7.0 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154768 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133957 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134946 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14359 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139514 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150074 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1405 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169189 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145537 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156988 s, 6.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152506 s, 6.9 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg248-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg248-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change localhost rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change clients rate=101 ost.OSS.ost_io.nrs_tbf_rule=change others rate=51 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.414878 s, 40.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0545673 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121989 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145052 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152298 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147828 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195595 s, 5.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223444 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19069 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.217371 s, 4.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123309 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19908 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.214143 s, 4.9 MB/s ldlm.namespaces.lustre-OST0001-osc-ffffa00411183000.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222271 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.279466 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.26679 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.243856 s, 4.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115662 s, 9.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104402 s, 10.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0691672 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0809254 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0813835 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0882693 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131757 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0917691 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0918622 s, 11.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0983029 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10271 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156218 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105326 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114991 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166277 s, 6.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffffa00411183000.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129837 s, 8.1 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg248-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg248-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop localhost ost.OSS.ost_io.nrs_tbf_rule=stop clients ost.OSS.ost_io.nrs_tbf_rule=stop others 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.31826 s, 52.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0918419 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.096318 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170313 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.25778 s, 4.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169686 s, 6.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172863 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.290694 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227681 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.281157 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.24417 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.279159 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.267317 s, 3.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.299796 s, 3.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.270589 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.263054 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.320462 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0333956 s, 31.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0626289 s, 16.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0424193 s, 24.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.035079 s, 29.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0245069 s, 42.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0452608 s, 23.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0453641 s, 23.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0333535 s, 31.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0519947 s, 20.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0774556 s, 13.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0402398 s, 26.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0485345 s, 21.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0521115 s, 20.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0544482 s, 19.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0548732 s, 19.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0590813 s, 17.7 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg248-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg248-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.348478 s, 48.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117709 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106546 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0958812 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0964646 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178801 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148787 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.231555 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.268946 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.240636 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.270514 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.299207 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.248867 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.248938 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.315558 s, 3.3 MB/s 1+0 records in 1+0 records out ldlm.namespaces.lustre-OST0000-osc-ffffa00411183000.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffffa00411183000.lock_unused_count=6 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.298307 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.29754 s, 3.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153488 s, 68.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0414023 s, 25.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135266 s, 77.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0321983 s, 32.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0351965 s, 29.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0215363 s, 48.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0342991 s, 30.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0556788 s, 18.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0355581 s, 29.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00877845 s, 119 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130458 s, 80.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0224458 s, 46.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00864889 s, 121 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00290593 s, 361 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00915807 s, 114 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00817802 s, 128 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg248-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg248-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' PASS 77e (18s) == sanityn test 77f: check TBF JobID nrs policy ========== 12:34:29 (1782491669) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start runas jobid={iozone.500 dd.500 tiotest.500} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start iozone_runas jobid={iozone.500} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg248-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.425473 s, 39.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0657784 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0686276 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0781029 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0722387 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237808 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211082 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.236762 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.258203 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.254297 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.247815 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.232127 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.235003 s, 4.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.247339 s, 4.2 MB/s execvp fails running declare (13): Permission denied ldlm.namespaces.lustre-OST0000-osc-ffffa00411183000.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffffa00411183000.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.285839 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220063 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.269471 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00295862 s, 354 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00293788 s, 357 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00536725 s, 195 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0991944 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.09094 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126271 s, 8.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0704197 s, 14.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00510659 s, 205 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.06742 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0629929 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0108809 s, 96.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0583834 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0828387 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0273759 s, 38.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137269 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.02267 s, 46.3 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg248-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg248-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change runas rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change iozone_runas rate=101 ost.OSS.ost_io.nrs_tbf_rule=change dd_runas rate=51 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg248-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.376623 s, 44.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0705277 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0870729 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122776 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216453 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171812 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200801 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166935 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207469 s, 5.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.255323 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.248142 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.326208 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.30644 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.27638 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.319492 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.339293 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.384809 s, 2.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffffa00411183000.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffffa00411183000.lock_unused_count=7 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00730893 s, 143 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00939733 s, 112 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129577 s, 80.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00749713 s, 140 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151783 s, 69.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0046293 s, 227 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0975344 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118119 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00411772 s, 255 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139499 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0750332 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0607318 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0713211 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0893065 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0333467 s, 31.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0289727 s, 36.2 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg248-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg248-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop runas ost.OSS.ost_io.nrs_tbf_rule=stop iozone_runas ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg248-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.346359 s, 48.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0565213 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0662345 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125163 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125263 s, 8.4 MB/s ldlm.namespaces.lustre-OST0001-osc-ffffa00411183000.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11575 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127635 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160468 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13926 s, 7.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19249 s, 5.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170157 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210043 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.225725 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174605 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.256465 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22183 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.247568 s, 4.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied ldlm.namespaces.lustre-OST0000-osc-ffffa00411183000.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0216829 s, 48.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0264357 s, 39.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167659 s, 62.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0171244 s, 61.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.048777 s, 21.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141472 s, 74.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.020034 s, 52.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0359538 s, 29.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00315124 s, 333 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0329047 s, 31.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0420252 s, 25.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0424068 s, 24.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00919028 s, 114 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0347795 s, 30.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0402218 s, 26.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0037262 s, 281 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg248-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg248-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_policies=fifo running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg248-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.353333 s, 47.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0672372 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0864536 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0621792 s, 16.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152052 s, 6.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162401 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193479 s, 5.4 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18339 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205516 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179912 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201464 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199858 s, 5.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.204012 s, 5.1 MB/sldlm.namespaces.lustre-OST0001-osc-ffffa00411183000.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181051 s, 5.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21979 s, 4.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19444 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179596 s, 5.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0255074 s, 41.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0556678 s, 18.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0212797 s, 49.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0521189 s, 20.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0234673 s, 44.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0583917 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0366718 s, 28.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0477027 s, 22.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0351591 s, 29.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0384171 s, 27.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0544924 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0571394 s, 18.4 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163393 s, 64.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0097152 s, 108 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0296164 s, 35.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0253268 s, 41.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffffa00411183000.lock_unused_count=3 removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg248-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg248-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' PASS 77f (17s) == sanityn test 77g: Change TBF type directly ============ 12:34:46 (1782491686) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 ost.OSS.ost_io.nrs_policies=fifo PASS 77g (8s) == sanityn test 77h: Wrong policy name should report error, not LBUG ========================================================== 12:34:54 (1782491694) oleg248-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=abc: No such file or directory oleg248-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='abc': No such file or directory pdsh@oleg248-client: oleg248-server: ssh exited with exit code 2 oleg248-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc: Invalid argument oleg248-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc': Invalid argument pdsh@oleg248-client: oleg248-server: ssh exited with exit code 22 oleg248-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf reg abc: Invalid argument oleg248-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf reg abc': Invalid argument pdsh@oleg248-client: oleg248-server: ssh exited with exit code 22 oleg248-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc efg: Invalid argument oleg248-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc efg': Invalid argument pdsh@oleg248-client: oleg248-server: ssh exited with exit code 22 PASS 77h (9s) == sanityn test 77i: Change rank of TBF rule ============= 12:35:04 (1782491704) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start before jobid={jobid} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start after jobid={jobid} rate=1000 rank=default ost.OSS.ost_io.nrs_tbf_rule=start target jobid={jobid} rate=1000 rank=after Move before itself ost.OSS.ost_io.nrs_tbf_rule=change target rank=target Move to higher rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=before Move to lower rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=after Move before default ost.OSS.ost_io.nrs_tbf_rule=change target rank=default ost.OSS.ost_io.nrs_policies=fifo PASS 77i (17s) == sanityn test 77j: check TBF-OPCode NRS policy ========= 12:35:22 (1782491722) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf opcode ost.OSS.ost_io.nrs_tbf_rule=start ost_r opcode={ost_read} rate=5 ost.OSS.ost_io.nrs_tbf_rule=start ost_w opcode={ost_write} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.467358 s, 35.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0391144 s, 26.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0844328 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.090378 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153145 s, 6.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127651 s, 8.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116851 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202407 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.219194 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161545 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190672 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.229979 s, 4.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20576 s, 5.1 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22371 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202975 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22017 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166187 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0116709 s, 89.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162154 s, 64.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.017795 s, 58.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0206506 s, 50.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0677958 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0268484 s, 39.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0657987 s, 15.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0297634 s, 35.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0322082 s, 32.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.042175 s, 24.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0372881 s, 28.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0389562 s, 26.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0779568 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0901003 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0394364 s, 26.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0480471 s, 21.8 MB/s removed '/mnt/lustre/d77j.sanityn/nrs_w_oleg248-client.virtnet' removed '/mnt/lustre/d77j.sanityn/nrs_r_oleg248-client.virtnet' removed directory '/mnt/lustre/d77j.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 5 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.41159 s, 16.4 MB/s Write runtime is 7 s, speed is 14.285714 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.5859 s, 5.4 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77j.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_r ost.OSS.ost_io.nrs_tbf_rule=stop ost_w ost.OSS.ost_io.nrs_policies=fifo PASS 77j (61s) == sanityn test 77ja: check TBF-UID/GID NRS policy ======= 12:36:23 (1782491783) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg248-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg248-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg248-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg248-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg248-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.32798 s, 12.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0591868 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106564 s, 9.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183271 s, 5.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141261 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15982 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140813 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17492 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205425 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114813 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.338086 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.254155 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.065751 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192746 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.229981 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185743 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207503 s, 5.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0818625 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0840378 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0994732 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10378 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0920962 s, 11.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0969826 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116865 s, 9.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0831301 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.082907 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122069 s, 8.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0987024 s, 10.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0988442 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117368 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110793 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121155 s, 8.7 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg248-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg248-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9321 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9447 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg248-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg248-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg248-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg248-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg248-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.28415 s, 13.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0767905 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110092 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107284 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0994011 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135002 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130805 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150997 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17157 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14559 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.262678 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.266494 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.254638 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.267744 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.219934 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.293961 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23661 s, 4.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0830374 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0790497 s, 13.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.045806 s, 22.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0558199 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0425363 s, 24.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0842277 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0671454 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0909705 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0774152 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105688 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0898848 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0744765 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0706693 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125074 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.085248 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0794395 s, 13.2 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg248-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg248-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8547 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9255 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77ja (149s) == sanityn test 77jb: check TBF-UID/GID NRS policy on files that don't belong to us ========================================================== 12:38:52 (1782491932) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg248-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg248-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg248-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg248-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg248-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.27445 s, 13.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0954588 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0886241 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121762 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11524 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129007 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176415 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180503 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177504 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127032 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162517 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18358 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186714 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1101 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201093 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223625 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.226249 s, 4.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0203326 s, 51.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0424338 s, 24.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0533416 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0320004 s, 32.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0353554 s, 29.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0494426 s, 21.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0392034 s, 26.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0392938 s, 26.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0612963 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0243393 s, 43.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0515507 s, 20.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0848592 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0878282 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0616438 s, 17.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0927988 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.076975 s, 13.6 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg248-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg248-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8701 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9397 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg248-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg248-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg248-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg248-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg248-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.29588 s, 12.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137565 s, 7.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106521 s, 9.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15105 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.092594 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182718 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17424 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.204832 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139744 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196305 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16116 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.234107 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18923 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.288253 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221206 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.313329 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227676 s, 4.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0195606 s, 53.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0503753 s, 20.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0793138 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0536362 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0416002 s, 25.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0449921 s, 23.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0576173 s, 18.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0798246 s, 13.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0542988 s, 19.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0548758 s, 19.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0593876 s, 17.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0800363 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0588639 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.053345 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0863963 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0689839 s, 15.2 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg248-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg248-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9374 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.917 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77jb (148s) == sanityn test 77k: check TBF policy with UID/GID/JobID/OPCode expression ========================================================== 12:41:20 (1782492080) ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg248-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.50454 s, 33.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0718341 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0583373 s, 18.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216362 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.228846 s, 4.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205681 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.238303 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.27131 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.272759 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.312417 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.322851 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.336609 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.325516 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.31384 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.369533 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.337858 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.37643 s, 2.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163091 s, 64.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0204632 s, 51.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0247881 s, 42.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0282594 s, 37.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0575309 s, 18.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0302741 s, 34.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0403556 s, 26.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0322315 s, 32.5 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0202912 s, 51.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.041119 s, 25.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0390067 s, 26.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.038641 s, 27.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0445447 s, 23.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0473384 s, 22.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.049914 s, 21.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0521522 s, 20.1 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg248-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg248-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.64812 s, 18.6 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.90669 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.202.48@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.202.48@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.42598 s, 39.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0523169 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0578203 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0399891 s, 26.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0555413 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0789771 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169723 s, 6.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181812 s, 5.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201883 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20891 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.235508 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175603 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.248897 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170367 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.269786 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199427 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213791 s, 4.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffffa00411183000.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0255524 s, 41.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00799495 s, 131 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00248892 s, 421 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0502186 s, 20.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00546154 s, 192 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0852887 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0629165 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0695572 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0571222 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0599152 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0554422 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0520385 s, 20.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0508243 s, 20.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0316871 s, 33.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0410494 s, 25.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0554737 s, 18.9 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg248-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg248-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.8809 s, 17.8 MB/s Write runtime is 7 s, speed is 14.285714 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.86301 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.202.48@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg248-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.449956 s, 37.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0224807 s, 46.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0715891 s, 14.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0604956 s, 17.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0598031 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104809 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109542 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139768 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0610185 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113694 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158475 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0347151 s, 30.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0784375 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116894 s, 9.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101839 s, 10.3 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133899 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129324 s, 8.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.019147 s, 54.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0357517 s, 29.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0247406 s, 42.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0372224 s, 28.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0647854 s, 16.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.075039 s, 14.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0386027 s, 27.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0451462 s, 23.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0557695 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0493046 s, 21.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0590662 s, 17.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0700688 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0827475 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0703027 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0769314 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0827685 s, 12.7 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg248-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg248-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.94777 s, 17.6 MB/s Write runtime is 7 s, speed is 14.285714 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.03712 s, 20.8 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg248-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.520958 s, 32.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0570091 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0591916 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106149 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142325 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15511 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164799 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182184 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175933 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167393 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.224926 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205489 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212139 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.204947 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.236462 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.224813 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.249181 s, 4.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0518113 s, 20.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0441073 s, 23.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0279577 s, 37.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0605763 s, 17.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0385413 s, 27.2 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0392005 s, 26.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0401047 s, 26.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0413287 s, 25.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.045026 s, 23.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0434258 s, 24.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0726871 s, 14.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0464338 s, 22.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0505902 s, 20.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.053692 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0587935 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0589472 s, 17.8 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg248-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg248-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0532 s, 10.4 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.89381 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.97868 s, 17.5 MB/s Write runtime is 7 s, speed is 14.285714 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.88099 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg248-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.933428 s, 18.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0687704 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0958392 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106493 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122763 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173261 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159908 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14797 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.252979 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.217514 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.256577 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.225665 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23248 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227671 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207657 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.277636 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21734 s, 4.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.025681 s, 40.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0232151 s, 45.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0345629 s, 30.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0317975 s, 33.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136441 s, 76.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0652353 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0424527 s, 24.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0802913 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0664034 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.038292 s, 27.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0785366 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0465237 s, 22.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0466189 s, 22.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.017755 s, 59.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00862008 s, 122 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0102258 s, 103 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg248-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg248-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.754 s, 5.3 MB/s Write runtime is 20 s, speed is 5.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.587 s, 5.4 MB/s Read runtime is 20 s, speed is 5.000000 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_ug ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg248-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.445823 s, 37.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0587115 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0731985 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0590366 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168493 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174156 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165191 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223552 s, 4.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183599 s, 5.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.241566 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19309 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223498 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.235851 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.2079 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177024 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.234177 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.277299 s, 3.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0377257 s, 27.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0206648 s, 50.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0824091 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0627265 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0318134 s, 33.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.086807 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0449007 s, 23.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0449731 s, 23.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0942118 s, 11.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0739556 s, 14.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.080234 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0485215 s, 21.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0773828 s, 13.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0763673 s, 13.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0849648 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0850478 s, 12.3 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg248-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg248-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.97933 s, 17.5 MB/s Write runtime is 7 s, speed is 14.285714 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.8796 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg248-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.570619 s, 29.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0485315 s, 21.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0417577 s, 25.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0827649 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142681 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163607 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169994 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167439 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.204864 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205119 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19023 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183648 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215006 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18481 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.236984 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223666 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.235749 s, 4.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0494678 s, 21.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0303396 s, 34.6 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0571589 s, 18.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.046219 s, 22.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0446959 s, 23.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0523371 s, 20.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0854759 s, 12.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0595956 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.060393 s, 17.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0772717 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0614439 s, 17.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0647175 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0630925 s, 16.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0590468 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0753145 s, 13.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.082526 s, 12.7 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg248-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg248-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.96995 s, 10.5 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.89005 s, 10.6 MB/s Read runtime is 10 s, speed is 10.000000 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0207 s, 10.5 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.8961 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_policies=fifo PASS 77k (419s) == sanityn test 77kb: Check different granular TBF type combination: jobid+opcode ========================================================== 12:48:19 (1782492499) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/nrs_r_oleg248-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.799331 s, 21.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0833817 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102671 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199149 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.224514 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198115 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.218377 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222797 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.268604 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.307575 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.34545 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.37752 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.36338 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.378596 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.399001 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.463613 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.500456 s, 2.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffffa00411183000.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffffa00411183000.lock_unused_count=7 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0352364 s, 29.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0735254 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10044 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0840393 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0891136 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0529914 s, 19.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0868403 s, 12.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123141 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0626078 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143869 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0711674 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109366 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0844798 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0872479 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0901527 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14788 s, 7.1 MB/s removed '/mnt/lustre/d77kb.sanityn/nrs_r_oleg248-client.virtnet' removed '/mnt/lustre/d77kb.sanityn/nrs_w_oleg248-client.virtnet' removed directory '/mnt/lustre/d77kb.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.60638 s, 13.8 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.2234 s, 10.3 MB/s Read runtime is 12 s, speed is 8.333333 IOPS cleanup_tbf /mnt/lustre/d77kb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kb (56s) == sanityn test 77kc: Check different granular TBF type combination: nid+opcode ========================================================== 12:49:15 (1782492555) ost.OSS.ost_io.nrs_policies=tbf nid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.202.48@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.202.48@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.14828 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.090253 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140767 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180266 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.380192 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.44847 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.387305 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.447634 s, 2.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.438612 s, 2.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.441087 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.439024 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.455049 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.470435 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.4412 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.522406 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.470968 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.470558 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139726 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192385 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146793 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174583 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147167 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12419 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179285 s, 5.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183158 s, 5.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187789 s, 5.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184565 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164287 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172402 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190738 s, 5.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183516 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203951 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.224488 s, 4.7 MB/s removed '/mnt/lustre/d77kc.sanityn/nrs_r_oleg248-client.virtnet' removed '/mnt/lustre/d77kc.sanityn/nrs_w_oleg248-client.virtnet' removed directory '/mnt/lustre/d77kc.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.79727 s, 10.7 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.094 s, 10.4 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kc.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kc (55s) == sanityn test 77kd: Check different granular TBF type combination: nid+jobid ========================================================== 12:50:10 (1782492610) ost.OSS.ost_io.nrs_policies=tbf nid+jobid ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.202.48@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/nrs_r_oleg248-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.05489 s, 15.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0799709 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14495 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101862 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.217275 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.254567 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.260029 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.397996 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.383238 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202084 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193324 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.391721 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.450807 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.309145 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.231555 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0994965 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.089776 s, 11.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied ldlm.namespaces.lustre-OST0000-osc-ffffa00411183000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffffa00411183000.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0934254 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0958083 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148588 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108356 s, 9.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141648 s, 7.4 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0600232 s, 17.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151799 s, 6.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107691 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114331 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0893879 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0902214 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0944954 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106844 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127899 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153762 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165037 s, 6.4 MB/s removed '/mnt/lustre/d77kd.sanityn/nrs_w_oleg248-client.virtnet' removed '/mnt/lustre/d77kd.sanityn/nrs_r_oleg248-client.virtnet' removed directory '/mnt/lustre/d77kd.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.79475 s, 13.5 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kd.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.24935 s, 20.0 MB/s Read runtime is 7 s, speed is 14.285714 IOPS cleanup_tbf /mnt/lustre/d77kd.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kd (52s) == sanityn test 77ke: Check different granular TBF type combination: jobid+opcode ========================================================== 12:51:02 (1782492662) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/nrs_r_oleg248-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.717886 s, 23.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0616357 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0709045 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0972589 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162277 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164435 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162978 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198506 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209064 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207589 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221374 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.235484 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.250694 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.240519 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237992 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.266191 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.272136 s, 3.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108276 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102807 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0758967 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106683 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.06937 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0869251 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0898989 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112289 s, 9.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0841782 s, 12.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.093397 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133324 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0854 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107039 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114284 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0962148 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125004 s, 8.4 MB/s removed '/mnt/lustre/d77ke.sanityn/nrs_w_oleg248-client.virtnet' removed '/mnt/lustre/d77ke.sanityn/nrs_r_oleg248-client.virtnet' removed directory '/mnt/lustre/d77ke.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 12.0305 s, 8.7 MB/s Write runtime is 13 s, speed is 7.692307 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ke.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0439 s, 10.4 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.57442 s, 13.8 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.93513 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77ke (97s) == sanityn test 77kf: Check different granular TBF type combination: uid+gid ========================================================== 12:52:39 (1782492759) ost.OSS.ost_io.nrs_policies=tbf uid+gid ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/nrs_r_oleg248-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.19969 s, 14.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0607748 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0676561 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200421 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192859 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21407 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.258326 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.254023 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.283017 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.245669 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.309053 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.282535 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.314535 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.347538 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.332542 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.393828 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.38336 s, 2.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0892511 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130252 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0780721 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0942161 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0796945 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0966021 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0867538 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0816965 s, 12.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128216 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0908793 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121279 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160586 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103619 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142421 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122083 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141352 s, 7.4 MB/s removed '/mnt/lustre/d77kf.sanityn/nrs_r_oleg248-client.virtnet' removed '/mnt/lustre/d77kf.sanityn/nrs_w_oleg248-client.virtnet' removed directory '/mnt/lustre/d77kf.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.1215 s, 5.2 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kf.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9869 s, 5.2 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77kf.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kf (76s) == sanityn test 77kg: check different granular TBF type combination: uid+opcode ========================================================== 12:53:55 (1782492835) ost.OSS.ost_io.nrs_policies=tbf uid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg248-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.54087 s, 31.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155456 s, 6.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107856 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100096 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114267 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0594246 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144604 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143551 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.299118 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.301421 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.353038 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.402729 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.417114 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.320436 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.320397 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.330222 s, 3.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffffa00411183000.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffffa00411183000.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.360618 s, 2.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100592 s, 10.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0254695 s, 41.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111843 s, 9.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0862167 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113236 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0487079 s, 21.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0525324 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0529819 s, 19.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0584342 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0629273 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0965899 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146739 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0762542 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0779194 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.082264 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0853838 s, 12.3 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg248-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg248-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.1888 s, 11.4 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.93552 s, 10.6 MB/s Read runtime is 12 s, speed is 8.333333 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg248-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.02488 s, 16.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14009 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.214652 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150088 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.290013 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139299 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199004 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.361862 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.274103 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.307795 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.263353 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.265056 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.357299 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.383973 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.46435 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.515787 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.521053 s, 2.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0569694 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.057908 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106001 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0593133 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119891 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157997 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0881518 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125633 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.084844 s, 12.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0897768 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.07759 s, 13.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0918478 s, 11.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110262 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14456 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114032 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0925646 s, 11.3 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg248-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg248-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.3123 s, 9.3 MB/s Write runtime is 13 s, speed is 7.692307 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.91346 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 12.393 s, 8.5 MB/s Write runtime is 14 s, speed is 7.142857 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.94157 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kg (146s) == sanityn test 77kh: Verify that Project ID support for NRS TBF rule ========================================================== 12:56:21 (1782492981) using SAVE_PROJECT_SUPPORTED=0 Stopping clients: oleg248-client.virtnet /mnt/lustre (opts:) Stopping client oleg248-client.virtnet /mnt/lustre opts: Stopping clients: oleg248-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg248-client.virtnet /mnt/lustre2 opts: Stopping /mnt/lustre-mds1 (opts:-f) on oleg248-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg248-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg248-server sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime) proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) devtmpfs on /dev type devtmpfs (rw,nosuid,mode=755) securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime) tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev) devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000) tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755) tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755) cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd) pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime) bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700) cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset) cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio) cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids) cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma) cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices) cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event) cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer) cgroup on /sys/fs/cgroup/misc type cgroup (rw,nosuid,nodev,noexec,relatime,misc) cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb) cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio) cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory) cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct) none on /sys/kernel/tracing type tracefs (rw,relatime) configfs on /sys/kernel/config type configfs (rw,relatime) /dev/nbd0 on / type ext4 (ro,relatime,stripe=32) rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) mqueue on /dev/mqueue type mqueue (rw,relatime) hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M) debugfs on /sys/kernel/debug type debugfs (rw,relatime) none on /mnt type ramfs (rw,relatime) none on /var/lib/stateless/writable type tmpfs (rw,relatime) /dev/vda on /home/green/git/lustre-release type squashfs (ro,relatime) none on /var/cache/man type tmpfs (rw,relatime) none on /var/log type tmpfs (rw,relatime) none on /var/lib/dbus type tmpfs (rw,relatime) none on /tmp type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) none on /var/tmp type tmpfs (rw,relatime) none on /var/lib/NetworkManager type tmpfs (rw,relatime) none on /var/lib/systemd/random-seed type tmpfs (rw,relatime) none on /var/spool type tmpfs (rw,relatime) none on /var/lib/nfs type tmpfs (rw,relatime) none on /var/lib/gssproxy type tmpfs (rw,relatime) none on /var/lib/logrotate type tmpfs (rw,relatime) none on /etc type tmpfs (rw,relatime) none on /var/lib/rsyslog type tmpfs (rw,relatime) none on /root/.ssh type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) 192.168.200.253:/exports/state/oleg248-client.virtnet on /var/lib/stateless/state type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.202.48,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg248-client.virtnet/boot on /boot type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.202.48,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg248-client.virtnet/etc/kdump.conf on /etc/kdump.conf type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.202.48,local_lock=none,addr=192.168.200.253) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=378180k,mode=700) 192.168.200.253:/exports/testreports/65593/testresults/sanityn-zfs-rocky8.10_x86_64-rocky8.10_x86_64 on /tmp/testlogs type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.202.48,local_lock=none,addr=192.168.200.253) /dev/vda on /usr/sbin/mount.lustre type squashfs (ro,relatime) Checking servers environments Checking clients oleg248-client.virtnet environments /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1045: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg248-server' oleg248-server: oleg248-server.virtnet: executing load_modules_local oleg248-server: Loading modules from /home/green/git/lustre-release/lustre oleg248-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1045: echo: write error: Device or resource busy oleg248-server: detected 4 online CPUs by sysfs oleg248-server: MODOPTS_LIBCFS= oleg248-server: Force libcfs to create 2 CPU partitions Setup mgs, mdt, osts Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg248-server: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg248-client: oleg248-server: ssh exited with exit code 1 Started lustre-MDT0000 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg248-server: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg248-client: oleg248-server: ssh exited with exit code 1 Started lustre-OST0000 Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg248-server: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg248-client: oleg248-server: ssh exited with exit code 1 Started lustre-OST0001 Starting client: oleg248-client.virtnet: -o user_xattr,flock 192.168.202.148@tcp:/lustre /mnt/lustre Starting client oleg248-client.virtnet: -o user_xattr,flock 192.168.202.148@tcp:/lustre /mnt/lustre Started clients oleg248-client.virtnet: 192.168.202.148@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) Starting client: oleg248-client.virtnet: -o user_xattr,flock 192.168.202.148@tcp:/lustre /mnt/lustre2 Starting client oleg248-client.virtnet: -o user_xattr,flock 192.168.202.148@tcp:/lustre /mnt/lustre2 Started clients oleg248-client.virtnet: 192.168.202.148@tcp:/lustre on /mnt/lustre2 type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffffa00407e45000.idle_timeout=debug osc.lustre-OST0000-osc-ffffa0040a81c000.idle_timeout=debug osc.lustre-OST0001-osc-ffffa00407e45000.idle_timeout=debug osc.lustre-OST0001-osc-ffffa0040a81c000.idle_timeout=debug disable quota as required ost.OSS.ost_io.nrs_policies=tbf projid ost.OSS.ost_io.nrs_tbf_rule=start ext_proj_rw projid={100} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.552415 s, 30.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.078391 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0605894 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0552367 s, 19.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.086963 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160146 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209606 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176311 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.328028 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.328393 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.36842 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.268999 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.390815 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.27239 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.298192 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.42368 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.332299 s, 3.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffffa0040a81c000.lock_unused_count=2 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.352274 s, 3.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.369038 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.383033 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.366889 s, 2.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.406086 s, 2.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.392625 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.377739 s, 2.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.403284 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.433827 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.407245 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.406404 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.392254 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.421792 s, 2.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.42407 s, 2.5 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.418695 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.410818 s, 2.6 MB/s removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg248-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg248-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.05655 s, 11.6 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.52459 s, 19.0 MB/s Read runtime is 7 s, speed is 14.285714 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_proj_rw ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf projid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w projid={100}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r projid={100}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.557323 s, 30.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0900746 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0582911 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133248 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.06234 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223763 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.285917 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180425 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21685 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.278611 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.289415 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.356654 s, 2.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.304625 s, 3.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.402036 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.363861 s, 2.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffffa0040a81c000.lock_unused_count=3 ldlm.namespaces.lustre-OST0001-osc-ffffa0040a81c000.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.393798 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.431202 s, 2.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207118 s, 5.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153237 s, 6.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153698 s, 6.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158596 s, 6.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159645 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159587 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161841 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168759 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176843 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182796 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18741 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190572 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199739 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198771 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210631 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216612 s, 4.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffffa0040a81c000.lock_unused_count=1 removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg248-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg248-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.71105 s, 10.8 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.9324 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77kh (237s) == sanityn test 77ki: Add rule with unsupported TBF type should fail for generic TBF ========================================================== 13:00:19 (1782493219) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 oleg248-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_fail projid={100}&opcode={ost_write} rate=20: Invalid argument oleg248-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_fail projid={100}&opcode={ost_write} rate=20': Invalid argument pdsh@oleg248-client: oleg248-server: ssh exited with exit code 22 ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77ki (22s) == sanityn test 77l: check the output of NRS policies for generic TBF ========================================================== 13:00:41 (1782493241) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_policies=fifo PASS 77l (12s) == sanityn test 77m: check NRS Delay slows write RPC processing ========================================================== 13:00:53 (1782493253) ost.OSS.ost_io.nrs_policies=delay ost.OSS.ost_io.nrs_delay_min=4 ost.OSS.ost_io.nrs_delay_max=4 ost.OSS.ost_io.nrs_delay_pct=100 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg248-client.virtnet] [bs=1M] [count=1] [oflag=direct] [conv=fdatasync] 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 3.40555 s, 308 kB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg248-client.virtnet] [bs=1M] [count=10] [oflag=direct] [conv=fdatasync] 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 40.0824 s, 262 kB/s ost.OSS.ost_io.nrs_policies=fifo PASS 77m (59s) == sanityn test 77n: check wildcard support for TBF JobID NRS policy ========================================================== 13:01:53 (1782493313) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={*.500} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.738828 s, 22.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0973029 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0599385 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150228 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0652896 s, 16.1 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.047662 s, 22.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166372 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0788645 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.262854 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223427 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103897 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174864 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190373 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132354 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.265229 s, 4.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffffa0040a81c000.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffffa0040a81c000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109367 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221228 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0950856 s, 11.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffffa0040a81c000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0571745 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110321 s, 9.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120868 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0616869 s, 17.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0117605 s, 89.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116005 s, 9.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0911262 s, 11.5 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0892993 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0922312 s, 11.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110654 s, 9.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0817886 s, 12.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127888 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0929649 s, 11.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0795513 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0969793 s, 10.8 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg248-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg248-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77n.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.1836 s, 14.6 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77n.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.52922 s, 19.0 MB/s Read runtime is 7 s, speed is 14.285714 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.*} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.5081 s, 33.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0764277 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0526568 s, 19.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0869278 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14995 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18953 s, 5.5 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184039 s, 5.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.228137 s, 4.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207742 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.276029 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.280697 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.319208 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.259981 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.354218 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.316258 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.375923 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.354424 s, 3.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffffa0040a81c000.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffffa0040a81c000.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.324124 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.335087 s, 3.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.383512 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.331111 s, 3.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.378822 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.335111 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.357342 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.402491 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.35874 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.349833 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.351097 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.427207 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.371292 s, 2.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.375986 s, 2.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.370463 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.386334 s, 2.7 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg248-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg248-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.52555 s, 12.3 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.3998 s, 19.4 MB/s Read runtime is 7 s, speed is 14.285714 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_policies=fifo PASS 77n (91s) == sanityn test 77o: Changing rank should not panic ====== 13:03:24 (1782493404) mds.MDS.mdt.nrs_policies=tbf nid mds.MDS.mdt.nrs_tbf_rule=start name nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=start name1 nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=change name1 rank=name mds.MDS.mdt.nrs_tbf_rule=stop name mds.MDS.mdt.nrs_policies=fifo PASS 77o (13s) == sanityn test 77q: Parallel TBF rule definitions should not panic ========================================================== 13:03:37 (1782493417) mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_policies=fifo PASS 77q (119s) == sanityn test 77p: Check validity of rule names for TBF policies ========================================================== 13:05:36 (1782493536) mds.MDS.mdt.nrs_policies=tbf oleg248-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_overflo uid={500} rate=500: File name too long oleg248-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_overflo uid={500} rate=500': File name too long pdsh@oleg248-client: oleg248-server: ssh exited with exit code 36 oleg248-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start uid={500} rate=500: Invalid argument oleg248-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start uid={500} rate=500': Invalid argument pdsh@oleg248-client: oleg248-server: ssh exited with exit code 22 oleg248-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_empty: Invalid argument oleg248-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_empty': Invalid argument pdsh@oleg248-client: oleg248-server: ssh exited with exit code 22 oleg248-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p@spec uid={500} rate=500: Invalid argument oleg248-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p@spec uid={500} rate=500': Invalid argument pdsh@oleg248-client: oleg248-server: ssh exited with exit code 22 oleg248-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p.spec uid={500} rate=500: Invalid argument oleg248-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p.spec uid={500} rate=500': Invalid argument pdsh@oleg248-client: oleg248-server: ssh exited with exit code 22 oleg248-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p~spec uid={500} rate=500: Invalid argument oleg248-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p~spec uid={500} rate=500': Invalid argument pdsh@oleg248-client: oleg248-server: ssh exited with exit code 22 oleg248-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p#spec uid={500} rate=500: Invalid argument oleg248-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p#spec uid={500} rate=500': Invalid argument pdsh@oleg248-client: oleg248-server: ssh exited with exit code 22 oleg248-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p/spec uid={500} rate=500: Invalid argument oleg248-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p/spec uid={500} rate=500': Invalid argument pdsh@oleg248-client: oleg248-server: ssh exited with exit code 22 oleg248-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p^spec uid={500} rate=500: Invalid argument oleg248-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p^spec uid={500} rate=500': Invalid argument pdsh@oleg248-client: oleg248-server: ssh exited with exit code 22 oleg248-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p%spec uid={500} rate=500: Invalid argument oleg248-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p%spec uid={500} rate=500': Invalid argument pdsh@oleg248-client: oleg248-server: ssh exited with exit code 22 oleg248-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p*spec uid={500} rate=500: Invalid argument oleg248-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p*spec uid={500} rate=500': Invalid argument pdsh@oleg248-client: oleg248-server: ssh exited with exit code 22 oleg248-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p;spec uid={500} rate=500: Invalid argument oleg248-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p;spec uid={500} rate=500': Invalid argument pdsh@oleg248-client: oleg248-server: ssh exited with exit code 22 oleg248-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p,spec uid={500} rate=500: Invalid argument oleg248-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p,spec uid={500} rate=500': Invalid argument pdsh@oleg248-client: oleg248-server: ssh exited with exit code 22 oleg248-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p?spec uid={500} rate=500: Invalid argument oleg248-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p?spec uid={500} rate=500': Invalid argument pdsh@oleg248-client: oleg248-server: ssh exited with exit code 22 oleg248-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77pspec uid={500} rate=500: Invalid argument oleg248-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p>spec uid={500} rate=500': Invalid argument pdsh@oleg248-client: oleg248-server: ssh exited with exit code 22 oleg248-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p:spec uid={500} rate=500: Invalid argument oleg248-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p:spec uid={500} rate=500': Invalid argument pdsh@oleg248-client: oleg248-server: ssh exited with exit code 22 oleg248-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p+spec uid={500} rate=500: Invalid argument oleg248-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p+spec uid={500} rate=500': Invalid argument pdsh@oleg248-client: oleg248-server: ssh exited with exit code 22 oleg248-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p=spec uid={500} rate=500: Invalid argument oleg248-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p=spec uid={500} rate=500': Invalid argument pdsh@oleg248-client: oleg248-server: ssh exited with exit code 22 oleg248-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p)spec uid={500} rate=500: Invalid argument oleg248-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p)spec uid={500} rate=500': Invalid argument pdsh@oleg248-client: oleg248-server: ssh exited with exit code 22 oleg248-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p(spec uid={500} rate=500: Invalid argument oleg248-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p(spec uid={500} rate=500': Invalid argument pdsh@oleg248-client: oleg248-server: ssh exited with exit code 22 oleg248-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p{spec uid={500} rate=500: Invalid argument oleg248-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p{spec uid={500} rate=500': Invalid argument pdsh@oleg248-client: oleg248-server: ssh exited with exit code 22 oleg248-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p}spec uid={500} rate=500: Invalid argument oleg248-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p}spec uid={500} rate=500': Invalid argument pdsh@oleg248-client: oleg248-server: ssh exited with exit code 22 oleg248-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p|spec uid={500} rate=500: Invalid argument oleg248-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p|spec uid={500} rate=500': Invalid argument pdsh@oleg248-client: oleg248-server: ssh exited with exit code 22 oleg248-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p[spec uid={500} rate=500: Invalid argument oleg248-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p[spec uid={500} rate=500': Invalid argument pdsh@oleg248-client: oleg248-server: ssh exited with exit code 22 oleg248-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p]spec uid={500} rate=500: Invalid argument oleg248-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p]spec uid={500} rate=500': Invalid argument pdsh@oleg248-client: oleg248-server: ssh exited with exit code 22 oleg248-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p!spec uid={500} rate=500: Invalid argument oleg248-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p!spec uid={500} rate=500': Invalid argument pdsh@oleg248-client: oleg248-server: ssh exited with exit code 22 oleg248-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p&spec uid={500} rate=500: Invalid argument oleg248-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p&spec uid={500} rate=500': Invalid argument pdsh@oleg248-client: oleg248-server: ssh exited with exit code 22 oleg248-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p$spec uid={500} rate=500: Invalid argument oleg248-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p$spec uid={500} rate=500': Invalid argument pdsh@oleg248-client: oleg248-server: ssh exited with exit code 22 oleg248-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p`spec uid={500} rate=500: Invalid argument oleg248-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p`spec uid={500} rate=500': Invalid argument pdsh@oleg248-client: oleg248-server: ssh exited with exit code 22 oleg248-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p\spec uid={500} rate=500: Invalid argument oleg248-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p\spec uid={500} rate=500': Invalid argument pdsh@oleg248-client: oleg248-server: ssh exited with exit code 22 mds.MDS.mdt.nrs_policies=fifo PASS 77p (43s) == sanityn test 77r: Change type of tbf policy at run time ========================================================== 13:06:19 (1782493579) mds.MDS.mdt.nrs_policies=tbf uid mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf gid mds.MDS.mdt.nrs_tbf_rule=start md_rule gid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start md_rule jobid={*.500} rate=1 mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=fifo check the number of requests in queue: fifo: 0 crrn: 0 tbf: 0 delay: 0 fifo: 0 crrn: 0 tbf: 0 delay: 0 cleanup 77r 269766 kill md_thread xargs PASS 77r (57s) == sanityn test 77s: Check TBF LRU shrinker ============== 13:07:17 (1782493637) jobid_name=%e.%p jobid_var=TEST_77s_JOBVAR mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start touch jobid={touch.*} rate=1000 mds.MDS.mdt.nrs_tbf_rule=start unlink jobid={unlink.*} rate=1000 touch jobid={touch.*} 1000, ref 10 touch jobid={touch.*} 1000, ref 0 unlink jobid={unlink.*} 1000, ref 10 unlink jobid={unlink.*} 1000, ref 0 mds.MDS.mdt.nrs_policies=fifo jobid_var=procname_uid jobid_name=%e.%u PASS 77s (69s) == sanityn test 78: Enable policy and specify tunings right away ========================================================== 13:08:25 (1782493705) ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_orr_quantum=1 ost.OSS.ost_io.nrs_policies=fifo PASS 78 (10s) == sanityn test 79: xattr: intent error ================== 13:08:35 (1782493715) File: /mnt/lustre/d79.sanityn Size: 512 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115238826935626 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-06-26 13:08:37.000000000 -0400 Modify: 2026-06-26 13:08:37.000000000 -0400 Change: 2026-06-26 13:08:37.000000000 -0400 Birth: 2026-06-26 13:08:36.000000000 -0400 fail_val=0 fail_loc=0x80000160 fail_val=0 fail_loc=0x80000131 # file: mnt/lustre/d79.sanityn trusted.name1="value1" PASS 79 (18s) == sanityn test 80a: migrate directory when some children is being opened ========================================================== 13:08:53 (1782493733) SKIP: sanityn test_80a needs >= 2 MDTs SKIP 80a (3s) == sanityn test 80b: Accessing directory during migration ========================================================== 13:08:57 (1782493737) SKIP: sanityn test_80b needs >= 2 MDTs SKIP 80b (3s) == sanityn test 81a: rename and stat under striped directory ========================================================== 13:09:00 (1782493740) SKIP: sanityn test_81a needs >= 2 MDTs SKIP 81a (3s) == sanityn test 81b: rename under striped directory doesn't deadlock ========================================================== 13:09:04 (1782493744) SKIP: sanityn test_81b We need at least 2 MDTs for this test SKIP 81b (4s) == sanityn test 81c: rename revoke LOOKUP lock for remote object ========================================================== 13:09:07 (1782493747) SKIP: sanityn test_81c needs >= 4 MDTs SKIP 81c (4s) == sanityn test 81d: parallel rename file cross-dir on same MDT ========================================================== 13:09:11 (1782493751) - open/close 614 (time 1782493767.22 total 10.02 last 61.30) - open/close 1320 (time 1782493777.23 total 20.03 last 70.53) - open/close 2035 (time 1782493787.24 total 30.04 last 71.40) total: 2500 open/close in 36.73 seconds: 68.07 ops/second snapshot_time 1782493888.472704338 secs.nsecs start_time 1782493753.951960133 secs.nsecs elapsed_time 134.520744205 secs.nsecs open 2505 samples [usecs] 217 31295 3849837 10822320257 close 2505 samples [usecs] 188 13327 923671 886082149 mknod 2501 samples [usecs] 582 31183 3386307 8711550471 unlink 2500 samples [usecs] 702 92635 6117996 88125225462 mkdir 2 samples [usecs] 1513 2949 4462 10985770 rmdir 2 samples [usecs] 1136 17237 18373 298404665 rename 2500 samples [usecs] 508 49513 16886666 240188836216 getattr 3761 samples [usecs] 6 8219 222088 282243750 parallel_rename_file 2500 samples [usecs] 508 49513 16886666 240188836216 crossdir_rename 2500 samples [usecs] 508 49513 16886666 240188836216 PASS 81d (144s) == sanityn test 82: fsetxattr and fgetxattr on orphan files ========================================================== 13:11:36 (1782493896) multiop /mnt/lustre/f82.sanityn vO_ac TMPPIPE=/tmp/multiop_open_wait_pipe.7587 multiop /mnt/lustre2/f82.sanityn vo_Ac TMPPIPE=/tmp/multiop_open_wait_pipe.7587 PASS 82 (9s) == sanityn test 83: access striped directory while it is being created/unlinked ========================================================== 13:11:44 (1782493904) SKIP: sanityn test_83 needs >= 2 MDTs SKIP 83 (4s) == sanityn test 84: 0-nlink race in lu_object_find() ===== 13:11:48 (1782493908) fail_loc=0x8000060b PASS 84 (13s) == sanityn test 85: Lustre API root cache race =========== 13:12:01 (1782493921) Starting: llapi_root_test /mnt/lustre/f85.sanityn /mnt/lustre2/f85.sanityn Starting test test1 at 1782493923 Finishing test test1 at 1782493928 PASS 85 (13s) == sanityn test 90: open/create and unlink striped directory ========================================================== 13:12:14 (1782493934) SKIP: sanityn test_90 needs >= 2 MDTs SKIP 90 (4s) == sanityn test 91: chmod and unlink striped directory === 13:12:18 (1782493938) SKIP: sanityn test_91 needs >= 2 MDTs SKIP 91 (4s) == sanityn test 92: create remote directory under orphan directory ========================================================== 13:12:22 (1782493942) SKIP: sanityn test_92 needs >= 2 MDTs SKIP 92 (4s) == sanityn test 93: alloc_rr should not allocate on same ost ========================================================== 13:12:26 (1782493946) fail_loc=0x00000163 fail_loc=0x0 /mnt/lustre/f93.sanityn-1/file1 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5763 0x1683 0x240000400 1 5726 0x165e 0x280000400 /mnt/lustre2/f93.sanityn-2/file2 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 5727 0x165f 0x280000400 0 5764 0x1684 0x240000400 rc1=0 and rc2=0 PASS 93 (17s) == sanityn test 94: signal vs CP callback race =========== 13:12:43 (1782493963) osc.lustre-OST0000-osc-ffffa00407e45000.idle_timeout=0 osc.lustre-OST0000-osc-ffffa0040a81c000.idle_timeout=0 osc.lustre-OST0001-osc-ffffa00407e45000.idle_timeout=0 osc.lustre-OST0001-osc-ffffa0040a81c000.idle_timeout=0 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.323111 s, 25.4 kB/s fail_val=5 fail_loc=0x80000312 fail_val=6 fail_loc=0x80000329 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 2.98284 s, 1.4 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 6178: 286524 Alarm clock dd if=/dev/zero of=$DIR/$tfile conv=notrunc oflag=append bs=4k count=1 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0656724 s, 62.4 kB/s osc.lustre-OST0000-osc-ffffa00407e45000.idle_timeout=debug osc.lustre-OST0000-osc-ffffa0040a81c000.idle_timeout=debug osc.lustre-OST0001-osc-ffffa00407e45000.idle_timeout=debug osc.lustre-OST0001-osc-ffffa0040a81c000.idle_timeout=debug PASS 94 (16s) == sanityn test 95a: Check readpage() on a page that was removed from page cache ========================================================== 13:12:59 (1782493979) llite.lustre-ffffa00407e45000.fast_read=0 llite.lustre-ffffa0040a81c000.fast_read=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0865008 s, 24.2 MB/s fail_loc=0x80001421 fail_val=7 fail_loc=0x80001422 fail_val=10 llite.lustre-ffffa00407e45000.fast_read=1 llite.lustre-ffffa0040a81c000.fast_read=1 PASS 95a (20s) == sanityn test 95b: Check readpage() on a page that is no longer uptodate ========================================================== 13:13:20 (1782494000) llite.lustre-ffffa00407e45000.fast_read=0 llite.lustre-ffffa0040a81c000.fast_read=0 1+0 records in 1+0 records out 12288 bytes (12 kB, 12 KiB) copied, 0.00582441 s, 2.1 MB/s fail_loc=0x80001424 fail_val=5 fail_loc=0 llite.lustre-ffffa00407e45000.fast_read=1 llite.lustre-ffffa0040a81c000.fast_read=1 PASS 95b (12s) == sanityn test 100a: DoM: glimpse RPCs for stat without IO lock (DoM only file) ========================================================== 13:13:33 (1782494013) SKIP: sanityn test_100a Reserved for glimpse-ahead SKIP 100a (3s) == sanityn test 100b: DoM: no glimpse RPC for stat with IO lock (DoM only file) ========================================================== 13:13:36 (1782494016) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00629564 s, 651 kB/s /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK PASS 100b (7s) == sanityn test 100c: DoM: write vs stat without IO lock (combined file) ========================================================== 13:13:43 (1782494023) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.137638 s, 15.2 MB/s /mnt/lustre/d100c.sanityn/dom has type file OK /mnt/lustre/d100c.sanityn/dom has size 2097152 OK PASS 100c (7s) == sanityn test 100d: DoM: write+truncate vs stat without IO lock (combined file) ========================================================== 13:13:50 (1782494030) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.213075 s, 9.8 MB/s /mnt/lustre/d100d.sanityn/dom has type file OK /mnt/lustre/d100d.sanityn/dom has size 4096 OK PASS 100d (8s) == sanityn test 100e: DoM: read on open and file size ==== 13:13:58 (1782494038) 1+0 records in 1+0 records out 12000 bytes (12 kB, 12 KiB) copied, 0.00528503 s, 2.3 MB/s PASS 100e (7s) == sanityn test 101a: Discard DoM data on unlink ========= 13:14:05 (1782494045) /mnt/lustre/f101a.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0116607 s, 351 kB/s PASS 101a (7s) == sanityn test 101b: Discard DoM data on rename ========= 13:14:12 (1782494052) /mnt/lustre/f101b.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0191649 s, 214 kB/s PASS 101b (7s) == sanityn test 101c: Discard DoM data on close-unlink === 13:14:19 (1782494059) /mnt/lustre/f101c.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0123988 s, 330 kB/s PASS 101c (8s) SKIP: sanityn test_102 skipping ALWAYS excluded test 102 == sanityn test 103: Test size correctness with lockahead ========================================================== 13:14:28 (1782494068) fail_loc=0x415 fail_loc=0x415 fail_val=2 Incorrect size expected (no glimpse fix): Starting test test23 at 1782494071 lockahead_test: lockahead_test.c:1171: test23: assertion 'sb.st_size == sb2.st_size' failed: size on /mnt/lustre/d103.sanityn/f103.sanityn and /mnt/lustre2/d103.sanityn/f103.sanityn differs: 0 vs 1048576 fail_loc=0 fail_loc=0x214 fail_val=2 Starting test test23 at 1782494075 Finishing test test23 at 1782494075 PASS 103 (15s) == sanityn test 104: Verify that MDS stores atime/mtime/ctime during close ========================================================== 13:14:43 (1782494083) SKIP: sanityn test_104 ldiskfs only test SKIP 104 (3s) == sanityn test 105: Glimpse and lock cancel race ======== 13:14:46 (1782494086) fail_loc=0x416 PASS 105 (27s) == sanityn test 106a: Verify the btime via statx() ======= 13:15:13 (1782494113) SKIP: sanityn test_106a Test only for ldiskfs and statx() supported SKIP 106a (3s) == sanityn test 106b: Glimpse RPCs test for statx ======== 13:15:16 (1782494116) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00889169 s, 115 kB/s File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238826938161 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-06-26 13:15:18.000000000 -0400 Modify: 2026-06-26 13:15:18.000000000 -0400 Change: 2026-06-26 13:15:18.000000000 -0400 Birth: 2026-06-26 13:15:17.000000000 -0400 /mnt/lustre/f106b.sanityn 144115238826938161 -rw-r--r-- 1 0 0 1782494117 1782494118 1782494118 File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238826938161 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-06-26 13:15:18.000000000 -0400 Modify: 2026-06-26 13:15:18.000000000 -0400 Change: 2026-06-26 13:15:18.000000000 -0400 Birth: 2026-06-26 13:15:17.000000000 -0400 1782494118 1024 1 PASS 106b (7s) == sanityn test 106c: Verify statx attributes mask ======= 13:15:23 (1782494123) PASS 106c (6s) == sanityn test 107a: Basic grouplock conflict =========== 13:15:29 (1782494129) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.342195 s, 30.6 MB/s /mnt/lustre/f107a.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5772 0x168c 0x240000400 multiop /mnt/lustre/f107a.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7587 multiop /mnt/lustre2/f107a.sanityn vO_G16022000r10g16022000c TMPPIPE=/tmp/multiop_open_wait_pipe.7587 First grouplock blocks second one PASS 107a (10s) == sanityn test 107b: Grouplock is added to the head of waiting list ========================================================== 13:15:39 (1782494139) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.324739 s, 32.3 MB/s /mnt/lustre/f107b.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 5734 0x1666 0x280000400 multiop /mnt/lustre/f107b.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7587 Grouplock blocks IO First grouplock blocks second one Second grouplock blocks IO PASS 107b (13s) == sanityn test 108a: lseek: parallel updates ============ 13:15:52 (1782494152) 1+0 records in 1+0 records out 10000 bytes (10 kB, 9.8 KiB) copied, 0.0114122 s, 876 kB/s llite.lustre-ffffa00407e45000.hybrid_io=0 llite.lustre-ffffa0040a81c000.hybrid_io=0 fail_val=4 fail_loc=0x80000414 1+0 records in 1+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 4.57631 s, 1.8 MB/s PASS 108a (12s) == sanityn test 109: Race with several mount instances on 1 node ========================================================== 13:16:04 (1782494164) 192.168.202.148@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg248-client.virtnet /mnt/lustre (opts:) 192.168.202.148@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg248-client.virtnet /mnt/lustre2 (opts:) Starting race between client mounts (duration 30) Iteration 0 Starting client: oleg248-client.virtnet: -o user_xattr,flock 192.168.202.148@tcp:/lustre /mnt/lustre Starting client: oleg248-client.virtnet: -o user_xattr,flock 192.168.202.148@tcp:/lustre /mnt/lustre2 192.168.202.148@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.148@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg248-client.virtnet /mnt/lustre (opts:) Stopping client oleg248-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1045: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg248-server' oleg248-server: oleg248-server.virtnet: executing load_modules_local oleg248-server: Loading modules from /home/green/git/lustre-release/lustre oleg248-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1045: echo: write error: Device or resource busy oleg248-server: detected 4 online CPUs by sysfs oleg248-server: MODOPTS_LIBCFS= oleg248-server: Force libcfs to create 2 CPU partitions Iteration 1 Starting client: oleg248-client.virtnet: -o user_xattr,flock 192.168.202.148@tcp:/lustre /mnt/lustre Starting client: oleg248-client.virtnet: -o user_xattr,flock 192.168.202.148@tcp:/lustre /mnt/lustre2 192.168.202.148@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.148@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg248-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg248-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1045: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg248-server' oleg248-server: oleg248-server.virtnet: executing load_modules_local oleg248-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1045: echo: write error: Device or resource busy oleg248-server: Loading modules from /home/green/git/lustre-release/lustre oleg248-server: detected 4 online CPUs by sysfs oleg248-server: MODOPTS_LIBCFS= oleg248-server: Force libcfs to create 2 CPU partitions Starting client: oleg248-client.virtnet: -o user_xattr,flock 192.168.202.148@tcp:/lustre /mnt/lustre Starting client: oleg248-client.virtnet: -o user_xattr,flock 192.168.202.148@tcp:/lustre /mnt/lustre2 PASS 109 (60s) == sanityn test 110: do not grant another lock on resend ========================================================== 13:17:04 (1782494224) fail_loc=0x525 fail_val=3 fail_loc=0x534 File: /mnt/lustre/f110.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238860488707 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-06-26 13:17:06.000000000 -0400 Modify: 2026-06-26 13:17:06.000000000 -0400 Change: 2026-06-26 13:17:06.000000000 -0400 Birth: 2026-06-26 13:17:05.000000000 -0400 fail_loc=0 fail_val=0 Connected clients: oleg248-client.virtnet oleg248-client.virtnet PASS 110 (169s) == sanityn test 111: A racy rename/link an open file should not cause fs corruption ========================================================== 13:19:53 (1782494393) SKIP: sanityn test_111 needs >= 2 MDTs SKIP 111 (4s) == sanityn test 112: update max-inherit in default LMV === 13:19:57 (1782494397) SKIP: sanityn test_112 We need at least 2 MDTs for this test SKIP 112 (3s) == sanityn test 113: check servers of specified fs ======= 13:20:01 (1782494401) lustre-OST0000-osc-ffffa00407d5e800 active. lustre-OST0001-osc-ffffa00407d5e800 active. lustre-OST0000-osc-ffffa00407d5e800 active. lustre-OST0001-osc-ffffa00407d5e800 active. MGC192.168.202.148@tcp active. PASS 113 (9s) == sanityn test 114: implicit default LMV inherit ======== 13:20:09 (1782494409) SKIP: sanityn test_114 We need at least 2 MDTs for this test SKIP 114 (3s) == sanityn test 115: ldiskfs doesn't check direntry for uniqueness ========================================================== 13:20:12 (1782494412) SKIP: sanityn test_115 ldiskfs only test SKIP 115 (4s) == sanityn test 116: DNE: Set default LMV layout from a remote client ========================================================== 13:20:16 (1782494416) SKIP: sanityn test_116 needs >= 2 MDTs SKIP 116 (2s) == sanityn test 121: trunc append race =================== 13:20:19 (1782494419) 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0165658 s, 1.2 MB/s fail_loc=0x80001436 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.09102 s, 225 kB/s 0000000 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0005000 0000 0000 0000 0000 0000 0000 0000 0000 * 000c800 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0011800 PASS 121 (8s) == sanityn test 200: service remains healthy while able to process request ========================================================== 13:20:27 (1782494427) at_max=10 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=9 fail_loc=0x1e1 fail_val=20 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1076 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1165 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1571 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.2012 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1896 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1836 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1966 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1275 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1849 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.2212 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.218 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.2573 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.2457 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.2506 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2629 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.3321 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.3911 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.4025 s, 0.1 kB/s at_max=600 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=116 oleg248-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffffa00407d5e800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffffa00407d5e800.ost_server_uuid in FULL state after 0 sec PASS 200 (81s) cleanup: ====================================================== == sanityn test complete, duration 9367 sec ============== 13:21:50 (1782494510) === sanityn: start cleanup 13:21:51 (1782494511) === Stopping clients: oleg248-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg248-client.virtnet /mnt/lustre2 opts: === sanityn: finish cleanup 13:25:42 (1782494742) ===