-----============= acceptance-small: sanityn ============----- Thu Sep 3 14:42:23 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg127-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanityn.*ex': No such file or directory excepting tests: 27 40a skipping tests SLOW=no: 33a === sanityn: start setup 14:42:47 (1788460967) === Starting client oleg127-client.virtnet: -o user_xattr,flock 192.168.201.127@tcp:/lustre /mnt/lustre2 Mount client oleg127-client.virtnet: mount -t lustre -o user_xattr,flock 192.168.201.127@tcp:/lustre /mnt/lustre2 Started clients oleg127-client.virtnet: 192.168.201.127@tcp:/lustre on /mnt/lustre2 type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) oleg127-client.virtnet: executing check_config_client /mnt/lustre oleg127-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg127-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8b4442ab6800.idle_timeout=debug osc.lustre-OST0000-osc-ffff8b44437aa800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8b4442ab6800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8b44437aa800.idle_timeout=debug disable quota as required oleg127-server: oleg127-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanityn: finish setup 14:43:22 (1788461002) === 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0330418 s, 31.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7649] == sanityn test 0: do_node_vp() and do_facet_vp() do the right thing ========================================================== 14:43:24 (1788461004) PASS 0 (10s) == sanityn test 1: Check attribute updates on 2 mount points ========================================================== 14:43:34 (1788461014) /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0777 OK /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0666 OK /mnt/lustre/f1.sanityn: absent OK PASS 1 (8s) == sanityn test 2a: check cached attribute updates on 2 mtpt's ================================================================== 14:43:42 (1788461022) -rw-r--r-- 1 root root 0 Sep 3 14:43 /mnt/lustre2/f2a /mnt/lustre/f2a has type file OK /mnt/lustre/f2a has perms 0777 OK PASS 2a (8s) == sanityn test 2b: check cached attribute updates on 2 mtpt's ================================================================== 14:43:50 (1788461030) -rw-r--r-- 1 root root 0 Sep 3 14:43 /mnt/lustre2/f2b /mnt/lustre2/f2b has type file OK /mnt/lustre2/f2b has perms 0777 OK PASS 2b (7s) == sanityn test 2c: check cached attribute updates on 2 mtpt's root ============================================================= 14:43:57 (1788461037) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0777 OK PASS 2c (6s) == sanityn test 2d: check cached attribute updates on 2 mtpt's root ============================================================= 14:44:04 (1788461044) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0755 OK PASS 2d (8s) == sanityn test 2e: check chmod on root is propagated to others ========================================================== 14:44:11 (1788461051) total 1 -rwxrwxrwx 1 root root 0 Sep 3 14:43 f2a -rwxrwxrwx 1 root root 0 Sep 3 14:43 f2b total 1 -rwxrwxrwx 1 root root 0 Sep 3 14:43 f2a -rwxrwxrwx 1 root root 0 Sep 3 14:43 f2b running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre2/f2e.sanityn] [count=1] 1+0 records in 1+0 records out 512 bytes copied, 0.00940938 s, 54.4 kB/s PASS 2e (8s) == sanityn test 2f: check attr/owner updates on DNE with 2 mtpt's ========================================================== 14:44:19 (1788461059) SKIP: sanityn test_2f needs >= 2 MDTs SKIP 2f (4s) == sanityn test 2g: check blocks update on sync write ==== 14:44:23 (1788461063) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.582858 s, 3.6 MB/s /mnt/lustre/f2g.sanityn has 4103 blocks /mnt/lustre2/f2g.sanityn has 4103 blocks PASS 2g (8s) == sanityn test 3: symlink on one mtpt, readlink on another ===================================================================== 14:44:31 (1788461071) PASS 3 (6s) == sanityn test 4: fstat validation on multiple mount points ==================================================================== 14:44:37 (1788461077) PASS 4 (8s) == sanityn test 5: create a file on one mount, truncate it on the other ========================================================== 14:44:45 (1788461085) /mnt/lustre/f5 has type file OK /mnt/lustre/f5 has size 100 OK PASS 5 (6s) == sanityn test 6: remove of open file on other node ============================================================================ 14:44:52 (1788461092) opening writing unlinking /mnt/lustre2/f6.sanityn accessing (1) seeking (1) accessing (2) fstat... reading comparing data truncating seeking (2) writing again seeking (3) reading again comparing data again closing SUCCESS - goto beer PASS 6 (7s) == sanityn test 7: remove of open directory on other node ======================================================================= 14:44:59 (1788461099) creating directory /mnt/lustre/d7 opening directory unlinking /mnt/lustre/d7 Ok, everything goes well. PASS 7 (6s) == sanityn test 8: remove of open special file on other node ==================================================================== 14:45:05 (1788461105) creating special file /mnt/lustre/f8.sanityn opening file unlinking /mnt/lustre/f8.sanityn Ok, everything goes well. PASS 8 (7s) == sanityn test 9a: append of file with sub-page size on multiple mounts ========================================================== 14:45:12 (1788461112) PASS 9a (7s) == sanityn test 9b: append to striped sparse file ======== 14:45:19 (1788461119) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.044471 s, 23.6 MB/s 3+0 records in 3+0 records out 3 bytes copied, 0.0747132 s, 0.0 kB/s Data read (expecting 'foo'): foo PASS 9b (8s) == sanityn test 10a: write of file with sub-page size on multiple mounts ========================================================== 14:45:27 (1788461127) 1+0 records in 1+0 records out 1 byte copied, 0.00640811 s, 0.2 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.011117 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00974592 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0176862 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0083565 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00699235 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00915222 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0106053 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00916006 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0154197 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00906432 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00813893 s, 0.1 kB/s PASS 10a (8s) == sanityn test 10b: write of file with sub-page size on multiple mounts ========================================================== 14:45:35 (1788461135) 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.0384721 s, 79.9 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0771395 s, 53.1 kB/s 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.000212295 s, 14.5 MB/s PASS 10b (9s) == sanityn test 11: execution of file opened for write should return error ============================================================== 14:45:44 (1788461144) multiop /mnt/lustre/d11/f vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7649 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 298: /mnt/lustre2/d11/f: Text file busy PASS 11 (9s) == sanityn test 12: test lock ordering (link, stat, unlink) ========================================================== 14:45:53 (1788461153) warning: '-runtime' deprecated, use '-t runtime' instead - link 347 (time 1788461165.83 total 10.00 last 34.70) - link 679 (time 1788461175.84 total 20.01 last 33.17) - link 979 (time 1788461185.85 total 30.03 last 29.95) - link 1332 (time 1788461195.88 total 40.05 last 35.21) - link 1644 (time 1788461205.88 total 50.05 last 31.19) - link 1994 (time 1788461215.90 total 60.07 last 34.95) - link 2338 (time 1788461225.90 total 70.07 last 34.39) - link 2720 (time 1788461235.91 total 80.08 last 38.16) - link 3051 (time 1788461245.91 total 90.09 last 33.09) total: 3404 link in 100.01 seconds: 34.04 ops/second using seed 3621309329 running for 100 seconds - stat 5562 (time 1788461167 ; total 11 ; last 11) - stat 10000 (time 1788461174 ; total 18 ; last 7) - stat 15132 (time 1788461185 ; total 29 ; last 11) - stat 20000 (time 1788461193 ; total 37 ; last 8) - stat 26460 (time 1788461204 ; total 48 ; last 11) - stat 30000 (time 1788461211 ; total 55 ; last 7) - stat 32909 (time 1788461222 ; total 66 ; last 11) - stat 36604 (time 1788461233 ; total 77 ; last 11) - stat 39972 (time 1788461244 ; total 88 ; last 11) - stat 40000 (time 1788461244 ; total 88 ; last 0) - stat 43407 (time 1788461255 ; total 99 ; last 11) total: 43754 stats in 100 seconds: 437.540009 stats/second - unlinked 0 (time 1788461206 ; total 0 ; last 0) unlink(/mnt/lustre2/lockdir/lockfile3404) error: No such file or directory total: 3403 unlinks in 86 seconds: 39.569767 unlinks/second /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 77: kill: (21448) - No such process /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 78: kill: (21451) - No such process PASS 12 (145s) == sanityn test 13: test directory page revocation ======= 14:48:18 (1788461298) f13 f13 PASS 13 (10s) == sanityn test 14aa: execution of file open for write returns -ETXTBSY ========================================================== 14:48:29 (1788461309) multiop /mnt/lustre/d14aa.sanityn/f14aa.sanityn vOw_c TMPPIPE=/tmp/multiop_open_wait_pipe.7649 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 333: /mnt/lustre2/d14aa.sanityn/f14aa.sanityn: Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 22635 Terminated $MULTIOP_PROG $FILE v$ARGS > $TMPPIPE (wd: ~) PASS 14aa (9s) == sanityn test 14ab: open(RDWR) of executing file returns -ETXTBSY ========================================================== 14:48:37 (1788461317) open(O_RDWR|O_CREAT): Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 23214 Terminated $DIR1/$tdir/sleep 60 (wd: ~) PASS 14ab (8s) == sanityn test 14b: truncate of executing file returns -ETXTBSY ================================================================ 14:48:45 (1788461325) truncate: cannot truncate '/mnt/lustre2/d14b.sanityn/sleep' to length 60: Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 348: 23792 Terminated $DIR1/$tdir/sleep 60 PASS 14b (7s) == sanityn test 14c: open(O_TRUNC) of executing file return -ETXTBSY ============================================================ 14:48:52 (1788461332) cp: cannot create regular file '/mnt/lustre2/d14c.sanityn/sleep': Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 360: 24374 Terminated $DIR1/$tdir/sleep 60 PASS 14c (9s) == sanityn test 14d: chmod of executing file is still possible ================================================================== 14:49:01 (1788461341) chmod /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 371: 24958 Terminated $DIR1/$tdir/sleep 60 PASS 14d (11s) == sanityn test 15: test out-of-space with multiple writers ===================================================================== 14:49:12 (1788461352) PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin: MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky sleep 5 for ZFS MDS Waiting for MDT destroys to complete fallocate on zfs doesn't consume space STRIPECOUNT=2 ORIGFREE=7530496 MAXFREE=800000 SKIP: oos2 test_15 oos2.sh: 7530496KB free > 800000KB, increase MAXFREE (or reduce fs size) sleep 5 for ZFS MDS Waiting for MDT destroys to complete SKIP 15 (60s) == sanityn test 16a: 500 iterations of dual-mount fsx ==== 14:50:12 (1788461412) keep default fallocate mode: 2 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.5463 s, 15.4 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.333488 s, 25.2 MB/s Chance of close/open is 1 in 50 Seed set to 1417 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read skipping zero length punch hole 000004[0] 1788461418.167600 mapwrite 0x84c822 thru 0x85bcf7 (0xf4d6 bytes) 000005[1] 1788461418.284711 read 0x29423f thru 0x29c8bb (0x867d bytes) 000011[1] 1788461418.369982 mapread 0x60a5ff thru 0x60fde2 (0x57e4 bytes) 000012[1] 1788461418.460604 write 0x0fdbe8 thru 0x1092f9 (0xb712 bytes) 000013[1] 1788461418.499655 write 0x09c69a thru 0x0a804a (0xb9b1 bytes) 000014[1] 1788461418.521347 mapwrite 0x4a66b0 thru 0x4a7f05 (0x1856 bytes) 000015[1] 1788461418.604834 read 0x658da5 thru 0x665102 (0xc35e bytes) 000017[0] 1788461418.632767 punch from 0x13dced to 0x1466ab (0x89bf bytes) 000020[1] 1788461418.726334 read 0x1820fa thru 0x182d33 (0xc3a bytes) 000021[1] 1788461418.743873 mapread 0x6d60ed thru 0x6d9da0 (0x3cb4 bytes) 000023[1] 1788461418.758245 punch from 0x10f740 to 0x112045 (0x2906 bytes) 000024[1] 1788461418.791225 mapwrite 0x1192dd thru 0x128fca (0xfcee bytes) 000025[1] 1788461418.866933 write 0x4b4379 thru 0x4bb3ac (0x7034 bytes) 000027[1] 1788461418.918186 mapread 0x370fce thru 0x375e57 (0x4e8a bytes) 000028[1] 1788461418.942065 mapread 0x0cf57f thru 0x0dcfb7 (0xda39 bytes) 000031[0] 1788461418.983885 punch from 0x0cf37f to 0x0d6b8d (0x780f bytes) 000033[0] 1788461419.032138 write 0x53b1a4 thru 0x54ac02 (0xfa5f bytes) 000034[1] 1788461419.076804 mapread 0x810669 thru 0x8131b4 (0x2b4c bytes) 000036[1] 1788461419.124196 write 0x6d70fa thru 0x6d73b2 (0x2b9 bytes) truncating to largest ever: 0x3f5eb7 000037[0] 1788461419.146764 trunc from 0x85bcf8 to 0x3f5eb6 (0x465e41 bytes) 000038[1] 1788461419.229555 read 0x167be9 thru 0x171335 (0x974d bytes) 000041[1] 1788461419.349817 mapread 0x090abe thru 0x098f26 (0x8469 bytes) 000042[1] 1788461419.360487 write 0x569914 thru 0x5718cc (0x7fb9 bytes) 000043[0] 1788461419.388312 mapread 0x450781 thru 0x4532e5 (0x2b65 bytes) 000046[0] 1788461419.465754 mapwrite 0x71f99a thru 0x723a0a (0x4071 bytes) 000047[0] 1788461419.606518 mapwrite 0x228588 thru 0x229b7f (0x15f8 bytes) 000050[1] 1788461419.756448 punch from 0x6b931b to 0x6bc003 (0x2ce9 bytes) 000053[0] 1788461419.828492 write 0x839164 thru 0x83fa3a (0x68d7 bytes) 000055[1] 1788461419.859407 read 0x5b7795 thru 0x5c3483 (0xbcef bytes) 000057[0] 1788461419.893157 mapwrite 0x83fa5a thru 0x84fa0e (0xffb5 bytes) 000058[0] 1788461420.105971 mapwrite 0x0552a9 thru 0x055940 (0x698 bytes) 000060[0] 1788461420.275637 punch from 0x44218c to 0x448675 (0x64ea bytes) 000061[0] 1788461420.285875 write 0x0515c3 thru 0x052f59 (0x1997 bytes) 000062[0] 1788461420.304540 punch from 0x2588ec to 0x25b2aa (0x29bf bytes) 000065[1] 1788461420.391749 punch from 0x41b9d3 to 0x42a769 (0xed97 bytes) 000067[0] 1788461420.447628 punch from 0x2a05ad to 0x2a2090 (0x1ae4 bytes) 000068[0] 1788461420.466895 mapwrite 0x4057bb thru 0x40c672 (0x6eb8 bytes) 000070[1] 1788461420.605815 read 0x4fd6f1 thru 0x508182 (0xaa92 bytes) 000073[1] 1788461420.656568 write 0x6debf1 thru 0x6e83d8 (0x97e8 bytes) 000074[1] 1788461420.718879 mapread 0x199c73 thru 0x19ad1c (0x10aa bytes) 000076[1] 1788461420.829168 read 0x37bb8d thru 0x37cfd2 (0x1446 bytes) 000078[1] 1788461420.896899 write 0x68a358 thru 0x69020b (0x5eb4 bytes) 000079[0] 1788461420.910427 mapwrite 0x51e1bf thru 0x52cff2 (0xee34 bytes) 000080[1] 1788461421.043883 punch from 0x0e1872 to 0x0f1312 (0xfaa1 bytes) 000081[1] 1788461421.085006 read 0x019dc2 thru 0x02945e (0xf69d bytes) 000082[0] 1788461421.127318 write 0x7f4aa9 thru 0x7f8ee7 (0x443f bytes) 000085[0] 1788461421.194759 punch from 0x454c02 to 0x458e7d (0x427c bytes) 000088[0] 1788461421.267953 mapread 0x2ecfa6 thru 0x2f6533 (0x958e bytes) 000089[1] 1788461421.317215 mapread 0x26390d thru 0x26b9ac (0x80a0 bytes) 000090[1] 1788461421.372876 mapread 0x59a6ff thru 0x5a3196 (0x8a98 bytes) 000092[0] 1788461421.417437 mapwrite 0x9d5c96 thru 0x9e51f0 (0xf55b bytes) 000093[0] 1788461421.549817 read 0x8a1f9a thru 0x8a99d7 (0x7a3e bytes) 000094[0] 1788461421.584527 trunc from 0x9e51f1 to 0x26499b (0x780855 bytes) 000095[0] 1788461421.607767 write 0x7562f3 thru 0x75e8cc (0x85da bytes) 000096[0] 1788461421.636790 read 0x4550ce thru 0x458469 (0x339c bytes) 000097[1] 1788461421.642939 mapwrite 0x33bdbc thru 0x3479cb (0xbc10 bytes) 000098[0] 1788461421.981709 mapread 0x403806 thru 0x40f31d (0xbb18 bytes) 000099[0] 1788461422.030554 read 0x2ebb79 thru 0x2fac8e (0xf116 bytes) truncating to largest ever: 0x4ad9f0 000103[1] 1788461422.098836 trunc from 0x75e8cd to 0x4ad9ef (0x2b0edd bytes) 000109[0] 1788461422.209952 mapwrite 0x20504d thru 0x20ee13 (0x9dc7 bytes) truncating to largest ever: 0x72cddb 000110[1] 1788461422.318748 trunc from 0x4ad9f0 to 0x72cdda (0x27f3eb bytes) 000113[0] 1788461422.372475 read 0x55a0f1 thru 0x561ffb (0x7f0b bytes) 000114[1] 1788461422.419011 punch from 0x33fa59 to 0x349a85 (0xa02d bytes) 000115[1] 1788461422.450343 mapread 0x1ef010 thru 0x1f00bf (0x10b0 bytes) 000116[1] 1788461422.480458 mapread 0x296625 thru 0x296f73 (0x94f bytes) 000117[1] 1788461422.503791 punch from 0x0814cc to 0x0907a3 (0xf2d8 bytes) 000119[0] 1788461422.527958 mapwrite 0x8d834b thru 0x8dccee (0x49a4 bytes) 000121[1] 1788461422.780159 write 0x25ad09 thru 0x262aa1 (0x7d99 bytes) 000122[1] 1788461422.832908 read 0x132f7e thru 0x1423b1 (0xf434 bytes) 000128[1] 1788461422.931367 write 0x4116b7 thru 0x419839 (0x8183 bytes) 000129[1] 1788461422.983685 read 0x8d1888 thru 0x8d68ef (0x5068 bytes) 000131[1] 1788461422.999851 read 0x1fc7ce thru 0x202d81 (0x65b4 bytes) 000132[0] 1788461423.047971 write 0x0a915a thru 0x0af3e5 (0x628c bytes) 000134[1] 1788461423.106414 mapread 0x0c095b thru 0x0c0c0d (0x2b3 bytes) 000135[1] 1788461423.155819 write 0x372b2c thru 0x382a79 (0xff4e bytes) 000136[0] 1788461423.201855 punch from 0x18aeca to 0x19138d (0x64c4 bytes) 000138[1] 1788461423.255058 write 0x97434f thru 0x97bc42 (0x78f4 bytes) 000139[0] 1788461423.286883 punch from 0x4bd19f to 0x4ccd03 (0xfb65 bytes) 000140[1] 1788461423.324278 mapread 0x3655f5 thru 0x3662b2 (0xcbe bytes) 000143[0] 1788461423.373514 punch from 0x3d8cd5 to 0x3dea54 (0x5d80 bytes) 000145[0] 1788461423.405295 read 0x07e33b thru 0x08a972 (0xc638 bytes) 000146[1] 1788461423.436705 write 0x84ba96 thru 0x85538d (0x98f8 bytes) 000149[0] 1788461423.498586 mapread 0x7ccbe1 thru 0x7d90e3 (0xc503 bytes) 000150[1] 1788461423.549796 write 0x84f55b thru 0x85d46d (0xdf13 bytes) 000151[0] 1788461423.566680 punch from 0x45dcf0 to 0x46d0e5 (0xf3f6 bytes) 000152[1] 1788461423.580840 read 0x0aeff1 thru 0x0b8097 (0x90a7 bytes) 000159[1] 1788461423.699366 read 0x3df593 thru 0x3ec84b (0xd2b9 bytes) 000160[0] 1788461423.722561 punch from 0x3ba323 to 0x3c84e8 (0xe1c6 bytes) 000163[1] 1788461423.800829 mapwrite 0x95ec6a thru 0x95fc30 (0xfc7 bytes) 000164[1] 1788461423.889102 write 0x3ac4ee thru 0x3b1757 (0x526a bytes) 000167[0] 1788461423.939928 mapwrite 0x610e0c thru 0x61398b (0x2b80 bytes) 000168[0] 1788461424.035145 mapwrite 0x497f05 thru 0x4a441e (0xc51a bytes) 000170[0] 1788461424.172757 read 0x760ecf thru 0x768c46 (0x7d78 bytes) 000171[0] 1788461424.196163 punch from 0x3a5db1 to 0x3ae0db (0x832b bytes) 000174[1] 1788461424.253901 mapwrite 0x5f2b75 thru 0x602810 (0xfc9c bytes) 000175[0] 1788461424.387112 punch from 0x1fe53d to 0x20958e (0xb052 bytes) 000176[1] 1788461424.404755 write 0x2be7ce thru 0x2c538f (0x6bc2 bytes) 000177[1] 1788461424.426423 read 0x27a91e thru 0x27ff12 (0x55f5 bytes) 000178[0] 1788461424.437862 mapwrite 0x54b9c4 thru 0x54debf (0x24fc bytes) 000180[0] 1788461424.653094 mapread 0x0f2144 thru 0x101475 (0xf332 bytes) 000182[1] 1788461424.799327 mapread 0x33160d thru 0x34065e (0xf052 bytes) 000183[0] 1788461424.844376 mapread 0x27f863 thru 0x28b640 (0xbdde bytes) 000185[1] 1788461424.905746 mapwrite 0x09908d thru 0x0a8fde (0xff52 bytes) 000186[0] 1788461425.021776 write 0x07bfcb thru 0x089aea (0xdb20 bytes) 000189[0] 1788461425.183896 punch from 0x7c63b4 to 0x7d228a (0xbed7 bytes) 000190[1] 1788461425.247437 mapwrite 0x0e57e3 thru 0x0e582b (0x49 bytes) 000191[1] 1788461425.363468 mapread 0x0b644c thru 0x0bc0eb (0x5ca0 bytes) 000193[1] 1788461425.443380 mapwrite 0x36afc1 thru 0x374536 (0x9576 bytes) 000194[1] 1788461425.631903 write 0x1399db thru 0x142e25 (0x944b bytes) truncating to largest ever: 0x856ac1 000196[0] 1788461425.704102 trunc from 0x97bc43 to 0x856ac0 (0x125182 bytes) 000199[1] 1788461425.852472 punch from 0x445aee to 0x44c88b (0x6d9e bytes) 000201[1] 1788461425.908324 punch from 0x09aa1e to 0x0a9102 (0xe6e5 bytes) 000202[1] 1788461425.975872 write 0x43ce82 thru 0x4483d1 (0xb550 bytes) 000205[0] 1788461425.992902 punch from 0x1dd7ed to 0x1e2044 (0x4858 bytes) 000208[1] 1788461426.050853 punch from 0x33bfbd to 0x34095d (0x49a1 bytes) 000209[0] 1788461426.126673 write 0x4dd581 thru 0x4e5ffa (0x8a7a bytes) 000211[0] 1788461426.206900 mapread 0x3aa1ba thru 0x3b5e01 (0xbc48 bytes) 000212[1] 1788461426.259336 punch from 0x3895df to 0x38b86f (0x2291 bytes) 000213[1] 1788461426.328768 punch from 0x750382 to 0x75e512 (0xe191 bytes) 000214[0] 1788461426.370166 punch from 0x29325a to 0x296cef (0x3a96 bytes) 000215[0] 1788461426.424942 punch from 0x7347c7 to 0x73ded6 (0x9710 bytes) 000216[0] 1788461426.449967 read 0x5996da thru 0x59e056 (0x497d bytes) 000217[0] 1788461426.513389 mapwrite 0x389151 thru 0x397a49 (0xe8f9 bytes) 000218[0] 1788461426.747354 trunc from 0x856ac1 to 0x550839 (0x306287 bytes) 000219[1] 1788461426.851017 write 0x987d21 thru 0x98a6be (0x299e bytes) 000223[0] 1788461426.939737 trunc from 0x98a6bf to 0x04d902 (0x93cdbc bytes) 000225[1] 1788461427.003563 read 0x015002 thru 0x01d2c1 (0x82c0 bytes) 000227[0] 1788461427.072838 mapread 0x01cd26 thru 0x02c0cd (0xf3a8 bytes) 000228[0] 1788461427.106518 read 0x032624 thru 0x03f323 (0xcd00 bytes) 000229[1] 1788461427.118563 punch from 0x02e98c to 0x0383c6 (0x9a3b bytes) 000231[0] 1788461427.170051 punch from 0x026fed to 0x030691 (0x96a5 bytes) 000232[0] 1788461427.191630 trunc from 0x04d903 to 0x2e2671 (0x294d6f bytes) 000234[1] 1788461427.238559 read 0x25090f thru 0x25da69 (0xd15b bytes) 000235[0] 1788461427.359432 punch from 0x035888 to 0x041646 (0xbdbf bytes) 000236[0] 1788461427.421867 trunc from 0x2e2672 to 0x16d1f9 (0x175478 bytes) 000238[1] 1788461427.558402 mapwrite 0x811812 thru 0x81d863 (0xc052 bytes) 000242[0] 1788461428.032554 mapread 0x2f285b thru 0x2f9294 (0x6a3a bytes) 000245[1] 1788461428.293148 read 0x15525d thru 0x15d13a (0x7ede bytes) 000247[1] 1788461428.320006 mapread 0x763d2e thru 0x765c26 (0x1ef9 bytes) 000250[0] 1788461428.334706 mapread 0x195402 thru 0x1a22aa (0xcea9 bytes) 000251[1] 1788461428.361314 mapwrite 0x4cf174 thru 0x4dd5a3 (0xe430 bytes) 000253[1] 1788461428.436796 trunc from 0x81d864 to 0x144c7c (0x6d8be7 bytes) 000255[0] 1788461428.523821 read 0x0ebf7f thru 0x0f70e3 (0xb165 bytes) truncating to largest ever: 0x94dfe6 000257[0] 1788461428.594625 trunc from 0x144c7d to 0x94dfe5 (0x809369 bytes) 000258[0] 1788461428.676834 mapread 0x878e8c thru 0x888d13 (0xfe88 bytes) 000259[0] 1788461428.867303 trunc from 0x94dfe6 to 0x83a680 (0x113965 bytes) 000260[0] 1788461428.920600 write 0x9c950c thru 0x9d5614 (0xc109 bytes) 000261[0] 1788461428.948304 mapread 0x67623c thru 0x67faf3 (0x98b8 bytes) 000262[1] 1788461428.971568 write 0x73a0e3 thru 0x73f69a (0x55b8 bytes) 000264[0] 1788461429.026579 mapwrite 0x9691e8 thru 0x97442e (0xb247 bytes) 000265[1] 1788461429.103680 mapwrite 0x56d71e thru 0x56e49a (0xd7d bytes) 000266[1] 1788461429.169259 trunc from 0x9d5615 to 0x5d16f2 (0x403f22 bytes) 000268[0] 1788461429.239502 mapwrite 0x698421 thru 0x6a00b3 (0x7c93 bytes) 000273[1] 1788461429.390775 punch from 0x56e3f6 to 0x57aba3 (0xc7ae bytes) 000277[0] 1788461429.496276 mapread 0x4d33a7 thru 0x4dce0b (0x9a65 bytes) 000279[0] 1788461429.588014 mapread 0x2727f0 thru 0x27cfe9 (0xa7fa bytes) 000283[0] 1788461429.736573 write 0x39e125 thru 0x3a5563 (0x743f bytes) 000284[1] 1788461429.755850 trunc from 0x6a00b4 to 0x52b11e (0x174f95 bytes) 000286[1] 1788461429.855500 trunc from 0x52b11f to 0x79359f (0x268481 bytes) 000288[1] 1788461429.920422 mapread 0x129756 thru 0x12d904 (0x41af bytes) 000289[1] 1788461429.946650 read 0x11392d thru 0x12235b (0xea2f bytes) 000293[1] 1788461429.990964 mapwrite 0x3caf50 thru 0x3d0c5a (0x5d0b bytes) 000295[0] 1788461430.085675 mapwrite 0x23673c thru 0x242d6a (0xc62f bytes) 000297[1] 1788461430.246559 mapwrite 0x570ff7 thru 0x5731f9 (0x2203 bytes) 000300[0] 1788461430.345672 trunc from 0x7935a0 to 0x829ed4 (0x96935 bytes) 000302[1] 1788461430.379323 mapread 0x197886 thru 0x199ef2 (0x266d bytes) 000304[1] 1788461430.427992 mapread 0x641bae thru 0x64a8d5 (0x8d28 bytes) 000305[0] 1788461430.471504 trunc from 0x829ed5 to 0x7d5fbf (0x53f15 bytes) 000306[1] 1788461430.515270 punch from 0x7c57ba to 0x7ccc89 (0x74d0 bytes) 000307[0] 1788461430.538568 write 0x3fc7a7 thru 0x3fcc5e (0x4b8 bytes) 000308[0] 1788461430.561116 punch from 0x77ae33 to 0x77e54e (0x371c bytes) 000310[1] 1788461430.581638 write 0x9d4436 thru 0x9e386f (0xf43a bytes) 000314[0] 1788461430.658577 read 0x167077 thru 0x16785e (0x7e8 bytes) 000315[1] 1788461430.709391 write 0x05d367 thru 0x05ed1e (0x19b8 bytes) 000316[0] 1788461430.746275 trunc from 0x9e3870 to 0x79765d (0x24c212 bytes) 000317[1] 1788461430.782514 punch from 0x606bf2 to 0x612e47 (0xc256 bytes) 000319[1] 1788461430.824935 mapwrite 0x6b4cd8 thru 0x6c1294 (0xc5bd bytes) 000320[0] 1788461430.955340 punch from 0x3296b5 to 0x32ccbd (0x3609 bytes) 000321[0] 1788461431.011151 read 0x118ed7 thru 0x11c2c4 (0x33ee bytes) 000324[0] 1788461431.081975 mapwrite 0x469388 thru 0x474c38 (0xb8b1 bytes) 000327[0] 1788461431.219676 trunc from 0x79765e to 0x8b53b2 (0x11dd55 bytes) 000328[1] 1788461431.268112 punch from 0x572d6b to 0x574c0d (0x1ea3 bytes) 000329[1] 1788461431.308316 write 0x39ce13 thru 0x39d7ed (0x9db bytes) 000330[1] 1788461431.327556 read 0x60e469 thru 0x60fe62 (0x19fa bytes) 000331[0] 1788461431.358882 mapwrite 0x50079b thru 0x504892 (0x40f8 bytes) 000332[0] 1788461431.438697 punch from 0x19b6f9 to 0x1a5b1b (0xa423 bytes) 000333[0] 1788461431.474952 trunc from 0x8b53b3 to 0x40fcd7 (0x4a56db bytes) 000336[0] 1788461431.568964 mapread 0x1ec771 thru 0x1f271d (0x5fad bytes) 000337[1] 1788461431.594353 write 0x98734f thru 0x993796 (0xc448 bytes) 000339[1] 1788461431.632452 trunc from 0x993797 to 0x137fb8 (0x85b7de bytes) 000344[1] 1788461431.730580 mapread 0x06228c thru 0x0713d4 (0xf149 bytes) 000346[0] 1788461431.805461 mapread 0x0c014b thru 0x0c78cf (0x7785 bytes) 000347[1] 1788461431.880661 mapread 0x0eb018 thru 0x0ec460 (0x1449 bytes) 000348[0] 1788461431.923924 punch from 0x0df622 to 0x0e92a2 (0x9c81 bytes) 000349[1] 1788461431.963554 trunc from 0x137fb9 to 0x63e9b1 (0x5069f9 bytes) 000351[0] 1788461432.036701 punch from 0x2bf865 to 0x2c9f9e (0xa73a bytes) 000356[0] 1788461432.150408 mapwrite 0x5c869d thru 0x5d0d07 (0x866b bytes) 000358[1] 1788461432.309534 trunc from 0x63e9b2 to 0x94b80b (0x30ce5a bytes) 000360[1] 1788461432.349952 mapwrite 0x486607 thru 0x491550 (0xaf4a bytes) 000361[0] 1788461432.418662 trunc from 0x94b80c to 0x27bac2 (0x6cfd49 bytes) 000363[0] 1788461432.509765 write 0x70361b thru 0x7102d1 (0xccb7 bytes) 000365[1] 1788461432.514430 read 0x0bfff4 thru 0x0cb6e7 (0xb6f4 bytes) 000366[0] 1788461432.571470 write 0x41c1f4 thru 0x41d56c (0x1379 bytes) 000367[0] 1788461432.579656 mapwrite 0x0d0978 thru 0x0d90f6 (0x877f bytes) 000368[0] 1788461432.679272 read 0x137ab8 thru 0x1463c0 (0xe909 bytes) 000373[1] 1788461432.744298 read 0x5a61e9 thru 0x5adb69 (0x7981 bytes) 000374[1] 1788461432.781472 read 0x691a0a thru 0x699d09 (0x8300 bytes) 000378[1] 1788461432.820220 read 0x55cdb1 thru 0x566cee (0x9f3e bytes) 000379[0] 1788461432.828607 trunc from 0x7102d2 to 0x6fa80e (0x15ac3 bytes) 000380[0] 1788461432.887236 mapwrite 0x8912a2 thru 0x89aaa5 (0x9804 bytes) 000381[1] 1788461432.982503 write 0x804dfe thru 0x80e7cd (0x99d0 bytes) 000382[1] 1788461433.016218 write 0x8062f3 thru 0x80b5f8 (0x5306 bytes) 000384[0] 1788461433.043579 write 0x9accf0 thru 0x9bc92b (0xfc3c bytes) 000387[1] 1788461433.091690 mapread 0x72b670 thru 0x739d50 (0xe6e1 bytes) 000388[0] 1788461433.147343 read 0x8e94ac thru 0x8f64a6 (0xcffb bytes) 000389[0] 1788461433.157596 read 0x1aa3a6 thru 0x1b300d (0x8c68 bytes) 000390[1] 1788461433.169151 punch from 0x07816f to 0x0833c7 (0xb259 bytes) 000392[1] 1788461433.197465 write 0x5f28a5 thru 0x6009b3 (0xe10f bytes) 000394[1] 1788461433.218576 write 0x172d59 thru 0x1795ce (0x6876 bytes) 000395[0] 1788461433.228350 mapwrite 0x779dbb thru 0x77a87b (0xac1 bytes) 000396[0] 1788461433.322619 mapread 0x502347 thru 0x504317 (0x1fd1 bytes) 000397[1] 1788461433.358765 punch from 0x1c9051 to 0x1d49da (0xb98a bytes) 000398[0] 1788461433.381951 read 0x7312aa thru 0x73e1c4 (0xcf1b bytes) 000401[0] 1788461433.432550 punch from 0x727628 to 0x733b86 (0xc55f bytes) 000403[1] 1788461433.469369 mapwrite 0x38c2a5 thru 0x38eb43 (0x289f bytes) 000404[1] 1788461433.558175 punch from 0x8ad6bf to 0x8b5a54 (0x8396 bytes) 000406[1] 1788461433.593164 write 0x5d95eb thru 0x5e8722 (0xf138 bytes) 000407[1] 1788461433.639323 punch from 0x15aab7 to 0x163d5f (0x92a9 bytes) 000411[1] 1788461433.732739 read 0x518564 thru 0x524e5b (0xc8f8 bytes) truncating to largest ever: 0x98ea20 000413[1] 1788461433.766957 trunc from 0x9bc92c to 0x98ea1f (0x2df0c bytes) 000414[0] 1788461433.810687 mapread 0x705860 thru 0x70cfcd (0x776e bytes) 000415[1] 1788461433.879386 mapread 0x1965a7 thru 0x19ee06 (0x8860 bytes) 000416[0] 1788461433.909779 mapwrite 0x79faaa thru 0x7a7122 (0x7679 bytes) 000418[1] 1788461434.018996 mapwrite 0x9fddee thru 0x9fffff (0x2212 bytes) 000419[1] 1788461434.170389 read 0x1b9892 thru 0x1b9cec (0x45b bytes) 000420[0] 1788461434.182643 trunc from 0xa00000 to 0x7328c3 (0x2cd73c bytes) 000422[0] 1788461434.289889 punch from 0x54be77 to 0x5525bf (0x6749 bytes) 000423[0] 1788461434.295152 punch from 0x393a94 to 0x39a0e3 (0x6650 bytes) 000425[1] 1788461434.320719 mapread 0x32e3e5 thru 0x33b0e9 (0xcd05 bytes) 000426[0] 1788461434.397013 trunc from 0x7328c4 to 0x283398 (0x4af52b bytes) 000430[1] 1788461434.459478 punch from 0x025fb3 to 0x0290ed (0x313b bytes) 000431[1] 1788461434.487473 punch from 0x00a1bb to 0x014e2d (0xac73 bytes) 000432[1] 1788461434.503793 mapread 0x12fa85 thru 0x13929b (0x9817 bytes) 000435[1] 1788461434.589739 punch from 0x08af49 to 0x09740a (0xc4c2 bytes) 000436[0] 1788461434.606188 write 0x27838a thru 0x27c57e (0x41f5 bytes) 000437[1] 1788461434.629759 mapread 0x058930 thru 0x05d4bc (0x4b8d bytes) 000439[1] 1788461434.704584 read 0x1f0f85 thru 0x1f998a (0x8a06 bytes) 000440[1] 1788461434.792715 trunc from 0x283399 to 0x023923 (0x25fa75 bytes) 000441[1] 1788461434.899269 mapread 0x0033fc thru 0x00e3c9 (0xafce bytes) 000442[0] 1788461434.994229 read 0x01bde3 thru 0x023923 (0x7b41 bytes) 000443[0] 1788461435.032377 trunc from 0x023924 to 0x528eda (0x5055b7 bytes) 000444[1] 1788461435.105385 punch from 0x0942ef to 0x0949b0 (0x6c2 bytes) 000445[1] 1788461435.132097 write 0x23a684 thru 0x23f835 (0x51b2 bytes) 000447[1] 1788461435.211828 mapread 0x30c29d thru 0x3112f3 (0x5057 bytes) 000448[1] 1788461435.240898 mapwrite 0x7da3a6 thru 0x7e728c (0xcee7 bytes) 000450[1] 1788461435.515741 mapread 0x0c248f thru 0x0c2b5b (0x6cd bytes) 000451[1] 1788461435.523655 write 0x1bca55 thru 0x1c1fc6 (0x5572 bytes) 000452[0] 1788461435.553362 trunc from 0x7e728d to 0x814bd5 (0x2d949 bytes) 000453[0] 1788461435.654847 mapwrite 0x32378e thru 0x32bd99 (0x860c bytes) 000454[0] 1788461435.792569 read 0x24e09d thru 0x25a20b (0xc16f bytes) 000457[1] 1788461435.821873 trunc from 0x814bd6 to 0x44e5a1 (0x3c6634 bytes) 000460[0] 1788461435.940126 mapread 0x3c1a2f thru 0x3cbea2 (0xa474 bytes) 000462[1] 1788461436.013560 write 0x4cb61c thru 0x4d356f (0x7f54 bytes) 000463[0] 1788461436.031854 mapwrite 0x527779 thru 0x530726 (0x8fae bytes) 000464[0] 1788461436.238700 read 0x2955a2 thru 0x29b317 (0x5d76 bytes) 000465[0] 1788461436.295106 read 0x3d5d73 thru 0x3d85c5 (0x2853 bytes) 000466[0] 1788461436.316583 mapread 0x1e7b18 thru 0x1f29af (0xae98 bytes) 000467[0] 1788461436.396553 trunc from 0x530727 to 0x522069 (0xe6bd bytes) 000468[1] 1788461436.460608 punch from 0x195c22 to 0x19e421 (0x8800 bytes) truncating to largest ever: 0x9e27e9 000469[1] 1788461436.542184 trunc from 0x52206a to 0x9e27e8 (0x4c077f bytes) 000470[0] 1788461436.628602 write 0x2f348d thru 0x300215 (0xcd89 bytes) 000471[1] 1788461436.679312 trunc from 0x9e27e9 to 0x50ad1f (0x4d7ac9 bytes) 000472[1] 1788461436.740657 trunc from 0x50ad20 to 0x110aa0 (0x3fa27f bytes) 000475[1] 1788461436.895190 trunc from 0x110aa1 to 0x7ed2a6 (0x6dc806 bytes) 000479[0] 1788461437.018013 trunc from 0x7ed2a7 to 0x8d3745 (0xe649f bytes) 000483[1] 1788461437.191920 punch from 0x1cbdaa to 0x1d27f1 (0x6a48 bytes) 000484[0] 1788461437.227161 trunc from 0x8d3746 to 0x387411 (0x54c334 bytes) 000485[0] 1788461437.261049 write 0x0a1691 thru 0x0ab6f5 (0xa065 bytes) 000486[1] 1788461437.272347 mapread 0x0dc4b7 thru 0x0e4cb7 (0x8801 bytes) 000488[0] 1788461437.387845 mapwrite 0x0b4d93 thru 0x0c4b43 (0xfdb1 bytes) 000489[0] 1788461437.564661 mapwrite 0x1c48a6 thru 0x1ce630 (0x9d8b bytes) 000490[0] 1788461437.668821 trunc from 0x387412 to 0x270d3f (0x1166d2 bytes) 000492[0] 1788461437.716993 punch from 0x01b780 to 0x024456 (0x8cd7 bytes) 000493[1] 1788461437.732624 mapwrite 0x5b1998 thru 0x5c1825 (0xfe8e bytes) 000495[0] 1788461438.044809 read 0x5291cc thru 0x535cd4 (0xcb09 bytes) 000496[0] 1788461438.185195 write 0x412b1d thru 0x41a47e (0x7962 bytes) 000498[1] 1788461438.250442 mapwrite 0x95ea6a thru 0x96b945 (0xcedc bytes) 000499[1] 1788461438.440473 punch from 0x82f104 to 0x83276a (0x3667 bytes) 000500[1] 1788461438.452491 trunc from 0x96b946 to 0x62cc75 (0x33ecd0 bytes) All operations completed A-OK! Chance of close/open is 1 in 50 Seed set to 1438 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported truncating to largest ever: 0x60a58f 000005[0] 1788461439.794592 trunc from 00000000 to 0x60a58e (0x60a58f bytes) 000007[0] 1788461439.866483 read_OD 0x214000 thru 0x220fff (0xd000 bytes) 000008[1] 1788461439.907289 mapread 0x32b000 thru 0x335ae3 (0xaae4 bytes) 000012[0] 1788461439.946648 punch from 0x1beaaf to 0x1ce7d5 (0xfd27 bytes) 000013[1] 1788461439.970779 write 0x417000 thru 0x4185d3 (0x15d4 bytes) 000017[0] 1788461440.029270 write_OD 0x55e000 thru 0x560fff (0x3000 bytes) 000019[1] 1788461440.155325 mapread 0x5c5000 thru 0x5cd2a4 (0x82a5 bytes) 000020[0] 1788461440.188161 mapwrite 0x4f2000 thru 0x4f7cea (0x5ceb bytes) 000022[1] 1788461440.323880 read 0x4c4000 thru 0x4d2491 (0xe492 bytes) 000023[1] 1788461440.359239 write 0x711000 thru 0x7142de (0x32df bytes) 000024[1] 1788461440.378494 read 0x5f4000 thru 0x603234 (0xf235 bytes) 000025[1] 1788461440.405957 punch from 0x5db3d8 to 0x5e4ba8 (0x97d1 bytes) 000026[1] 1788461440.412008 mapwrite 0x5cb000 thru 0x5cefd1 (0x3fd2 bytes) 000030[1] 1788461440.743723 write 0x94b000 thru 0x94c1b5 (0x11b6 bytes) 000031[0] 1788461440.745046 mapread 0x03e000 thru 0x04aa05 (0xca06 bytes) 000033[1] 1788461440.819826 mapread 0x3fa000 thru 0x3ff08d (0x508e bytes) 000034[0] 1788461440.878153 read_OD 0x508000 thru 0x50dfff (0x6000 bytes) 000036[1] 1788461440.897488 write 0x55c000 thru 0x55d0b8 (0x10b9 bytes) 000037[1] 1788461440.928690 write 0x889000 thru 0x88f6fe (0x66ff bytes) 000038[0] 1788461440.965981 mapread 0x5a7000 thru 0x5b14ae (0xa4af bytes) 000041[1] 1788461441.047766 punch from 0x200ce5 to 0x2019ea (0xd06 bytes) 000043[0] 1788461441.114951 mapread 0x05f000 thru 0x066012 (0x7013 bytes) 000045[1] 1788461441.167831 mapwrite 0x523000 thru 0x52dd04 (0xad05 bytes) 000046[1] 1788461441.317292 mapwrite 0x7b6000 thru 0x7b9f4f (0x3f50 bytes) 000048[0] 1788461441.476588 punch from 0x1cefe9 to 0x1dc07c (0xd094 bytes) 000051[1] 1788461441.610805 mapread 0x24e000 thru 0x24e8b0 (0x8b1 bytes) 000055[0] 1788461441.683098 punch from 0x7e2feb to 0x7e9f08 (0x6f1e bytes) 000056[1] 1788461441.746847 mapwrite 0x423000 thru 0x42fcf7 (0xccf8 bytes) 000057[0] 1788461441.937612 punch from 0x198aa9 to 0x19fd75 (0x72cd bytes) 000059[1] 1788461442.034787 mapread 0x6a3000 thru 0x6ab8f0 (0x88f1 bytes) 000060[1] 1788461442.089236 mapread 0x086000 thru 0x094c57 (0xec58 bytes) 000061[1] 1788461442.129611 read 0x5f5000 thru 0x5fda88 (0x8a89 bytes) 000062[1] 1788461442.140496 read 0x6a6000 thru 0x6a7a6a (0x1a6b bytes) 000064[0] 1788461442.157434 mapread 0x17e000 thru 0x18339c (0x539d bytes) 000066[1] 1788461442.197586 trunc from 0x94c1b6 to 0x5d5aec (0x3766c9 bytes) 000069[0] 1788461442.251828 mapread 0x211000 thru 0x213cc8 (0x2cc9 bytes) 000070[1] 1788461442.278638 write 0x653000 thru 0x65cb68 (0x9b69 bytes) 000071[1] 1788461442.290299 mapwrite 0x72e000 thru 0x7315eb (0x35ec bytes) 000072[0] 1788461442.381559 trunc from 0x7315ec to 0x1c90d6 (0x568515 bytes) 000073[1] 1788461442.405072 punch from 0x0eb2b3 to 0x0f0d9d (0x5aeb bytes) 000074[1] 1788461442.429176 read 0x07a000 thru 0x0888d9 (0xe8da bytes) 000075[0] 1788461442.478925 read_OD 0x128000 thru 0x12afff (0x3000 bytes) 000076[1] 1788461442.508826 punch from 0x167691 to 0x16e84a (0x71ba bytes) 000077[1] 1788461442.530266 punch from 0x0c4750 to 0x0c563b (0xeec bytes) 000080[0] 1788461442.575441 read_OD 0x07a000 thru 0x07bfff (0x2000 bytes) 000081[1] 1788461442.600523 punch from 0x198c0d to 0x19b42d (0x2821 bytes) 000082[1] 1788461442.611934 write 0x24d000 thru 0x252d31 (0x5d32 bytes) 000083[0] 1788461442.629301 mapread 0x0b3000 thru 0x0b6952 (0x3953 bytes) 000084[1] 1788461442.734681 write 0x276000 thru 0x280c8c (0xac8d bytes) 000086[1] 1788461442.810205 punch from 0x24cd77 to 0x24f71a (0x29a4 bytes) 000087[1] 1788461442.843385 punch from 0x07f5d8 to 0x082947 (0x3370 bytes) 000089[0] 1788461442.883466 punch from 0x030a17 to 0x03bdd0 (0xb3ba bytes) 000091[1] 1788461442.942259 mapread 0x23e000 thru 0x24547f (0x7480 bytes) 000092[1] 1788461443.013599 mapwrite 0x515000 thru 0x519754 (0x4755 bytes) truncating to largest ever: 0x8d04cd 000093[1] 1788461443.186257 trunc from 0x519755 to 0x8d04cc (0x3b6d78 bytes) 000094[0] 1788461443.198300 mapwrite 0x19c000 thru 0x19f468 (0x3469 bytes) 000096[1] 1788461443.355204 punch from 0x399153 to 0x3994d9 (0x387 bytes) 000097[0] 1788461443.373397 punch from 0x6e2e9d to 0x6e521c (0x2380 bytes) 000098[0] 1788461443.405834 write_OD 0x319000 thru 0x31dfff (0x5000 bytes) 000099[0] 1788461443.453643 read_OD 0x68d000 thru 0x696fff (0xa000 bytes) 000100[0] 1788461443.464505 mapwrite 0x849000 thru 0x84a39b (0x139c bytes) 000103[1] 1788461443.541746 punch from 0x218338 to 0x2211cd (0x8e96 bytes) 000105[0] 1788461443.566113 mapread 0x7d5000 thru 0x7e28d4 (0xd8d5 bytes) 000106[1] 1788461443.620060 trunc from 0x8d04cd to 0x23c14b (0x694381 bytes) truncating to largest ever: 0x91150f 000111[0] 1788461443.736511 trunc from 0x23c14c to 0x91150e (0x6d53c3 bytes) 000112[1] 1788461443.783080 mapread 0x290000 thru 0x2947b0 (0x47b1 bytes) 000113[0] 1788461443.829227 write_OD 0x0de000 thru 0x0e2fff (0x5000 bytes) 000114[0] 1788461443.873747 mapread 0x0f8000 thru 0x101afd (0x9afe bytes) 000116[0] 1788461443.898547 trunc from 0x91150f to 0x4069ee (0x50ab20 bytes) 000119[0] 1788461443.967477 mapwrite 0x63f000 thru 0x64eacb (0xfacc bytes) 000120[1] 1788461444.061507 mapwrite 0x3b5000 thru 0x3c3112 (0xe113 bytes) 000121[1] 1788461444.233704 write 0x706000 thru 0x70faae (0x9aaf bytes) 000122[1] 1788461444.262935 trunc from 0x70faaf to 0x55d0b6 (0x1b29f8 bytes) 000127[0] 1788461444.505133 write_OD 0x3ab000 thru 0x3abfff (0x1000 bytes) 000131[1] 1788461444.638811 write 0x052000 thru 0x05b9fe (0x99ff bytes) 000136[0] 1788461444.762001 read_OD 0x09d000 thru 0x09efff (0x2000 bytes) 000139[0] 1788461444.825642 read_OD 0x10e000 thru 0x112fff (0x5000 bytes) 000140[1] 1788461444.844780 trunc from 0x55d0b7 to 0x864650 (0x30759a bytes) 000141[1] 1788461444.894319 mapwrite 0x19c000 thru 0x1ab5ed (0xf5ee bytes) 000142[0] 1788461445.054627 write_OD 0x11c000 thru 0x127fff (0xc000 bytes) 000143[0] 1788461445.142379 trunc from 0x864651 to 0x7bfe4f (0xa4801 bytes) 000145[0] 1788461445.186304 punch from 0x69d66f to 0x6a756e (0x9f00 bytes) 000146[1] 1788461445.217095 mapread 0x4e7000 thru 0x4e7e21 (0xe22 bytes) 000147[0] 1788461445.241357 trunc from 0x7bfe50 to 0x3559fa (0x46a455 bytes) 000148[1] 1788461445.293186 mapwrite 0x79f000 thru 0x7a8120 (0x9121 bytes) 000149[1] 1788461445.407682 punch from 0x47ca71 to 0x47d5d2 (0xb62 bytes) 000150[1] 1788461445.412411 write 0x4af000 thru 0x4b5b73 (0x6b74 bytes) 000153[1] 1788461445.445416 trunc from 0x7a8121 to 0x3e5df7 (0x3c2329 bytes) 000155[0] 1788461445.488952 mapwrite 0x469000 thru 0x473e42 (0xae43 bytes) 000156[1] 1788461445.635759 punch from 0x208118 to 0x20d263 (0x514c bytes) 000157[1] 1788461445.661276 punch from 0x268daf to 0x276cb9 (0xdf0b bytes) truncating to largest ever: 0x99dbb4 000159[0] 1788461445.710492 trunc from 0x473e43 to 0x99dbb3 (0x529d71 bytes) 000160[1] 1788461445.772787 mapread 0x87d000 thru 0x8865ba (0x95bb bytes) 000161[0] 1788461445.858754 mapwrite 0x708000 thru 0x70cb1b (0x4b1c bytes) 000165[1] 1788461445.961338 mapwrite 0x72e000 thru 0x735c3b (0x7c3c bytes) 000166[1] 1788461446.083587 punch from 0x40757a to 0x4151f3 (0xdc7a bytes) 000168[0] 1788461446.092177 punch from 0x109e04 to 0x116ca8 (0xcea5 bytes) 000169[0] 1788461446.131485 write_OD 0x27b000 thru 0x286fff (0xc000 bytes) 000170[1] 1788461446.237702 write 0x4b1000 thru 0x4b1d26 (0xd27 bytes) 000172[0] 1788461446.310714 write_OD 0x664000 thru 0x667fff (0x4000 bytes) 000173[0] 1788461446.468936 trunc from 0x99dbb4 to 0x8bd464 (0xe074f bytes) 000174[1] 1788461446.547250 write 0x346000 thru 0x354a4c (0xea4d bytes) 000176[1] 1788461446.586949 read 0x22a000 thru 0x22b116 (0x1117 bytes) 000177[1] 1788461446.592007 mapwrite 0x988000 thru 0x9902d2 (0x82d3 bytes) 000178[1] 1788461446.687571 mapread 0x8fd000 thru 0x901dbe (0x4dbf bytes) 000179[0] 1788461446.717370 write_OD 0x0f6000 thru 0x103fff (0xe000 bytes) 000181[0] 1788461446.765721 punch from 0x3cbe5f to 0x3db07b (0xf21d bytes) 000182[0] 1788461446.783475 write_OD 0x1e5000 thru 0x1e5fff (0x1000 bytes) 000183[0] 1788461446.830229 trunc from 0x9902d3 to 0x864cec (0x12b5e6 bytes) 000184[0] 1788461446.878168 punch from 0x260bbc to 0x267c31 (0x7076 bytes) 000185[0] 1788461446.892949 punch from 0x28a7b9 to 0x294272 (0x9aba bytes) 000188[0] 1788461446.901949 mapread 0x62d000 thru 0x62f750 (0x2751 bytes) 000189[0] 1788461446.920012 mapwrite 0x3ef000 thru 0x3fe9b7 (0xf9b8 bytes) 000190[0] 1788461447.012188 read_OD 0x60a000 thru 0x60efff (0x5000 bytes) 000191[1] 1788461447.035836 write 0x844000 thru 0x84dcc0 (0x9cc1 bytes) 000192[1] 1788461447.055182 read 0x0c5000 thru 0x0c8c79 (0x3c7a bytes) 000193[1] 1788461447.060214 trunc from 0x864ced to 0x868da4 (0x40b8 bytes) 000196[0] 1788461447.121472 mapwrite 0x495000 thru 0x49ae43 (0x5e44 bytes) 000198[0] 1788461447.227980 write_OD 0x129000 thru 0x12afff (0x2000 bytes) 000201[1] 1788461447.269903 punch from 0x2390b6 to 0x245317 (0xc262 bytes) 000203[0] 1788461447.302645 punch from 0x24a251 to 0x259414 (0xf1c4 bytes) 000204[1] 1788461447.334475 mapread 0x041000 thru 0x0435b3 (0x25b4 bytes) 000206[0] 1788461447.380765 mapread 0x48b000 thru 0x48d6e4 (0x26e5 bytes) 000207[1] 1788461447.388049 read 0x5b5000 thru 0x5bf2af (0xa2b0 bytes) 000209[0] 1788461447.394895 mapread 0x3a8000 thru 0x3b2a29 (0xaa2a bytes) 000210[0] 1788461447.408942 trunc from 0x868da5 to 0x3ef289 (0x479b1b bytes) 000211[0] 1788461447.440338 mapread 0x1b9000 thru 0x1c0870 (0x7871 bytes) 000212[0] 1788461447.498451 mapwrite 0x916000 thru 0x925b91 (0xfb92 bytes) 000215[0] 1788461447.667558 punch from 0x6631cd to 0x6634cb (0x2ff bytes) 000217[1] 1788461447.701178 write 0x307000 thru 0x307a34 (0xa35 bytes) 000218[0] 1788461447.729357 punch from 0x1fb319 to 0x208acc (0xd7b4 bytes) 000220[1] 1788461447.766372 mapwrite 0x6af000 thru 0x6b296c (0x396d bytes) 000221[1] 1788461447.870831 mapread 0x8db000 thru 0x8e7d8f (0xcd90 bytes) 000224[0] 1788461447.908159 mapwrite 0x0fa000 thru 0x0fda20 (0x3a21 bytes) 000225[1] 1788461447.971532 punch from 0x7aeb90 to 0x7b5a3e (0x6eaf bytes) 000229[0] 1788461448.005878 mapread 0x0de000 thru 0x0e9b1f (0xbb20 bytes) 000231[0] 1788461448.037646 punch from 0x68371e to 0x683901 (0x1e4 bytes) 000232[0] 1788461448.063983 read_OD 0x830000 thru 0x832fff (0x3000 bytes) 000233[1] 1788461448.077882 punch from 0x73ecf0 to 0x743449 (0x475a bytes) 000237[1] 1788461448.116913 trunc from 0x925b92 to 0x1f7445 (0x72e74c bytes) 000240[0] 1788461448.133992 mapwrite 0x141000 thru 0x14c1cf (0xb1d0 bytes) 000242[1] 1788461448.223179 write 0x91d000 thru 0x9238c9 (0x68ca bytes) 000243[1] 1788461448.239103 mapwrite 0x5cb000 thru 0x5d4fed (0x9fee bytes) 000245[1] 1788461448.298283 write 0x2d7000 thru 0x2e3c13 (0xcc14 bytes) 000246[0] 1788461448.304186 trunc from 0x9238ca to 0x399580 (0x58a349 bytes) 000247[0] 1788461448.359084 punch from 0x0509c3 to 0x05be5f (0xb49d bytes) truncating to largest ever: 0x9c57ca 000249[1] 1788461448.396152 trunc from 0x399581 to 0x9c57c9 (0x62c249 bytes) 000250[1] 1788461448.430695 write 0x183000 thru 0x18753e (0x453f bytes) 000254[1] 1788461448.453239 read 0x089000 thru 0x090ad8 (0x7ad9 bytes) 000255[0] 1788461448.470296 punch from 0x17bee7 to 0x1829bc (0x6ad6 bytes) 000258[1] 1788461448.515411 trunc from 0x9c57ca to 0x7cf83d (0x1f5f8c bytes) 000260[1] 1788461448.562488 write 0x3f9000 thru 0x3fffc2 (0x6fc3 bytes) 000261[0] 1788461448.580674 punch from 0x6ebb62 to 0x6f8c46 (0xd0e5 bytes) 000264[1] 1788461448.616631 read 0x048000 thru 0x052923 (0xa924 bytes) 000265[0] 1788461448.635586 mapread 0x0e1000 thru 0x0e5969 (0x496a bytes) 000267[1] 1788461448.645492 read 0x741000 thru 0x74915a (0x815b bytes) 000268[1] 1788461448.652357 mapwrite 0x568000 thru 0x568624 (0x625 bytes) 000269[0] 1788461448.715790 read_OD 0x5db000 thru 0x5e9fff (0xf000 bytes) 000272[1] 1788461448.740680 mapwrite 0x3f2000 thru 0x3f2584 (0x585 bytes) 000273[0] 1788461448.792864 mapwrite 0x837000 thru 0x83b613 (0x4614 bytes) 000275[1] 1788461448.890699 read 0x521000 thru 0x52dc70 (0xcc71 bytes) 000277[1] 1788461448.914429 write 0x3c7000 thru 0x3cae83 (0x3e84 bytes) 000278[0] 1788461448.938841 mapread 0x3b3000 thru 0x3ba35f (0x7360 bytes) 000282[1] 1788461448.979523 mapread 0x09a000 thru 0x0a6117 (0xc118 bytes) 000283[1] 1788461449.000427 trunc from 0x83b614 to 0x0fad2d (0x7408e6 bytes) 000286[1] 1788461449.079117 write 0x562000 thru 0x5644f4 (0x24f5 bytes) 000287[1] 1788461449.099075 write 0x0af000 thru 0x0b4c63 (0x5c64 bytes) 000288[1] 1788461449.116566 read 0x460000 thru 0x4632b1 (0x32b2 bytes) 000291[0] 1788461449.136851 punch from 0x29c8d5 to 0x29ce95 (0x5c1 bytes) 000296[1] 1788461449.211515 trunc from 0x5644f5 to 0x04e5bf (0x515f35 bytes) 000297[0] 1788461449.248273 punch from 0x032b76 to 0x035d2e (0x31b9 bytes) 000298[0] 1788461449.273371 trunc from 0x04e5c0 to 0x8995d3 (0x84b014 bytes) 000299[0] 1788461449.324080 trunc from 0x8995d4 to 0x3604b9 (0x53911a bytes) 000302[1] 1788461449.409264 trunc from 0x3604ba to 0x009d90 (0x356729 bytes) 000305[1] 1788461449.487190 read 0x002000 thru 0x008f8e (0x6f8f bytes) 000308[1] 1788461449.525544 read 0x009000 thru 0x009b6a (0xb6b bytes) 000310[1] 1788461449.526340 write 0x7be000 thru 0x7c6a63 (0x8a64 bytes) 000311[0] 1788461449.532153 mapread 0x487000 thru 0x48b309 (0x430a bytes) 000313[0] 1788461449.576337 mapwrite 0x380000 thru 0x38ff1c (0xff1d bytes) 000314[1] 1788461449.728299 mapread 0x3e7000 thru 0x3f6aa1 (0xfaa2 bytes) 000315[1] 1788461449.850052 mapwrite 0x50f000 thru 0x51ea94 (0xfa95 bytes) 000316[1] 1788461450.038527 read 0x199000 thru 0x1a4619 (0xb61a bytes) 000318[1] 1788461450.059449 mapwrite 0x97c000 thru 0x97c3b2 (0x3b3 bytes) 000320[1] 1788461450.167913 mapread 0x50c000 thru 0x518395 (0xc396 bytes) 000322[0] 1788461450.187503 read_OD 0x34d000 thru 0x34dfff (0x1000 bytes) 000323[0] 1788461450.214369 mapread 0x657000 thru 0x661110 (0xa111 bytes) 000326[1] 1788461450.232336 punch from 0x4d2d61 to 0x4dc57f (0x981f bytes) 000327[0] 1788461450.249896 read_OD 0x86b000 thru 0x871fff (0x7000 bytes) 000328[0] 1788461450.269962 trunc from 0x97c3b3 to 0x157c3c (0x824776 bytes) 000329[1] 1788461450.295865 mapwrite 0x67e000 thru 0x685b77 (0x7b78 bytes) 000330[1] 1788461450.371257 punch from 0x565865 to 0x572c57 (0xd3f3 bytes) 000331[1] 1788461450.375530 read 0x3a4000 thru 0x3a53fa (0x13fb bytes) 000333[1] 1788461450.410821 mapread 0x5d2000 thru 0x5df433 (0xd434 bytes) 000334[1] 1788461450.446377 mapread 0x547000 thru 0x54e3c2 (0x73c3 bytes) 000337[1] 1788461450.490894 mapread 0x3bc000 thru 0x3c3911 (0x7912 bytes) 000339[0] 1788461450.538516 read_OD 0x274000 thru 0x27cfff (0x9000 bytes) 000341[0] 1788461450.562220 mapwrite 0x5a6000 thru 0x5b0d2a (0xad2b bytes) 000342[0] 1788461450.745786 punch from 0x4d373e to 0x4d555b (0x1e1e bytes) truncating to largest ever: 0x9c5f25 000343[0] 1788461450.753498 trunc from 0x685b78 to 0x9c5f24 (0x3403ad bytes) 000344[0] 1788461450.822148 read_OD 0x8cd000 thru 0x8d9fff (0xd000 bytes) 000345[1] 1788461450.854914 trunc from 0x9c5f25 to 0x24d04f (0x778ed5 bytes) 000346[0] 1788461450.915148 punch from 0x0a2c38 to 0x0a74c5 (0x488e bytes) 000348[1] 1788461450.996871 mapread 0x1c7000 thru 0x1cc90e (0x590f bytes) 000350[1] 1788461451.081361 mapread 0x210000 thru 0x219fbc (0x9fbd bytes) 000353[0] 1788461451.120297 trunc from 0x24d050 to 0x4586ce (0x20b67f bytes) 000356[0] 1788461451.245654 mapread 0x208000 thru 0x21579c (0xd79d bytes) 000359[0] 1788461451.352683 mapwrite 0x719000 thru 0x7197f4 (0x7f5 bytes) 000361[1] 1788461451.476878 punch from 0x2ee305 to 0x2f481f (0x651b bytes) 000366[0] 1788461451.602775 mapwrite 0x939000 thru 0x940335 (0x7336 bytes) 000367[1] 1788461451.758997 trunc from 0x940336 to 0x16a090 (0x7d62a5 bytes) 000368[1] 1788461451.855580 mapwrite 0x310000 thru 0x31bbdd (0xbbde bytes) 000369[0] 1788461452.039099 write_OD 0x187000 thru 0x191fff (0xb000 bytes) 000370[1] 1788461452.174838 trunc from 0x31bbde to 0x225d03 (0xf5eda bytes) 000371[1] 1788461452.215477 mapread 0x1c5000 thru 0x1cd34a (0x834b bytes) 000372[1] 1788461452.245215 mapwrite 0x381000 thru 0x38a14e (0x914f bytes) 000373[1] 1788461452.386921 punch from 0x08c8e2 to 0x0956d8 (0x8df7 bytes) 000374[1] 1788461452.395907 trunc from 0x38a14f to 0x57b6bc (0x1f156e bytes) 000376[1] 1788461452.443750 punch from 0x049c8c to 0x0585e1 (0xe956 bytes) 000377[1] 1788461452.450556 mapread 0x3c6000 thru 0x3ca34f (0x4350 bytes) 000379[1] 1788461452.466416 mapwrite 0x5e9000 thru 0x5f65d6 (0xd5d7 bytes) 000381[1] 1788461452.568164 write 0x178000 thru 0x17c481 (0x4482 bytes) 000382[0] 1788461452.599378 read_OD 0x3b3000 thru 0x3b9fff (0x7000 bytes) 000383[1] 1788461452.629495 trunc from 0x5f65d7 to 0x71630b (0x11fd35 bytes) 000384[1] 1788461452.673875 mapread 0x4ed000 thru 0x4f504c (0x804d bytes) 000386[1] 1788461452.705969 mapread 0x595000 thru 0x59a154 (0x5155 bytes) 000387[1] 1788461452.717838 mapwrite 0x1cd000 thru 0x1d9bc3 (0xcbc4 bytes) 000391[0] 1788461452.804017 mapread 0x3eb000 thru 0x3f0acb (0x5acc bytes) 000392[1] 1788461452.842402 punch from 0x3a3226 to 0x3a5ef9 (0x2cd4 bytes) 000393[0] 1788461452.855718 punch from 0x595b5d to 0x599fee (0x4492 bytes) 000394[1] 1788461452.869665 write 0x006000 thru 0x0114dc (0xb4dd bytes) 000395[1] 1788461452.895048 trunc from 0x71630c to 0x4f1cc2 (0x224649 bytes) 000396[1] 1788461452.925576 punch from 0x16a040 to 0x177e17 (0xddd8 bytes) 000397[1] 1788461452.943206 read 0x4b6000 thru 0x4c5397 (0xf398 bytes) 000398[0] 1788461452.962742 mapwrite 0x980000 thru 0x987657 (0x7658 bytes) 000399[0] 1788461453.051782 trunc from 0x987658 to 0x9347fe (0x52e59 bytes) 000401[1] 1788461453.087159 punch from 0x368601 to 0x3696a0 (0x10a0 bytes) 000402[1] 1788461453.115918 write 0x32c000 thru 0x33bdaf (0xfdb0 bytes) 000403[1] 1788461453.137066 mapwrite 0x208000 thru 0x2085b9 (0x5ba bytes) 000405[0] 1788461453.189499 read_OD 0x348000 thru 0x349fff (0x2000 bytes) 000406[0] 1788461453.209105 punch from 0x6cd230 to 0x6d663f (0x9410 bytes) 000407[1] 1788461453.221635 write 0x853000 thru 0x8610b2 (0xe0b3 bytes) 000408[1] 1788461453.249219 read 0x56d000 thru 0x56f09c (0x209d bytes) 000410[1] 1788461453.277989 read 0x1d1000 thru 0x1d5b11 (0x4b12 bytes) 000411[1] 1788461453.287407 trunc from 0x9347ff to 0x92d02c (0x77d2 bytes) 000412[1] 1788461453.304973 mapwrite 0x5da000 thru 0x5e14f2 (0x74f3 bytes) 000414[1] 1788461453.379633 mapread 0x486000 thru 0x4889ba (0x29bb bytes) 000415[1] 1788461453.395825 punch from 0x6bc267 to 0x6c6a93 (0xa82d bytes) 000416[0] 1788461453.399216 read_OD 0x287000 thru 0x290fff (0xa000 bytes) 000417[1] 1788461453.420622 mapread 0x45b000 thru 0x45bcce (0xccf bytes) 000420[1] 1788461453.431735 trunc from 0x92d02d to 0x2a5034 (0x687ff8 bytes) 000421[1] 1788461453.515913 read 0x243000 thru 0x2503b8 (0xd3b9 bytes) 000422[0] 1788461453.595765 write_OD 0x492000 thru 0x49afff (0x9000 bytes) 000423[1] 1788461453.683894 read 0x10c000 thru 0x10e9ae (0x29af bytes) 000426[0] 1788461453.795967 punch from 0x360935 to 0x36bf33 (0xb5ff bytes) 000427[0] 1788461453.833773 write_OD 0x0af000 thru 0x0b0fff (0x2000 bytes) 000428[1] 1788461453.887393 read 0x110000 thru 0x1192e9 (0x92ea bytes) 000429[0] 1788461453.904254 mapwrite 0x510000 thru 0x518831 (0x8832 bytes) 000430[0] 1788461454.029409 punch from 0x3127fd to 0x31f23f (0xca43 bytes) 000431[0] 1788461454.049463 read_OD 0x1c2000 thru 0x1c3fff (0x2000 bytes) 000432[0] 1788461454.071799 trunc from 0x518832 to 0x068723 (0x4b010e bytes) 000433[0] 1788461454.112439 read_OD 0x012000 thru 0x015fff (0x4000 bytes) 000436[1] 1788461454.130857 trunc from 0x068724 to 0x142fef (0xda8cc bytes) 000437[1] 1788461454.163445 trunc from 0x142ff0 to 0x486539 (0x34354a bytes) 000441[1] 1788461454.196965 mapwrite 0x85a000 thru 0x863938 (0x9939 bytes) 000442[0] 1788461454.315756 punch from 0x089a13 to 0x0944ef (0xaadd bytes) 000443[0] 1788461454.342983 punch from 0x494fef to 0x4a4258 (0xf26a bytes) 000444[0] 1788461454.364330 trunc from 0x863939 to 0x9c5b49 (0x162211 bytes) 000446[0] 1788461454.393137 mapread 0x3e4000 thru 0x3e4730 (0x731 bytes) 000447[0] 1788461454.470684 mapwrite 0x7f9000 thru 0x7ff86b (0x686c bytes) 000448[0] 1788461454.637281 punch from 0x7aa0f4 to 0x7b1e57 (0x7d64 bytes) 000451[1] 1788461454.670920 read 0x88b000 thru 0x899764 (0xe765 bytes) 000454[1] 1788461454.716763 punch from 0x43ae3f to 0x43deee (0x30b0 bytes) 000458[1] 1788461454.786139 trunc from 0x9c5b4a to 0x5d610d (0x3efa3c bytes) 000459[0] 1788461454.810786 mapread 0x156000 thru 0x157ec6 (0x1ec7 bytes) 000462[1] 1788461454.839271 trunc from 0x5d610e to 0x2bec3c (0x3174d1 bytes) 000463[0] 1788461454.869950 punch from 0x0b0d38 to 0x0b2bc2 (0x1e8b bytes) 000464[1] 1788461454.900946 punch from 0x1daa54 to 0x1e843e (0xd9eb bytes) 000465[1] 1788461454.924539 mapwrite 0x9d1000 thru 0x9d8d14 (0x7d15 bytes) 000466[1] 1788461455.013425 trunc from 0x9d8d15 to 0x21f058 (0x7b9cbc bytes) 000467[1] 1788461455.058526 mapread 0x1ea000 thru 0x1eb143 (0x1144 bytes) 000468[1] 1788461455.078558 mapread 0x08e000 thru 0x0900b6 (0x20b7 bytes) 000469[1] 1788461455.109004 mapread 0x120000 thru 0x126f0d (0x6f0e bytes) 000473[1] 1788461455.164429 write 0x0f1000 thru 0x0fbe94 (0xae95 bytes) 000476[0] 1788461455.187533 punch from 0x0f8429 to 0x1060bb (0xdc93 bytes) 000477[0] 1788461455.208395 punch from 0x1e4c75 to 0x1e65d1 (0x195d bytes) 000478[1] 1788461455.211814 write 0x1bd000 thru 0x1be4cb (0x14cc bytes) 000479[0] 1788461455.222546 mapwrite 0x22e000 thru 0x236ab6 (0x8ab7 bytes) 000480[0] 1788461455.338715 write_OD 0x588000 thru 0x590fff (0x9000 bytes) 000484[0] 1788461455.391940 mapread 0x2d3000 thru 0x2dd6ac (0xa6ad bytes) 000485[0] 1788461455.434350 mapwrite 0x9a5000 thru 0x9ab7db (0x67dc bytes) 000486[1] 1788461455.579153 write 0x9f0000 thru 0x9f22d6 (0x22d7 bytes) 000490[1] 1788461455.655667 read 0x241000 thru 0x249b65 (0x8b66 bytes) 000494[1] 1788461455.676847 mapread 0x5b9000 thru 0x5bef11 (0x5f12 bytes) 000495[1] 1788461455.707264 write 0x6b7000 thru 0x6b91f7 (0x21f8 bytes) 000498[1] 1788461455.750612 trunc from 0x9f22d7 to 0x178198 (0x87a13e bytes) All operations completed A-OK! PASS 16a (48s) == sanityn test 16b: 500 iterations of dual-mount fsx at small size ========================================================== 14:51:00 (1788461460) keep default fallocate mode: 2 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.568831 s, 14.7 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.448988 s, 18.7 MB/s Chance of close/open is 1 in 50 Seed set to 1465 fd 0: /mnt/lustre/f16b.sanityn fd 1: /mnt/lustre2/f16b.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000001[0] 1788461466.269570 write 0x052640 thru 0x05265c (0x1d bytes) 000003[1] 1788461466.312235 write 0x7d50d3 thru 0x7d6ce8 (0x1c16 bytes) 000004[1] 1788461466.424174 read 0x3913cf thru 0x39241c (0x104e bytes) 000005[0] 1788461466.551946 mapread 0x4144d7 thru 0x415600 (0x112a bytes) 000010[1] 1788461466.644174 mapwrite 0x60cc0f thru 0x60cf30 (0x322 bytes) 000011[0] 1788461466.811578 punch from 0x5d7127 to 0x5d865a (0x1534 bytes) 000012[1] 1788461466.857734 mapwrite 0x5c89c9 thru 0x5c8e23 (0x45b bytes) 000013[1] 1788461467.102284 punch from 0x2715e3 to 0x27226a (0xc88 bytes) truncating to largest ever: 0x20010b 000014[0] 1788461467.149454 trunc from 0x7d6ce9 to 0x20010a (0x5d6bde bytes) 000015[0] 1788461467.236866 punch from 0x002e23 to 0x003e3f (0x101d bytes) 000019[0] 1788461467.311491 trunc from 0x20010b to 0x1954e2 (0x6ac28 bytes) 000020[0] 1788461467.361716 read 0x0496c9 thru 0x04ad8d (0x16c5 bytes) 000021[1] 1788461467.367455 mapwrite 0x860cbc thru 0x862053 (0x1398 bytes) 000022[1] 1788461467.599952 punch from 0x6d3c1c to 0x6d5108 (0x14ed bytes) 000023[0] 1788461467.628143 mapwrite 0x1f3167 thru 0x1f3655 (0x4ef bytes) 000024[1] 1788461467.716648 mapwrite 0x1d9173 thru 0x1d9c15 (0xaa3 bytes) 000027[0] 1788461467.885836 punch from 0x0e6677 to 0x0e80f0 (0x1a7a bytes) 000028[1] 1788461467.966643 mapwrite 0x7e0f36 thru 0x7e29a4 (0x1a6f bytes) 000030[1] 1788461468.112270 mapwrite 0x280f19 thru 0x28281c (0x1904 bytes) 000032[0] 1788461468.271773 trunc from 0x862054 to 0x153455 (0x70ebfe bytes) 000033[1] 1788461468.329780 read 0x14e0b2 thru 0x1500a7 (0x1ff6 bytes) 000034[1] 1788461468.352368 read 0x03c611 thru 0x03c836 (0x226 bytes) 000036[0] 1788461468.407944 mapread 0x0b835b thru 0x0b8ecd (0xb73 bytes) 000038[0] 1788461468.426962 punch from 0x0fc998 to 0x0fe85f (0x1ec8 bytes) 000039[0] 1788461468.461483 mapread 0x0e8b8e thru 0x0ea4c1 (0x1934 bytes) 000042[1] 1788461468.493817 punch from 0x140e07 to 0x141b3d (0xd37 bytes) truncating to largest ever: 0x80037a 000043[1] 1788461468.522166 trunc from 0x153456 to 0x800379 (0x6acf24 bytes) 000044[0] 1788461468.569496 write 0x435879 thru 0x435ffe (0x786 bytes) 000046[1] 1788461468.617945 mapread 0x7eef30 thru 0x7f0daa (0x1e7b bytes) 000048[0] 1788461468.780768 read 0x14cd79 thru 0x14e590 (0x1818 bytes) 000049[0] 1788461468.807306 punch from 0x295735 to 0x295a40 (0x30c bytes) 000051[1] 1788461468.845324 punch from 0x1a2b08 to 0x1a4260 (0x1759 bytes) 000052[0] 1788461468.870375 punch from 0x7226ae to 0x722dc5 (0x718 bytes) 000053[1] 1788461468.907146 punch from 0x37e679 to 0x37e6ba (0x42 bytes) 000056[1] 1788461468.945851 write 0x206cd4 thru 0x2080a3 (0x13d0 bytes) 000057[0] 1788461468.966078 read 0x05619d thru 0x056d03 (0xb67 bytes) 000058[1] 1788461468.986149 read 0x6a0cbe thru 0x6a1ded (0x1130 bytes) 000059[0] 1788461469.003547 read 0x29555a thru 0x296d9f (0x1846 bytes) 000060[1] 1788461469.008634 punch from 0x3745cf to 0x375252 (0xc84 bytes) 000062[1] 1788461469.036782 mapwrite 0x2a2b3b thru 0x2a364b (0xb11 bytes) 000063[1] 1788461469.091958 read 0x08c41f thru 0x08d3b5 (0xf97 bytes) 000065[1] 1788461469.126376 write 0x42cc43 thru 0x42e8cb (0x1c89 bytes) 000066[0] 1788461469.163665 punch from 0x3efefc to 0x3f0df7 (0xefc bytes) 000067[1] 1788461469.191557 write 0x8ed671 thru 0x8ef66c (0x1ffc bytes) 000069[0] 1788461469.244737 mapwrite 0x9f3cf0 thru 0x9f46c2 (0x9d3 bytes) 000071[0] 1788461469.371838 mapread 0x45ff31 thru 0x4600f0 (0x1c0 bytes) 000074[1] 1788461469.410703 read 0x326d64 thru 0x328037 (0x12d4 bytes) truncating to largest ever: 0x8b82bb 000075[0] 1788461469.439894 trunc from 0x9f46c3 to 0x8b82ba (0x13c408 bytes) 000079[1] 1788461469.543941 read 0x05e7bf thru 0x05f546 (0xd88 bytes) 000080[1] 1788461469.566149 punch from 0x2d11ba to 0x2d2165 (0xfac bytes) 000081[0] 1788461469.609129 read 0x16afc8 thru 0x16c839 (0x1872 bytes) 000083[1] 1788461469.639330 trunc from 0x8b82bb to 0x00f1ef (0x8a90cb bytes) 000084[0] 1788461469.709171 trunc from 0x00f1f0 to 0x6dac01 (0x6cba12 bytes) 000086[1] 1788461469.773970 mapread 0x56fdbd thru 0x56fe2f (0x73 bytes) 000090[1] 1788461469.818974 write 0x15aefb thru 0x15cef1 (0x1ff7 bytes) 000091[0] 1788461469.840078 write 0x98095f thru 0x98203e (0x16e0 bytes) 000092[0] 1788461469.875426 mapread 0x1c0125 thru 0x1c110d (0xfe9 bytes) 000096[1] 1788461469.922560 mapread 0x084cea thru 0x085305 (0x61c bytes) 000097[1] 1788461469.948397 punch from 0x16332e to 0x1652b0 (0x1f83 bytes) truncating to largest ever: 0x9e2e14 000099[1] 1788461469.962345 trunc from 0x98203f to 0x9e2e13 (0x60dd5 bytes) 000100[0] 1788461469.988713 trunc from 0x9e2e14 to 0x904486 (0xde98d bytes) 000103[1] 1788461470.073802 punch from 0x0eb989 to 0x0ec4f9 (0xb71 bytes) 000106[0] 1788461470.135077 trunc from 0x904487 to 0x01b349 (0x8e913d bytes) 000109[1] 1788461470.203796 punch from 0x006c74 to 0x007e8c (0x1219 bytes) 000110[0] 1788461470.231483 mapwrite 0x0e3bb4 thru 0x0e57e3 (0x1c30 bytes) 000114[0] 1788461470.356969 write 0x66bc00 thru 0x66bf2e (0x32f bytes) 000116[1] 1788461470.427949 read 0x1314fc thru 0x1326b2 (0x11b7 bytes) 000119[0] 1788461470.488909 punch from 0x3844ee to 0x384c37 (0x74a bytes) 000120[0] 1788461470.510067 punch from 0x3f2b6a to 0x3f3378 (0x80f bytes) 000121[1] 1788461470.514353 trunc from 0x66bf2f to 0x9a4b09 (0x338bdb bytes) 000122[0] 1788461470.573755 mapwrite 0x75322c thru 0x7534ec (0x2c1 bytes) 000123[1] 1788461470.646709 mapread 0x249903 thru 0x24b518 (0x1c16 bytes) 000124[1] 1788461470.701630 trunc from 0x9a4b0a to 0x366e9d (0x63dc6c bytes) 000125[1] 1788461470.766400 trunc from 0x366e9e to 0x7034ee (0x39c651 bytes) 000126[1] 1788461470.802821 punch from 0x531978 to 0x53285a (0xee3 bytes) 000128[1] 1788461470.813992 mapwrite 0x33ffeb thru 0x340fe2 (0xff8 bytes) 000129[0] 1788461470.864612 mapwrite 0x5bc72e thru 0x5bde46 (0x1719 bytes) 000131[1] 1788461470.988933 trunc from 0x7034ef to 0x01c6bb (0x6e6e33 bytes) 000132[0] 1788461471.037832 read 0x019922 thru 0x01a802 (0xee1 bytes) 000133[1] 1788461471.069500 mapread 0x01094d thru 0x0126d2 (0x1d86 bytes) 000134[0] 1788461471.095002 trunc from 0x01c6bc to 0x7b2135 (0x795a7a bytes) 000135[0] 1788461471.154940 mapwrite 0x4b39f5 thru 0x4b55f6 (0x1c02 bytes) 000137[1] 1788461471.367760 write 0x9bd457 thru 0x9bf2aa (0x1e54 bytes) 000139[1] 1788461471.412743 trunc from 0x9bf2ab to 0x67e835 (0x340a75 bytes) 000141[0] 1788461471.472116 read 0x57c194 thru 0x57dc42 (0x1aaf bytes) 000142[1] 1788461471.495972 punch from 0x08868f to 0x088fc2 (0x934 bytes) 000144[0] 1788461471.514589 mapread 0x16932d thru 0x1695b1 (0x285 bytes) 000146[1] 1788461471.540414 trunc from 0x67e836 to 0x258f81 (0x4258b4 bytes) 000147[0] 1788461471.621422 mapread 0x0c8032 thru 0x0c99cc (0x199b bytes) 000149[1] 1788461471.677480 write 0x3fba88 thru 0x3fda38 (0x1fb1 bytes) 000150[1] 1788461471.695496 punch from 0x21cc2f to 0x21d93c (0xd0e bytes) 000151[0] 1788461471.720602 mapwrite 0x090716 thru 0x091b12 (0x13fd bytes) 000154[1] 1788461471.839427 write 0x514030 thru 0x5153a2 (0x1373 bytes) 000155[0] 1788461471.854489 write 0x7e3c22 thru 0x7e5191 (0x1570 bytes) 000156[0] 1788461471.871279 mapwrite 0x745565 thru 0x745e47 (0x8e3 bytes) 000159[0] 1788461471.949234 read 0x50d1a4 thru 0x50d37d (0x1da bytes) 000160[1] 1788461471.957719 read 0x4acac5 thru 0x4ad8c0 (0xdfc bytes) 000161[0] 1788461471.988646 read 0x0ef430 thru 0x0efb8c (0x75d bytes) 000162[1] 1788461471.998635 write 0x5f136c thru 0x5f1fa3 (0xc38 bytes) 000163[0] 1788461472.035908 read 0x035209 thru 0x03704a (0x1e42 bytes) 000164[0] 1788461472.061789 punch from 0x5d0847 to 0x5d08dd (0x97 bytes) 000165[1] 1788461472.082483 read 0x685528 thru 0x685598 (0x71 bytes) 000166[1] 1788461472.098797 write 0x683539 thru 0x684516 (0xfde bytes) 000167[0] 1788461472.135152 mapread 0x07c855 thru 0x07cf94 (0x740 bytes) 000168[1] 1788461472.160927 punch from 0x584a90 to 0x58692a (0x1e9b bytes) 000169[1] 1788461472.187412 mapread 0x471d5d thru 0x4730b6 (0x135a bytes) 000174[1] 1788461472.283427 trunc from 0x7e5192 to 0x66c50d (0x178c84 bytes) 000175[0] 1788461472.314139 write 0x2a9967 thru 0x2a9af0 (0x18a bytes) 000176[0] 1788461472.348427 punch from 0x00bde4 to 0x00c0d0 (0x2ed bytes) 000177[1] 1788461472.370992 trunc from 0x66c50e to 0x11f25e (0x54d2af bytes) 000179[1] 1788461472.441777 punch from 0x032e06 to 0x0335fc (0x7f7 bytes) 000180[1] 1788461472.449601 punch from 0x0f8695 to 0x0f8974 (0x2e0 bytes) 000182[0] 1788461472.475651 write 0x4dd046 thru 0x4defe2 (0x1f9d bytes) 000183[0] 1788461472.500577 mapwrite 0x75c92c thru 0x75d4cc (0xba1 bytes) 000189[1] 1788461472.680261 trunc from 0x75d4cd to 0x0c9933 (0x693b99 bytes) 000195[1] 1788461472.790609 read 0x0affc1 thru 0x0b0072 (0xb2 bytes) 000196[0] 1788461472.838805 mapwrite 0x49f151 thru 0x49f1ae (0x5e bytes) 000197[1] 1788461472.918774 mapread 0x47b9f4 thru 0x47d3f6 (0x1a03 bytes) 000198[1] 1788461472.971943 mapwrite 0x2c4ac7 thru 0x2c6263 (0x179d bytes) 000204[0] 1788461473.144796 mapwrite 0x9b248a thru 0x9b3aaf (0x1626 bytes) 000205[0] 1788461473.248082 punch from 0x82cc97 to 0x82e076 (0x13e0 bytes) 000206[0] 1788461473.252336 punch from 0x6442f5 to 0x645d90 (0x1a9c bytes) 000207[0] 1788461473.281914 punch from 0x757581 to 0x75859a (0x101a bytes) 000209[1] 1788461473.288996 trunc from 0x9b3ab0 to 0x7a24af (0x211600 bytes) 000210[1] 1788461473.350412 mapread 0x1e5ce6 thru 0x1e75a6 (0x18c1 bytes) 000211[0] 1788461473.373550 trunc from 0x7a24b0 to 0x5dae32 (0x1c767d bytes) 000213[0] 1788461473.440551 mapwrite 0x326e28 thru 0x328c58 (0x1e31 bytes) 000215[1] 1788461473.546119 write 0x5259ed thru 0x527313 (0x1927 bytes) 000216[1] 1788461473.605637 write 0x755d85 thru 0x75766b (0x18e7 bytes) 000220[1] 1788461473.655462 mapread 0x21b4c3 thru 0x21cde9 (0x1927 bytes) 000222[1] 1788461473.686860 write 0x50cf80 thru 0x50e6ad (0x172e bytes) 000223[1] 1788461473.687958 trunc from 0x75766c to 0x2516eb (0x505f80 bytes) 000226[0] 1788461473.787697 punch from 0x127797 to 0x128cad (0x1517 bytes) 000228[1] 1788461473.828432 mapwrite 0x1ab320 thru 0x1abaf5 (0x7d6 bytes) 000230[1] 1788461473.922957 mapwrite 0x125a0a thru 0x125b5a (0x151 bytes) 000231[1] 1788461474.010849 read 0x230550 thru 0x231c78 (0x1729 bytes) 000232[1] 1788461474.136415 punch from 0x24f6ce to 0x2508b8 (0x11eb bytes) 000233[1] 1788461474.168800 write 0x820b8b thru 0x82143a (0x8b0 bytes) 000235[0] 1788461474.207844 mapwrite 0x04a9d1 thru 0x04bce9 (0x1319 bytes) 000238[1] 1788461474.398799 write 0x18c2bc thru 0x18e0a6 (0x1deb bytes) 000240[0] 1788461474.491105 mapread 0x0f998b thru 0x0fa456 (0xacc bytes) 000241[1] 1788461474.528260 punch from 0x4aa969 to 0x4aac7c (0x314 bytes) 000242[0] 1788461474.543811 mapwrite 0x8391fa thru 0x83a69e (0x14a5 bytes) 000243[1] 1788461474.634148 trunc from 0x83a69f to 0x1536aa (0x6e6ff4 bytes) 000247[0] 1788461474.746573 mapwrite 0x4addcb thru 0x4af73b (0x1971 bytes) 000248[0] 1788461474.844174 mapwrite 0x9fd3e6 thru 0x9fd663 (0x27e bytes) 000249[1] 1788461474.914052 trunc from 0x9fd664 to 0x62f4c7 (0x3ce19c bytes) 000251[0] 1788461474.992322 mapread 0x11ea9a thru 0x11fcb4 (0x121b bytes) 000252[1] 1788461475.037735 trunc from 0x62f4c8 to 0x204fe5 (0x42a4e2 bytes) 000256[0] 1788461475.117487 write 0x2e5a66 thru 0x2e6e72 (0x140d bytes) 000257[1] 1788461475.142125 punch from 0x03f612 to 0x0404c1 (0xeb0 bytes) 000258[0] 1788461475.191552 punch from 0x281b9e to 0x281c74 (0xd7 bytes) 000259[0] 1788461475.232892 punch from 0x0c9e36 to 0x0cb948 (0x1b13 bytes) 000262[0] 1788461475.281858 mapread 0x147330 thru 0x147e60 (0xb31 bytes) 000263[0] 1788461475.310837 mapwrite 0x58a671 thru 0x58bd88 (0x1718 bytes) 000265[1] 1788461475.425609 read 0x1c4700 thru 0x1c4847 (0x148 bytes) 000266[1] 1788461475.444641 mapwrite 0x6c6af8 thru 0x6c7dab (0x12b4 bytes) 000267[0] 1788461475.655247 mapread 0x3db438 thru 0x3dc0b3 (0xc7c bytes) 000268[0] 1788461475.732807 mapwrite 0x884d67 thru 0x8863db (0x1675 bytes) 000269[0] 1788461475.917338 mapwrite 0x392302 thru 0x394149 (0x1e48 bytes) 000270[1] 1788461476.005524 mapwrite 0x7b1e9a thru 0x7b3617 (0x177e bytes) 000271[1] 1788461476.115602 trunc from 0x8863dc to 0x7bd246 (0xc9195 bytes) 000272[0] 1788461476.199532 read 0x789fc7 thru 0x78ac58 (0xc92 bytes) 000274[0] 1788461476.248410 read 0x265365 thru 0x26716a (0x1e06 bytes) 000275[0] 1788461476.267986 trunc from 0x7bd247 to 0x7528a1 (0x6a9a5 bytes) 000276[1] 1788461476.367385 mapread 0x3ffef5 thru 0x3fff1d (0x29 bytes) 000280[1] 1788461476.440915 write 0x0ac15f thru 0x0ac301 (0x1a3 bytes) 000281[1] 1788461476.475598 punch from 0x4054c3 to 0x406705 (0x1243 bytes) 000282[0] 1788461476.496776 trunc from 0x7528a2 to 0x11ed60 (0x633b41 bytes) 000284[1] 1788461476.572736 read 0x06ee9f thru 0x0703ba (0x151c bytes) 000285[0] 1788461476.629478 write 0x6f2a05 thru 0x6f4230 (0x182c bytes) 000287[1] 1788461476.646642 read 0x3785ea thru 0x3797b0 (0x11c7 bytes) 000288[1] 1788461476.671003 read 0x45bfd9 thru 0x45c77f (0x7a7 bytes) 000289[0] 1788461476.733948 mapread 0x2c3825 thru 0x2c3ed5 (0x6b1 bytes) 000290[1] 1788461476.746646 mapwrite 0x6a820f thru 0x6a98cd (0x16bf bytes) 000292[1] 1788461476.885573 trunc from 0x6f4231 to 0x15288f (0x5a19a1 bytes) 000293[1] 1788461476.954742 trunc from 0x152890 to 0x4a215b (0x34f8cc bytes) 000294[1] 1788461476.997569 write 0x6252a7 thru 0x6252cf (0x29 bytes) 000296[1] 1788461477.049985 trunc from 0x6252d0 to 0x7e4703 (0x1bf434 bytes) 000299[1] 1788461477.089877 read 0x3d393f thru 0x3d4dbd (0x147f bytes) 000300[0] 1788461477.111898 write 0x8030db thru 0x804ead (0x1dd3 bytes) 000301[1] 1788461477.139671 read 0x00a91c thru 0x00ac14 (0x2f9 bytes) 000302[1] 1788461477.173552 write 0x96eae6 thru 0x96f217 (0x732 bytes) 000303[1] 1788461477.211704 mapwrite 0x7d04ee thru 0x7d1d45 (0x1858 bytes) 000304[1] 1788461477.277502 read 0x77999f thru 0x779bc9 (0x22b bytes) 000305[1] 1788461477.306944 mapwrite 0x6070ed thru 0x6074d4 (0x3e8 bytes) 000308[0] 1788461477.379682 write 0x84bdf2 thru 0x84d33c (0x154b bytes) 000309[0] 1788461477.421883 punch from 0x351063 to 0x3527e9 (0x1787 bytes) 000310[1] 1788461477.443229 trunc from 0x96f218 to 0x22f2e2 (0x73ff35 bytes) 000311[1] 1788461477.503531 mapwrite 0x011596 thru 0x0116e8 (0x153 bytes) 000312[1] 1788461477.568870 punch from 0x2097db to 0x209db3 (0x5d9 bytes) 000313[1] 1788461477.573795 punch from 0x17904c to 0x17adde (0x1d93 bytes) 000314[1] 1788461477.578459 mapwrite 0x364b5d thru 0x3658e9 (0xd8d bytes) 000316[1] 1788461477.743523 write 0x1e2e9d thru 0x1e3ebd (0x1021 bytes) 000317[0] 1788461477.754237 mapwrite 0x711620 thru 0x7123f0 (0xdd1 bytes) 000319[0] 1788461478.044183 write 0x8c93d0 thru 0x8c955d (0x18e bytes) 000321[1] 1788461478.121645 read 0x412c64 thru 0x4149d7 (0x1d74 bytes) 000322[1] 1788461478.162338 mapread 0x773b09 thru 0x77475c (0xc54 bytes) 000323[1] 1788461478.221926 mapwrite 0x8340ce thru 0x83597c (0x18af bytes) 000324[1] 1788461478.353819 mapread 0x8a9fe1 thru 0x8ab0bc (0x10dc bytes) 000326[0] 1788461478.401632 read 0x3bf7cf thru 0x3bfe3d (0x66f bytes) 000327[1] 1788461478.433591 mapwrite 0x94ba3a thru 0x94bda9 (0x370 bytes) 000329[0] 1788461478.743704 punch from 0x919a9e to 0x91a450 (0x9b3 bytes) 000330[0] 1788461478.797818 read 0x372f95 thru 0x37384a (0x8b6 bytes) 000332[1] 1788461478.842794 write 0x24f75b thru 0x24fb4d (0x3f3 bytes) 000333[0] 1788461478.875928 read 0x44453d thru 0x446379 (0x1e3d bytes) 000336[0] 1788461479.004743 punch from 0x484925 to 0x486558 (0x1c34 bytes) 000340[0] 1788461479.144498 mapread 0x55a4b7 thru 0x55bcbc (0x1806 bytes) 000341[0] 1788461479.186249 read 0x1633e9 thru 0x1653a0 (0x1fb8 bytes) 000343[0] 1788461479.233531 trunc from 0x94bdaa to 0x39d148 (0x5aec61 bytes) 000344[1] 1788461479.282601 trunc from 0x39d149 to 0x499686 (0xfc53e bytes) 000347[1] 1788461479.338435 mapwrite 0x832a58 thru 0x833c8f (0x1238 bytes) 000351[1] 1788461479.495982 read 0x65de51 thru 0x65f5c1 (0x1771 bytes) 000352[0] 1788461479.512750 trunc from 0x833c90 to 0x75c21c (0xd7a73 bytes) 000353[1] 1788461479.561556 trunc from 0x75c21d to 0x579ffa (0x1e2222 bytes) 000355[0] 1788461479.677734 trunc from 0x579ffb to 0x786ed3 (0x20ced9 bytes) 000356[0] 1788461479.734163 read 0x4fd3f0 thru 0x4fefc9 (0x1bda bytes) 000357[0] 1788461479.759955 mapread 0x597f1f thru 0x597fd2 (0xb4 bytes) 000359[1] 1788461479.804982 trunc from 0x786ed4 to 0x2e29b2 (0x4a4521 bytes) 000362[1] 1788461479.880742 mapread 0x001348 thru 0x002a1c (0x16d5 bytes) 000366[1] 1788461479.936708 mapwrite 0x7fd977 thru 0x7fe09f (0x729 bytes) 000369[1] 1788461480.039636 trunc from 0x7fe0a0 to 0x845767 (0x476c8 bytes) 000370[0] 1788461480.060446 read 0x6c79eb thru 0x6c9177 (0x178d bytes) 000371[0] 1788461480.079202 punch from 0x41b1f3 to 0x41cb40 (0x194e bytes) 000373[1] 1788461480.153163 punch from 0x09e72a to 0x09f1d5 (0xaac bytes) 000374[1] 1788461480.179491 mapwrite 0x52c614 thru 0x52c729 (0x116 bytes) 000375[0] 1788461480.341520 write 0x506caa thru 0x5080ef (0x1446 bytes) 000377[1] 1788461480.433885 mapread 0x107c95 thru 0x109b69 (0x1ed5 bytes) 000378[0] 1788461480.525238 mapread 0x44a2bd thru 0x44c26b (0x1faf bytes) 000379[0] 1788461480.596538 mapread 0x025c7f thru 0x026e17 (0x1199 bytes) 000381[0] 1788461480.707741 mapread 0x823ab5 thru 0x8249c1 (0xf0d bytes) 000382[0] 1788461480.726014 mapwrite 0x51f833 thru 0x520388 (0xb56 bytes) 000384[1] 1788461480.824965 write 0x380c48 thru 0x3813a6 (0x75f bytes) 000385[1] 1788461480.878691 mapwrite 0x3f1f85 thru 0x3f3bfe (0x1c7a bytes) 000386[1] 1788461481.003985 punch from 0x37a518 to 0x37c491 (0x1f7a bytes) 000387[0] 1788461481.026451 mapread 0x426be7 thru 0x4278d4 (0xcee bytes) 000389[0] 1788461481.083994 write 0x970478 thru 0x97234c (0x1ed5 bytes) 000392[0] 1788461481.122850 write 0x4c86e3 thru 0x4c9e7e (0x179c bytes) 000393[1] 1788461481.165828 trunc from 0x97234d to 0x6159cf (0x35c97d bytes) 000394[0] 1788461481.223754 write 0x65a7ff thru 0x65c2c8 (0x1aca bytes) 000396[1] 1788461481.313655 punch from 0x189301 to 0x18ae76 (0x1b76 bytes) 000399[1] 1788461481.387134 trunc from 0x65c2c9 to 0x5e3cb5 (0x78613 bytes) 000400[1] 1788461481.433522 mapwrite 0x76788e thru 0x768a47 (0x11ba bytes) 000401[0] 1788461481.532670 write 0x1a7d9c thru 0x1a85fa (0x85f bytes) 000403[1] 1788461481.619072 trunc from 0x768a48 to 0x5bb4a1 (0x1ad5a6 bytes) 000404[0] 1788461481.706834 mapread 0x0ddce8 thru 0x0de155 (0x46e bytes) 000405[0] 1788461481.841232 mapwrite 0x067436 thru 0x068d1b (0x18e6 bytes) 000406[0] 1788461482.032654 read 0x2b65af thru 0x2b6bf3 (0x645 bytes) 000408[1] 1788461482.078957 mapread 0x4a776d thru 0x4a7c58 (0x4ec bytes) 000409[1] 1788461482.110012 trunc from 0x5bb4a2 to 0x0f3a7b (0x4c7a26 bytes) 000410[1] 1788461482.143625 write 0x7d2b14 thru 0x7d3794 (0xc81 bytes) 000411[1] 1788461482.147814 write 0x67fda0 thru 0x681047 (0x12a8 bytes) 000412[0] 1788461482.180382 read 0x19a2d9 thru 0x19c142 (0x1e6a bytes) 000413[0] 1788461482.229251 mapread 0x516f6a thru 0x518433 (0x14ca bytes) 000414[1] 1788461482.297562 write 0x36c201 thru 0x36dff5 (0x1df5 bytes) 000415[1] 1788461482.323908 punch from 0x769dd2 to 0x76b13b (0x136a bytes) 000416[1] 1788461482.363127 mapwrite 0x330b6e thru 0x331159 (0x5ec bytes) 000418[1] 1788461482.427721 mapread 0x417955 thru 0x418f6e (0x161a bytes) 000419[1] 1788461482.436647 punch from 0x5270b2 to 0x528061 (0xfb0 bytes) 000420[1] 1788461482.441843 read 0x21066d thru 0x212172 (0x1b06 bytes) 000426[0] 1788461482.587448 mapwrite 0x52d034 thru 0x52ebcb (0x1b98 bytes) 000427[1] 1788461482.681568 read 0x0ae643 thru 0x0b03e5 (0x1da3 bytes) 000429[1] 1788461482.714351 trunc from 0x7d3795 to 0x76b61b (0x68179 bytes) 000430[0] 1788461482.767409 read 0x07e2af thru 0x07e5f2 (0x344 bytes) 000432[1] 1788461482.789887 read 0x01bf19 thru 0x01db85 (0x1c6d bytes) 000436[0] 1788461482.834720 read 0x3bf6fa thru 0x3bfc95 (0x59c bytes) 000438[0] 1788461482.857926 trunc from 0x76b61c to 0x6d6bbc (0x94a5f bytes) 000440[1] 1788461482.938252 punch from 0x13e80d to 0x14008a (0x187e bytes) 000442[1] 1788461482.980348 trunc from 0x6d6bbd to 0x898a10 (0x1c1e54 bytes) 000443[0] 1788461483.024973 write 0x1580f0 thru 0x1593f7 (0x1308 bytes) 000445[1] 1788461483.081134 read 0x3d3af0 thru 0x3d49d8 (0xee9 bytes) 000450[0] 1788461483.152685 mapread 0x367b91 thru 0x3697ee (0x1c5e bytes) 000451[1] 1788461483.189943 punch from 0x4dc9b0 to 0x4de64b (0x1c9c bytes) 000453[0] 1788461483.195858 mapread 0x48f8fd thru 0x4916b1 (0x1db5 bytes) 000454[0] 1788461483.240923 mapread 0x63bb37 thru 0x63d578 (0x1a42 bytes) 000457[0] 1788461483.269857 mapwrite 0x9e2dc6 thru 0x9e42f3 (0x152e bytes) 000458[1] 1788461483.427918 trunc from 0x9e42f4 to 0x70ce7c (0x2d7477 bytes) 000459[1] 1788461483.467110 mapwrite 0x1f3019 thru 0x1f4376 (0x135e bytes) 000460[1] 1788461483.506588 punch from 0x668337 to 0x66873d (0x407 bytes) 000462[0] 1788461483.534518 read 0x08cadc thru 0x08debf (0x13e4 bytes) 000463[0] 1788461483.562006 mapwrite 0x94e58f thru 0x950372 (0x1de4 bytes) 000466[1] 1788461483.680845 mapwrite 0x4793b6 thru 0x479a67 (0x6b2 bytes) 000467[1] 1788461483.755516 mapwrite 0x6ca391 thru 0x6cac99 (0x909 bytes) 000469[0] 1788461483.823277 mapread 0x42ae88 thru 0x42c0eb (0x1264 bytes) 000471[1] 1788461483.868622 punch from 0x848b49 to 0x8497d6 (0xc8e bytes) 000472[1] 1788461483.894642 mapread 0x8e08e8 thru 0x8e2728 (0x1e41 bytes) 000474[1] 1788461483.927887 punch from 0x69e2ec to 0x69ee87 (0xb9c bytes) 000478[1] 1788461483.994539 write 0x2e9437 thru 0x2e99a0 (0x56a bytes) 000479[1] 1788461484.000956 read 0x32e1b8 thru 0x32ef6d (0xdb6 bytes) 000481[1] 1788461484.023840 mapread 0x9446c0 thru 0x9451ea (0xb2b bytes) 000482[0] 1788461484.055997 write 0x39a923 thru 0x39ab61 (0x23f bytes) 000483[0] 1788461484.088482 punch from 0x0b1463 to 0x0b2c3c (0x17da bytes) 000487[1] 1788461484.146975 punch from 0x05b756 to 0x05ca89 (0x1334 bytes) 000489[1] 1788461484.186774 trunc from 0x950373 to 0x35908c (0x5f72e6 bytes) 000490[1] 1788461484.211169 write 0x42e073 thru 0x42f2b9 (0x1247 bytes) 000491[1] 1788461484.234971 trunc from 0x42f2ba to 0x438084 (0x8dcb bytes) 000492[1] 1788461484.255752 punch from 0x137286 to 0x1388c6 (0x1641 bytes) 000493[1] 1788461484.277488 mapread 0x0d0c6b thru 0x0d1f2e (0x12c4 bytes) 000495[0] 1788461484.304942 trunc from 0x438085 to 0x90ba07 (0x4d3983 bytes) 000496[1] 1788461484.346708 read 0x71892c thru 0x718d6e (0x443 bytes) 000497[1] 1788461484.376616 mapwrite 0x16aac0 thru 0x16afd1 (0x512 bytes) 000498[0] 1788461484.442203 write 0x10d1ce thru 0x10d925 (0x758 bytes) 000500[0] 1788461484.463362 read 0x21b64d thru 0x21bbc4 (0x578 bytes) All operations completed A-OK! PASS 16b (30s) == sanityn test 16c: verify data consistency on ldiskfs with cache disabled (b=17397) ========================================================== 14:51:31 (1788461491) SKIP: sanityn test_16c dio on ldiskfs only SKIP 16c (5s) == sanityn test 16d: Verify DIO and buffer IO with two clients ========================================================== 14:51:35 (1788461495) 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.44506 s, 18.8 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.308211 s, 27.2 MB/s ldlm.namespaces.MGC192.168.201.127@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8b4442ab6800.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8b44437aa800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8b4442ab6800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8b44437aa800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8b4442ab6800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8b44437aa800.lru_size=clear 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 13.0181 s, 8.1 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 3.9103 s, 26.8 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 2.74166 s, 38.2 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.8025 s, 15.4 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.16679 s, 11.4 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.69247 s, 22.3 MB/s PASS 16d (56s) == sanityn test 16e: Verify size consistency for O_DIRECT write ========================================================== 14:52:31 (1788461551) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.197066 s, 53.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106677 s, 9.8 MB/s PASS 16e (8s) == sanityn test 16f: rw sequential consistency vs drop_caches ========================================================== 14:52:39 (1788461559) TIMEOUT PASS 16f (29s) == sanityn test 16g: mmap rw sequential consistency vs drop_caches ========================================================== 14:53:09 (1788461589) TIMEOUT PASS 16g (28s) == sanityn test 16h: mmap read after truncate file ======= 14:53:37 (1788461617) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16h (9s) == sanityn test 16i: read after truncate file ============ 14:53:46 (1788461626) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16i (7s) == sanityn test 16j: race dio with buffered i/o ========== 14:53:54 (1788461634) 1+0 records in 1+0 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.5149 s, 17.3 MB/s bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.4137 s, 18.5 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.61256 s, 10.0 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.88114 s, 13.9 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.41091 s, 10.9 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.5534 s, 10.3 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.98492 s, 8.8 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.34242 s, 19.5 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.67475 s, 15.7 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.14637 s, 8.3 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.24178 s, 8.1 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.32298 s, 11.3 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.51291 s, 10.4 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.75384 s, 9.5 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.90522 s, 9.0 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.958 s, 8.9 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.32436 s, 7.9 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.53935 s, 7.4 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.71633 s, 7.1 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK PASS 16j (39s) == sanityn test 16k: Parallel FSX and drop caches should not panic ========================================================== 14:54:32 (1788461672) + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.1 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.2 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.3 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.4 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.5 -l 67108864 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read skipping zero size read skipping zero size read truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! + fsx 1 done: rc=0 All operations completed A-OK! Thu Sep 3 14:55:28 EDT 2026 + fsx 2 done: rc=0 Thu Sep 3 14:55:28 EDT 2026 + fsx 3 done: rc=0 Thu Sep 3 14:55:28 EDT 2026 + fsx 4 done: rc=0 Thu Sep 3 14:55:28 EDT 2026 + fsx 5 done: rc=0 Thu Sep 3 14:55:28 EDT 2026 PASS 16k (69s) == sanityn test 17: resource creation/LVB creation race ========================================================================= 14:55:41 (1788461741) fail_loc=0x8000030a PASS 17 (12s) == sanityn test 18: mmap sanity check =========================================================================================== 14:55:53 (1788461753) mmap test1: basic mmap operation (PASS, 0.18404s) mmap test2: MAP_PRIVATE not write back (PASS, 0.17799s) mmap test3: concurrent mmap ops on two nodes (PASS, 10.363s) mmap test4: c1 write to f1 from mmapped f2, c2 write to f1 from mmapped f1 (PASS, 4.6664s) mmap test5: read/write file to/from the buffer which mmapped to just this file (PASS, 0.13345s) mmap test6: check mmap write/read content on two nodes (PASS, 0.21693s) mmap test7: file i/o with an unmapped buffer (SKIPPED, 0s) mmap test8: SIGBUS for beyond file size (SKIPPED, 0s) mmap test9: SIGBUS for negative file offset (SKIPPED, 0s) mmap test10: multi-thread mmap access (PASS, 1.3042s) mmap test11: multi-thread shared mmap (PASS, 1.5496s) mmap test10: mtime not change for readonly mmap access (PASS, 1.1495s) PASS 18 (30s) == sanityn test 19: test concurrent uncached read races ========================================================================= 14:56:23 (1788461783) oleg127-server: error: get_param: param_path 'osd-*/lustre-OST*/read_cache_enable': No such file or directory pdsh@oleg127-client: oleg127-server: ssh exited with exit code 2 SKIP: sanityn test_19 not cache-capable obdfilter SKIP 19 (4s) == sanityn test 20: test extra readahead page left in cache ============================================================== 14:56:28 (1788461788) PASS 20 (9s) == sanityn test 21: Try to remove mountpoint on another dir ============================================================== 14:56:36 (1788461796) rmdir: removing directory, '/mnt/lustre/d21.sanityn' rmdir: failed to remove '/mnt/lustre/d21.sanityn': Device or resource busy rmdir: removing directory, '/mnt/lustre2/d21.sanityn' Removed mounted directory from another mountpoint, needs to be fixed PASS 21 (8s) == sanityn test 23: others should see updated atime while another read============================================================== 14:56:44 (1788461804) now is 1788461808 starting reads multiop /mnt/lustre/f23.sanityn vor20_c TMPPIPE=/tmp/multiop_open_wait_pipe.7649 new atime is 1788461869 PASS 23 (73s) == sanityn test 24a: lfs df [-ih] [path] test =================================================================================== 14:57:57 (1788461877) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 5120 3764224 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 5120 3764224 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre2 UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 115.3K 403 114.9K 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 114.5K 399 114.1K 1% /mnt/lustre[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.3K 403 114.9K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 399 114.1K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre2 UUID bytes Used Available Use% Mounted on lustre-MDT0000_UUID 2.1G 2.1M 2.1G 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3.6G 5.0M 3.6G 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3.6G 32.0M 3.6G 1% /mnt/lustre[OST:1] filesystem_summary: 7.2G 37.0M 7.2G 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118099 403 117696 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117231 399 116832 1% /mnt/lustre2[OST:1] filesystem_summary: 234914 386 234528 1% /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 5120 3764224 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.3K 403 114.9K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 399 114.1K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 118099 403 117696 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 117231 399 116832 1% /mnt/lustre[OST:1] filesystem_summary: 234914 386 234528 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118099 403 117696 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117231 399 116832 1% /mnt/lustre2[OST:1] filesystem_summary: 234914 386 234528 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 5120 3764224 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 5120 3764224 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre2 PASS 24a (8s) == sanityn test 24b: lfs df should show both filesystems ========================================================================= 14:58:05 (1788461885) PASS 24b (8s) == sanityn test 25a: change ACL on one mountpoint be seen on another ============================================================= 14:58:13 (1788461893) running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] PASS 25a (8s) == sanityn test 25b: change ACL under remote dir on one mountpoint be seen on another ========================================================== 14:58:22 (1788461902) SKIP: sanityn test_25b needs >= 2 MDTs SKIP 25b (4s) == sanityn test 26a: allow mtime to get older ============ 14:58:25 (1788461905) utime: good mknod times 1788461906 <= 1788461907 <= 1788461907 for /mnt/lustre/f26a utime: good mknod times 1788461906 <= 1788461907 <= 1788461907 for /mnt/lustre2/f26a utime: good utime mtimes 100000, atime 200000 utime: updated times for /mnt/lustre2/f26a PASS 26a (7s) == sanityn test 26b: sync mtime between ost and mds ====== 14:58:33 (1788461913) PASS 26b (9s) == sanityn test 26c: set-in-past on open file is not lost on close ========================================================== 14:58:42 (1788461922) multiop /mnt/lustre/f26c.sanityn vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7649 File: /mnt/lustre/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502398 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-09-03 14:58:44.000000000 -0400 Birth: 2026-09-03 14:58:44.000000000 -0400 File: /mnt/lustre2/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502398 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-09-03 14:58:44.000000000 -0400 Birth: 2026-09-03 14:58:44.000000000 -0400 PASS 26c (8s) SKIP: sanityn test_27 skipping excluded test 27 == sanityn test 30: recreate file race =================== 14:58:52 (1788461932) 0 PASS 30 (9s) == sanityn test 31a: voluntary cancel / blocking ast race======================================================================== 14:59:01 (1788461941) fail_loc=0x314 PASS 31a (9s) == sanityn test 31b: voluntary OST cancel / blocking ast race======================================================================== 14:59:10 (1788461950) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg127-server mds-ost sync done. sleep 5 for ZFS MDS Waiting for MDT destroys to complete fail_loc=0x314 fail_loc=0x316 fail_loc=0 fail_loc=0 Filesystem 1K-blocks Used Available Use% Mounted on 192.168.201.127@tcp:/lustre 7542784 38912 7496704 1% /mnt/lustre2 PASS 31b (27s) == sanityn test 31r: open-rename(replace) race =========== 14:59:37 (1788461977) -rw-r--r-- 1 root root 0 Sep 3 14:59 /mnt/lustre/f31r.sanityn.target fail_loc=0x1419 fail_val=3 PASS 31r (11s) == sanityn test 31s: open should not revalidate invalid dentry ========================================================== 14:59:48 (1788461988) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 1205: 43198 Terminated tail -f $tfile PASS 31s (9s) == sanityn test 31t: getattr should not revalidate invalid dentry ========================================================== 14:59:57 (1788461997) PASS 31t (9s) == sanityn test 32b: lockless i/o ======================== 15:00:06 (1788462006) SKIP: sanityn test_32b max_nolock_bytes is removed >= 2.17.53 SKIP 32b (3s) == sanityn test 32c: contention detection ================ 15:00:09 (1788462009) ldlm.namespaces.filter-lustre-OST0000_UUID.contended_locks=2 ldlm.namespaces.filter-lustre-OST0001_UUID.contended_locks=2 ldlm.namespaces.filter-lustre-OST0000_UUID.contention_events=0 ldlm.namespaces.filter-lustre-OST0001_UUID.contention_events=0 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.50264 s, 34.9 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.11315 s, 47.1 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.86418 s, 28.1 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.15518 s, 24.3 MB/s contention_seconds 4, contention_events 17 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.06972 s, 25.3 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.35751 s, 22.2 MB/s ldlm.namespaces.filter-lustre-OST0000_UUID.contention_events=0 ldlm.namespaces.filter-lustre-OST0001_UUID.contention_events=0 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.43955 s, 36.4 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.5122 s, 34.7 MB/s PASS 32c (28s) SKIP: sanityn test_33a skipping SLOW test 33a == sanityn test 33b: COS: cross create/delete, 2 clients, benchmark under remote dir ========================================================== 15:00:39 (1788462039) SKIP: sanityn test_33b Need two or more clients, have 1 SKIP 33b (3s) == sanityn test 33c: Cancel cross-MDT lock should trigger Sync-on-Lock-Cancel ========================================================== 15:00:43 (1788462043) SKIP: sanityn test_33c needs >= 2 MDTs SKIP 33c (3s) == sanityn test 33d: dependent transactions should trigger COS ========================================================== 15:00:46 (1788462046) SKIP: sanityn test_33d needs >= 2 MDTs SKIP 33d (3s) == sanityn test 33e: independent transactions shouldn't trigger COS ========================================================== 15:00:49 (1788462049) SKIP: sanityn test_33e needs >= 2 MDTs SKIP 33e (4s) == sanityn test 34: no lock timeout under IO ============= 15:00:53 (1788462053) fail_loc=0x512 fail_loc=0x512 lock should not expire writing on client1 reading on client2 success fail_loc=0x511 fail_loc=0x511 lock should expire writing on client1 reading on client2 success oleg127-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8b4442ab6800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8b4442ab6800.ost_server_uuid in IDLE state after 0 sec oleg127-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8b4442ab6800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8b4442ab6800.ost_server_uuid in IDLE state after 0 sec oleg127-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8b4442ab6800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8b4442ab6800.ost_server_uuid in IDLE state after 0 sec oleg127-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8b4442ab6800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8b4442ab6800.ost_server_uuid in IDLE state after 0 sec oleg127-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8b4442ab6800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8b4442ab6800.ost_server_uuid in IDLE state after 0 sec oleg127-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8b4442ab6800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8b4442ab6800.ost_server_uuid in IDLE state after 0 sec PASS 34 (145s) == sanityn test 35: -EINTR cp_ast vs. bl_ast race does not evict client ========================================================== 15:03:19 (1788462199) fail_loc=0x80000317 Race attempt 0 Wait for 50523 50535 for 60 sec... /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 1855: 50523 Killed createmany -o $MOUNT2/$tdir/a 4000 fail_loc=0x0 Filesystem Size Used Avail Use% Mounted on 192.168.201.127@tcp:/lustre 7.2G 39M 7.2G 1% /mnt/lustre 192.168.201.127@tcp:/lustre 7.2G 39M 7.2G 1% /mnt/lustre2 - open/close 895 (time 1788462217.15 total 10.00 last 89.46) - open/close 1420 (time 1788462227.16 total 20.02 last 52.44) - open/close 1778 (time 1788462237.18 total 30.04 last 35.71) - open/close 2388 (time 1788462247.19 total 40.05 last 60.94) - open/close 2857 (time 1788462257.21 total 50.07 last 46.82) - open/close 3579 (time 1788462267.23 total 60.09 last 72.06) total: 4000 open/close in 67.83 seconds: 58.97 ops/second PASS 35 (76s) == sanityn test 36: handle ESTALE/open-unlink correctly == 15:04:35 (1788462275) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg127-server mds-ost sync done. Waiting for MDT destroys to complete 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.25741 s, 41.7 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7649 sleep 5 for ZFS MDS Waiting for MDT destroys to complete *** cycle(0) *** before(7498752) after_dd(7438336) after(7489536) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.802314 s, 65.3 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7649 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(1) *** before(7489536) after_dd(7438336) after(7489536) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.675926 s, 77.6 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7649 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(2) *** before(7489536) after_dd(7440384) after(7491584) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.744512 s, 70.4 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7649 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(3) *** before(7491584) after_dd(7433216) after(7498752) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.78513 s, 66.8 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7649 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(4) *** before(7498752) after_dd(7426048) after(7498752) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.730797 s, 71.7 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7649 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(5) *** before(7498752) after_dd(7433216) after(7484416) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.01636 s, 51.6 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7649 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(6) *** before(7484416) after_dd(7440384) after(7491584) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.783261 s, 66.9 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7649 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(7) *** before(7491584) after_dd(7433216) after(7484416) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.716923 s, 73.1 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7649 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(8) *** before(7484416) after_dd(7435264) after(7498752) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.832046 s, 63.0 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7649 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(9) *** before(7498752) after_dd(7433216) after(7484416) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.749865 s, 69.9 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7649 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(10) *** before(7484416) after_dd(7437312) after(7488512) PASS 36 (334s) == sanityn test 37: check i_size is not updated for directory on close (bug 18695) ======================================================================== 15:10:09 (1788462609) multiop /mnt/lustre/d37.sanityn vD_c TMPPIPE=/tmp/multiop_open_wait_pipe.7649 - create 1574 (time 1788462623.06 total 10.01 last 157.26) - create 2918 (time 1788462633.07 total 20.02 last 134.27) - create 4380 (time 1788462643.07 total 30.02 last 146.20) - create 6009 (time 1788462653.08 total 40.03 last 162.68) - create 7545 (time 1788462663.09 total 50.04 last 153.50) - create 8966 (time 1788462673.09 total 60.04 last 142.03) total: 10000 create in 69.50 seconds: 143.88 ops/second PASS 37 (81s) == sanityn test 39a: file mtime does not change after rename ========================================================== 15:11:30 (1788462690) repeat after cancel_lru_locks PASS 39a (8s) == sanityn test 39b: file mtime the same on clients with/out lock ========================================================== 15:11:38 (1788462698) repeat after cancel_lru_locks PASS 39b (10s) == sanityn test 39c: check truncate mtime update ================================================================================ 15:11:48 (1788462708) repeat after cancel_lru_locks PASS 39c (11s) == sanityn test 39d: sync write should update mtime ====== 15:12:00 (1788462720) fail_loc=0x411 fail_loc=0 PASS 39d (8s) SKIP: sanityn test_40a skipping ALWAYS excluded test 40a == sanityn test 40b: pdirops: open|create and others ======================================================================== 15:12:10 (1788462730) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40b (17s) == sanityn test 40c: pdirops: link and others ======================================================================== 15:12:27 (1788462747) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40c (18s) == sanityn test 40d: pdirops: unlink and others ======================================================================== 15:12:45 (1788462765) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40d (15s) == sanityn test 40e: pdirops: rename and others ======================================================================== 15:13:00 (1788462780) No conflict No conflict No conflict No conflict No conflict No conflict PASS 40e (16s) == sanityn test 41a: pdirops: create vs mkdir ======================================================================== 15:13:16 (1788462796) mkdir: cannot create directory '/mnt/lustre2/f41a.sanityn': File exists Conflict PASS 41a (13s) == sanityn test 41b: pdirops: create vs create ======================================================================== 15:13:29 (1788462809) open: File exists Conflict PASS 41b (13s) == sanityn test 41c: pdirops: create vs link ======================================================================== 15:13:42 (1788462822) link: cannot create link '/mnt/lustre2/f41c.sanityn' to '/mnt/lustre2/f41c.sanityn-2': File exists Conflict PASS 41c (20s) == sanityn test 41d: pdirops: create vs unlink ======================================================================== 15:14:02 (1788462842) Conflict PASS 41d (16s) == sanityn test 41e: pdirops: create and rename (tgt) ======================================================================== 15:14:18 (1788462858) Conflict PASS 41e (16s) == sanityn test 41f: pdirops: create and rename (src) ======================================================================== 15:14:34 (1788462874) Conflict PASS 41f (14s) == sanityn test 41g: pdirops: create vs getattr ======================================================================== 15:14:48 (1788462888) Conflict PASS 41g (12s) == sanityn test 41h: pdirops: create vs readdir ======================================================================== 15:15:00 (1788462900) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 2427: lctl set_param -n fail_loc=0 2>/dev/null || true: No such file or directory Conflict PASS 41h (14s) == sanityn test 41i: reint_open: create vs create ======== 15:15:14 (1788462914) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 41i (1112s) == sanityn test 42a: pdirops: mkdir vs mkdir ======================================================================== 15:33:46 (1788464026) mkdir: cannot create directory '/mnt/lustre2/f42a.sanityn': File exists Conflict PASS 42a (14s) == sanityn test 42b: pdirops: mkdir vs create ======================================================================== 15:34:00 (1788464040) open: File exists Conflict PASS 42b (13s) == sanityn test 42c: pdirops: mkdir vs link ======================================================================== 15:34:13 (1788464053) link: cannot create link '/mnt/lustre2/f42c.sanityn' to '/mnt/lustre2/f42c.sanityn-2': File exists Conflict PASS 42c (15s) == sanityn test 42d: pdirops: mkdir vs unlink ======================================================================== 15:34:28 (1788464068) Conflict PASS 42d (14s) == sanityn test 42e: pdirops: mkdir and rename (tgt) ======================================================================== 15:34:42 (1788464082) mv: cannot overwrite directory '/mnt/lustre2/f42e.sanityn' with non-directory Conflict PASS 42e (14s) == sanityn test 42f: pdirops: mkdir and rename (src) ======================================================================== 15:34:56 (1788464096) Conflict PASS 42f (14s) == sanityn test 42g: pdirops: mkdir vs getattr ======================================================================== 15:35:10 (1788464110) Conflict PASS 42g (15s) == sanityn test 42h: pdirops: mkdir vs readdir ======================================================================== 15:35:26 (1788464126) Conflict PASS 42h (15s) == sanityn test 43a: rmdir,mkdir doesn't return -EEXIST ======================================================================== 15:35:40 (1788464140) PASS 43a (59s) == sanityn test 43b: pdirops: unlink vs create ======================================================================== 15:36:39 (1788464199) Conflict PASS 43b (16s) == sanityn test 43c: pdirops: unlink vs link ======================================================================== 15:36:55 (1788464215) Conflict PASS 43c (16s) == sanityn test 43d: pdirops: unlink vs unlink ======================================================================== 15:37:11 (1788464231) rm: cannot remove '/mnt/lustre2/f43d.sanityn': No such file or directory Conflict PASS 43d (15s) == sanityn test 43e: pdirops: unlink and rename (tgt) ======================================================================== 15:37:26 (1788464246) Conflict PASS 43e (15s) == sanityn test 43f: pdirops: unlink and rename (src) ======================================================================== 15:37:41 (1788464261) mv: cannot stat '/mnt/lustre2/f43f.sanityn': No such file or directory Conflict PASS 43f (16s) == sanityn test 43g: pdirops: unlink vs getattr ======================================================================== 15:37:57 (1788464277) stat: cannot statx '/mnt/lustre2/f43g.sanityn': No such file or directory Conflict PASS 43g (17s) == sanityn test 43h: pdirops: unlink vs readdir ======================================================================== 15:38:14 (1788464294) Conflict PASS 43h (14s) == sanityn test 43i: pdirops: unlink vs remote mkdir ===== 15:38:28 (1788464308) SKIP: sanityn test_43i needs >= 2 MDTs SKIP 43i (4s) == sanityn test 43j: racy mkdir return EEXIST ======================================================================== 15:38:32 (1788464312) mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists PASS 43j (123s) == sanityn test 43k: unlink vs create ==================== 15:40:35 (1788464435) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 43k (1306s) == sanityn test 44a: pdirops: rename tgt vs mkdir ======================================================================== 16:02:21 (1788465741) mkdir: cannot create directory '/mnt/lustre2/f44a.sanityn': File exists Conflict Thu Sep 3 16:02:29 EDT 2026 PASS 44a (11s) == sanityn test 44b: pdirops: rename tgt vs create ======================================================================== 16:02:33 (1788465753) open: File exists Conflict PASS 44b (16s) == sanityn test 44c: pdirops: rename tgt vs link ======================================================================== 16:02:48 (1788465768) link: cannot create link '/mnt/lustre2/f44c.sanityn' to '/mnt/lustre2/f44c.sanityn-3': File exists Conflict PASS 44c (16s) == sanityn test 44d: pdirops: rename tgt vs unlink ======================================================================== 16:03:04 (1788465784) Conflict PASS 44d (13s) == sanityn test 44e: pdirops: rename tgt and rename (tgt) ======================================================================== 16:03:17 (1788465797) Conflict PASS 44e (11s) == sanityn test 44f: pdirops: rename tgt and rename (src) ======================================================================== 16:03:28 (1788465808) Conflict PASS 44f (13s) == sanityn test 44g: pdirops: rename tgt vs getattr ======================================================================== 16:03:41 (1788465821) Conflict PASS 44g (12s) == sanityn test 44h: pdirops: rename tgt vs readdir ======================================================================== 16:03:54 (1788465834) Conflict PASS 44h (13s) == sanityn test 44i: pdirops: rename tgt vs remote mkdir ========================================================== 16:04:07 (1788465847) SKIP: sanityn test_44i needs >= 2 MDTs SKIP 44i (3s) == sanityn test 45a: rename,mkdir doesn't return -EEXIST ======================================================================== 16:04:10 (1788465850) PASS 45a (98s) == sanityn test 45b: pdirops: rename src vs create ======================================================================== 16:05:48 (1788465948) Conflict PASS 45b (14s) == sanityn test 45c: pdirops: rename src vs link ======================================================================== 16:06:02 (1788465962) Conflict PASS 45c (13s) == sanityn test 45d: pdirops: rename src vs unlink ======================================================================== 16:06:15 (1788465975) rm: cannot remove '/mnt/lustre2/f45d.sanityn': No such file or directory Conflict PASS 45d (16s) == sanityn test 45e: pdirops: rename src and rename (tgt) ======================================================================== 16:06:31 (1788465991) Conflict PASS 45e (16s) == sanityn test 45f: pdirops: rename src and rename (src) ======================================================================== 16:06:47 (1788466007) mv: cannot stat '/mnt/lustre2/f45f.sanityn': No such file or directory Conflict PASS 45f (19s) == sanityn test 45g: pdirops: rename src vs getattr ======================================================================== 16:07:06 (1788466026) stat: cannot statx '/mnt/lustre2/f45g.sanityn': No such file or directory Conflict PASS 45g (22s) == sanityn test 45h: pdirops: unlink vs readdir ======================================================================== 16:07:28 (1788466048) Conflict PASS 45h (15s) == sanityn test 45i: pdirops: rename src vs remote mkdir ========================================================== 16:07:44 (1788466064) SKIP: sanityn test_45i needs >= 2 MDTs SKIP 45i (5s) == sanityn test 45j: read vs rename ====================== 16:07:48 (1788466068) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 45j (1162s) == sanityn test 46a: pdirops: link vs mkdir ======================================================================== 16:27:10 (1788467230) mkdir: cannot create directory '/mnt/lustre2/f46a.sanityn': File exists Conflict PASS 46a (11s) == sanityn test 46b: pdirops: link vs create ======================================================================== 16:27:21 (1788467241) open: File exists Conflict PASS 46b (11s) == sanityn test 46c: pdirops: link vs link ======================================================================== 16:27:32 (1788467252) link: cannot create link '/mnt/lustre2/f46c.sanityn' to '/mnt/lustre2/f46c.sanityn': File exists Conflict PASS 46c (12s) == sanityn test 46d: pdirops: link vs unlink ======================================================================== 16:27:44 (1788467264) Conflict PASS 46d (12s) == sanityn test 46e: pdirops: link and rename (tgt) ======================================================================== 16:27:57 (1788467277) Conflict PASS 46e (11s) == sanityn test 46f: pdirops: link and rename (src) ======================================================================== 16:28:07 (1788467287) Conflict PASS 46f (12s) == sanityn test 46g: pdirops: link vs getattr ======================================================================== 16:28:19 (1788467299) Conflict PASS 46g (10s) == sanityn test 46h: pdirops: link vs readdir ======================================================================== 16:28:29 (1788467309) Conflict PASS 46h (12s) == sanityn test 46i: pdirops: link vs remote mkdir ======= 16:28:41 (1788467321) SKIP: sanityn test_46i needs >= 2 MDTs SKIP 46i (3s) == sanityn test 47a: pdirops: remote mkdir vs mkdir ====== 16:28:44 (1788467324) SKIP: sanityn test_47a needs >= 2 MDTs SKIP 47a (4s) == sanityn test 47b: pdirops: remote mkdir vs create ===== 16:28:48 (1788467328) SKIP: sanityn test_47b needs >= 2 MDTs SKIP 47b (3s) == sanityn test 47c: pdirops: remote mkdir vs link ======= 16:28:51 (1788467331) SKIP: sanityn test_47c needs >= 2 MDTs SKIP 47c (2s) == sanityn test 47d: pdirops: remote mkdir vs unlink ===== 16:28:54 (1788467334) SKIP: sanityn test_47d needs >= 2 MDTs SKIP 47d (2s) == sanityn test 47e: pdirops: remote mkdir and rename (tgt) ========================================================== 16:28:56 (1788467336) SKIP: sanityn test_47e needs >= 2 MDTs SKIP 47e (3s) == sanityn test 47f: pdirops: remote mkdir and rename (src) ========================================================== 16:28:59 (1788467339) SKIP: sanityn test_47f needs >= 2 MDTs SKIP 47f (3s) == sanityn test 47g: pdirops: remote mkdir vs getattr ==== 16:29:02 (1788467342) SKIP: sanityn test_47g needs >= 2 MDTs SKIP 47g (2s) == sanityn test 50: osc lvb attrs: enqueue vs. CP AST ======================================================================== 16:29:04 (1788467344) 10+0 records in 10+0 records out 10240 bytes (10 kB, 10 KiB) copied, 0.0992628 s, 103 kB/s fail_loc=0x410 fail_loc=0x0 PASS 50 (11s) == sanityn test 51a: layout lock: refresh layout should work ========================================================== 16:29:15 (1788467355) 0+1 records in 0+1 records out 158 bytes copied, 0.0317245 s, 5.0 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 3690: kill: (218817) - No such process PASS 51a (8s) == sanityn test 51b: layout lock: glimpse should be able to restart if layout changed ========================================================== 16:29:23 (1788467363) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00829164 s, 123 kB/s fail_loc=0x1404 fail_val=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0482748 s, 21.7 MB/s 3145728 PASS 51b (18s) == sanityn test 51c: layout lock: IT_LAYOUT blocked and correct layout can be returned ========================================================== 16:29:42 (1788467382) fail_loc=0x172 Setting layout to have 2 stripes ... 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0195896 s, 52.3 kB/s PASS 51c (11s) == sanityn test 51d: layout lock: losing layout lock should clean up memory map region ========================================================== 16:29:53 (1788467393) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.02564 s, 40.9 MB/s Before revoking layout lock: 1024 KB mapped PASS 51d (8s) == sanityn test 51e: lfs getstripe does not break leases, part 2 ========================================================== 16:30:01 (1788467401) /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 2455 0x997 0x240000400 write lease(2) has applied. /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 2455 0x997 0x240000400 read lease(1) has applied. PASS 51e (7s) == sanityn test 54: rename locking ======================= 16:30:08 (1788467408) llite.lustre-ffff8b4442ab6800.xattr_cache=1 llite.lustre-ffff8b44437aa800.xattr_cache=1 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505014 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-03 16:30:09.000000000 -0400 Modify: 2026-09-03 16:30:09.000000000 -0400 Change: 2026-09-03 16:30:09.000000000 -0400 Birth: 2026-09-03 16:30:10.000000000 -0400 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000154 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505017 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-03 16:30:15.000000000 -0400 Modify: 2026-09-03 16:30:16.000000000 -0400 Change: 2026-09-03 16:30:16.000000000 -0400 Birth: 2026-09-03 16:30:16.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505020 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-03 16:30:21.000000000 -0400 Modify: 2026-09-03 16:30:21.000000000 -0400 Change: 2026-09-03 16:30:21.000000000 -0400 Birth: 2026-09-03 16:30:22.000000000 -0400 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505019 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-03 16:30:21.000000000 -0400 Modify: 2026-09-03 16:30:22.000000000 -0400 Change: 2026-09-03 16:30:22.000000000 -0400 Birth: 2026-09-03 16:30:22.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000154 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505022 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-03 16:30:27.000000000 -0400 Modify: 2026-09-03 16:30:28.000000000 -0400 Change: 2026-09-03 16:30:28.000000000 -0400 Birth: 2026-09-03 16:30:28.000000000 -0400 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505023 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-03 16:30:27.000000000 -0400 Modify: 2026-09-03 16:30:28.000000000 -0400 Change: 2026-09-03 16:30:28.000000000 -0400 Birth: 2026-09-03 16:30:28.000000000 -0400 PASS 54 (29s) == sanityn test 55a: rename vs unlink target dir ========= 16:30:37 (1788467437) fail_loc=0x80000156 mv: cannot move '/mnt/lustre/d55a.sanityn/d1/d2' to '/mnt/lustre/d55a.sanityn/d3/d2': No such file or directory PASS 55a (13s) == sanityn test 55b: rename vs unlink source dir ========= 16:30:50 (1788467450) fail_loc=0x80000156 mv: cannot stat '/mnt/lustre/d55b.sanityn/d1/d2': No such file or directory PASS 55b (11s) == sanityn test 55c: rename vs unlink orphan target dir == 16:31:01 (1788467461) fail_loc=0x156 mv: cannot move '/mnt/lustre/d55c.sanityn/d1/d2' to '/mnt/lustre/d55c.sanityn/d3/d2': No such file or directory PASS 55c (16s) == sanityn test 55d: rename file vs link ================= 16:31:18 (1788467478) fail_loc=0x155 mv: '/mnt/lustre/d55d.sanityn/f1' and '/mnt/lustre/d55d.sanityn/d55d.sanityn/f1' are the same file PASS 55d (14s) == sanityn test 55e: rename race AB/BA under the same parent dir ========================================================== 16:31:31 (1788467491) SKIP: sanityn test_55e needs >= 2 MDTs SKIP 55e (2s) == sanityn test 55f: rename: (P1/A -> P2/B) race with (P2/B -> P1/A) ========================================================== 16:31:33 (1788467493) fail_loc=0x155 We survived after P1A_P2B/P2B_P1A race test fail_loc=0x0 PASS 55f (20s) == sanityn test 55g: rename: race with trylock =========== 16:31:53 (1788467513) mdt.lustre-MDT0000.enable_rename_trylock=1 fail_loc=0x155 mv: cannot move '/mnt/lustre2/d55g.sanityn/b1/b2' to a subdirectory of itself, '/mnt/lustre2/d55g.sanityn/a1/a2/a3/a4/b2' We survived race with trylock test fail_loc=0x0 PASS 55g (22s) == sanityn test 56a: test llverdev with single large stripe ========================================================== 16:32:15 (1788467535) run_llverdev /mnt/lustre/f56a.sanityn -p -s 16000000 -c 4k llverdev: /mnt/lustre/f56a.sanityn is 16777216000000 bytes (15625 GB) in size timestamp: 1788467536 chunksize: 4096 size: 16777216000000 write offset: 16384000000kB 0.911031 MB/s read offset: 16384000000kB inf MB/s llverdev: data verified successfully PASS 56a (73s) == sanityn test 56b: test llverdev and partial verify of wide stripe file ========================================================== 16:33:28 (1788467608) run_llverdev /mnt/lustre/f56b.sanityn -p -s 32000000 -c 4k llverdev: /mnt/lustre/f56b.sanityn is 33554432000000 bytes (31250 GB) in size timestamp: 1788467611 chunksize: 4096 size: 33554432000000 write offset: 32768000000kB 1.60624 MB/s read offset: 32768000000kB inf MB/s llverdev: data verified successfully PASS 56b (86s) == sanityn test 60: Verify data_version behaviour ======== 16:34:54 (1788467694) PASS 60 (8s) == sanityn test 70a: cd directory && rm directory ======== 16:35:02 (1788467702) PASS 70a (6s) == sanityn test 70b: remove files after calling rm_entry ========================================================== 16:35:08 (1788467708) PASS 70b (8s) == sanityn test 71a: correct file map just after write operation is finished ========================================================== 16:35:16 (1788467716) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.00885618 s, 7.4 MB/s SKIP: sanityn test_71a ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71a (3s) == sanityn test 71b: check fiemap support for stripecount > 1 ========================================================== 16:35:19 (1788467719) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0216351 s, 3.0 MB/s SKIP: sanityn test_71b ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71b (3s) == sanityn test 71c: check FIEMAP_EXTENT_LAST flag with different extents number ========================================================== 16:35:22 (1788467722) SKIP: sanityn test_71c support only ldiskfs ost SKIP 71c (3s) == sanityn test 71d: fiemap corruption test with fm_extent_count=0 ========================================================== 16:35:26 (1788467726) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0443412 s, 1.5 MB/s SKIP: sanityn test_71d ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71d (2s) == sanityn test 72: getxattr/setxattr cache should be consistent between nodes ========================================================== 16:35:29 (1788467729) llite.lustre-ffff8b4442ab6800.xattr_cache=1 llite.lustre-ffff8b44437aa800.xattr_cache=1 getfattr: Removing leading '/' from absolute path names user.attr1="value1" getfattr: Removing leading '/' from absolute path names user.attr1="value2" getfattr: Removing leading '/' from absolute path names getfattr: Removing leading '/' from absolute path names b362740ad371eae580e7f85452e8830b - 870e072da1ebfc20ea8c0cd3d6bdf3b6 - PASS 72 (6s) == sanityn test 73: getxattr should not cause xattr lock cancellation ========================================================== 16:35:35 (1788467735) llite.lustre-ffff8b4442ab6800.xattr_cache=1 llite.lustre-ffff8b44437aa800.xattr_cache=1 getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" PASS 73 (7s) == sanityn test 74: flock deadlock: different mounts ======================================================================== 16:35:42 (1788467742) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0152782 s, 67.0 kB/s 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0319848 s, 32.0 kB/s 231617: get lock2 231620: get lock1 231620: done 231620: get lock2 231617: done 231617: get lock1 fcntl cmd 7 failed: Resource deadlock avoided 231617: cannot get lock1: Resource deadlock avoided 231620: done 231620: exit rc=0 231617: done 231617: exit rc=0 PASS 74 (11s) == sanityn test 75: osc: upcall after unuse lock============================================================================= 16:35:53 (1788467753) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0418274 s, 50.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0180914 s, 58.0 MB/s fail_loc=0x31d /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 4373: 232201 Killed stat -c %s $DIR1/$tfile fail_loc=0 PASS 75 (11s) == sanityn test 76: Verify MDT open_files listing ======== 16:36:04 (1788467764) opening files: ........ closing files: ........ PASS 76 (52s) == sanityn test 77a: check FIFO NRS policy =============== 16:36:56 (1788467816) ost.OSS.ost.nrs_policies=fifo ost.OSS.ost_create.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_out.nrs_policies=fifo ost.OSS.ost_seq.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.221575 s, 75.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0249223 s, 42.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0240447 s, 43.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0301859 s, 34.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0662376 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104603 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134363 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157837 s, 6.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8b4442ab6800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193424 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139163 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160764 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178086 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203967 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237548 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202351 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216717 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.239498 s, 4.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8b4442ab6800.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffff8b4442ab6800.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0254101 s, 41.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.010446 s, 100 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0840573 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0634257 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00382222 s, 274 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0576309 s, 18.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0993971 s, 10.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.053092 s, 19.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0830107 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0607342 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0298799 s, 35.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0305333 s, 34.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0202877 s, 51.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136288 s, 76.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0269764 s, 38.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00916176 s, 114 MB/s removed '/mnt/lustre/d77a.sanityn/nrs_w_oleg127-client.virtnet' removed '/mnt/lustre/d77a.sanityn/nrs_r_oleg127-client.virtnet' removed directory '/mnt/lustre/d77a.sanityn' PASS 77a (8s) == sanityn test 77b: check CRR-N NRS policy ============== 16:37:04 (1788467824) ost.OSS.ost.nrs_policies=crrn ost.OSS.ost_create.nrs_policies=crrn ost.OSS.ost_io.nrs_policies=crrn ost.OSS.ost_out.nrs_policies=crrn ost.OSS.ost_seq.nrs_policies=crrn ost.OSS.ost.nrs_crrn_quantum=1 ost.OSS.ost_create.nrs_crrn_quantum=1 ost.OSS.ost_io.nrs_crrn_quantum=1 ost.OSS.ost_out.nrs_crrn_quantum=1 ost.OSS.ost_seq.nrs_crrn_quantum=1 policy: crr-n, crrn_quantum 1 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.344827 s, 48.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0351195 s, 29.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0910908 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124854 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105355 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139458 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0194075 s, 54.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167915 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120562 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193912 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.251112 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191811 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.266207 s, 3.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.272845 s, 3.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0814183 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208445 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.272462 s, 3.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8b4442ab6800.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff8b4442ab6800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11312 s, 9.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0759901 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123717 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0711533 s, 14.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0717954 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0755558 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0957985 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140025 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121522 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139892 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111241 s, 9.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118834 s, 8.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117052 s, 9.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124255 s, 8.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131918 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181394 s, 5.8 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg127-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg127-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost.nrs_crrn_quantum=64 ost.OSS.ost_create.nrs_crrn_quantum=64 ost.OSS.ost_io.nrs_crrn_quantum=64 ost.OSS.ost_out.nrs_crrn_quantum=64 ost.OSS.ost_seq.nrs_crrn_quantum=64 policy: crr-n, crrn_quantum 64 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.238289 s, 70.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0504795 s, 20.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.043149 s, 24.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147729 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137313 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163365 s, 6.4 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8b4442ab6800.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123341 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203836 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170113 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203549 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211066 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.226893 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.243236 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.259384 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203855 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.271617 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.387452 s, 2.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8b4442ab6800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0043853 s, 239 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00260383 s, 403 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00423742 s, 247 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00220423 s, 476 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150953 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126924 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106009 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146427 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0737191 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00522964 s, 201 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0337725 s, 31.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0628685 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0375112 s, 28.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0241456 s, 43.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0198113 s, 52.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0295644 s, 35.5 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg127-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg127-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77b (14s) == sanityn test 77c: check ORR NRS policy ================ 16:37:18 (1788467838) ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_orr_quantum=1 ost.OSS.ost_io.nrs_orr_offset_type=physical ost.OSS.ost_io.nrs_orr_supported=reads policy: orr, orr_quantum 1, orr_offset_type physical, orr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.241462 s, 69.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153683 s, 68.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0268983 s, 39.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0351117 s, 29.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0508513 s, 20.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.044431 s, 23.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.048837 s, 21.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0630775 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0266283 s, 39.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0324599 s, 32.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8b4442ab6800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155488 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173379 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223504 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215102 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.285708 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.311948 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.27681 s, 3.8 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8b4442ab6800.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0115437 s, 90.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00783331 s, 134 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0944477 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182074 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136271 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0319403 s, 32.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0727112 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140761 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0800056 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0453431 s, 23.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00490084 s, 214 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0071932 s, 146 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015241 s, 68.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.009107 s, 115 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0606901 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0114157 s, 91.9 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg127-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg127-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=writes ost.OSS.ost_io.nrs_orr_quantum=64 policy: orr, orr_quantum 64, orr_offset_type physical, orr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.240861 s, 69.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0276817 s, 37.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0485867 s, 21.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0697138 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0733571 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0484972 s, 21.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0480639 s, 21.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0185896 s, 56.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106333 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0991613 s, 10.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0834818 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132322 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158493 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105549 s, 9.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8b4442ab6800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff8b4442ab6800.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159244 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156745 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.254033 s, 4.1 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8b4442ab6800.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0888114 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0103832 s, 101 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0363793 s, 28.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0692221 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103224 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0773005 s, 13.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0450629 s, 23.3 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0901268 s, 11.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0923189 s, 11.4 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0576993 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10796 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106938 s, 9.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107842 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0636853 s, 16.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1136 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116436 s, 9.0 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg127-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg127-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=reads_and_writes ost.OSS.ost_io.nrs_orr_offset_type=logical policy: orr, orr_quantum 64, orr_offset_type logical, orr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.158722 s, 106 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0182827 s, 57.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0274546 s, 38.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.053249 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0639588 s, 16.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0325336 s, 32.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0510777 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0709226 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.022559 s, 46.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0654498 s, 16.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0859586 s, 12.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112459 s, 9.3 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137302 s, 7.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0906446 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117392 s, 8.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118037 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124129 s, 8.4 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8b4442ab6800.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0837828 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0804866 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0891255 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.064157 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0985539 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0556196 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0657532 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0604501 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0328177 s, 32.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0854792 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0590405 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0880071 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0927872 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0739564 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0750416 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0833847 s, 12.6 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg127-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg127-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77c (13s) == sanityn test 77d: check TRR nrs policy ================ 16:37:31 (1788467851) ost.OSS.ost_io.nrs_policies=trr ost.OSS.ost_io.nrs_trr_quantum=1 ost.OSS.ost_io.nrs_trr_offset_type=physical ost.OSS.ost_io.nrs_trr_supported=reads policy: trr, trr_quantum 1, trr_offset_type physical, trr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.316611 s, 53.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0481513 s, 21.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.049693 s, 21.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114758 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169582 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0697166 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10514 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183866 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138863 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110162 s, 9.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119942 s, 8.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184502 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203447 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168936 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205761 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180384 s, 5.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8b4442ab6800.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff8b4442ab6800.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13528 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0788227 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.062883 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0441862 s, 23.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0659465 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0623688 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0538868 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0730903 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0699942 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0472151 s, 22.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.059602 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0624712 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.05728 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0727379 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0784366 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0725308 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0668216 s, 15.7 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg127-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg127-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=writes ost.OSS.ost_io.nrs_trr_quantum=64 policy: trr, trr_quantum 64, trr_offset_type physical, trr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.161686 s, 104 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0496227 s, 21.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0721442 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1019 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172508 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18093 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154076 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135473 s, 7.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164926 s, 6.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203654 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180212 s, 5.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215257 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223747 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215108 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.226632 s, 4.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200836 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.235175 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0438918 s, 23.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0748781 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0528437 s, 19.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0561 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0749673 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0583498 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0730778 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0538292 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0939762 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0815883 s, 12.9 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0958736 s, 10.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0316189 s, 33.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0482852 s, 21.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0879396 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0705986 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.060183 s, 17.4 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg127-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg127-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=reads_and_writes ost.OSS.ost_io.nrs_trr_offset_type=logical policy: trr, trr_quantum 64, trr_offset_type logical, trr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.315636 s, 53.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0302861 s, 34.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0402367 s, 26.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0585004 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119449 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0884658 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0975174 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147963 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132859 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160746 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160938 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168905 s, 6.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170734 s, 6.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162642 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215745 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207359 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.225897 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0672237 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0462923 s, 22.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0577347 s, 18.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0685531 s, 15.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0327509 s, 32.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0672941 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0718471 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0290331 s, 36.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0250563 s, 41.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0583065 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0470715 s, 22.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0594071 s, 17.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0443491 s, 23.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0435661 s, 24.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.047749 s, 22.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0496566 s, 21.1 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg127-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg127-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77d (12s) == sanityn test 77e: check TBF NID nrs policy ============ 16:37:44 (1788467864) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_tbf_rule=start localhost nid={0@lo} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start clients nid={192.168.201.27@tcp} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start others nid={*.*.*.*@tcp} rate=50 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.228045 s, 73.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.041769 s, 25.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0432508 s, 24.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.080652 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0641805 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117006 s, 9.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124494 s, 8.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0960867 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131972 s, 7.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12217 s, 8.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127771 s, 8.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105307 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128235 s, 8.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10513 s, 10.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109391 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148677 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173581 s, 6.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0554859 s, 18.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0367365 s, 28.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0296064 s, 35.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.040454 s, 25.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0339206 s, 30.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0399637 s, 26.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0332735 s, 31.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0304572 s, 34.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0307446 s, 34.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.043111 s, 24.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.033878 s, 31.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0577617 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0452904 s, 23.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0301763 s, 34.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0631488 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0453204 s, 23.1 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg127-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg127-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change localhost rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change clients rate=101 ost.OSS.ost_io.nrs_tbf_rule=change others rate=51 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.175362 s, 95.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164933 s, 63.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0292991 s, 35.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0386618 s, 27.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142153 s, 73.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.060583 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171961 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16975 s, 6.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8b4442ab6800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123326 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146175 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137676 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211942 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202731 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222107 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222559 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17207 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.256759 s, 4.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8b4442ab6800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff8b4442ab6800.lock_unused_count=6 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.230923 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151011 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141275 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156268 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157856 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165345 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.24162 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169862 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198411 s, 5.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222153 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.256823 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179735 s, 5.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.218141 s, 4.8 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191896 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189052 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194902 s, 5.4 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg127-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg127-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop localhost ost.OSS.ost_io.nrs_tbf_rule=stop clients ost.OSS.ost_io.nrs_tbf_rule=stop others 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.252984 s, 66.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0196848 s, 53.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0254041 s, 41.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0337764 s, 31.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0231641 s, 45.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0564692 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0433421 s, 24.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165476 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109947 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173942 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174178 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194444 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161998 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165378 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22191 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223938 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209522 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00656225 s, 160 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00240119 s, 437 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0108777 s, 96.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00991146 s, 106 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168345 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.225201 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168734 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196353 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.219897 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.248022 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130446 s, 80.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.273362 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120522 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0646904 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0370182 s, 28.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0339102 s, 30.9 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg127-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg127-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.265279 s, 63.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0194989 s, 53.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0452825 s, 23.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.062199 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.036703 s, 28.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109719 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10531 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0675577 s, 15.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0631539 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0672704 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160468 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10414 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110717 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159214 s, 6.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8b4442ab6800.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffff8b4442ab6800.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121747 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151533 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15284 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0124182 s, 84.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0746868 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.050075 s, 20.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0911526 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00177645 s, 590 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0771179 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0577984 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.031861 s, 32.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0420167 s, 25.0 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0378908 s, 27.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0321103 s, 32.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0214387 s, 48.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.017945 s, 58.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154505 s, 67.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.030601 s, 34.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0282614 s, 37.1 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg127-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg127-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' PASS 77e (20s) == sanityn test 77f: check TBF JobID nrs policy ========== 16:38:03 (1788467883) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start runas jobid={iozone.500 dd.500 tiotest.500} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start iozone_runas jobid={iozone.500} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg127-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.318295 s, 52.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0300777 s, 34.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0763158 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0474861 s, 22.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0794139 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102158 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.088207 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171397 s, 6.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186423 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146143 s, 7.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200716 s, 5.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.238037 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.224164 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132429 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139453 s, 7.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172287 s, 6.1 MB/srunning as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.32151 s, 3.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00533867 s, 196 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00639421 s, 164 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0044996 s, 233 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00289162 s, 363 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0927721 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.093835 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10941 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0968717 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.097987 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0786991 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00697425 s, 150 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0700589 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148739 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105539 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0320256 s, 32.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0220476 s, 47.6 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg127-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg127-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change runas rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change iozone_runas rate=101 ost.OSS.ost_io.nrs_tbf_rule=change dd_runas rate=51 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg127-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.302114 s, 55.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0398006 s, 26.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0327045 s, 32.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114326 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116454 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149137 s, 7.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127711 s, 8.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167617 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174356 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189755 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150427 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178783 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193177 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192909 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20876 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213243 s, 4.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8b4442ab6800.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff8b4442ab6800.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.292869 s, 3.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0858301 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.093344 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0470412 s, 22.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0879506 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0375989 s, 27.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0641715 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0614532 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103201 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139018 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0706461 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152702 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0821741 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0650308 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0773063 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0841319 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178659 s, 5.9 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg127-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg127-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop runas ost.OSS.ost_io.nrs_tbf_rule=stop iozone_runas ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg127-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.32272 s, 52.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0377657 s, 27.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0506974 s, 20.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0782672 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111599 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105638 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0882794 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111952 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143156 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130801 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117851 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161562 s, 6.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8b4442ab6800.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff8b4442ab6800.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171962 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195309 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188004 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206748 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.258954 s, 4.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied ldlm.namespaces.lustre-OST0000-osc-ffff8b4442ab6800.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff8b4442ab6800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00274409 s, 382 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0177433 s, 59.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0238583 s, 44.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0437929 s, 23.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11447 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0431082 s, 24.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0493525 s, 21.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0531989 s, 19.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100658 s, 10.4 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0961207 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0612548 s, 17.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112891 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00486326 s, 216 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183523 s, 57.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013537 s, 77.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0269092 s, 39.0 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg127-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg127-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_policies=fifo running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg127-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.31326 s, 53.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0790733 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0666196 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126252 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200458 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220649 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10597 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.244593 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222505 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188722 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147435 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20009 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195366 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220728 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162585 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.249397 s, 4.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.316383 s, 3.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8b4442ab6800.lock_unused_count=3 ldlm.namespaces.lustre-OST0001-osc-ffff8b4442ab6800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0992336 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00255696 s, 410 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130793 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105705 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175779 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101697 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0857694 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1441 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0209526 s, 50.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0709662 s, 14.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0563637 s, 18.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0955922 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100307 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0732091 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0626601 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0529255 s, 19.8 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg127-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg127-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' PASS 77f (27s) == sanityn test 77g: Change TBF type directly ============ 16:38:30 (1788467910) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 ost.OSS.ost_io.nrs_policies=fifo PASS 77g (13s) == sanityn test 77h: Wrong policy name should report error, not LBUG ========================================================== 16:38:43 (1788467923) oleg127-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=abc: No such file or directory oleg127-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='abc': No such file or directory pdsh@oleg127-client: oleg127-server: ssh exited with exit code 2 oleg127-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc: Invalid argument oleg127-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc': Invalid argument pdsh@oleg127-client: oleg127-server: ssh exited with exit code 22 oleg127-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf reg abc: Invalid argument oleg127-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf reg abc': Invalid argument pdsh@oleg127-client: oleg127-server: ssh exited with exit code 22 oleg127-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc efg: Invalid argument oleg127-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc efg': Invalid argument pdsh@oleg127-client: oleg127-server: ssh exited with exit code 22 PASS 77h (10s) == sanityn test 77i: Change rank of TBF rule ============= 16:38:53 (1788467933) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start before jobid={jobid} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start after jobid={jobid} rate=1000 rank=default ost.OSS.ost_io.nrs_tbf_rule=start target jobid={jobid} rate=1000 rank=after Move before itself ost.OSS.ost_io.nrs_tbf_rule=change target rank=target Move to higher rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=before Move to lower rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=after Move before default ost.OSS.ost_io.nrs_tbf_rule=change target rank=default ost.OSS.ost_io.nrs_policies=fifo PASS 77i (19s) == sanityn test 77j: check TBF-OPCode NRS policy ========= 16:39:12 (1788467952) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf opcode ost.OSS.ost_io.nrs_tbf_rule=start ost_r opcode={ost_read} rate=5 ost.OSS.ost_io.nrs_tbf_rule=start ost_w opcode={ost_write} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.326794 s, 51.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0301003 s, 34.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0702482 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0546019 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.05922 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0401354 s, 26.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0926765 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155612 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186554 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177123 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185441 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196904 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174679 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.250719 s, 4.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201316 s, 5.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199103 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203389 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0429439 s, 24.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0397015 s, 26.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0543012 s, 19.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0509295 s, 20.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0560509 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0669546 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0618226 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0647924 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0703046 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0582993 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0778468 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0720643 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0897063 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0765644 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122633 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0899137 s, 11.7 MB/s removed '/mnt/lustre/d77j.sanityn/nrs_w_oleg127-client.virtnet' removed '/mnt/lustre/d77j.sanityn/nrs_r_oleg127-client.virtnet' removed directory '/mnt/lustre/d77j.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 5 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.66581 s, 15.7 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.5775 s, 5.4 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77j.sanityn sleep 5 for ZFS MDS Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_r ost.OSS.ost_io.nrs_tbf_rule=stop ost_w ost.OSS.ost_io.nrs_policies=fifo PASS 77j (49s) == sanityn test 77ja: check TBF-UID/GID NRS policy ======= 16:40:01 (1788468001) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg127-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg127-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg127-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg127-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg127-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.492195 s, 34.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0417409 s, 25.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0759214 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0863958 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0644555 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0830173 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10658 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111096 s, 9.4 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0326389 s, 32.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.099833 s, 10.5 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118946 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111054 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14681 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167191 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198422 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193597 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.214752 s, 4.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0508888 s, 20.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.080893 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0769621 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0712856 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0928114 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0527614 s, 19.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0870101 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0768719 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0934823 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0993986 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.065441 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.096512 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0717233 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.069303 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0735805 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0870414 s, 12.0 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg127-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg127-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.1192 s, 5.2 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9779 s, 5.2 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg127-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg127-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg127-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg127-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg127-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.26172 s, 13.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112648 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108765 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120899 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0908055 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209485 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.240824 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.274718 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.248075 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.262199 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.231495 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.236794 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.28762 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.306221 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.484734 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.477166 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.521772 s, 2.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0354356 s, 29.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0351574 s, 29.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0380716 s, 27.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0442439 s, 23.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0767937 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0831169 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.049917 s, 21.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0476784 s, 22.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0499281 s, 21.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0672074 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10069 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.058018 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0713886 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0923296 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.083423 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0821826 s, 12.8 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg127-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg127-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.1474 s, 5.2 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9436 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77ja (155s) == sanityn test 77jb: check TBF-UID/GID NRS policy on files that don't belong to us ========================================================== 16:42:36 (1788468156) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg127-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg127-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg127-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg127-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg127-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.26919 s, 13.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0959182 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160732 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163548 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1262 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160399 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210821 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173542 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.204542 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.27941 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223875 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200729 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.331072 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.329443 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.272711 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.294997 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.279348 s, 3.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0744228 s, 14.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0194755 s, 53.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0591504 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0497952 s, 21.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0740665 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0649827 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0351445 s, 29.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0578847 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00462732 s, 227 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100876 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0526254 s, 19.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0773017 s, 13.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163235 s, 64.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0849684 s, 12.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0883696 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00687717 s, 152 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg127-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg127-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.041 s, 5.2 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.866 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg127-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg127-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg127-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg127-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg127-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.25955 s, 13.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0566344 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0576979 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110928 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125361 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130475 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165436 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152415 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149963 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21645 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.231821 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.258569 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.257273 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.27846 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.253264 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.29214 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.293556 s, 3.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0922517 s, 11.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0952191 s, 11.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.098688 s, 10.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12979 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126539 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0934066 s, 11.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133586 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140883 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146414 s, 7.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144871 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144183 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161931 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150448 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159446 s, 6.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124347 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190764 s, 5.5 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg127-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg127-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.2203 s, 5.2 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.0425 s, 5.2 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77jb (156s) == sanityn test 77k: check TBF policy with UID/GID/JobID/OPCode expression ========================================================== 16:45:12 (1788468312) ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg127-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.539249 s, 31.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0853778 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0401188 s, 26.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0358872 s, 29.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128545 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0457229 s, 22.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0947793 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.427118 s, 2.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.365582 s, 2.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.272165 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.426044 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.459577 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.437182 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.334141 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.362681 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.411191 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.560067 s, 1.9 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8b4442ab6800.lock_unused_count=4 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160944 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0573336 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118734 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.05813 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177094 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0659023 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0688923 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0713012 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0789153 s, 13.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195902 s, 5.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151688 s, 6.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0849904 s, 12.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0998986 s, 10.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0934893 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103109 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149992 s, 7.0 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg127-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg127-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.5857 s, 9.9 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0086 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.201.27@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.201.27@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.218306 s, 76.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.045078 s, 23.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.051769 s, 20.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0303018 s, 34.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0573089 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0969207 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135459 s, 7.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0831342 s, 12.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166408 s, 6.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139288 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131976 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187484 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127372 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111933 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161925 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148501 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139668 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169581 s, 61.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0424859 s, 24.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.071363 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0269505 s, 38.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0366065 s, 28.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0332261 s, 31.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0578985 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0384012 s, 27.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0696828 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101124 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0517201 s, 20.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0575284 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0604499 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0553491 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0671343 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.071574 s, 14.7 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg127-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg127-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.87911 s, 13.3 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.93702 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.201.27@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg127-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.347504 s, 48.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0267055 s, 39.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0902256 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0772903 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164878 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191161 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19446 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155866 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207248 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.25137 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.230694 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162748 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.261618 s, 4.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213825 s, 4.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20161 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.241732 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.319139 s, 3.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0359134 s, 29.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0602947 s, 17.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0212766 s, 49.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0728477 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00988929 s, 106 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0178984 s, 58.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0734683 s, 14.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132939 s, 78.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0455347 s, 23.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0870862 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0915058 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10858 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00979683 s, 107 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103821 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0838909 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164037 s, 63.9 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg127-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg127-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.39927 s, 12.5 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.50456 s, 19.0 MB/s Read runtime is 7 s, speed is 14.285714 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg127-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.790276 s, 21.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0734947 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0329646 s, 31.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0243764 s, 43.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133104 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0968746 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.240219 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150132 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187684 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160571 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.254221 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.326282 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.231058 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.263318 s, 4.0 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210232 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.250699 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.363594 s, 2.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0217116 s, 48.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0286255 s, 36.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0263723 s, 39.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0362865 s, 28.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0301146 s, 34.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0527174 s, 19.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.045876 s, 22.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0642172 s, 16.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0492171 s, 21.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0516579 s, 20.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0922232 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0623478 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0987279 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0686655 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0678282 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119712 s, 8.8 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg127-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg127-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 12.4017 s, 8.5 MB/s Write runtime is 14 s, speed is 7.142857 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0319 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0063 s, 10.5 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0065 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg127-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.326528 s, 51.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0381969 s, 27.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0611064 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114382 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154663 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138371 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120761 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13607 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163031 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143229 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0988181 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160979 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181657 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.217002 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.214795 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191489 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18882 s, 5.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0328524 s, 31.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0269088 s, 39.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0489463 s, 21.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0110123 s, 95.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00248561 s, 422 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132608 s, 79.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0197944 s, 53.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0306794 s, 34.2 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0310008 s, 33.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0318538 s, 32.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137698 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103981 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103513 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138355 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0263291 s, 39.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0071997 s, 146 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg127-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg127-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.1727 s, 5.2 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.6374 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_ug ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg127-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.300263 s, 55.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0461072 s, 22.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0756323 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0451845 s, 23.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107016 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0513229 s, 20.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130133 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143696 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135413 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122511 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164696 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137515 s, 7.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1922 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.226523 s, 4.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199102 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194859 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.228432 s, 4.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0274423 s, 38.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0360534 s, 29.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0880413 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0495886 s, 21.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0383408 s, 27.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0412369 s, 25.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0446859 s, 23.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0406215 s, 25.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112033 s, 9.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0580064 s, 18.1 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0598719 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0632795 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0689238 s, 15.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0714223 s, 14.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0705806 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0740205 s, 14.2 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg127-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg127-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.55054 s, 16.0 MB/s Write runtime is 7 s, speed is 14.285714 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.93689 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg127-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.239302 s, 70.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0353072 s, 29.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0540951 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0712777 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12686 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14888 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147115 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116662 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131702 s, 8.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158477 s, 6.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146833 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15072 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136132 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150447 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173876 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145361 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154339 s, 6.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00254382 s, 412 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00251831 s, 416 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00248127 s, 423 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00642382 s, 163 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00252374 s, 415 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0025167 s, 417 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00358456 s, 293 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.002683 s, 391 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104344 s, 10.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131302 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0839327 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100456 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140997 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0350275 s, 29.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.031123 s, 33.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157304 s, 66.7 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg127-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg127-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.4564 s, 10.0 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.92266 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 12.2436 s, 8.6 MB/s Write runtime is 14 s, speed is 7.142857 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.88102 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_policies=fifo PASS 77k (438s) == sanityn test 77kb: Check different granular TBF type combination: jobid+opcode ========================================================== 16:52:30 (1788468750) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/nrs_r_oleg127-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.281936 s, 59.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0353544 s, 29.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0723746 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133973 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117555 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131622 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0881435 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0902686 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139651 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143235 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156251 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148854 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145351 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223602 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.228603 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208155 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.239311 s, 4.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00651709 s, 161 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0313821 s, 33.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149327 s, 70.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0544687 s, 19.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0107348 s, 97.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00289318 s, 362 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00469921 s, 223 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00341728 s, 307 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0080992 s, 129 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00456617 s, 230 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00631331 s, 166 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0078497 s, 134 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00231816 s, 452 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0791566 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0754284 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00745786 s, 141 MB/s removed '/mnt/lustre/d77kb.sanityn/nrs_r_oleg127-client.virtnet' removed '/mnt/lustre/d77kb.sanityn/nrs_w_oleg127-client.virtnet' removed directory '/mnt/lustre/d77kb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.4909 s, 14.0 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.99461 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kb (51s) == sanityn test 77kc: Check different granular TBF type combination: nid+opcode ========================================================== 16:53:21 (1788468801) ost.OSS.ost_io.nrs_policies=tbf nid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.201.27@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.201.27@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.438777 s, 38.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0437028 s, 24.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0408107 s, 25.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158797 s, 66.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.045781 s, 22.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0531316 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0117418 s, 89.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0444626 s, 23.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01346 s, 77.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0201321 s, 52.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00977905 s, 107 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129881 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0460317 s, 22.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140035 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165333 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121132 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116508 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0586784 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0788007 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0308039 s, 34.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0356114 s, 29.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0737677 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0421525 s, 24.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0496237 s, 21.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0466726 s, 22.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0600074 s, 17.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0634266 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0648351 s, 16.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0683414 s, 15.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0705856 s, 14.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0709741 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0764067 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137135 s, 7.6 MB/s removed '/mnt/lustre/d77kc.sanityn/nrs_r_oleg127-client.virtnet' removed '/mnt/lustre/d77kc.sanityn/nrs_w_oleg127-client.virtnet' removed directory '/mnt/lustre/d77kc.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.33905 s, 14.3 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.94654 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kc.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kc (51s) == sanityn test 77kd: Check different granular TBF type combination: nid+jobid ========================================================== 16:54:12 (1788468852) ost.OSS.ost_io.nrs_policies=tbf nid+jobid ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.201.27@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/nrs_r_oleg127-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.46127 s, 36.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0487974 s, 21.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0381632 s, 27.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0904436 s, 11.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117915 s, 8.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114471 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154726 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180819 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140223 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115061 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137697 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15936 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208177 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227005 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210061 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237641 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22011 s, 4.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0654653 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0848708 s, 12.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0530577 s, 19.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0593469 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0995799 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0546862 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.077256 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0680811 s, 15.4 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0749502 s, 14.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0777916 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107579 s, 9.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0919862 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0934245 s, 11.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150511 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132408 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143589 s, 7.3 MB/s removed '/mnt/lustre/d77kd.sanityn/nrs_w_oleg127-client.virtnet' removed '/mnt/lustre/d77kd.sanityn/nrs_r_oleg127-client.virtnet' removed directory '/mnt/lustre/d77kd.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.38395 s, 12.5 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kd.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.25328 s, 20.0 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kd.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kd (48s) == sanityn test 77ke: Check different granular TBF type combination: jobid+opcode ========================================================== 16:55:00 (1788468900) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/nrs_r_oleg127-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.594647 s, 28.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.053881 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0402665 s, 26.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.072078 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0360787 s, 29.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0958424 s, 10.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0606814 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0896784 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118235 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17101 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.2698 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0806904 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0978939 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208662 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144428 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.274478 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.331912 s, 3.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00855567 s, 123 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108917 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103962 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0111231 s, 94.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13841 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0240489 s, 43.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114979 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180559 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00360359 s, 291 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126098 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170966 s, 61.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161247 s, 65.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162195 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0200059 s, 52.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00369603 s, 284 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0541099 s, 19.4 MB/s removed '/mnt/lustre/d77ke.sanityn/nrs_r_oleg127-client.virtnet' removed '/mnt/lustre/d77ke.sanityn/nrs_w_oleg127-client.virtnet' removed directory '/mnt/lustre/d77ke.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 12.5068 s, 8.4 MB/s Write runtime is 14 s, speed is 7.142857 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ke.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.99347 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.35304 s, 14.3 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.95811 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77ke (98s) == sanityn test 77kf: Check different granular TBF type combination: uid+gid ========================================================== 16:56:38 (1788468998) ost.OSS.ost_io.nrs_policies=tbf uid+gid ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/nrs_r_oleg127-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.15135 s, 14.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0293132 s, 35.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0874234 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101646 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138409 s, 75.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0260802 s, 40.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0498304 s, 21.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136263 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.252002 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151434 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159966 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135614 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.219253 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0852866 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.065428 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.32099 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.366918 s, 2.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00243731 s, 430 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0545125 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0474994 s, 22.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0368998 s, 28.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149271 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135568 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0680594 s, 15.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00232497 s, 451 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169004 s, 6.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200897 s, 5.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190233 s, 5.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0475085 s, 22.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00237673 s, 441 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23438 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.246573 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0648893 s, 16.2 MB/s removed '/mnt/lustre/d77kf.sanityn/nrs_w_oleg127-client.virtnet' removed '/mnt/lustre/d77kf.sanityn/nrs_r_oleg127-client.virtnet' removed directory '/mnt/lustre/d77kf.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.3923 s, 5.1 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kf.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9739 s, 5.2 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77kf.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kf (78s) == sanityn test 77kg: check different granular TBF type combination: uid+opcode ========================================================== 16:57:56 (1788469076) ost.OSS.ost_io.nrs_policies=tbf uid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg127-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.335098 s, 50.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0387172 s, 27.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0800303 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135724 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126686 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119913 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162663 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123564 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165725 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118347 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156845 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122935 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.234157 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208728 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.330941 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.258145 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.270693 s, 3.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0623171 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0320749 s, 32.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0510801 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0768648 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0920582 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0384482 s, 27.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0424464 s, 24.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0664724 s, 15.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0521539 s, 20.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0578598 s, 18.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0658459 s, 15.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0535607 s, 19.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0758321 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0637538 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0704705 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0959976 s, 10.9 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg127-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg127-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.96588 s, 13.2 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0136 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg127-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.717052 s, 23.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0321439 s, 32.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0930278 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0884906 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0999509 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0967224 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13504 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158232 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151902 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100059 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146029 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130325 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154501 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.232496 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201044 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197789 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.218003 s, 4.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0593729 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0645064 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0564374 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101538 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0663702 s, 15.8 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0697108 s, 15.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0646356 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0663323 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0595415 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0613614 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0834423 s, 12.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.082388 s, 12.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0903713 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0794321 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0977074 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0819438 s, 12.8 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg127-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg127-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.9264 s, 8.8 MB/s Write runtime is 13 s, speed is 7.692307 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.9722 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 16.7951 s, 6.2 MB/s Write runtime is 18 s, speed is 5.555555 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.92628 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kg (152s) == sanityn test 77kh: Verify that Project ID support for NRS TBF rule ========================================================== 17:00:28 (1788469228) using SAVE_PROJECT_SUPPORTED=0 Stopping clients: oleg127-client.virtnet /mnt/lustre (opts:) Stopping client oleg127-client.virtnet /mnt/lustre opts: Stopping clients: oleg127-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg127-client.virtnet /mnt/lustre2 opts: Stopping /mnt/lustre-mds1 (opts:-f) on oleg127-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg127-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg127-server sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime) proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) devtmpfs on /dev type devtmpfs (rw,nosuid,mode=755) securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime) tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev) devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000) tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755) tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755) cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd) pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime) bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700) cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma) cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event) cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio) cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct) cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio) cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids) cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices) cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer) cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory) cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset) cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb) cgroup on /sys/fs/cgroup/misc type cgroup (rw,nosuid,nodev,noexec,relatime,misc) none on /sys/kernel/tracing type tracefs (rw,relatime) configfs on /sys/kernel/config type configfs (rw,relatime) /dev/nbd0 on / type ext4 (ro,relatime,stripe=32) rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) debugfs on /sys/kernel/debug type debugfs (rw,relatime) mqueue on /dev/mqueue type mqueue (rw,relatime) hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M) none on /mnt type ramfs (rw,relatime) /dev/vda on /home/green/git/lustre-release type squashfs (ro,relatime) none on /var/lib/stateless/writable type tmpfs (rw,relatime) none on /var/cache/man type tmpfs (rw,relatime) none on /var/log type tmpfs (rw,relatime) none on /var/lib/dbus type tmpfs (rw,relatime) none on /tmp type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) none on /var/tmp type tmpfs (rw,relatime) none on /var/lib/NetworkManager type tmpfs (rw,relatime) none on /var/lib/systemd/random-seed type tmpfs (rw,relatime) none on /var/spool type tmpfs (rw,relatime) none on /var/lib/nfs type tmpfs (rw,relatime) none on /var/lib/gssproxy type tmpfs (rw,relatime) none on /var/lib/logrotate type tmpfs (rw,relatime) none on /etc type tmpfs (rw,relatime) none on /var/lib/rsyslog type tmpfs (rw,relatime) none on /root/.ssh type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) 192.168.200.253:/exports/state/oleg127-client.virtnet on /var/lib/stateless/state type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.201.27,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg127-client.virtnet/boot on /boot type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.201.27,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg127-client.virtnet/etc/kdump.conf on /etc/kdump.conf type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.201.27,local_lock=none,addr=192.168.200.253) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=378180k,mode=700) 192.168.200.253:/exports/testreports/69423/testresults/sanityn-zfs-rocky8.10_x86_64-rocky8.10_x86_64 on /tmp/testlogs type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.201.27,local_lock=none,addr=192.168.200.253) /dev/vda on /usr/sbin/mount.lustre type squashfs (ro,relatime) Checking servers environments Checking clients oleg127-client.virtnet environments Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg127-server' oleg127-server: oleg127-server.virtnet: executing load_modules_local oleg127-server: kmemleak disabled oleg127-server: Loading modules from /home/green/git/lustre-release/lustre oleg127-server: detected 4 online CPUs by sysfs oleg127-server: MODOPTS_LIBCFS= oleg127-server: Force libcfs to create 2 CPU partitions oleg127-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Setup mgs, mdt, osts Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg127-server: oleg127-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg127-client: oleg127-server: ssh exited with exit code 1 Started lustre-MDT0000 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg127-server: oleg127-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg127-client: oleg127-server: ssh exited with exit code 1 Started lustre-OST0000 Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg127-server: oleg127-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg127-client: oleg127-server: ssh exited with exit code 1 Started lustre-OST0001 Starting client: oleg127-client.virtnet: -o user_xattr,flock 192.168.201.127@tcp:/lustre /mnt/lustre Starting client oleg127-client.virtnet: -o user_xattr,flock 192.168.201.127@tcp:/lustre /mnt/lustre Started clients oleg127-client.virtnet: 192.168.201.127@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Starting client: oleg127-client.virtnet: -o user_xattr,flock 192.168.201.127@tcp:/lustre /mnt/lustre2 Starting client oleg127-client.virtnet: -o user_xattr,flock 192.168.201.127@tcp:/lustre /mnt/lustre2 Started clients oleg127-client.virtnet: 192.168.201.127@tcp:/lustre on /mnt/lustre2 type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8b44461e4000.idle_timeout=debug osc.lustre-OST0000-osc-ffff8b447d034800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8b44461e4000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8b447d034800.idle_timeout=debug disable quota as required ost.OSS.ost_io.nrs_policies=tbf projid ost.OSS.ost_io.nrs_tbf_rule=start ext_proj_rw projid={100} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.502807 s, 33.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0825504 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0683452 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117375 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101468 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193093 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.30138 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242782 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.27128 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.259038 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.287969 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.351386 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.348573 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.278139 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.329976 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.370907 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.293486 s, 3.6 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8b447d034800.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10207 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108543 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0760333 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0880386 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197713 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145546 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0877512 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0911632 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.097902 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159875 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100281 s, 10.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123568 s, 8.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120568 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18915 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13136 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144388 s, 7.3 MB/s removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg127-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg127-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.16112 s, 14.6 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.15705 s, 20.3 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_proj_rw ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf projid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w projid={100}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r projid={100}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.667602 s, 25.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0385931 s, 27.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0823451 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0872957 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185866 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206568 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205393 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207476 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.224474 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207628 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.261123 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177372 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213858 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.249432 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.269808 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.278971 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.275312 s, 3.8 MB/sldlm.namespaces.lustre-OST0000-osc-ffff8b447d034800.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffff8b447d034800.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143889 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166299 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155417 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162442 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160191 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177072 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16477 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200331 s, 5.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172074 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172373 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209661 s, 5.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185059 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187437 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208367 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193713 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20693 s, 5.1 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8b447d034800.lock_unused_count=3 removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg127-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg127-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.23784 s, 16.8 MB/s Write runtime is 7 s, speed is 14.285714 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.93321 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77kh (228s) == sanityn test 77ki: Add rule with unsupported TBF type should fail for generic TBF ========================================================== 17:04:16 (1788469456) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 oleg127-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_fail projid={100}&opcode={ost_write} rate=20: Invalid argument oleg127-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_fail projid={100}&opcode={ost_write} rate=20': Invalid argument pdsh@oleg127-client: oleg127-server: ssh exited with exit code 22 ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77ki (18s) == sanityn test 77kj: Verify nodemap support for NRS TBF rule ========================================================== 17:04:34 (1788469474) On MGS 192.168.201.127, active = nodemap.active=1 On MGS 192.168.201.127, TBF = ost.OSS.ost_io.nrs_policies=tbf nodemap ost.OSS.ost_io.nrs_tbf_rule=start ext_nm_rw nodemap={TBF} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.704878 s, 23.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.098145 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0729701 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0673121 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126375 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.218648 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.239878 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.239502 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.228186 s, 4.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8b447d034800.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.252864 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.311954 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.293716 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.419692 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.44635 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.462899 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.486929 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.490544 s, 2.1 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8b447d034800.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.323903 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.339854 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.342709 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.249895 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.294154 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.257513 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.349944 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.27855 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.303461 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.325328 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.268133 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.347226 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.369573 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.287792 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.329681 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.294423 s, 3.6 MB/s removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg127-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg127-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.89224 s, 11.8 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.47321 s, 19.2 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete On MGS 192.168.201.127, TBF.id = On MGS 192.168.201.127, active = nodemap.active=0 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.595866 s, 28.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0699922 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0794571 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116648 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167812 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180359 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155895 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208702 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.244833 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.38623 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.387669 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.452441 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.45132 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.31829 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.422051 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.360264 s, 2.9 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8b447d034800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.353767 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0858575 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0845932 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0311209 s, 33.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0346379 s, 30.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105736 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0413513 s, 25.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0474899 s, 22.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0445685 s, 23.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.05093 s, 20.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0439912 s, 23.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110563 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113442 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114155 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105019 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0673819 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0752473 s, 13.9 MB/s removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg127-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg127-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 1 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.08515 s, 11.5 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 3.45003 s, 30.4 MB/s Read runtime is 4 s, speed is 25.000000 IOPS read rate (25.000000) should exceed limit (20 * 1) cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_nm_rw ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf nodemap+opcode oleg127-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_w nodemap={TBF}&opcode={ost_write} rate=20: Invalid argument oleg127-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_w nodemap={TBF}&opcode={ost_write} rate=20': Invalid argument oleg127-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_r nodemap={TBF}&opcode={ost_read} rate=10: Invalid argument oleg127-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_r nodemap={TBF}&opcode={ost_read} rate=10': Invalid argument pdsh@oleg127-client: oleg127-server: ssh exited with exit code 22 On MGS 192.168.201.127, active = nodemap.active=1 On MGS 192.168.201.127, TBF = ost.OSS.ost_io.nrs_tbf_rule=start ext_w nodemap={TBF}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nodemap={TBF}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.891127 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0645444 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0918861 s, 11.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133131 s, 7.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176984 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.218166 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141677 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.245584 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.224422 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0658406 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111845 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190976 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.453754 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.326293 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.458687 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.501801 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.294353 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145194 s, 7.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145916 s, 7.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147076 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179589 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175085 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180277 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203203 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213915 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202535 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200933 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209262 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210972 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.226151 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.324851 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.251579 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.235751 s, 4.4 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8b447d034800.lock_unused_count=2 removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg127-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg127-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 12.506 s, 8.4 MB/s Write runtime is 14 s, speed is 7.142857 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0347 s, 10.4 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo On MGS 192.168.201.127, TBF.id = On MGS 192.168.201.127, active = nodemap.active=0 PASS 77kj (206s) == sanityn test 77l: check the output of NRS policies for generic TBF ========================================================== 17:08:00 (1788469680) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_policies=fifo PASS 77l (11s) == sanityn test 77m: check NRS Delay slows write RPC processing ========================================================== 17:08:11 (1788469691) ost.OSS.ost_io.nrs_policies=delay ost.OSS.ost_io.nrs_delay_min=4 ost.OSS.ost_io.nrs_delay_max=4 ost.OSS.ost_io.nrs_delay_pct=100 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg127-client.virtnet] [bs=1M] [count=1] [oflag=direct] [conv=fdatasync] 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 3.565 s, 294 kB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg127-client.virtnet] [bs=1M] [count=10] [oflag=direct] [conv=fdatasync] 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 40.0781 s, 262 kB/s ost.OSS.ost_io.nrs_policies=fifo PASS 77m (58s) == sanityn test 77n: check wildcard support for TBF JobID NRS policy ========================================================== 17:09:09 (1788469749) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={*.500} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.577792 s, 29.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0703801 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.087697 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0879078 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135068 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201041 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195488 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0876655 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135794 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109048 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192076 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.341899 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.283976 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175641 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.353094 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.314781 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.314193 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136573 s, 76.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0457038 s, 22.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0827708 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0716837 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0315922 s, 33.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.031029 s, 33.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0364375 s, 28.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0374351 s, 28.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0427229 s, 24.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0476883 s, 22.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100863 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0179909 s, 58.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0549694 s, 19.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0642426 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0712496 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.071293 s, 14.7 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg127-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg127-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77n.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.42881 s, 14.1 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77n.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.4769 s, 19.1 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.*} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.660667 s, 25.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0387668 s, 27.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0610089 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0503981 s, 20.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104149 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0782249 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137339 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104496 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.364204 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.35578 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.38749 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.413095 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.49031 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.504158 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.514712 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.563147 s, 1.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8b447d034800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff8b447d034800.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.648123 s, 1.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0494529 s, 21.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0753147 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0649947 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153263 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156138 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0964105 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100529 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106544 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113065 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168597 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199048 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122622 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108879 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119099 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147157 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.236341 s, 4.4 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg127-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg127-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.22644 s, 11.4 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.36972 s, 19.5 MB/s Read runtime is 7 s, speed is 14.285714 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_policies=fifo PASS 77n (93s) == sanityn test 77o: Changing rank should not panic ====== 17:10:42 (1788469842) mds.MDS.mdt.nrs_policies=tbf nid mds.MDS.mdt.nrs_tbf_rule=start name nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=start name1 nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=change name1 rank=name mds.MDS.mdt.nrs_tbf_rule=stop name mds.MDS.mdt.nrs_policies=fifo PASS 77o (13s) == sanityn test 77q: Parallel TBF rule definitions should not panic ========================================================== 17:10:55 (1788469855) mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_policies=fifo PASS 77q (144s) == sanityn test 77p: Check validity of rule names for TBF policies ========================================================== 17:13:19 (1788469999) mds.MDS.mdt.nrs_policies=tbf oleg127-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_overflo uid={500} rate=500: File name too long oleg127-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_overflo uid={500} rate=500': File name too long pdsh@oleg127-client: oleg127-server: ssh exited with exit code 36 oleg127-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start uid={500} rate=500: Invalid argument oleg127-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start uid={500} rate=500': Invalid argument pdsh@oleg127-client: oleg127-server: ssh exited with exit code 22 oleg127-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_empty: Invalid argument oleg127-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_empty': Invalid argument pdsh@oleg127-client: oleg127-server: ssh exited with exit code 22 oleg127-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p@spec uid={500} rate=500: Invalid argument oleg127-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p@spec uid={500} rate=500': Invalid argument pdsh@oleg127-client: oleg127-server: ssh exited with exit code 22 oleg127-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p.spec uid={500} rate=500: Invalid argument oleg127-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p.spec uid={500} rate=500': Invalid argument pdsh@oleg127-client: oleg127-server: ssh exited with exit code 22 oleg127-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p~spec uid={500} rate=500: Invalid argument oleg127-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p~spec uid={500} rate=500': Invalid argument pdsh@oleg127-client: oleg127-server: ssh exited with exit code 22 oleg127-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p#spec uid={500} rate=500: Invalid argument oleg127-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p#spec uid={500} rate=500': Invalid argument pdsh@oleg127-client: oleg127-server: ssh exited with exit code 22 oleg127-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p/spec uid={500} rate=500: Invalid argument oleg127-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p/spec uid={500} rate=500': Invalid argument pdsh@oleg127-client: oleg127-server: ssh exited with exit code 22 oleg127-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p^spec uid={500} rate=500: Invalid argument oleg127-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p^spec uid={500} rate=500': Invalid argument pdsh@oleg127-client: oleg127-server: ssh exited with exit code 22 oleg127-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p%spec uid={500} rate=500: Invalid argument oleg127-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p%spec uid={500} rate=500': Invalid argument pdsh@oleg127-client: oleg127-server: ssh exited with exit code 22 oleg127-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p*spec uid={500} rate=500: Invalid argument oleg127-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p*spec uid={500} rate=500': Invalid argument pdsh@oleg127-client: oleg127-server: ssh exited with exit code 22 oleg127-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p;spec uid={500} rate=500: Invalid argument oleg127-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p;spec uid={500} rate=500': Invalid argument pdsh@oleg127-client: oleg127-server: ssh exited with exit code 22 oleg127-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p,spec uid={500} rate=500: Invalid argument oleg127-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p,spec uid={500} rate=500': Invalid argument pdsh@oleg127-client: oleg127-server: ssh exited with exit code 22 oleg127-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p?spec uid={500} rate=500: Invalid argument oleg127-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p?spec uid={500} rate=500': Invalid argument pdsh@oleg127-client: oleg127-server: ssh exited with exit code 22 oleg127-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77pspec uid={500} rate=500: Invalid argument oleg127-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p>spec uid={500} rate=500': Invalid argument pdsh@oleg127-client: oleg127-server: ssh exited with exit code 22 oleg127-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p:spec uid={500} rate=500: Invalid argument oleg127-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p:spec uid={500} rate=500': Invalid argument pdsh@oleg127-client: oleg127-server: ssh exited with exit code 22 oleg127-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p+spec uid={500} rate=500: Invalid argument oleg127-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p+spec uid={500} rate=500': Invalid argument pdsh@oleg127-client: oleg127-server: ssh exited with exit code 22 oleg127-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p=spec uid={500} rate=500: Invalid argument oleg127-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p=spec uid={500} rate=500': Invalid argument pdsh@oleg127-client: oleg127-server: ssh exited with exit code 22 oleg127-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p)spec uid={500} rate=500: Invalid argument oleg127-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p)spec uid={500} rate=500': Invalid argument pdsh@oleg127-client: oleg127-server: ssh exited with exit code 22 oleg127-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p(spec uid={500} rate=500: Invalid argument oleg127-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p(spec uid={500} rate=500': Invalid argument pdsh@oleg127-client: oleg127-server: ssh exited with exit code 22 oleg127-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p{spec uid={500} rate=500: Invalid argument oleg127-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p{spec uid={500} rate=500': Invalid argument pdsh@oleg127-client: oleg127-server: ssh exited with exit code 22 oleg127-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p}spec uid={500} rate=500: Invalid argument oleg127-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p}spec uid={500} rate=500': Invalid argument pdsh@oleg127-client: oleg127-server: ssh exited with exit code 22 oleg127-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p|spec uid={500} rate=500: Invalid argument oleg127-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p|spec uid={500} rate=500': Invalid argument pdsh@oleg127-client: oleg127-server: ssh exited with exit code 22 oleg127-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p[spec uid={500} rate=500: Invalid argument oleg127-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p[spec uid={500} rate=500': Invalid argument pdsh@oleg127-client: oleg127-server: ssh exited with exit code 22 oleg127-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p]spec uid={500} rate=500: Invalid argument oleg127-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p]spec uid={500} rate=500': Invalid argument pdsh@oleg127-client: oleg127-server: ssh exited with exit code 22 oleg127-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p!spec uid={500} rate=500: Invalid argument oleg127-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p!spec uid={500} rate=500': Invalid argument pdsh@oleg127-client: oleg127-server: ssh exited with exit code 22 oleg127-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p&spec uid={500} rate=500: Invalid argument oleg127-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p&spec uid={500} rate=500': Invalid argument pdsh@oleg127-client: oleg127-server: ssh exited with exit code 22 oleg127-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p$spec uid={500} rate=500: Invalid argument oleg127-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p$spec uid={500} rate=500': Invalid argument pdsh@oleg127-client: oleg127-server: ssh exited with exit code 22 oleg127-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p`spec uid={500} rate=500: Invalid argument oleg127-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p`spec uid={500} rate=500': Invalid argument pdsh@oleg127-client: oleg127-server: ssh exited with exit code 22 oleg127-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p\spec uid={500} rate=500: Invalid argument oleg127-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p\spec uid={500} rate=500': Invalid argument pdsh@oleg127-client: oleg127-server: ssh exited with exit code 22 mds.MDS.mdt.nrs_policies=fifo PASS 77p (49s) == sanityn test 77r: Change type of tbf policy at run time ========================================================== 17:14:08 (1788470048) mds.MDS.mdt.nrs_policies=tbf uid mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf gid mds.MDS.mdt.nrs_tbf_rule=start md_rule gid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start md_rule jobid={*.500} rate=1 mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=fifo check the number of requests in queue: fifo: 0 crrn: 0 tbf: 0 delay: 0 fifo: 0 crrn: 0 tbf: 0 delay: 0 cleanup 77r 273079 kill md_thread xargs PASS 77r (61s) == sanityn test 77s: Check TBF LRU shrinker ============== 17:15:10 (1788470110) jobid_name=%e.%p jobid_var=TEST_77s_JOBVAR mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start touch jobid={touch.*} rate=1000 mds.MDS.mdt.nrs_tbf_rule=start unlink jobid={unlink.*} rate=1000 touch jobid={touch.*} 1000, ref 10 touch jobid={touch.*} 1000, ref 0 unlink jobid={unlink.*} 1000, ref 10 unlink jobid={unlink.*} 1000, ref 0 mds.MDS.mdt.nrs_policies=fifo jobid_var=procname_uid jobid_name=%e.%u PASS 77s (79s) == sanityn test 78: Enable policy and specify tunings right away ========================================================== 17:16:30 (1788470190) oleg127-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_orr_quantum=1: No such device oleg127-server: error: set_param: setting 'ost/OSS/*/nrs_orr_quantum'='1': No such device pdsh@oleg127-client: oleg127-server: ssh exited with exit code 19 ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_policies=fifo PASS 78 (11s) == sanityn test 79: xattr: intent error ================== 17:16:41 (1788470201) File: /mnt/lustre/d79.sanityn Size: 16384 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115238826935638 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-03 17:16:43.000000000 -0400 Modify: 2026-09-03 17:16:43.000000000 -0400 Change: 2026-09-03 17:16:43.000000000 -0400 Birth: 2026-09-03 17:16:44.000000000 -0400 fail_val=0 fail_loc=0x80000160 fail_val=0 fail_loc=0x80000131 # file: mnt/lustre/d79.sanityn trusted.name1="value1" PASS 79 (22s) == sanityn test 80a: migrate directory when some children is being opened ========================================================== 17:17:03 (1788470223) SKIP: sanityn test_80a needs >= 2 MDTs SKIP 80a (5s) == sanityn test 80b: Accessing directory during migration ========================================================== 17:17:08 (1788470228) SKIP: sanityn test_80b needs >= 2 MDTs SKIP 80b (4s) == sanityn test 81a: rename and stat under striped directory ========================================================== 17:17:12 (1788470232) SKIP: sanityn test_81a needs >= 2 MDTs SKIP 81a (4s) == sanityn test 81b: rename under striped directory doesn't deadlock ========================================================== 17:17:16 (1788470236) SKIP: sanityn test_81b We need at least 2 MDTs for this test SKIP 81b (5s) == sanityn test 81c: rename revoke LOOKUP lock for remote object ========================================================== 17:17:21 (1788470241) SKIP: sanityn test_81c needs >= 4 MDTs SKIP 81c (4s) == sanityn test 81d: parallel rename file cross-dir on same MDT ========================================================== 17:17:26 (1788470246) - open/close 590 (time 1788470264.51 total 10.01 last 58.95) - open/close 1101 (time 1788470274.52 total 20.01 last 51.08) - open/close 1596 (time 1788470284.52 total 30.02 last 49.49) - open/close 2413 (time 1788470294.52 total 40.02 last 81.67) total: 2500 open/close in 41.34 seconds: 60.47 ops/second snapshot_time 1788470386.196201296 secs.nsecs start_time 1788470251.412969652 secs.nsecs elapsed_time 134.783231644 secs.nsecs open 2505 samples [usecs] 126 28032 4479799 18404640725 close 2505 samples [usecs] 163 52157 1042858 4518874928 mknod 2501 samples [usecs] 556 24587 3891405 14200888027 unlink 2500 samples [usecs] 672 100701 5123405 45604737297 mkdir 2 samples [usecs] 1135 5441 6576 30892706 rmdir 2 samples [usecs] 9688 11122 20810 217556228 rename 2500 samples [usecs] 504 82219 17183679 275161718507 getattr 3768 samples [usecs] 7 14353 272474 816904244 parallel_rename_file 2500 samples [usecs] 504 82219 17183679 275161718507 crossdir_rename 2500 samples [usecs] 504 82219 17183679 275161718507 readpage 21 samples [usecs] 4626 64268 434995 12520316981 PASS 81d (144s) == sanityn test 82: fsetxattr and fgetxattr on orphan files ========================================================== 17:19:50 (1788470390) multiop /mnt/lustre/f82.sanityn vO_ac TMPPIPE=/tmp/multiop_open_wait_pipe.7649 multiop /mnt/lustre2/f82.sanityn vo_Ac TMPPIPE=/tmp/multiop_open_wait_pipe.7649 PASS 82 (7s) == sanityn test 83: access striped directory while it is being created/unlinked ========================================================== 17:19:57 (1788470397) SKIP: sanityn test_83 needs >= 2 MDTs SKIP 83 (3s) == sanityn test 84: 0-nlink race in lu_object_find() ===== 17:20:00 (1788470400) fail_loc=0x8000060b PASS 84 (13s) == sanityn test 85: Lustre API root cache race =========== 17:20:13 (1788470413) Starting: llapi_root_test /mnt/lustre/f85.sanityn /mnt/lustre2/f85.sanityn Starting test test1 at 1788470415 Finishing test test1 at 1788470419 PASS 85 (11s) == sanityn test 90: open/create and unlink striped directory ========================================================== 17:20:24 (1788470424) SKIP: sanityn test_90 needs >= 2 MDTs SKIP 90 (3s) == sanityn test 91: chmod and unlink striped directory === 17:20:27 (1788470427) SKIP: sanityn test_91 needs >= 2 MDTs SKIP 91 (3s) == sanityn test 92: create remote directory under orphan directory ========================================================== 17:20:31 (1788470431) SKIP: sanityn test_92 needs >= 2 MDTs SKIP 92 (4s) == sanityn test 93: alloc_rr should not allocate on same ost ========================================================== 17:20:34 (1788470434) fail_loc=0x00000163 fail_loc=0x0 /mnt/lustre/f93.sanityn-1/file1 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5767 0x1687 0x240000400 1 5760 0x1680 0x280000400 /mnt/lustre2/f93.sanityn-2/file2 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 5761 0x1681 0x280000400 0 5768 0x1688 0x240000400 rc1=0 and rc2=0 PASS 93 (16s) == sanityn test 94: signal vs CP callback race =========== 17:20:51 (1788470451) osc.lustre-OST0000-osc-ffff8b44461e4000.idle_timeout=0 osc.lustre-OST0000-osc-ffff8b447d034800.idle_timeout=0 osc.lustre-OST0001-osc-ffff8b44461e4000.idle_timeout=0 osc.lustre-OST0001-osc-ffff8b447d034800.idle_timeout=0 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.29827 s, 27.5 kB/s fail_val=5 fail_loc=0x80000312 fail_val=6 fail_loc=0x80000329 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 3.00516 s, 1.4 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 6389: 289544 Alarm clock dd if=/dev/zero of=$DIR/$tfile conv=notrunc oflag=append bs=4k count=1 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0909117 s, 45.1 kB/s osc.lustre-OST0000-osc-ffff8b44461e4000.idle_timeout=debug osc.lustre-OST0000-osc-ffff8b447d034800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8b44461e4000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8b447d034800.idle_timeout=debug PASS 94 (17s) == sanityn test 95a: Check readpage() on a page that was removed from page cache ========================================================== 17:21:07 (1788470467) llite.lustre-ffff8b44461e4000.fast_read=0 llite.lustre-ffff8b447d034800.fast_read=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0770386 s, 27.2 MB/s fail_loc=0x80001421 fail_val=7 fail_loc=0x80001422 fail_val=10 llite.lustre-ffff8b44461e4000.fast_read=1 llite.lustre-ffff8b447d034800.fast_read=1 PASS 95a (26s) == sanityn test 95b: Check readpage() on a page that is no longer uptodate ========================================================== 17:21:33 (1788470493) llite.lustre-ffff8b44461e4000.fast_read=0 llite.lustre-ffff8b447d034800.fast_read=0 1+0 records in 1+0 records out 12288 bytes (12 kB, 12 KiB) copied, 0.00577521 s, 2.1 MB/s fail_loc=0x80001424 fail_val=5 fail_loc=0 llite.lustre-ffff8b44461e4000.fast_read=1 llite.lustre-ffff8b447d034800.fast_read=1 PASS 95b (18s) == sanityn test 100a: DoM: glimpse RPCs for stat without IO lock (DoM only file) ========================================================== 17:21:51 (1788470511) SKIP: sanityn test_100a Reserved for glimpse-ahead SKIP 100a (4s) == sanityn test 100b: DoM: no glimpse RPC for stat with IO lock (DoM only file) ========================================================== 17:21:55 (1788470515) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00358988 s, 1.1 MB/s /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK PASS 100b (8s) == sanityn test 100c: DoM: write vs stat without IO lock (combined file) ========================================================== 17:22:03 (1788470523) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.190397 s, 11.0 MB/s /mnt/lustre/d100c.sanityn/dom has type file OK /mnt/lustre/d100c.sanityn/dom has size 2097152 OK PASS 100c (8s) == sanityn test 100d: DoM: write+truncate vs stat without IO lock (combined file) ========================================================== 17:22:11 (1788470531) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.172751 s, 12.1 MB/s /mnt/lustre/d100d.sanityn/dom has type file OK /mnt/lustre/d100d.sanityn/dom has size 4096 OK PASS 100d (6s) == sanityn test 100e: DoM: read on open and file size ==== 17:22:17 (1788470537) 1+0 records in 1+0 records out 12000 bytes (12 kB, 12 KiB) copied, 0.00565741 s, 2.1 MB/s PASS 100e (7s) == sanityn test 101a: Discard DoM data on unlink ========= 17:22:24 (1788470544) /mnt/lustre/f101a.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00356837 s, 1.1 MB/s PASS 101a (7s) == sanityn test 101b: Discard DoM data on rename ========= 17:22:31 (1788470551) /mnt/lustre/f101b.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0193124 s, 212 kB/s PASS 101b (7s) == sanityn test 101c: Discard DoM data on close-unlink === 17:22:38 (1788470558) /mnt/lustre/f101c.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00590314 s, 694 kB/s PASS 101c (8s) == sanityn test 102: Test open by handle of unlinked file ========================================================== 17:22:46 (1788470566) file: /mnt/lustre/f102.sanityn fh_bytes: 32 fh_type: 151 fh_data: d1 0b 00 00 02 00 00 00 37 0f 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 file's parent FID is [0:0x0:0x0] file FID is [0x200000bd1:0xf37:0x0] access via mount point '/mnt/lustre2' - OK access after unlink - OK check_fhandle_syscalls test Passed! PASS 102 (8s) == sanityn test 103: Test size correctness with lockahead ========================================================== 17:22:54 (1788470574) fail_loc=0x415 fail_loc=0x415 fail_val=2 Incorrect size expected (no glimpse fix): Starting test test23 at 1788470577 lockahead_test: lockahead_test.c:1171: test23: assertion 'sb.st_size == sb2.st_size' failed: size on /mnt/lustre/d103.sanityn/f103.sanityn and /mnt/lustre2/d103.sanityn/f103.sanityn differs: 0 vs 1048576 fail_loc=0 fail_loc=0x214 fail_val=2 Starting test test23 at 1788470580 Finishing test test23 at 1788470581 PASS 103 (13s) == sanityn test 104: Verify that MDS stores atime/mtime/ctime during close ========================================================== 17:23:07 (1788470587) SKIP: sanityn test_104 ldiskfs only test SKIP 104 (3s) == sanityn test 105: Glimpse and lock cancel race ======== 17:23:11 (1788470591) fail_loc=0x416 PASS 105 (30s) == sanityn test 106a: Verify the btime via statx() ======= 17:23:40 (1788470620) SKIP: sanityn test_106a Test only for ldiskfs and statx() supported SKIP 106a (3s) == sanityn test 106b: Glimpse RPCs test for statx ======== 17:23:43 (1788470623) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00797283 s, 128 kB/s File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238826938174 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-03 17:23:45.000000000 -0400 Modify: 2026-09-03 17:23:45.000000000 -0400 Change: 2026-09-03 17:23:45.000000000 -0400 Birth: 2026-09-03 17:23:46.000000000 -0400 /mnt/lustre/f106b.sanityn 144115238826938174 -rw-r--r-- 1 0 0 1788470626 1788470625 1788470625 File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238826938174 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-03 17:23:45.000000000 -0400 Modify: 2026-09-03 17:23:45.000000000 -0400 Change: 2026-09-03 17:23:45.000000000 -0400 Birth: 2026-09-03 17:23:46.000000000 -0400 1788470625 1024 1 PASS 106b (8s) == sanityn test 106c: Verify statx attributes mask ======= 17:23:51 (1788470631) PASS 106c (9s) == sanityn test 107a: Basic grouplock conflict =========== 17:24:00 (1788470640) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.465115 s, 22.5 MB/s /mnt/lustre/f107a.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 5768 0x1688 0x280000400 multiop /mnt/lustre/f107a.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7649 multiop /mnt/lustre2/f107a.sanityn vO_G16022000r10g16022000c TMPPIPE=/tmp/multiop_open_wait_pipe.7649 First grouplock blocks second one PASS 107a (10s) == sanityn test 107b: Grouplock is added to the head of waiting list ========================================================== 17:24:10 (1788470650) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.329333 s, 31.8 MB/s /mnt/lustre/f107b.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5777 0x1691 0x240000400 multiop /mnt/lustre/f107b.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7649 Grouplock blocks IO First grouplock blocks second one Second grouplock blocks IO PASS 107b (15s) == sanityn test 108a: lseek: parallel updates ============ 17:24:25 (1788470665) 1+0 records in 1+0 records out 10000 bytes (10 kB, 9.8 KiB) copied, 0.020422 s, 490 kB/s llite.lustre-ffff8b44461e4000.hybrid_io=0 llite.lustre-ffff8b447d034800.hybrid_io=0 fail_val=4 fail_loc=0x80000414 1+0 records in 1+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 4.70813 s, 1.8 MB/s PASS 108a (13s) == sanityn test 109: Race with several mount instances on 1 node ========================================================== 17:24:38 (1788470678) 192.168.201.127@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg127-client.virtnet /mnt/lustre (opts:) 192.168.201.127@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg127-client.virtnet /mnt/lustre2 (opts:) Starting race between client mounts (duration 30) Iteration 0 Starting client: oleg127-client.virtnet: -o user_xattr,flock 192.168.201.127@tcp:/lustre /mnt/lustre2 Starting client: oleg127-client.virtnet: -o user_xattr,flock 192.168.201.127@tcp:/lustre /mnt/lustre 192.168.201.127@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.127@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg127-client.virtnet /mnt/lustre (opts:) Stopping client oleg127-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg127-server' oleg127-server: oleg127-server.virtnet: executing load_modules_local oleg127-server: kmemleak disabled oleg127-server: Loading modules from /home/green/git/lustre-release/lustre oleg127-server: detected 4 online CPUs by sysfs oleg127-server: MODOPTS_LIBCFS= oleg127-server: Force libcfs to create 2 CPU partitions oleg127-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Starting client: oleg127-client.virtnet: -o user_xattr,flock 192.168.201.127@tcp:/lustre /mnt/lustre Starting client: oleg127-client.virtnet: -o user_xattr,flock 192.168.201.127@tcp:/lustre /mnt/lustre2 PASS 109 (50s) == sanityn test 110: do not grant another lock on resend ========================================================== 17:25:28 (1788470728) fail_loc=0x525 fail_val=3 fail_loc=0x534 File: /mnt/lustre/f110.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238860488707 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-03 17:25:30.000000000 -0400 Modify: 2026-09-03 17:25:30.000000000 -0400 Change: 2026-09-03 17:25:30.000000000 -0400 Birth: 2026-09-03 17:25:31.000000000 -0400 fail_loc=0 fail_val=0 Connected clients: oleg127-client.virtnet oleg127-client.virtnet PASS 110 (92s) == sanityn test 111: A racy rename/link an open file should not cause fs corruption ========================================================== 17:27:01 (1788470821) SKIP: sanityn test_111 needs >= 2 MDTs SKIP 111 (4s) == sanityn test 112: update max-inherit in default LMV === 17:27:05 (1788470825) SKIP: sanityn test_112 We need at least 2 MDTs for this test SKIP 112 (3s) == sanityn test 113: check servers of specified fs ======= 17:27:08 (1788470828) lustre-OST0000-osc-ffff8b44462e7800 active. lustre-OST0001-osc-ffff8b44462e7800 active. lustre-OST0000-osc-ffff8b44462e7800 active. lustre-OST0001-osc-ffff8b44462e7800 active. MGC192.168.201.127@tcp active. PASS 113 (8s) == sanityn test 114: implicit default LMV inherit ======== 17:27:16 (1788470836) SKIP: sanityn test_114 We need at least 2 MDTs for this test SKIP 114 (3s) == sanityn test 115: ldiskfs doesn't check direntry for uniqueness ========================================================== 17:27:20 (1788470840) SKIP: sanityn test_115 ldiskfs only test SKIP 115 (3s) == sanityn test 116: DNE: Set default LMV layout from a remote client ========================================================== 17:27:24 (1788470844) SKIP: sanityn test_116 needs >= 2 MDTs SKIP 116 (4s) == sanityn test 121: trunc append race =================== 17:27:27 (1788470847) 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0483377 s, 424 kB/s fail_loc=0x80001436 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.100412 s, 204 kB/s 0000000 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0005000 0000 0000 0000 0000 0000 0000 0000 0000 * 000c800 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0011800 PASS 121 (11s) == sanityn test 122: directory size is consistent across mounts ========================================================== 17:27:38 (1788470858) PASS 122 (11s) == sanityn test 200: service remains healthy while able to process request ========================================================== 17:27:49 (1788470869) at_max=10 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=9 fail_loc=0x1e1 fail_val=20 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1834 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.2794 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.2347 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.2104 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.2073 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.2454 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.2941 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.085 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.2467 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.2719 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.2864 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.3175 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.2388 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.2279 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2857 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.3911 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.3427 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.3275 s, 0.1 kB/s at_max=600 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=116 oleg127-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8b44462e7800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8b44462e7800.ost_server_uuid in FULL state after 0 sec PASS 200 (87s) cleanup: ====================================================== == sanityn test complete, duration 10013 sec ============= 17:29:18 (1788470958) === sanityn: start cleanup 17:29:20 (1788470960) === Stopping clients: oleg127-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg127-client.virtnet /mnt/lustre2 opts: === sanityn: finish cleanup 17:33:37 (1788471217) ===