-----============= acceptance-small: sanityn ============----- Thu Sep 10 10:06:07 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg341-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanityn.*ex': No such file or directory excepting tests: 27 40a skipping tests SLOW=no: 33a === sanityn: start setup 10:06:25 (1789049185) === Starting client oleg341-client.virtnet: -o user_xattr,flock 192.168.203.141@tcp:/lustre /mnt/lustre2 Mount client oleg341-client.virtnet: mount -t lustre -o user_xattr,flock 192.168.203.141@tcp:/lustre /mnt/lustre2 Started clients oleg341-client.virtnet: 192.168.203.141@tcp:/lustre on /mnt/lustre2 type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) oleg341-client.virtnet: executing check_config_client /mnt/lustre oleg341-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg341-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9668d169d000.idle_timeout=debug osc.lustre-OST0000-osc-ffff9668d8375800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9668d169d000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9668d8375800.idle_timeout=debug disable quota as required oleg341-server: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanityn: finish setup 10:06:58 (1789049218) === 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0220301 s, 47.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7620] == sanityn test 0: do_node_vp() and do_facet_vp() do the right thing ========================================================== 10:07:01 (1789049221) PASS 0 (8s) == sanityn test 1: Check attribute updates on 2 mount points ========================================================== 10:07:10 (1789049230) /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0777 OK /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0666 OK /mnt/lustre/f1.sanityn: absent OK PASS 1 (6s) == sanityn test 2a: check cached attribute updates on 2 mtpt's ================================================================== 10:07:17 (1789049237) -rw-r--r-- 1 root root 0 Sep 10 10:07 /mnt/lustre2/f2a /mnt/lustre/f2a has type file OK /mnt/lustre/f2a has perms 0777 OK PASS 2a (8s) == sanityn test 2b: check cached attribute updates on 2 mtpt's ================================================================== 10:07:25 (1789049245) -rw-r--r-- 1 root root 0 Sep 10 10:07 /mnt/lustre2/f2b /mnt/lustre2/f2b has type file OK /mnt/lustre2/f2b has perms 0777 OK PASS 2b (6s) == sanityn test 2c: check cached attribute updates on 2 mtpt's root ============================================================= 10:07:31 (1789049251) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0777 OK PASS 2c (7s) == sanityn test 2d: check cached attribute updates on 2 mtpt's root ============================================================= 10:07:38 (1789049258) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0755 OK PASS 2d (6s) == sanityn test 2e: check chmod on root is propagated to others ========================================================== 10:07:44 (1789049264) total 1 -rwxrwxrwx 1 root root 0 Sep 10 10:07 f2a -rwxrwxrwx 1 root root 0 Sep 10 10:07 f2b total 1 -rwxrwxrwx 1 root root 0 Sep 10 10:07 f2a -rwxrwxrwx 1 root root 0 Sep 10 10:07 f2b running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre2/f2e.sanityn] [count=1] 1+0 records in 1+0 records out 512 bytes copied, 0.0228314 s, 22.4 kB/s PASS 2e (6s) == sanityn test 2f: check attr/owner updates on DNE with 2 mtpt's ========================================================== 10:07:50 (1789049270) SKIP: sanityn test_2f needs >= 2 MDTs SKIP 2f (3s) == sanityn test 2g: check blocks update on sync write ==== 10:07:53 (1789049273) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.334198 s, 6.3 MB/s /mnt/lustre/f2g.sanityn has 4103 blocks /mnt/lustre2/f2g.sanityn has 4103 blocks PASS 2g (7s) == sanityn test 3: symlink on one mtpt, readlink on another ===================================================================== 10:08:00 (1789049280) PASS 3 (6s) == sanityn test 4: fstat validation on multiple mount points ==================================================================== 10:08:07 (1789049287) PASS 4 (6s) == sanityn test 5: create a file on one mount, truncate it on the other ========================================================== 10:08:13 (1789049293) /mnt/lustre/f5 has type file OK /mnt/lustre/f5 has size 100 OK PASS 5 (6s) == sanityn test 6: remove of open file on other node ============================================================================ 10:08:19 (1789049299) opening writing unlinking /mnt/lustre2/f6.sanityn accessing (1) seeking (1) accessing (2) fstat... reading comparing data truncating seeking (2) writing again seeking (3) reading again comparing data again closing SUCCESS - goto beer PASS 6 (5s) == sanityn test 7: remove of open directory on other node ======================================================================= 10:08:25 (1789049305) creating directory /mnt/lustre/d7 opening directory unlinking /mnt/lustre/d7 Ok, everything goes well. PASS 7 (6s) == sanityn test 8: remove of open special file on other node ==================================================================== 10:08:30 (1789049310) creating special file /mnt/lustre/f8.sanityn opening file unlinking /mnt/lustre/f8.sanityn Ok, everything goes well. PASS 8 (5s) == sanityn test 9a: append of file with sub-page size on multiple mounts ========================================================== 10:08:35 (1789049315) PASS 9a (6s) == sanityn test 9b: append to striped sparse file ======== 10:08:41 (1789049321) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0551379 s, 19.0 MB/s 3+0 records in 3+0 records out 3 bytes copied, 0.0469102 s, 0.1 kB/s Data read (expecting 'foo'): foo PASS 9b (5s) == sanityn test 10a: write of file with sub-page size on multiple mounts ========================================================== 10:08:46 (1789049326) 1+0 records in 1+0 records out 1 byte copied, 0.00717844 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.018557 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0163192 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0140621 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0101567 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0120952 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.010904 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0152265 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0159251 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0300234 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00758584 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0153498 s, 0.1 kB/s PASS 10a (9s) == sanityn test 10b: write of file with sub-page size on multiple mounts ========================================================== 10:08:55 (1789049335) 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.0285261 s, 108 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0428426 s, 95.6 kB/s 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.000388329 s, 7.9 MB/s PASS 10b (6s) == sanityn test 11: execution of file opened for write should return error ============================================================== 10:09:02 (1789049342) multiop /mnt/lustre/d11/f vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7620 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 298: /mnt/lustre2/d11/f: Text file busy PASS 11 (6s) == sanityn test 12: test lock ordering (link, stat, unlink) ========================================================== 10:09:07 (1789049347) warning: '-runtime' deprecated, use '-t runtime' instead - link 391 (time 1789049360.33 total 10.01 last 39.06) - link 774 (time 1789049370.33 total 20.01 last 38.29) - link 1155 (time 1789049380.36 total 30.04 last 38.01) - link 1483 (time 1789049390.38 total 40.06 last 32.74) - link 1812 (time 1789049400.41 total 50.09 last 32.80) - link 2138 (time 1789049410.41 total 60.09 last 32.60) - link 2545 (time 1789049420.43 total 70.11 last 40.61) - link 2903 (time 1789049430.47 total 80.15 last 35.65) - link 3215 (time 1789049440.49 total 90.17 last 31.14) total: 3527 link in 100.00 seconds: 35.27 ops/second using seed 2472768868 running for 100 seconds - stat 5975 (time 1789049362 ; total 11 ; last 11) - stat 10000 (time 1789049368 ; total 17 ; last 6) - stat 16537 (time 1789049379 ; total 28 ; last 11) - stat 20000 (time 1789049385 ; total 34 ; last 6) - stat 26345 (time 1789049396 ; total 45 ; last 11) - stat 30000 (time 1789049406 ; total 55 ; last 10) - stat 32995 (time 1789049417 ; total 66 ; last 11) - stat 36223 (time 1789049428 ; total 77 ; last 11) - stat 39126 (time 1789049439 ; total 88 ; last 11) - stat 40000 (time 1789049442 ; total 91 ; last 3) total: 42635 stats in 100 seconds: 426.350006 stats/second - unlinked 0 (time 1789049401 ; total 0 ; last 0) unlink(/mnt/lustre2/lockdir/lockfile3527) error: No such file or directory total: 3526 unlinks in 91 seconds: 38.747253 unlinks/second /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 77: kill: (21420) - No such process /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 78: kill: (21422) - No such process PASS 12 (150s) == sanityn test 13: test directory page revocation ======= 10:11:37 (1789049497) f13 f13 PASS 13 (8s) == sanityn test 14aa: execution of file open for write returns -ETXTBSY ========================================================== 10:11:45 (1789049505) multiop /mnt/lustre/d14aa.sanityn/f14aa.sanityn vOw_c TMPPIPE=/tmp/multiop_open_wait_pipe.7620 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 333: /mnt/lustre2/d14aa.sanityn/f14aa.sanityn: Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5116: 22607 Terminated $MULTIOP_PROG $FILE v$ARGS > $TMPPIPE (wd: ~) PASS 14aa (7s) == sanityn test 14ab: open(RDWR) of executing file returns -ETXTBSY ========================================================== 10:11:52 (1789049512) open(O_RDWR|O_CREAT): Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5116: 23186 Terminated $DIR1/$tdir/sleep 60 (wd: ~) PASS 14ab (6s) == sanityn test 14b: truncate of executing file returns -ETXTBSY ================================================================ 10:11:58 (1789049518) truncate: cannot truncate '/mnt/lustre2/d14b.sanityn/sleep' to length 60: Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 348: 23764 Terminated $DIR1/$tdir/sleep 60 PASS 14b (6s) == sanityn test 14c: open(O_TRUNC) of executing file return -ETXTBSY ============================================================ 10:12:04 (1789049524) cp: cannot create regular file '/mnt/lustre2/d14c.sanityn/sleep': Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 360: 24346 Terminated $DIR1/$tdir/sleep 60 PASS 14c (5s) == sanityn test 14d: chmod of executing file is still possible ================================================================== 10:12:09 (1789049529) chmod /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 371: 24928 Terminated $DIR1/$tdir/sleep 60 PASS 14d (7s) == sanityn test 15: test out-of-space with multiple writers ===================================================================== 10:12:16 (1789049536) PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin: MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky sleep 5 for ZFS MDS Waiting for MDT destroys to complete fallocate on zfs doesn't consume space STRIPECOUNT=2 ORIGFREE=7531520 MAXFREE=800000 SKIP: oos2 test_15 oos2.sh: 7531520KB free > 800000KB, increase MAXFREE (or reduce fs size) sleep 5 for ZFS MDS Waiting for MDT destroys to complete SKIP 15 (45s) == sanityn test 16a: 500 iterations of dual-mount fsx ==== 10:13:01 (1789049581) keep default fallocate mode: 2 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.542829 s, 15.5 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.303675 s, 27.6 MB/s Chance of close/open is 1 in 50 Seed set to 9585 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000001[1] 1789049585.996384 write 0x2b4a66 thru 0x2b6aca (0x2065 bytes) 000002[1] 1789049586.045189 write 0x983597 thru 0x98839b (0x4e05 bytes) truncating to largest ever: 0x632b00 000003[1] 1789049586.083092 trunc from 0x98839c to 0x632aff (0x35589c bytes) 000004[1] 1789049586.131172 mapread 0x58ae2a thru 0x592f2a (0x8101 bytes) 000005[0] 1789049586.160111 read 0x1fd901 thru 0x209a3b (0xc13b bytes) 000006[1] 1789049586.203165 punch from 0x0e9964 to 0x0f58c7 (0xbf64 bytes) 000008[1] 1789049586.251528 read 0x11aa19 thru 0x11df5b (0x3543 bytes) 000011[1] 1789049586.316036 punch from 0x57a3f8 to 0x57e2c3 (0x3ecc bytes) 000012[0] 1789049586.357538 trunc from 0x632b00 to 0x136094 (0x4fca6b bytes) 000013[1] 1789049586.430531 mapread 0x0e4ecc thru 0x0e8f5b (0x4090 bytes) 000014[1] 1789049586.477390 trunc from 0x136095 to 0x1180a7 (0x1dfed bytes) 000015[0] 1789049586.515929 trunc from 0x1180a8 to 0x1c7810 (0xaf769 bytes) truncating to largest ever: 0x91640c 000016[0] 1789049586.553432 trunc from 0x1c7811 to 0x91640b (0x74ebfb bytes) 000017[0] 1789049586.592894 trunc from 0x91640c to 0x41d44d (0x4f8fbe bytes) 000019[0] 1789049586.648366 read 0x11469f thru 0x118abf (0x4421 bytes) 000022[0] 1789049586.681496 read 0x0e9c3f thru 0x0f4a9e (0xae60 bytes) 000023[0] 1789049586.694574 mapwrite 0x72ad81 thru 0x72cc8f (0x1f0f bytes) 000024[1] 1789049586.789869 read 0x251b90 thru 0x254e4e (0x32bf bytes) 000026[0] 1789049586.811576 punch from 0x370e79 to 0x374bbd (0x3d45 bytes) 000027[1] 1789049586.844278 write 0x24505f thru 0x245bd4 (0xb76 bytes) 000028[0] 1789049586.877831 read 0x477dc1 thru 0x47c4f0 (0x4730 bytes) 000029[1] 1789049586.900585 mapwrite 0x0b7cad thru 0x0bd220 (0x5574 bytes) 000030[0] 1789049586.993829 read 0x12528e thru 0x12e7a3 (0x9516 bytes) 000031[1] 1789049587.060176 mapwrite 0x12ea11 thru 0x139b46 (0xb136 bytes) 000034[0] 1789049587.273640 mapread 0x2e0da4 thru 0x2edd33 (0xcf90 bytes) 000035[0] 1789049587.305312 read 0x57f5bc thru 0x585f4a (0x698f bytes) 000037[1] 1789049587.324328 punch from 0x4d42cc to 0x4d81dc (0x3f11 bytes) 000038[1] 1789049587.340166 trunc from 0x72cc90 to 0x015352 (0x71793d bytes) 000039[1] 1789049587.386924 mapwrite 0x57a5ed thru 0x587927 (0xd33b bytes) 000041[0] 1789049587.592518 trunc from 0x587928 to 0x26a72a (0x31d1fd bytes) 000042[0] 1789049587.669615 mapread 0x0d9366 thru 0x0dca3a (0x36d5 bytes) 000045[1] 1789049587.714134 mapwrite 0x9a8a55 thru 0x9b4fad (0xc559 bytes) 000046[0] 1789049587.929945 read 0x9534bb thru 0x957cef (0x4835 bytes) 000047[0] 1789049587.974886 write 0x0572b8 thru 0x059d39 (0x2a82 bytes) 000049[1] 1789049588.050293 write 0x8e12db thru 0x8e20cd (0xdf3 bytes) 000050[1] 1789049588.105353 trunc from 0x9b4fae to 0x29885d (0x71c750 bytes) 000051[1] 1789049588.150657 mapread 0x1eb013 thru 0x1f2d14 (0x7d02 bytes) 000052[0] 1789049588.168024 mapread 0x1cd5d5 thru 0x1dd5a1 (0xffcd bytes) 000053[0] 1789049588.219575 read 0x137cde thru 0x1438f8 (0xbc1b bytes) 000054[1] 1789049588.315136 read 0x17f310 thru 0x18c009 (0xccfa bytes) 000058[0] 1789049588.401220 write 0x9b202d thru 0x9b5539 (0x350d bytes) 000059[1] 1789049588.456358 trunc from 0x9b553a to 0x704be2 (0x2b0957 bytes) 000060[1] 1789049588.541267 write 0x745503 thru 0x74b2f4 (0x5df2 bytes) 000061[0] 1789049588.549534 mapread 0x653d8f thru 0x65c730 (0x89a2 bytes) 000062[1] 1789049588.701210 mapread 0x09dacf thru 0x0ac8ff (0xee31 bytes) 000063[1] 1789049588.827780 read 0x5cf82f thru 0x5d07a1 (0xf73 bytes) 000065[0] 1789049588.837408 punch from 0x59ecd6 to 0x5a8a04 (0x9d2f bytes) 000067[1] 1789049588.888441 mapwrite 0x46841b thru 0x46d1ee (0x4dd4 bytes) 000071[1] 1789049589.065494 punch from 0x5e458b to 0x5e8ada (0x4550 bytes) 000073[1] 1789049589.072091 write 0x4ef3e0 thru 0x4ef553 (0x174 bytes) 000074[1] 1789049589.107560 read 0x3fa297 thru 0x408f1d (0xec87 bytes) 000075[1] 1789049589.115480 read 0x07af71 thru 0x08912f (0xe1bf bytes) 000077[0] 1789049589.137412 read 0x3c6d25 thru 0x3cd341 (0x661d bytes) 000079[1] 1789049589.187851 punch from 0x313f65 to 0x31decb (0x9f67 bytes) 000081[1] 1789049589.287755 mapwrite 0x68f92c thru 0x6954e3 (0x5bb8 bytes) 000083[1] 1789049589.461540 punch from 0x35c8b9 to 0x366e4a (0xa592 bytes) 000084[0] 1789049589.512323 read 0x1109e5 thru 0x111c89 (0x12a5 bytes) 000087[1] 1789049589.586785 punch from 0x231ad9 to 0x236d61 (0x5289 bytes) 000088[1] 1789049589.606713 trunc from 0x74b2f5 to 0x7eeefa (0xa3c06 bytes) 000090[0] 1789049589.630380 mapread 0x70cc49 thru 0x70e2e8 (0x16a0 bytes) 000091[1] 1789049589.694110 trunc from 0x7eeefb to 0x771ba8 (0x7d352 bytes) 000093[0] 1789049589.778894 mapwrite 0x44c2d5 thru 0x45c09f (0xfdcb bytes) 000094[1] 1789049589.929797 trunc from 0x771ba9 to 0x1915c9 (0x5e05df bytes) 000095[0] 1789049590.012909 mapwrite 0x0eb3ed thru 0x0f1490 (0x60a4 bytes) 000098[0] 1789049590.177601 write 0x97654d thru 0x984e2c (0xe8e0 bytes) 000099[1] 1789049590.197239 mapwrite 0x0a17bd thru 0x0a3b78 (0x23bc bytes) 000101[0] 1789049590.323426 read 0x6b4e1a thru 0x6b5ca0 (0xe87 bytes) 000104[0] 1789049590.398667 write 0x813f7e thru 0x818c0a (0x4c8d bytes) 000106[0] 1789049590.429561 mapwrite 0x078508 thru 0x07b5f2 (0x30eb bytes) 000109[1] 1789049590.581277 trunc from 0x984e2d to 0x4a41a2 (0x4e0c8a bytes) 000110[0] 1789049590.633943 mapread 0x0fa95a thru 0x106846 (0xbeed bytes) 000111[0] 1789049590.690382 write 0x04043a thru 0x04fd06 (0xf8cd bytes) 000113[1] 1789049590.754003 mapwrite 0x32aef2 thru 0x32d2d3 (0x23e2 bytes) 000115[0] 1789049590.873229 punch from 0x37788a to 0x386236 (0xe9ad bytes) 000116[0] 1789049590.899068 punch from 0x1416c2 to 0x148d30 (0x766f bytes) 000119[0] 1789049590.941755 mapread 0x2de4c4 thru 0x2ea6ff (0xc23c bytes) 000120[1] 1789049590.983956 read 0x1a1c3f thru 0x1a4a3e (0x2e00 bytes) 000122[1] 1789049591.020267 mapread 0x320724 thru 0x321e42 (0x171f bytes) 000123[0] 1789049591.029816 punch from 0x108469 to 0x108c70 (0x808 bytes) 000124[0] 1789049591.069424 mapwrite 0x415f95 thru 0x41fc08 (0x9c74 bytes) 000127[0] 1789049591.210787 punch from 0x3c3885 to 0x3cb7f8 (0x7f74 bytes) 000128[1] 1789049591.254344 write 0x95d723 thru 0x96733b (0x9c19 bytes) 000131[0] 1789049591.353469 write 0x959aee thru 0x965684 (0xbb97 bytes) 000132[0] 1789049591.439369 mapwrite 0x67ff17 thru 0x68e986 (0xea70 bytes) 000133[1] 1789049591.589556 punch from 0x63db87 to 0x64dab7 (0xff31 bytes) truncating to largest ever: 0x9a7664 000134[1] 1789049591.609579 trunc from 0x96733c to 0x9a7663 (0x40328 bytes) truncating to largest ever: 0x9cb86e 000138[1] 1789049591.751898 trunc from 0x9a7664 to 0x9cb86d (0x2420a bytes) 000144[1] 1789049591.851334 mapwrite 0x358f51 thru 0x35a7a9 (0x1859 bytes) 000147[1] 1789049592.022802 trunc from 0x9cb86e to 0x973c76 (0x57bf7 bytes) 000150[0] 1789049592.072133 trunc from 0x973c77 to 0x666ad9 (0x30d19d bytes) 000151[0] 1789049592.129649 read 0x1c9e54 thru 0x1d773a (0xd8e7 bytes) 000153[0] 1789049592.142263 punch from 0x5c112b to 0x5ca8aa (0x9780 bytes) 000154[1] 1789049592.179439 trunc from 0x666ada to 0x36a42d (0x2fc6ac bytes) 000156[0] 1789049592.245318 mapread 0x2ba514 thru 0x2bcbca (0x26b7 bytes) 000160[0] 1789049592.334444 trunc from 0x36a42e to 0x156079 (0x2143b4 bytes) 000162[1] 1789049592.402883 read 0x08dfde thru 0x08e041 (0x64 bytes) 000163[1] 1789049592.476513 read 0x05598f thru 0x05713d (0x17af bytes) 000164[0] 1789049592.489944 punch from 0x00f966 to 0x015e10 (0x64ab bytes) truncating to largest ever: 0x9eb429 000165[1] 1789049592.504579 trunc from 0x15607a to 0x9eb428 (0x8953af bytes) 000166[1] 1789049592.560463 mapwrite 0x208738 thru 0x208e0f (0x6d8 bytes) 000168[1] 1789049592.726124 mapread 0x75de27 thru 0x76733b (0x9515 bytes) 000169[0] 1789049592.796508 punch from 0x4aba53 to 0x4b7963 (0xbf11 bytes) 000170[0] 1789049592.840230 punch from 0x0b306a to 0x0b650b (0x34a2 bytes) 000171[0] 1789049592.893258 punch from 0x222139 to 0x226e72 (0x4d3a bytes) 000173[1] 1789049592.912498 mapwrite 0x8fb008 thru 0x907f66 (0xcf5f bytes) 000174[1] 1789049593.047884 mapread 0x8f2cee thru 0x8fa839 (0x7b4c bytes) 000178[0] 1789049593.145992 mapread 0x7a78ac thru 0x7b4620 (0xcd75 bytes) 000179[0] 1789049593.186790 trunc from 0x9eb429 to 0x61e815 (0x3ccc13 bytes) 000180[1] 1789049593.257058 punch from 0x5f38cf to 0x5f7c1c (0x434e bytes) 000183[1] 1789049593.323655 mapwrite 0x361dde thru 0x366cdf (0x4f02 bytes) 000184[0] 1789049593.422485 mapread 0x27ebae thru 0x28ae85 (0xc2d8 bytes) 000185[1] 1789049593.479525 read 0x3bf57e thru 0x3cce28 (0xd8ab bytes) 000187[0] 1789049593.531537 trunc from 0x61e816 to 0x8f4ba9 (0x2d6394 bytes) 000188[0] 1789049593.587606 write 0x12142a thru 0x122863 (0x143a bytes) 000189[1] 1789049593.597995 mapwrite 0x793345 thru 0x7a2138 (0xedf4 bytes) 000190[1] 1789049593.755477 write 0x0c166e thru 0x0ceca9 (0xd63c bytes) 000191[0] 1789049593.793906 mapwrite 0x09fb0b thru 0x0a9be4 (0xa0da bytes) 000192[0] 1789049593.937187 write 0x32a431 thru 0x3315a3 (0x7173 bytes) 000193[0] 1789049593.960119 punch from 0x494b9d to 0x49a832 (0x5c96 bytes) 000194[0] 1789049593.973440 write 0x6a37ee thru 0x6a762f (0x3e42 bytes) 000195[0] 1789049593.981379 mapwrite 0x34898e thru 0x348cd7 (0x34a bytes) 000198[0] 1789049594.047315 punch from 0x5a143a to 0x5aaed5 (0x9a9c bytes) 000199[1] 1789049594.058089 read 0x5fb59c thru 0x6093d1 (0xde36 bytes) 000202[0] 1789049594.181032 mapwrite 0x0acf57 thru 0x0b9163 (0xc20d bytes) 000206[1] 1789049594.311135 mapread 0x04dd4f thru 0x04ef1e (0x11d0 bytes) 000207[0] 1789049594.338409 punch from 0x4dc6e7 to 0x4e865e (0xbf78 bytes) 000208[0] 1789049594.352364 write 0x68a0e8 thru 0x68d528 (0x3441 bytes) 000209[1] 1789049594.361674 write 0x65143d thru 0x6546be (0x3282 bytes) 000211[1] 1789049594.382829 mapread 0x86b7c0 thru 0x872042 (0x6883 bytes) 000212[1] 1789049594.424884 trunc from 0x8f4baa to 0x3b46f3 (0x5404b6 bytes) 000213[1] 1789049594.485545 mapwrite 0x81d7a4 thru 0x8235fa (0x5e57 bytes) 000214[0] 1789049594.619011 mapwrite 0x8bade2 thru 0x8c6dc6 (0xbfe5 bytes) 000215[0] 1789049594.808515 mapread 0x39b9e3 thru 0x3ab4d6 (0xfaf4 bytes) 000216[1] 1789049594.882667 read 0x7019f5 thru 0x70948b (0x7a97 bytes) 000219[0] 1789049594.934637 punch from 0x2e4dac to 0x2e6c3e (0x1e93 bytes) 000220[1] 1789049594.943307 mapwrite 0x81df72 thru 0x823d7a (0x5e09 bytes) 000221[0] 1789049595.101112 write 0x537593 thru 0x5423f3 (0xae61 bytes) 000222[1] 1789049595.162121 read 0x5330d4 thru 0x539071 (0x5f9e bytes) 000225[0] 1789049595.221875 trunc from 0x8c6dc7 to 0x9097af (0x429e9 bytes) 000226[1] 1789049595.279034 write 0x1311f8 thru 0x13e45f (0xd268 bytes) 000227[0] 1789049595.317219 trunc from 0x9097b0 to 0x93d5af (0x33e00 bytes) 000228[0] 1789049595.353155 punch from 0x08d2ec to 0x09ba3c (0xe751 bytes) 000230[1] 1789049595.386582 mapwrite 0x883c9d thru 0x88f7a0 (0xbb04 bytes) 000231[1] 1789049595.530492 read 0x923675 thru 0x931c9d (0xe629 bytes) 000233[1] 1789049595.597907 mapread 0x2ce16a thru 0x2d5851 (0x76e8 bytes) 000234[0] 1789049595.650748 trunc from 0x93d5b0 to 0x1dd3fc (0x7601b3 bytes) 000235[1] 1789049595.685607 write 0x186d77 thru 0x19211f (0xb3a9 bytes) 000236[0] 1789049595.715506 punch from 0x1cda59 to 0x1d1ea8 (0x4450 bytes) 000237[0] 1789049595.784310 punch from 0x157b82 to 0x159e4f (0x22ce bytes) 000238[1] 1789049595.791018 read 0x1343b2 thru 0x13ee0b (0xaa5a bytes) 000240[0] 1789049595.858552 write 0x0fcff9 thru 0x1035de (0x65e6 bytes) 000241[1] 1789049595.895345 punch from 0x0bc1b9 to 0x0c0bc4 (0x4a0c bytes) 000242[0] 1789049595.933351 trunc from 0x1dd3fd to 0x7129ea (0x5355ee bytes) 000243[1] 1789049595.988138 punch from 0x3bdaa9 to 0x3cd065 (0xf5bd bytes) 000244[0] 1789049596.021220 punch from 0x4ddb52 to 0x4ebfcd (0xe47c bytes) 000245[1] 1789049596.054374 write 0x250dd2 thru 0x26044e (0xf67d bytes) 000247[0] 1789049596.154259 punch from 0x5f643c to 0x5fd8b4 (0x7479 bytes) 000253[0] 1789049596.275199 trunc from 0x7129eb to 0x01c84c (0x6f619e bytes) 000254[1] 1789049596.330045 punch from 0x017135 to 0x01c84c (0x5718 bytes) 000255[0] 1789049596.380646 mapread 0x001f71 thru 0x011498 (0xf528 bytes) 000257[1] 1789049596.469419 write 0x46d248 thru 0x471f85 (0x4d3e bytes) 000258[1] 1789049596.494448 mapwrite 0x3702b4 thru 0x379ce4 (0x9a31 bytes) 000261[0] 1789049596.593138 mapread 0x15aa98 thru 0x165fc4 (0xb52d bytes) 000262[1] 1789049596.641231 punch from 0x1d949e to 0x1e3d49 (0xa8ac bytes) 000264[1] 1789049596.682926 read 0x0e4354 thru 0x0f05db (0xc288 bytes) 000267[0] 1789049596.714282 mapwrite 0x6ca3d4 thru 0x6cb1cf (0xdfc bytes) 000269[1] 1789049596.848460 write 0x206d38 thru 0x20f116 (0x83df bytes) 000270[0] 1789049596.886296 mapread 0x341342 thru 0x34662f (0x52ee bytes) 000272[0] 1789049596.961630 read 0x176690 thru 0x184711 (0xe082 bytes) 000273[1] 1789049596.984953 punch from 0x1f202c to 0x201fad (0xff82 bytes) 000276[1] 1789049597.031167 mapwrite 0x63315b thru 0x63f15f (0xc005 bytes) 000280[0] 1789049597.183557 trunc from 0x6cb1d0 to 0x77f596 (0xb43c7 bytes) 000281[1] 1789049597.276457 write 0x8ea1b0 thru 0x8f4888 (0xa6d9 bytes) 000283[0] 1789049597.317953 mapwrite 0x945682 thru 0x94993e (0x42bd bytes) 000285[1] 1789049597.505787 mapread 0x3c6a10 thru 0x3d4f2e (0xe51f bytes) 000287[0] 1789049597.653636 write 0x33f420 thru 0x34c2ad (0xce8e bytes) 000288[1] 1789049597.706598 punch from 0x8bbef8 to 0x8c543d (0x9546 bytes) 000289[0] 1789049597.746325 punch from 0x675333 to 0x67aedd (0x5bab bytes) 000290[0] 1789049597.775188 write 0x436881 thru 0x43eda3 (0x8523 bytes) 000291[1] 1789049597.800972 mapread 0x6d3f51 thru 0x6d92b1 (0x5361 bytes) 000292[0] 1789049597.853326 trunc from 0x94993f to 0x1cc02f (0x77d90f bytes) 000294[1] 1789049597.895505 mapwrite 0x9cee21 thru 0x9d77a6 (0x8986 bytes) 000298[1] 1789049598.056633 trunc from 0x9d77a7 to 0x10a2c9 (0x8cd4dd bytes) 000299[1] 1789049598.097646 punch from 0x0d7fad to 0x0dbee1 (0x3f35 bytes) 000300[0] 1789049598.115267 punch from 0x059dba to 0x062251 (0x8498 bytes) 000301[1] 1789049598.149309 mapread 0x097eac thru 0x0999d3 (0x1b28 bytes) 000302[1] 1789049598.184194 write 0x94acd9 thru 0x94cc67 (0x1f8f bytes) 000303[1] 1789049598.197909 mapread 0x6ee6cb thru 0x6f00dc (0x1a12 bytes) 000304[1] 1789049598.239432 mapread 0x291dec thru 0x298f0f (0x7124 bytes) 000305[0] 1789049598.305283 trunc from 0x94cc68 to 0x31fd13 (0x62cf54 bytes) 000307[0] 1789049598.397330 trunc from 0x31fd14 to 0x2399e6 (0xe632d bytes) 000311[1] 1789049598.437996 mapread 0x1c6203 thru 0x1d3239 (0xd037 bytes) 000313[0] 1789049598.519525 trunc from 0x2399e7 to 0x345517 (0x10bb31 bytes) 000315[1] 1789049598.549102 trunc from 0x345518 to 0x094225 (0x2b12f2 bytes) 000319[0] 1789049598.658495 trunc from 0x094226 to 0x20eee5 (0x17acc0 bytes) 000320[1] 1789049598.712585 trunc from 0x20eee6 to 0x4a5cde (0x296df9 bytes) 000324[1] 1789049598.778338 trunc from 0x4a5cdf to 0x233e38 (0x271ea6 bytes) 000325[0] 1789049598.820485 mapwrite 0x89df3d thru 0x8aa9d2 (0xca96 bytes) 000326[0] 1789049598.915507 write 0x529917 thru 0x536408 (0xcaf2 bytes) 000327[1] 1789049598.924973 trunc from 0x8aa9d3 to 0x60915f (0x2a1873 bytes) 000328[0] 1789049598.974906 mapread 0x35b5c3 thru 0x35bc52 (0x690 bytes) 000329[1] 1789049599.001332 mapread 0x43b552 thru 0x43f38b (0x3e3a bytes) 000334[1] 1789049599.097476 punch from 0x4c2c1d to 0x4d1ea9 (0xf28d bytes) 000335[1] 1789049599.113313 punch from 0x597d69 to 0x5a7527 (0xf7bf bytes) 000336[0] 1789049599.128491 punch from 0x04abdb to 0x058075 (0xd49b bytes) 000337[0] 1789049599.173331 read 0x228f7b thru 0x233665 (0xa6eb bytes) 000338[1] 1789049599.221121 mapwrite 0x15398c thru 0x15f536 (0xbbab bytes) 000339[1] 1789049599.480446 punch from 0x251d89 to 0x25703b (0x52b3 bytes) 000340[0] 1789049599.495929 write 0x5b8669 thru 0x5c1493 (0x8e2b bytes) 000341[0] 1789049599.551491 mapwrite 0x6fe866 thru 0x709722 (0xaebd bytes) 000343[1] 1789049599.782846 punch from 0x67ba02 to 0x68671c (0xad1b bytes) 000344[0] 1789049599.835549 write 0x4e0f73 thru 0x4e3518 (0x25a6 bytes) 000346[0] 1789049599.908582 mapwrite 0x082c14 thru 0x090331 (0xd71e bytes) 000347[0] 1789049600.013467 punch from 0x4178ac to 0x422c7a (0xb3cf bytes) 000348[0] 1789049600.024540 mapread 0x0c8f5b thru 0x0cd81e (0x48c4 bytes) 000349[0] 1789049600.037454 read 0x414f15 thru 0x4225b6 (0xd6a2 bytes) 000351[0] 1789049600.049413 mapwrite 0x74b20e thru 0x753f5f (0x8d52 bytes) 000352[1] 1789049600.156317 mapread 0x1bb27c thru 0x1c1e3a (0x6bbf bytes) 000353[0] 1789049600.201545 write 0x2644d0 thru 0x26db87 (0x96b8 bytes) 000354[0] 1789049600.223102 write 0x7a43c0 thru 0x7b05fd (0xc23e bytes) 000356[1] 1789049600.257656 read 0x0356a7 thru 0x043427 (0xdd81 bytes) 000357[1] 1789049600.293981 write 0x846a06 thru 0x84d56d (0x6b68 bytes) 000364[1] 1789049600.456209 mapwrite 0x6fc241 thru 0x706a9a (0xa85a bytes) 000367[0] 1789049600.585751 trunc from 0x84d56e to 0x1f1bd7 (0x65b996 bytes) 000368[0] 1789049600.652920 trunc from 0x1f1bd8 to 0x718570 (0x526999 bytes) 000369[0] 1789049600.693584 mapwrite 0x399e5b thru 0x3a8638 (0xe7de bytes) 000372[0] 1789049600.802153 trunc from 0x718571 to 0x5d4c5f (0x143911 bytes) 000373[1] 1789049600.833524 mapread 0x03ed82 thru 0x0451ab (0x642a bytes) 000374[0] 1789049600.899767 trunc from 0x5d4c60 to 0x39eb13 (0x23614c bytes) 000375[0] 1789049600.963883 trunc from 0x39eb14 to 0x8351ea (0x4966d7 bytes) 000378[0] 1789049601.084236 mapwrite 0x2e4a46 thru 0x2e7572 (0x2b2d bytes) 000380[1] 1789049601.143287 write 0x57b2af thru 0x57f616 (0x4368 bytes) 000382[1] 1789049601.204068 mapwrite 0x8830c2 thru 0x88e9dd (0xb91c bytes) 000383[1] 1789049601.369211 mapwrite 0x9dd47f thru 0x9de7a7 (0x1329 bytes) 000386[0] 1789049601.450182 read 0x2df7e4 thru 0x2ec415 (0xcc32 bytes) 000387[0] 1789049601.471042 trunc from 0x9de7a8 to 0x6a18d5 (0x33ced2 bytes) 000388[0] 1789049601.546421 mapread 0x60ade0 thru 0x614d77 (0x9f98 bytes) 000391[1] 1789049601.577573 mapread 0x5a7f0b thru 0x5a8135 (0x22b bytes) 000392[1] 1789049601.628381 read 0x5e4330 thru 0x5ed1b2 (0x8e83 bytes) 000393[0] 1789049601.654581 mapwrite 0x577a29 thru 0x584859 (0xce31 bytes) 000397[1] 1789049601.808598 punch from 0x0b4e62 to 0x0b807f (0x321e bytes) 000401[0] 1789049601.912038 read 0x4a26b7 thru 0x4a33b1 (0xcfb bytes) 000403[1] 1789049601.957489 mapwrite 0x805d24 thru 0x814509 (0xe7e6 bytes) 000404[1] 1789049602.101299 punch from 0x60279a to 0x60ac48 (0x84af bytes) 000405[1] 1789049602.108096 mapread 0x2c9375 thru 0x2cbe02 (0x2a8e bytes) 000407[1] 1789049602.170757 write 0x94b244 thru 0x94d584 (0x2341 bytes) 000409[1] 1789049602.173439 punch from 0x75903a to 0x7627bf (0x9786 bytes) 000411[0] 1789049602.225098 write 0x54191b thru 0x54d1e0 (0xb8c6 bytes) 000415[0] 1789049602.352161 trunc from 0x94d585 to 0x0844cb (0x8c90b9 bytes) 000416[0] 1789049602.403109 mapread 0x01105f thru 0x01f786 (0xe728 bytes) 000417[0] 1789049602.437340 write 0x755ac6 thru 0x759288 (0x37c3 bytes) 000418[0] 1789049602.480388 write 0x8bf2e5 thru 0x8c09d7 (0x16f3 bytes) 000422[1] 1789049602.579477 mapread 0x881381 thru 0x887fe1 (0x6c61 bytes) 000423[1] 1789049602.634014 mapwrite 0x279c75 thru 0x27dcf1 (0x407d bytes) 000426[0] 1789049602.784959 read 0x70867d thru 0x715294 (0xcc18 bytes) 000430[1] 1789049602.898088 punch from 0x1665ac to 0x17618b (0xfbe0 bytes) 000432[0] 1789049602.933830 read 0x716e3d thru 0x719df7 (0x2fbb bytes) 000434[1] 1789049602.984591 punch from 0x840da0 to 0x8412d4 (0x535 bytes) 000435[0] 1789049603.003561 trunc from 0x8c09d8 to 0x2d1bfc (0x5eeddb bytes) 000436[0] 1789049603.040079 mapread 0x21bf4f thru 0x220c94 (0x4d46 bytes) 000439[0] 1789049603.134971 mapread 0x17b892 thru 0x18075c (0x4ecb bytes) truncating to largest ever: 0x9fd5b9 000440[1] 1789049603.145461 trunc from 0x2d1bfd to 0x9fd5b8 (0x72b9bc bytes) 000443[1] 1789049603.248880 punch from 0x979b3e to 0x9872fb (0xd7be bytes) 000444[0] 1789049603.280061 punch from 0x56723a to 0x573b27 (0xc8ee bytes) 000446[0] 1789049603.358913 trunc from 0x9fd5b9 to 0x3d9721 (0x623e97 bytes) 000447[1] 1789049603.452537 punch from 0x250c60 to 0x2583b8 (0x7759 bytes) 000448[1] 1789049603.482521 mapwrite 0x5b32d5 thru 0x5b89d0 (0x56fc bytes) 000449[0] 1789049603.560438 mapread 0x46b231 thru 0x476c27 (0xb9f7 bytes) 000451[1] 1789049603.663328 write 0x7d35a8 thru 0x7d9ac2 (0x651b bytes) 000452[1] 1789049603.698013 write 0x0030a4 thru 0x00dd92 (0xacef bytes) 000454[1] 1789049603.717516 trunc from 0x7d9ac3 to 0x049499 (0x790629 bytes) 000455[1] 1789049603.748906 trunc from 0x04949a to 0x7445cd (0x6fb134 bytes) 000457[0] 1789049603.803622 mapwrite 0x5a1bea thru 0x5a3b8d (0x1fa4 bytes) 000458[0] 1789049603.927209 mapwrite 0x795d21 thru 0x79ca1c (0x6cfc bytes) 000459[1] 1789049604.110548 write 0x403fcf thru 0x406062 (0x2094 bytes) 000460[0] 1789049604.191270 mapwrite 0x701fcf thru 0x70283c (0x86e bytes) 000461[0] 1789049604.457934 read 0x0b1c41 thru 0x0bd36f (0xb72f bytes) 000464[1] 1789049604.508376 write 0x33b436 thru 0x33cd52 (0x191d bytes) 000465[0] 1789049604.558265 trunc from 0x79ca1d to 0x82554d (0x88b31 bytes) 000466[0] 1789049604.654498 read 0x2e3e08 thru 0x2ea660 (0x6859 bytes) 000468[1] 1789049604.706469 trunc from 0x82554e to 0x0807b1 (0x7a4d9c bytes) 000469[1] 1789049604.784046 mapread 0x00d2c9 thru 0x017a44 (0xa77c bytes) 000471[1] 1789049604.809298 write 0x8fe25e thru 0x9034c7 (0x526a bytes) 000472[1] 1789049604.837559 write 0x14a88f thru 0x14c7bc (0x1f2e bytes) 000473[0] 1789049604.869112 trunc from 0x9034c8 to 0x3896a5 (0x579e22 bytes) 000474[0] 1789049604.918321 write 0x93fa28 thru 0x943d4d (0x4326 bytes) 000478[0] 1789049604.964298 mapwrite 0x42137a thru 0x42443f (0x30c6 bytes) 000481[0] 1789049605.088102 mapread 0x520e74 thru 0x529818 (0x89a5 bytes) 000482[0] 1789049605.147625 mapwrite 0x023e34 thru 0x028863 (0x4a30 bytes) 000485[1] 1789049605.274123 mapread 0x4b6bd3 thru 0x4bff5e (0x938c bytes) 000486[0] 1789049605.464291 mapread 0x07a022 thru 0x0838de (0x98bd bytes) 000487[1] 1789049605.483943 trunc from 0x943d4e to 0x8679f0 (0xdc35d bytes) 000488[1] 1789049605.552274 write 0x14ef01 thru 0x1534af (0x45af bytes) 000489[0] 1789049605.591321 mapwrite 0x03b12e thru 0x048461 (0xd334 bytes) 000492[1] 1789049605.800924 trunc from 0x8679f1 to 0x56a09c (0x2fd954 bytes) 000495[0] 1789049605.887333 read 0x4b2791 thru 0x4bf43e (0xccae bytes) 000497[0] 1789049605.932297 mapwrite 0x4a1835 thru 0x4ab0c1 (0x988d bytes) All operations completed A-OK! Chance of close/open is 1 in 50 Seed set to 9606 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000001[0] 1789049606.866345 write_OD 0x243000 thru 0x24efff (0xc000 bytes) 000002[1] 1789049606.923300 mapwrite 0x04b000 thru 0x04dfb9 (0x2fba bytes) 000004[1] 1789049607.011469 mapread 0x0ef000 thru 0x0f39b8 (0x49b9 bytes) truncating to largest ever: 0x545f08 000005[1] 1789049607.029395 trunc from 0x24f000 to 0x545f07 (0x2f6f08 bytes) 000006[0] 1789049607.049833 punch from 0x1e7009 to 0x1ec4ee (0x54e6 bytes) 000007[0] 1789049607.074514 mapread 0x458000 thru 0x465619 (0xd61a bytes) 000008[1] 1789049607.120936 read 0x479000 thru 0x48524d (0xc24e bytes) 000009[0] 1789049607.144820 punch from 0x40728c to 0x40acaa (0x3a1f bytes) 000010[1] 1789049607.160563 punch from 0x505163 to 0x50bc9f (0x6b3d bytes) 000011[0] 1789049607.192012 write_OD 0x058000 thru 0x060fff (0x9000 bytes) truncating to largest ever: 0x60b982 000013[1] 1789049607.257165 trunc from 0x545f08 to 0x60b981 (0xc5a7a bytes) 000015[1] 1789049607.319013 mapread 0x249000 thru 0x249245 (0x246 bytes) 000016[0] 1789049607.345670 read_OD 0x5fc000 thru 0x5fffff (0x4000 bytes) 000020[0] 1789049607.379916 read_OD 0x1d1000 thru 0x1d4fff (0x4000 bytes) 000021[0] 1789049607.389869 write_OD 0x816000 thru 0x818fff (0x3000 bytes) 000027[1] 1789049607.483261 mapwrite 0x58b000 thru 0x5911e3 (0x61e4 bytes) 000028[1] 1789049607.577499 mapwrite 0x7fe000 thru 0x804515 (0x6516 bytes) 000030[0] 1789049607.675387 mapread 0x33c000 thru 0x340233 (0x4234 bytes) 000032[1] 1789049607.725413 mapwrite 0x5da000 thru 0x5e6226 (0xc227 bytes) 000034[1] 1789049607.829585 mapwrite 0x41f000 thru 0x4214db (0x24dc bytes) 000036[0] 1789049607.890894 trunc from 0x819000 to 0x100f6d (0x718092 bytes) 000037[0] 1789049607.915264 mapwrite 0x7ac000 thru 0x7ad543 (0x1544 bytes) 000038[0] 1789049607.979251 read_OD 0x5b7000 thru 0x5c0fff (0xa000 bytes) 000039[0] 1789049607.996320 trunc from 0x7ad544 to 0x3b722f (0x3f6314 bytes) 000040[1] 1789049608.033480 trunc from 0x3b7230 to 0x45c303 (0xa50d4 bytes) 000042[0] 1789049608.070259 mapread 0x418000 thru 0x42011d (0x811e bytes) 000044[0] 1789049608.173163 mapread 0x23e000 thru 0x240e33 (0x2e34 bytes) 000045[1] 1789049608.210158 punch from 0x071994 to 0x077193 (0x5800 bytes) 000046[1] 1789049608.235501 write 0x642000 thru 0x64c944 (0xa945 bytes) 000047[0] 1789049608.260443 read_OD 0x5dc000 thru 0x5e9fff (0xe000 bytes) 000048[1] 1789049608.302995 write 0x555000 thru 0x564180 (0xf181 bytes) 000049[0] 1789049608.322181 mapwrite 0x696000 thru 0x69a677 (0x4678 bytes) 000050[1] 1789049608.423453 write 0x78a000 thru 0x796fe1 (0xcfe2 bytes) 000051[0] 1789049608.436442 read_OD 0x0cb000 thru 0x0d0fff (0x6000 bytes) 000057[0] 1789049608.491218 trunc from 0x796fe2 to 0x05b1f9 (0x73bde8 bytes) 000058[0] 1789049608.523074 punch from 0x032476 to 0x040bb0 (0xe73b bytes) 000059[1] 1789049608.541970 mapwrite 0x57e000 thru 0x57f04b (0x104c bytes) 000060[0] 1789049608.662452 write_OD 0x33d000 thru 0x349fff (0xd000 bytes) 000063[0] 1789049608.743510 write_OD 0x24f000 thru 0x259fff (0xb000 bytes) 000064[0] 1789049608.810801 trunc from 0x57f04c to 0x23c794 (0x3428b7 bytes) 000066[1] 1789049608.864170 mapwrite 0x81a000 thru 0x823d34 (0x9d35 bytes) 000071[1] 1789049608.976198 read 0x3f0000 thru 0x3f09ab (0x9ac bytes) 000075[1] 1789049609.004167 trunc from 0x823d35 to 0x4cefd3 (0x354d61 bytes) 000077[1] 1789049609.036435 punch from 0x214f72 to 0x21dcbf (0x8d4e bytes) truncating to largest ever: 0x72779f 000080[1] 1789049609.066241 trunc from 0x4cefd4 to 0x72779e (0x2587cb bytes) 000081[1] 1789049609.097160 mapwrite 0x588000 thru 0x58e486 (0x6487 bytes) 000083[0] 1789049609.152999 write_OD 0x0e3000 thru 0x0eafff (0x8000 bytes) 000085[1] 1789049609.216192 punch from 0x461280 to 0x46ac07 (0x9988 bytes) 000086[0] 1789049609.253523 read_OD 0x579000 thru 0x579fff (0x1000 bytes) 000087[1] 1789049609.277525 mapwrite 0x0dc000 thru 0x0e7eed (0xbeee bytes) 000088[1] 1789049609.424000 punch from 0x5e55c4 to 0x5e82d5 (0x2d12 bytes) 000089[1] 1789049609.428996 mapwrite 0x208000 thru 0x20a078 (0x2079 bytes) 000090[1] 1789049609.517442 punch from 0x06e381 to 0x072c2c (0x48ac bytes) 000094[1] 1789049609.556300 read 0x62a000 thru 0x636de4 (0xcde5 bytes) 000095[1] 1789049609.565834 mapread 0x047000 thru 0x04ddc4 (0x6dc5 bytes) 000096[0] 1789049609.594652 write_OD 0x45f000 thru 0x468fff (0xa000 bytes) 000097[0] 1789049609.638899 write_OD 0x290000 thru 0x298fff (0x9000 bytes) 000098[0] 1789049609.683667 mapread 0x060000 thru 0x06601e (0x601f bytes) 000099[0] 1789049609.702947 mapwrite 0x7e0000 thru 0x7e2009 (0x200a bytes) 000100[1] 1789049609.804585 punch from 0x07be67 to 0x07f8e5 (0x3a7f bytes) 000101[0] 1789049609.829143 mapwrite 0x523000 thru 0x5267e8 (0x37e9 bytes) 000104[0] 1789049609.929878 write_OD 0x22d000 thru 0x239fff (0xd000 bytes) 000106[1] 1789049610.000929 read 0x215000 thru 0x21afea (0x5feb bytes) 000110[0] 1789049610.021543 trunc from 0x7e200a to 0x2a98e8 (0x538721 bytes) 000111[1] 1789049610.073292 mapread 0x0cb000 thru 0x0d6c37 (0xbc38 bytes) 000112[1] 1789049610.122250 punch from 0x1f9ec4 to 0x209376 (0xf4b3 bytes) 000114[0] 1789049610.136080 trunc from 0x2a98e9 to 0x4f2ff0 (0x249708 bytes) 000116[0] 1789049610.188367 read_OD 0x467000 thru 0x46efff (0x8000 bytes) 000117[0] 1789049610.205680 mapread 0x229000 thru 0x2386c6 (0xf6c7 bytes) 000118[0] 1789049610.243014 trunc from 0x4f2ff1 to 0x45c92f (0x966c1 bytes) 000119[1] 1789049610.286404 punch from 0x02459b to 0x02a09b (0x5b01 bytes) 000121[1] 1789049610.312378 write 0x6b4000 thru 0x6b7708 (0x3709 bytes) 000122[1] 1789049610.337934 punch from 0x2c8d90 to 0x2d5a51 (0xccc2 bytes) 000123[0] 1789049610.342549 mapwrite 0x3e8000 thru 0x3f5c64 (0xdc65 bytes) 000125[1] 1789049610.547822 mapread 0x28a000 thru 0x28f970 (0x5971 bytes) 000126[1] 1789049610.611651 mapwrite 0x8a6000 thru 0x8af31a (0x931b bytes) 000129[0] 1789049610.726546 read_OD 0x54d000 thru 0x54efff (0x2000 bytes) 000130[1] 1789049610.748115 write 0x05b000 thru 0x0604fa (0x54fb bytes) 000132[0] 1789049610.762650 mapwrite 0x42d000 thru 0x432fea (0x5feb bytes) 000133[1] 1789049610.859826 mapwrite 0x8b6000 thru 0x8bbf5f (0x5f60 bytes) 000139[0] 1789049610.960308 punch from 0x0a76fa to 0x0b5cc4 (0xe5cb bytes) 000141[1] 1789049610.995364 trunc from 0x8bbf60 to 0x353b63 (0x5683fc bytes) 000142[0] 1789049611.027300 punch from 0x1b6872 to 0x1bff9f (0x972e bytes) 000143[0] 1789049611.036357 trunc from 0x353b64 to 0x4c894c (0x174de9 bytes) 000144[1] 1789049611.054792 mapwrite 0x4cc000 thru 0x4db9b3 (0xf9b4 bytes) 000145[0] 1789049611.169481 punch from 0x4d49e7 to 0x4db9b3 (0x6fcd bytes) 000146[1] 1789049611.184076 write 0x2f2000 thru 0x2fca4c (0xaa4d bytes) 000147[1] 1789049611.201286 trunc from 0x4db9b4 to 0x5cab5b (0xef1a8 bytes) 000149[1] 1789049611.219852 mapread 0x575000 thru 0x58410a (0xf10b bytes) 000156[0] 1789049611.289087 write_OD 0x2ab000 thru 0x2b3fff (0x9000 bytes) 000160[0] 1789049611.367344 punch from 0x5b9fb5 to 0x5c90ac (0xf0f8 bytes) 000165[1] 1789049611.439287 write 0x6de000 thru 0x6e37e1 (0x57e2 bytes) 000166[0] 1789049611.476433 mapread 0x683000 thru 0x68ed65 (0xbd66 bytes) 000167[1] 1789049611.546093 mapwrite 0x291000 thru 0x299a0a (0x8a0b bytes) 000168[0] 1789049611.634995 write_OD 0x84a000 thru 0x851fff (0x8000 bytes) 000169[0] 1789049611.705386 trunc from 0x852000 to 0x3a75ae (0x4aaa51 bytes) 000170[0] 1789049611.751000 write_OD 0x411000 thru 0x41ffff (0xf000 bytes) 000171[0] 1789049611.814550 mapwrite 0x22f000 thru 0x233ff9 (0x4ffa bytes) truncating to largest ever: 0x996476 000175[0] 1789049611.932773 trunc from 0x420000 to 0x996475 (0x576476 bytes) 000178[0] 1789049611.978496 read_OD 0x55e000 thru 0x565fff (0x8000 bytes) 000179[0] 1789049612.009458 write_OD 0x62a000 thru 0x62cfff (0x3000 bytes) 000181[0] 1789049612.066367 mapwrite 0x121000 thru 0x12bbdc (0xabdd bytes) 000182[1] 1789049612.166483 trunc from 0x996476 to 0x62a43d (0x36c038 bytes) 000183[0] 1789049612.212250 mapread 0x35e000 thru 0x36db81 (0xfb82 bytes) 000184[0] 1789049612.271953 mapread 0x1b6000 thru 0x1c0789 (0xa78a bytes) 000185[1] 1789049612.299570 trunc from 0x62a43e to 0x82c948 (0x20250b bytes) 000186[1] 1789049612.356542 mapwrite 0x5f0000 thru 0x5f1afd (0x1afe bytes) 000187[1] 1789049612.422812 mapread 0x459000 thru 0x462eb4 (0x9eb5 bytes) 000188[1] 1789049612.469015 trunc from 0x82c949 to 0x3a8727 (0x484221 bytes) 000189[0] 1789049612.496332 mapwrite 0x3b0000 thru 0x3bf8ba (0xf8bb bytes) 000191[0] 1789049612.643265 punch from 0x218e4a to 0x21fa06 (0x6bbd bytes) 000193[1] 1789049612.652342 write 00000000 thru 0x009987 (0x9988 bytes) 000194[1] 1789049612.684554 mapwrite 0x78d000 thru 0x7932e4 (0x62e5 bytes) 000198[0] 1789049612.894140 mapread 0x19f000 thru 0x1aad9f (0xbda0 bytes) 000201[0] 1789049612.996329 mapread 0x499000 thru 0x4a6ca2 (0xdca3 bytes) 000203[1] 1789049613.070070 punch from 0x613658 to 0x62106c (0xda15 bytes) 000204[0] 1789049613.107271 trunc from 0x7932e5 to 0x035918 (0x75d9cc bytes) 000205[0] 1789049613.187365 mapwrite 0x53a000 thru 0x543b97 (0x9b98 bytes) 000206[0] 1789049613.338250 punch from 0x44f7d8 to 0x457857 (0x8080 bytes) 000208[0] 1789049613.375556 write_OD 0x8b4000 thru 0x8bffff (0xc000 bytes) 000209[0] 1789049613.475969 write_OD 0x6f9000 thru 0x702fff (0xa000 bytes) 000211[0] 1789049613.534492 mapwrite 0x3c4000 thru 0x3cea02 (0xaa03 bytes) 000212[1] 1789049613.655658 mapwrite 0x3d5000 thru 0x3d95a7 (0x45a8 bytes) 000213[1] 1789049613.728921 mapwrite 0x1f0000 thru 0x1fbf92 (0xbf93 bytes) 000214[1] 1789049613.793504 mapread 0x88b000 thru 0x890892 (0x5893 bytes) 000215[1] 1789049613.813602 punch from 0x5ed3af to 0x5f237e (0x4fd0 bytes) 000217[1] 1789049613.828539 read 0x018000 thru 0x0231db (0xb1dc bytes) 000218[1] 1789049613.869169 read 0x3ea000 thru 0x3eaa32 (0xa33 bytes) 000219[1] 1789049613.876235 read 0x6b6000 thru 0x6c1bdf (0xbbe0 bytes) 000221[1] 1789049613.884311 mapread 0x83a000 thru 0x8462a5 (0xc2a6 bytes) 000224[0] 1789049613.904745 write_OD 0x15c000 thru 0x169fff (0xe000 bytes) 000228[0] 1789049613.954164 punch from 0x60fcd3 to 0x614abb (0x4de9 bytes) 000229[1] 1789049613.970482 punch from 0x3984bd to 0x3a3728 (0xb26c bytes) 000231[1] 1789049614.010677 read 0x559000 thru 0x55dd7b (0x4d7c bytes) 000232[1] 1789049614.018292 trunc from 0x8c0000 to 0x7f9617 (0xc69e8 bytes) 000233[0] 1789049614.037154 write_OD 0x93c000 thru 0x93cfff (0x1000 bytes) 000236[0] 1789049614.084560 write_OD 0x5fd000 thru 0x605fff (0x9000 bytes) 000239[1] 1789049614.132298 mapread 0x6d4000 thru 0x6da8a4 (0x68a5 bytes) 000240[0] 1789049614.149622 mapread 0x768000 thru 0x76b0a1 (0x30a2 bytes) 000241[0] 1789049614.162136 write_OD 0x3f2000 thru 0x3f8fff (0x7000 bytes) 000242[0] 1789049614.214942 punch from 0x082211 to 0x08b8fc (0x96ec bytes) 000244[1] 1789049614.226912 mapread 0x77f000 thru 0x7831e0 (0x41e1 bytes) 000245[1] 1789049614.257968 write 0x974000 thru 0x977efd (0x3efe bytes) 000246[0] 1789049614.310178 punch from 0x6727dd to 0x67e06e (0xb892 bytes) 000248[1] 1789049614.346820 write 0x629000 thru 0x631eb8 (0x8eb9 bytes) 000254[1] 1789049614.434403 mapread 0x0cb000 thru 0x0d2da1 (0x7da2 bytes) 000255[1] 1789049614.487386 punch from 0x90c080 to 0x9195ea (0xd56b bytes) 000257[0] 1789049614.493376 punch from 0x18c92d to 0x18faef (0x31c3 bytes) 000258[1] 1789049614.522399 read 0x5f4000 thru 0x5fd9d7 (0x99d8 bytes) 000261[1] 1789049614.545863 write 0x4cf000 thru 0x4d539b (0x639c bytes) 000262[0] 1789049614.562300 mapread 0x0d0000 thru 0x0de78c (0xe78d bytes) 000263[1] 1789049614.630024 read 0x307000 thru 0x309def (0x2df0 bytes) 000266[0] 1789049614.637658 trunc from 0x977efe to 0x72c586 (0x24b977 bytes) 000267[0] 1789049614.663470 punch from 0x39e003 to 0x3a828d (0xa28b bytes) 000270[1] 1789049614.681532 write 0x6a6000 thru 0x6b5504 (0xf505 bytes) 000272[0] 1789049614.723599 mapwrite 0x30d000 thru 0x312e6e (0x5e6f bytes) 000273[1] 1789049614.854307 mapread 0x626000 thru 0x62fc36 (0x9c37 bytes) 000275[1] 1789049614.923880 trunc from 0x72c587 to 0x382d37 (0x3a984f bytes) 000276[1] 1789049614.961386 mapwrite 0x16c000 thru 0x16d237 (0x1238 bytes) 000277[1] 1789049615.027164 trunc from 0x382d38 to 0x31a34e (0x689e9 bytes) 000278[0] 1789049615.058194 trunc from 0x31a34f to 0x2a99b5 (0x70999 bytes) 000279[1] 1789049615.101358 punch from 0x0d7ac1 to 0x0d90d1 (0x1611 bytes) 000281[0] 1789049615.145135 mapread 0x1ae000 thru 0x1b1d72 (0x3d73 bytes) 000282[1] 1789049615.193105 write 0x1f4000 thru 0x1fa44d (0x644e bytes) 000284[1] 1789049615.234400 mapwrite 0x297000 thru 0x297314 (0x315 bytes) 000285[0] 1789049615.314397 trunc from 0x2a99b6 to 0x2534aa (0x5650b bytes) 000286[0] 1789049615.378500 mapwrite 0x278000 thru 0x281846 (0x9847 bytes) 000288[1] 1789049615.490284 trunc from 0x281847 to 0x214c65 (0x6cbe1 bytes) 000289[0] 1789049615.518464 trunc from 0x214c66 to 0x3d097e (0x1bbd19 bytes) 000290[0] 1789049615.566246 trunc from 0x3d097f to 0x7fa7fc (0x429e7e bytes) 000291[1] 1789049615.585469 mapwrite 0x3fd000 thru 0x3fe17b (0x117c bytes) 000292[1] 1789049615.674830 mapread 0x345000 thru 0x352e24 (0xde25 bytes) 000293[0] 1789049615.695236 read_OD 0x7f5000 thru 0x7f6fff (0x2000 bytes) 000295[1] 1789049615.719247 read 0x31c000 thru 0x31c62a (0x62b bytes) 000297[1] 1789049615.724179 write 0x82d000 thru 0x83aa72 (0xda73 bytes) 000298[0] 1789049615.763759 mapread 0x453000 thru 0x4592f3 (0x62f4 bytes) 000301[0] 1789049615.828235 punch from 0x7ceb75 to 0x7db1a3 (0xc62f bytes) 000303[0] 1789049615.859283 write_OD 0x563000 thru 0x570fff (0xe000 bytes) 000304[1] 1789049615.909656 punch from 0x219d3d to 0x222c9a (0x8f5e bytes) 000305[0] 1789049615.935194 read_OD 0x07a000 thru 0x085fff (0xc000 bytes) 000306[1] 1789049615.959159 mapwrite 0x854000 thru 0x86357a (0xf57b bytes) 000307[1] 1789049616.097870 read 0x249000 thru 0x255f55 (0xcf56 bytes) 000310[0] 1789049616.148587 read_OD 0x6cb000 thru 0x6cdfff (0x3000 bytes) 000311[1] 1789049616.180923 mapwrite 0x93d000 thru 0x947b44 (0xab45 bytes) 000312[1] 1789049616.343798 write 0x7e2000 thru 0x7eaf70 (0x8f71 bytes) 000314[1] 1789049616.353203 write 0x7f8000 thru 0x803f68 (0xbf69 bytes) 000323[1] 1789049616.422186 punch from 0x01f176 to 0x029747 (0xa5d2 bytes) 000324[1] 1789049616.426228 trunc from 0x947b45 to 0x376dc7 (0x5d0d7d bytes) 000327[0] 1789049616.447128 punch from 0x070dd5 to 0x079c88 (0x8eb4 bytes) 000329[1] 1789049616.477200 punch from 0x367415 to 0x36c669 (0x5255 bytes) 000330[1] 1789049616.501483 write 0x840000 thru 0x84458c (0x458d bytes) 000331[0] 1789049616.527367 punch from 0x357975 to 0x35f779 (0x7e05 bytes) 000334[0] 1789049616.581471 punch from 0x39a78e to 0x3a7137 (0xc9aa bytes) 000335[0] 1789049616.588380 trunc from 0x84458d to 0x107835 (0x73cd57 bytes) 000338[0] 1789049616.613426 mapwrite 0x697000 thru 0x6a425b (0xd25c bytes) 000339[1] 1789049616.761052 punch from 0x3bc3ea to 0x3c3d0f (0x7926 bytes) 000340[1] 1789049616.814296 punch from 0x3a0f1f to 0x3a21de (0x12c0 bytes) 000344[1] 1789049616.917454 mapwrite 0x6f6000 thru 0x703e87 (0xde88 bytes) 000350[0] 1789049617.171161 write_OD 0x090000 thru 0x091fff (0x2000 bytes) 000352[1] 1789049617.251471 mapread 0x4d7000 thru 0x4dbd9d (0x4d9e bytes) 000353[0] 1789049617.286143 mapread 0x610000 thru 0x615a47 (0x5a48 bytes) 000356[1] 1789049617.322475 mapread 0x04f000 thru 0x05a532 (0xb533 bytes) 000357[0] 1789049617.370701 mapwrite 0x6de000 thru 0x6e1f9a (0x3f9b bytes) 000358[0] 1789049617.457928 read_OD 0x387000 thru 0x393fff (0xd000 bytes) 000359[0] 1789049617.477633 mapwrite 0x8d1000 thru 0x8de247 (0xd248 bytes) 000362[0] 1789049617.633058 punch from 0x536103 to 0x53b44a (0x5348 bytes) 000363[0] 1789049617.646475 mapread 0x32b000 thru 0x332c96 (0x7c97 bytes) 000364[0] 1789049617.667874 mapread 0x7ba000 thru 0x7c1c83 (0x7c84 bytes) 000366[0] 1789049617.709334 punch from 0x627f74 to 0x62d152 (0x51df bytes) 000369[1] 1789049617.751373 write 0x5c2000 thru 0x5c3363 (0x1364 bytes) 000370[0] 1789049617.771235 mapwrite 0x509000 thru 0x515eea (0xceeb bytes) 000371[1] 1789049617.945386 mapwrite 0x039000 thru 0x04380f (0xa810 bytes) 000373[0] 1789049618.142954 mapwrite 0x7b0000 thru 0x7bcd8b (0xcd8c bytes) 000374[1] 1789049618.233048 punch from 0x0fab47 to 0x100546 (0x5a00 bytes) 000375[0] 1789049618.262881 read_OD 0x0ae000 thru 0x0b4fff (0x7000 bytes) 000376[0] 1789049618.289870 trunc from 0x8de248 to 0x367dae (0x576499 bytes) 000377[0] 1789049618.320066 read_OD 0x309000 thru 0x311fff (0x9000 bytes) 000378[1] 1789049618.332681 punch from 0x094145 to 0x099b10 (0x59cc bytes) 000380[1] 1789049618.361865 write 0x841000 thru 0x843c30 (0x2c31 bytes) 000382[0] 1789049618.365178 punch from 0x283ee8 to 0x28c917 (0x8a30 bytes) 000384[1] 1789049618.382430 punch from 0x19d00f to 0x1a8afc (0xbaee bytes) 000385[1] 1789049618.395488 trunc from 0x843c31 to 0x46439b (0x3df895 bytes) 000386[0] 1789049618.428676 read_OD 0x353000 thru 0x35cfff (0xa000 bytes) 000387[1] 1789049618.455314 punch from 0x26f24d to 0x27224d (0x3001 bytes) 000389[1] 1789049618.481343 read 0x44a000 thru 0x451b6d (0x7b6e bytes) 000395[1] 1789049618.538812 punch from 0x33cc6c to 0x343942 (0x6cd7 bytes) 000396[1] 1789049618.570584 write 0x881000 thru 0x882590 (0x1591 bytes) 000397[0] 1789049618.589523 mapwrite 0x2d6000 thru 0x2e347c (0xd47d bytes) 000402[0] 1789049618.721064 punch from 0x15f991 to 0x16bc6d (0xc2dd bytes) 000404[0] 1789049618.765940 write_OD 0x449000 thru 0x455fff (0xd000 bytes) 000405[1] 1789049618.853689 mapread 0x393000 thru 0x39ee54 (0xbe55 bytes) 000406[0] 1789049618.900054 trunc from 0x882591 to 0x0bba86 (0x7c6b0a bytes) 000407[1] 1789049618.969868 punch from 0x082296 to 0x08b2f2 (0x905d bytes) 000408[0] 1789049619.025953 read_OD 0x098000 thru 0x0a1fff (0xa000 bytes) 000409[0] 1789049619.050306 read_OD 0x0b4000 thru 0x0bafff (0x7000 bytes) 000414[0] 1789049619.066520 mapread 0x086000 thru 0x08b643 (0x5644 bytes) 000416[0] 1789049619.090930 trunc from 0x0bba87 to 0x4fabb7 (0x43f131 bytes) 000417[0] 1789049619.155369 mapread 0x2cc000 thru 0x2d9de1 (0xdde2 bytes) 000421[1] 1789049619.281115 mapread 0x229000 thru 0x229f89 (0xf8a bytes) 000423[0] 1789049619.315933 read_OD 0x4ee000 thru 0x4f0fff (0x3000 bytes) 000425[1] 1789049619.332308 read 0x21c000 thru 0x22984c (0xd84d bytes) 000426[0] 1789049619.341477 write_OD 0x351000 thru 0x35ffff (0xf000 bytes) 000427[1] 1789049619.424496 mapread 0x015000 thru 0x01dcc7 (0x8cc8 bytes) 000430[0] 1789049619.465484 read_OD 0x422000 thru 0x422fff (0x1000 bytes) 000433[1] 1789049619.479817 write 0x8ef000 thru 0x8fa09c (0xb09d bytes) 000434[1] 1789049619.520075 mapread 0x7d9000 thru 0x7dbdfd (0x2dfe bytes) 000436[1] 1789049619.551432 write 0x054000 thru 0x0562ea (0x22eb bytes) 000437[0] 1789049619.560391 read_OD 0x67c000 thru 0x688fff (0xd000 bytes) 000440[0] 1789049619.597073 read_OD 0x7b4000 thru 0x7b9fff (0x6000 bytes) 000442[1] 1789049619.609477 mapread 0x59f000 thru 0x5add8b (0xed8c bytes) 000444[0] 1789049619.663013 read_OD 0x5d4000 thru 0x5e0fff (0xd000 bytes) 000445[1] 1789049619.677174 write 0x070000 thru 0x07f18e (0xf18f bytes) 000446[0] 1789049619.709107 trunc from 0x8fa09d to 0x282374 (0x677d28 bytes) 000448[0] 1789049619.774382 punch from 0x121b79 to 0x126493 (0x491b bytes) 000449[0] 1789049619.804788 mapread 0x06d000 thru 0x079e9a (0xce9b bytes) 000450[1] 1789049619.867939 mapread 0x220000 thru 0x221c7b (0x1c7c bytes) 000451[1] 1789049619.915900 punch from 0x22a2c9 to 0x234dfb (0xab33 bytes) 000452[0] 1789049619.967968 punch from 0x105559 to 0x10c72a (0x71d2 bytes) 000454[0] 1789049620.026239 write_OD 0x279000 thru 0x27cfff (0x4000 bytes) 000457[0] 1789049620.172243 read_OD 0x127000 thru 0x129fff (0x3000 bytes) 000458[1] 1789049620.181457 trunc from 0x282375 to 0x4cb5c4 (0x249250 bytes) 000459[0] 1789049620.271852 read_OD 0x348000 thru 0x351fff (0xa000 bytes) 000460[0] 1789049620.332556 read_OD 0x45d000 thru 0x462fff (0x6000 bytes) 000461[1] 1789049620.373392 write 0x5d8000 thru 0x5e1e7f (0x9e80 bytes) 000463[0] 1789049620.431499 punch from 0x036274 to 0x043f73 (0xdd00 bytes) 000464[1] 1789049620.486925 read 0x55a000 thru 0x568367 (0xe368 bytes) 000465[1] 1789049620.535012 write 0x326000 thru 0x3281aa (0x21ab bytes) 000466[0] 1789049620.603538 mapwrite 0x757000 thru 0x75a5a9 (0x35aa bytes) 000468[0] 1789049620.782366 mapread 0x343000 thru 0x34cb3a (0x9b3b bytes) 000469[1] 1789049620.812324 trunc from 0x75a5aa to 0x31523b (0x44536e bytes) 000471[0] 1789049620.847143 read_OD 0x15b000 thru 0x163fff (0x9000 bytes) 000472[1] 1789049620.878951 mapread 0x30c000 thru 0x30df26 (0x1f27 bytes) 000475[0] 1789049620.894694 punch from 0x17a765 to 0x17ae09 (0x6a5 bytes) 000480[0] 1789049620.951771 read_OD 0x1a9000 thru 0x1acfff (0x4000 bytes) 000483[1] 1789049620.969478 write 0x0f5000 thru 0x0fa140 (0x5141 bytes) 000484[0] 1789049620.988321 mapread 0x035000 thru 0x03ac7c (0x5c7d bytes) 000486[0] 1789049621.078797 mapread 0x308000 thru 0x31440f (0xc410 bytes) 000487[0] 1789049621.079376 punch from 0x1fe015 to 0x20bdd2 (0xddbe bytes) 000488[1] 1789049621.097036 mapread 0x27c000 thru 0x282754 (0x6755 bytes) 000491[0] 1789049621.126101 mapread 0x028000 thru 0x0335a4 (0xb5a5 bytes) 000492[1] 1789049621.170400 punch from 0x1980b7 to 0x19a66c (0x25b6 bytes) 000494[1] 1789049621.196141 trunc from 0x31523c to 0x96bf9e (0x656d63 bytes) 000495[1] 1789049621.228326 read 0x7d8000 thru 0x7db7bf (0x37c0 bytes) 000496[0] 1789049621.237424 punch from 0x430f91 to 0x43860c (0x767c bytes) 000497[0] 1789049621.261507 write_OD 0x2fe000 thru 0x305fff (0x8000 bytes) 000498[1] 1789049621.307169 trunc from 0x96bf9f to 0x81f82a (0x14c774 bytes) 000500[1] 1789049621.334743 punch from 0x2ad603 to 0x2bb0c3 (0xdac1 bytes) All operations completed A-OK! PASS 16a (45s) == sanityn test 16b: 500 iterations of dual-mount fsx at small size ========================================================== 10:13:46 (1789049626) keep default fallocate mode: 2 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.429029 s, 19.6 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.309029 s, 27.1 MB/s Chance of close/open is 1 in 50 Seed set to 9629 fd 0: /mnt/lustre/f16b.sanityn fd 1: /mnt/lustre2/f16b.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported truncating to largest ever: 0x53ad55 000004[1] 1789049630.124909 trunc from 00000000 to 0x53ad54 (0x53ad55 bytes) truncating to largest ever: 0x7ba1b6 000005[0] 1789049630.202385 trunc from 0x53ad55 to 0x7ba1b5 (0x27f461 bytes) 000008[0] 1789049630.298385 trunc from 0x7ba1b6 to 0x5c09d6 (0x1f97df bytes) 000009[1] 1789049630.353325 mapread 0x088936 thru 0x08a806 (0x1ed1 bytes) 000010[0] 1789049630.401767 mapread 0x45a286 thru 0x45aa16 (0x791 bytes) 000013[0] 1789049630.437794 write 0x606fa8 thru 0x60896d (0x19c6 bytes) 000014[1] 1789049630.439006 read 0x360b3c thru 0x361e7d (0x1342 bytes) 000016[1] 1789049630.445323 punch from 0x472990 to 0x472ed4 (0x545 bytes) 000017[1] 1789049630.465121 write 0x3fc1ab thru 0x3fdb1e (0x1974 bytes) 000019[1] 1789049630.487041 read 0x1eb50c thru 0x1ec0f7 (0xbec bytes) 000023[1] 1789049630.523668 write 0x32ad71 thru 0x32be25 (0x10b5 bytes) 000024[0] 1789049630.569421 read 0x5ad6fc thru 0x5aef15 (0x181a bytes) 000026[1] 1789049630.593314 read 0x46fad9 thru 0x471769 (0x1c91 bytes) 000027[1] 1789049630.618514 trunc from 0x60896e to 0x302726 (0x306247 bytes) 000028[0] 1789049630.710261 write 0x49669a thru 0x497d30 (0x1697 bytes) 000029[0] 1789049630.775593 write 0x69ad25 thru 0x69c369 (0x1645 bytes) 000033[0] 1789049630.851638 trunc from 0x69c36a to 0x392909 (0x309a60 bytes) 000036[0] 1789049630.910914 trunc from 0x39290a to 0x0b7397 (0x2db572 bytes) 000037[0] 1789049630.934697 trunc from 0x0b7398 to 0x5ba7ed (0x503456 bytes) 000043[1] 1789049631.078599 read 0x0fb844 thru 0x0fd270 (0x1a2d bytes) 000044[1] 1789049631.174325 punch from 0x3d35a2 to 0x3d4cc0 (0x171f bytes) 000045[0] 1789049631.205593 mapwrite 0x6b9710 thru 0x6bac34 (0x1525 bytes) 000046[1] 1789049631.331897 mapread 0x2610d7 thru 0x261efa (0xe24 bytes) 000047[1] 1789049631.369759 read 0x390169 thru 0x390935 (0x7cd bytes) 000048[1] 1789049631.425856 read 0x41654d thru 0x416d80 (0x834 bytes) 000049[1] 1789049631.448581 punch from 0x0e303d to 0x0e38de (0x8a2 bytes) 000050[0] 1789049631.468504 read 0x65bdd7 thru 0x65c56a (0x794 bytes) 000051[0] 1789049631.484921 mapwrite 0x6546bf thru 0x656270 (0x1bb2 bytes) 000053[0] 1789049631.617383 punch from 0x20a10b to 0x20a99e (0x894 bytes) 000055[1] 1789049631.683239 write 0x30667a thru 0x307645 (0xfcc bytes) 000056[1] 1789049631.734032 punch from 0x13cab4 to 0x13dd32 (0x127f bytes) truncating to largest ever: 0x84d4cf 000057[1] 1789049631.756711 trunc from 0x6bac35 to 0x84d4ce (0x19289a bytes) 000059[1] 1789049631.845258 trunc from 0x84d4cf to 0x7ab47c (0xa2052 bytes) 000061[0] 1789049631.868544 mapwrite 0x9588da thru 0x959b43 (0x126a bytes) 000062[1] 1789049631.955854 read 0x56e0d8 thru 0x56e267 (0x190 bytes) 000065[1] 1789049632.090865 write 0x18fc4f thru 0x1902e3 (0x695 bytes) 000066[0] 1789049632.147184 punch from 0x21c760 to 0x21e607 (0x1ea8 bytes) 000068[0] 1789049632.185131 mapwrite 0x5d4221 thru 0x5d4f6e (0xd4e bytes) 000069[0] 1789049632.400199 write 0x99c289 thru 0x99dbc9 (0x1941 bytes) 000070[0] 1789049632.437333 read 0x2e8308 thru 0x2e8aed (0x7e6 bytes) 000071[1] 1789049632.476068 write 0x4d418f thru 0x4d47f1 (0x663 bytes) 000073[1] 1789049632.534965 write 0x69889a thru 0x699ea8 (0x160f bytes) 000076[1] 1789049632.624082 punch from 0x1cea65 to 0x1cffc8 (0x1564 bytes) 000081[0] 1789049632.739925 mapread 0x3c153c thru 0x3c2f6b (0x1a30 bytes) 000082[0] 1789049632.819139 read 0x7c34f7 thru 0x7c5382 (0x1e8c bytes) 000083[0] 1789049632.850304 mapread 0x15d80e thru 0x15e245 (0xa38 bytes) 000084[1] 1789049632.861252 write 0x14eb78 thru 0x14f040 (0x4c9 bytes) 000087[1] 1789049632.933193 punch from 0x17c24b to 0x17db53 (0x1909 bytes) 000088[1] 1789049633.000181 punch from 0x76fa7b to 0x77125b (0x17e1 bytes) 000090[0] 1789049633.040422 write 0x61547e thru 0x615c51 (0x7d4 bytes) 000091[1] 1789049633.083234 read 0x012a41 thru 0x014354 (0x1914 bytes) 000093[0] 1789049633.127999 mapwrite 0x656160 thru 0x65717c (0x101d bytes) 000094[1] 1789049633.205423 read 0x7918ed thru 0x792acf (0x11e3 bytes) 000098[1] 1789049633.244419 mapread 0x7c70da thru 0x7c8272 (0x1199 bytes) 000099[0] 1789049633.276270 mapread 0x2f8c55 thru 0x2f94bc (0x868 bytes) 000100[0] 1789049633.327493 write 0x147fb9 thru 0x1483b6 (0x3fe bytes) 000101[1] 1789049633.364148 mapwrite 0x81cc96 thru 0x81d567 (0x8d2 bytes) 000104[1] 1789049633.510015 mapread 0x24ed5b thru 0x2500a2 (0x1348 bytes) 000105[0] 1789049633.544414 trunc from 0x99dbca to 0x4f5bd8 (0x4a7ff1 bytes) 000106[1] 1789049633.600034 punch from 0x40c8e9 to 0x40d1ae (0x8c6 bytes) 000107[1] 1789049633.634654 write 0x1fa5a9 thru 0x1faebf (0x917 bytes) 000108[1] 1789049633.677674 punch from 0x494ddb to 0x49691b (0x1b41 bytes) 000109[1] 1789049633.682003 mapwrite 0x4e00b4 thru 0x4e08e4 (0x831 bytes) 000110[0] 1789049633.743231 punch from 0x341f4e to 0x343a5c (0x1b0f bytes) 000112[1] 1789049633.789351 mapwrite 0x8517ed thru 0x85261c (0xe30 bytes) 000114[0] 1789049633.913216 mapwrite 0x46c93d thru 0x46d2cb (0x98f bytes) 000116[0] 1789049634.017029 trunc from 0x85261d to 0x689201 (0x1c941b bytes) 000119[1] 1789049634.085649 mapread 0x4df95f thru 0x4e05e2 (0xc84 bytes) 000120[1] 1789049634.121806 mapread 0x2423a3 thru 0x242568 (0x1c6 bytes) 000121[1] 1789049634.157806 read 0x13d536 thru 0x13ecb6 (0x1781 bytes) 000123[1] 1789049634.173614 read 0x3fb38b thru 0x3fba6d (0x6e3 bytes) 000124[1] 1789049634.185985 read 0x1424e2 thru 0x1425e6 (0x105 bytes) 000126[1] 1789049634.194802 mapread 0x5960e9 thru 0x596a9c (0x9b4 bytes) 000127[1] 1789049634.200933 write 0x556b96 thru 0x556d01 (0x16c bytes) 000130[0] 1789049634.250768 read 0x166b68 thru 0x167941 (0xdda bytes) 000131[1] 1789049634.287542 write 0x6d2520 thru 0x6d29c3 (0x4a4 bytes) 000132[1] 1789049634.289133 read 0x5b90dd thru 0x5b9d4d (0xc71 bytes) 000133[0] 1789049634.323171 mapread 0x2b8505 thru 0x2ba4ec (0x1fe8 bytes) 000135[0] 1789049634.394648 punch from 0x128afb to 0x128e38 (0x33e bytes) 000137[0] 1789049634.447052 trunc from 0x6d29c4 to 0x503365 (0x1cf65e bytes) 000138[1] 1789049634.493151 mapread 0x109bfe thru 0x10a116 (0x519 bytes) 000140[0] 1789049634.543099 mapread 0x445222 thru 0x446d8f (0x1b6e bytes) 000141[1] 1789049634.565278 mapwrite 0x8bb381 thru 0x8bcf04 (0x1b84 bytes) 000143[1] 1789049634.703058 trunc from 0x8bcf05 to 0x58d1ae (0x32fd56 bytes) 000144[0] 1789049634.756386 write 0x73812d thru 0x738d6c (0xc40 bytes) 000145[1] 1789049634.812685 read 0x221575 thru 0x222f92 (0x1a1e bytes) 000146[1] 1789049634.835956 mapread 0x5fe090 thru 0x5ff8f0 (0x1861 bytes) 000150[0] 1789049634.888949 punch from 0x61dc14 to 0x61e3b4 (0x7a1 bytes) 000151[1] 1789049634.932153 write 0x0d436c thru 0x0d5b20 (0x17b5 bytes) 000152[0] 1789049634.967199 trunc from 0x738d6d to 0x6b1965 (0x87407 bytes) 000153[0] 1789049635.047574 trunc from 0x6b1966 to 0x75e76f (0xace0a bytes) 000154[1] 1789049635.094370 punch from 0x137451 to 0x138abe (0x166e bytes) 000156[0] 1789049635.156154 mapread 0x18e6a9 thru 0x18facd (0x1425 bytes) 000158[0] 1789049635.228699 mapwrite 0x06c4c9 thru 0x06d721 (0x1259 bytes) 000159[1] 1789049635.340197 read 0x09c93e thru 0x09da9d (0x1160 bytes) 000161[1] 1789049635.415906 mapread 0x598c81 thru 0x599dff (0x117f bytes) truncating to largest ever: 0x9b4cf5 000163[0] 1789049635.481772 trunc from 0x75e770 to 0x9b4cf4 (0x256585 bytes) 000164[0] 1789049635.548292 read 0x863a3e thru 0x864b79 (0x113c bytes) 000166[0] 1789049635.603541 write 0x02fa91 thru 0x03031d (0x88d bytes) 000168[0] 1789049635.620663 mapread 0x97965e thru 0x97a15e (0xb01 bytes) 000170[1] 1789049635.738792 mapread 0x340c56 thru 0x3427dc (0x1b87 bytes) 000171[0] 1789049635.862265 mapwrite 0x4961da thru 0x496e48 (0xc6f bytes) 000173[0] 1789049635.993434 read 0x3c587f thru 0x3c5f14 (0x696 bytes) 000174[1] 1789049636.023316 mapwrite 0x6481db thru 0x649aff (0x1925 bytes) 000175[1] 1789049636.212120 punch from 0x812d2d to 0x81483f (0x1b13 bytes) 000178[0] 1789049636.324224 mapread 0x2b78dd thru 0x2b882f (0xf53 bytes) 000180[0] 1789049636.356154 read 0x82b454 thru 0x82c034 (0xbe1 bytes) 000181[0] 1789049636.372599 write 0x687567 thru 0x688562 (0xffc bytes) 000182[1] 1789049636.401366 trunc from 0x9b4cf5 to 0x8e8150 (0xccba4 bytes) 000183[1] 1789049636.459153 write 0x1b3f05 thru 0x1b4e80 (0xf7c bytes) 000184[0] 1789049636.500554 mapwrite 0x1dfc20 thru 0x1e00f8 (0x4d9 bytes) 000188[1] 1789049636.658973 punch from 0x05276a to 0x05343c (0xcd3 bytes) 000189[1] 1789049636.695534 trunc from 0x8e8151 to 0x40137e (0x4e6dd2 bytes) 000190[1] 1789049636.744350 mapread 0x297330 thru 0x297dbc (0xa8d bytes) 000192[1] 1789049636.788161 read 0x04f729 thru 0x05095d (0x1235 bytes) 000193[0] 1789049636.829554 mapread 0x0099ee thru 0x00a37e (0x991 bytes) 000195[0] 1789049636.899614 punch from 0x01b28d to 0x01ca40 (0x17b4 bytes) 000196[0] 1789049636.931243 trunc from 0x40137f to 0x2fe466 (0x102f18 bytes) 000197[0] 1789049636.992664 write 0x90b179 thru 0x90caac (0x1934 bytes) 000198[1] 1789049637.013109 punch from 0x31a969 to 0x31b416 (0xaae bytes) 000201[1] 1789049637.125642 read 0x1e8eb8 thru 0x1e98cd (0xa16 bytes) 000202[0] 1789049637.156522 mapread 0x0c1314 thru 0x0c2885 (0x1572 bytes) 000203[0] 1789049637.261179 write 0x6da8e9 thru 0x6dc8a0 (0x1fb8 bytes) 000204[0] 1789049637.282124 trunc from 0x90caad to 0x0d93e0 (0x8336cc bytes) 000205[1] 1789049637.359178 mapwrite 0x092c38 thru 0x093636 (0x9ff bytes) 000206[1] 1789049637.456771 write 0x617224 thru 0x617e65 (0xc42 bytes) 000207[1] 1789049637.469093 read 0x57dff6 thru 0x57eec3 (0xece bytes) 000209[0] 1789049637.496147 mapread 0x2d2c9c thru 0x2d47ec (0x1b51 bytes) 000211[1] 1789049637.612615 mapwrite 0x178413 thru 0x179793 (0x1381 bytes) 000212[1] 1789049637.753187 mapread 0x3b57f2 thru 0x3b750c (0x1d1b bytes) 000213[0] 1789049637.797288 trunc from 0x617e66 to 0x4c68c1 (0x1515a4 bytes) 000214[0] 1789049637.850250 punch from 0x34e031 to 0x34ff84 (0x1f54 bytes) 000217[1] 1789049637.873205 trunc from 0x4c68c2 to 0x1766f6 (0x3501cb bytes) 000218[1] 1789049637.937142 mapwrite 0x0de178 thru 0x0df57a (0x1403 bytes) 000219[0] 1789049638.009456 punch from 0x15ab63 to 0x15c1b1 (0x164f bytes) 000221[0] 1789049638.056936 read 0x02087b thru 0x022751 (0x1ed7 bytes) 000222[1] 1789049638.104331 read 0x0dc1a0 thru 0x0dd126 (0xf87 bytes) 000224[1] 1789049638.159109 write 0x3cfc3d thru 0x3d0dfe (0x11c2 bytes) 000226[1] 1789049638.195307 mapread 0x11f953 thru 0x120d83 (0x1431 bytes) 000227[0] 1789049638.258215 trunc from 0x3d0dff to 0x41862c (0x4782e bytes) 000228[1] 1789049638.309391 mapwrite 0x410be1 thru 0x411ba0 (0xfc0 bytes) 000229[1] 1789049638.397714 mapread 0x3e7541 thru 0x3e83c0 (0xe80 bytes) 000230[0] 1789049638.432409 trunc from 0x41862d to 0x77653b (0x35df0f bytes) 000231[1] 1789049638.499957 mapread 0x3572ed thru 0x357897 (0x5ab bytes) 000232[0] 1789049638.562333 mapread 0x44a319 thru 0x44bdf4 (0x1adc bytes) 000233[1] 1789049638.679630 trunc from 0x77653c to 0x4c926c (0x2ad2cf bytes) 000234[1] 1789049638.795411 trunc from 0x4c926d to 0x3638b9 (0x1659b3 bytes) 000235[0] 1789049638.810580 trunc from 0x3638ba to 0x862cab (0x4ff3f2 bytes) 000236[1] 1789049638.863970 mapwrite 0x22f63f thru 0x231330 (0x1cf2 bytes) 000238[1] 1789049639.031257 trunc from 0x862cac to 0x572601 (0x2f06aa bytes) 000239[0] 1789049639.071593 punch from 0x153c79 to 0x155917 (0x1c9f bytes) 000240[0] 1789049639.121353 read 0x235c6f thru 0x235fdb (0x36d bytes) 000241[1] 1789049639.146865 punch from 0x24bee0 to 0x24c3e7 (0x508 bytes) 000244[1] 1789049639.169306 mapwrite 0x3e2af4 thru 0x3e3826 (0xd33 bytes) 000245[1] 1789049639.271821 mapread 0x4fee2b thru 0x500c7f (0x1e55 bytes) 000248[1] 1789049639.342434 mapread 0x34a922 thru 0x34bbfd (0x12dc bytes) 000249[0] 1789049639.353099 trunc from 0x572602 to 0x5279d7 (0x4ac2a bytes) 000251[0] 1789049639.449638 trunc from 0x5279d8 to 0x70846d (0x1e0a96 bytes) 000252[0] 1789049639.508089 mapwrite 0x7a6849 thru 0x7a7dee (0x15a6 bytes) 000254[0] 1789049639.645933 mapwrite 0x1a20cb thru 0x1a2731 (0x667 bytes) 000255[1] 1789049639.788958 punch from 0x549b13 to 0x549e59 (0x347 bytes) 000256[1] 1789049639.809894 write 0x6921f3 thru 0x693c0c (0x1a1a bytes) 000257[0] 1789049639.850307 read 0x230bba thru 0x230c1f (0x66 bytes) 000258[0] 1789049639.866375 mapwrite 0x2be6b5 thru 0x2bff61 (0x18ad bytes) 000259[0] 1789049639.991557 read 0x07b3fb thru 0x07b76f (0x375 bytes) 000260[1] 1789049640.026632 trunc from 0x7a7def to 0x545500 (0x2628ee bytes) 000264[0] 1789049640.140297 mapread 0x1a06bf thru 0x1a1da0 (0x16e2 bytes) 000265[1] 1789049640.168636 read 0x4086c8 thru 0x40a020 (0x1959 bytes) 000266[1] 1789049640.173213 mapwrite 0x39c6c0 thru 0x39d79d (0x10de bytes) 000267[1] 1789049640.243321 read 0x4881c9 thru 0x488ffb (0xe33 bytes) 000269[0] 1789049640.278340 trunc from 0x545501 to 0x1dcf06 (0x3685fa bytes) 000272[0] 1789049640.378565 trunc from 0x1dcf07 to 0x31c0c2 (0x13f1bc bytes) 000273[1] 1789049640.401709 read 0x142eef thru 0x144278 (0x138a bytes) 000274[0] 1789049640.487067 mapread 0x0a5792 thru 0x0a603b (0x8aa bytes) 000275[1] 1789049640.497076 read 0x2f3f3d thru 0x2f4d5c (0xe20 bytes) 000278[0] 1789049640.534023 read 0x12188f thru 0x121fa6 (0x718 bytes) 000279[0] 1789049640.651920 trunc from 0x31c0c3 to 0x493509 (0x177447 bytes) 000283[1] 1789049640.768025 read 0x0a4264 thru 0x0a4bb9 (0x956 bytes) 000284[0] 1789049640.794323 punch from 0x3c12e0 to 0x3c2b66 (0x1887 bytes) 000286[0] 1789049640.822268 mapwrite 0x99838c thru 0x999e35 (0x1aaa bytes) 000289[1] 1789049641.169273 mapwrite 0x1bd0ac thru 0x1bda92 (0x9e7 bytes) 000290[1] 1789049641.286361 write 0x3ea30c thru 0x3eae6a (0xb5f bytes) 000291[1] 1789049641.306523 punch from 0x224f40 to 0x2264ae (0x156f bytes) 000292[0] 1789049641.326503 punch from 0x958038 to 0x959387 (0x1350 bytes) 000294[1] 1789049641.361562 mapread 0x143080 thru 0x143834 (0x7b5 bytes) 000295[1] 1789049641.401172 trunc from 0x999e36 to 0x2a0a29 (0x6f940c bytes) 000296[1] 1789049641.471701 punch from 0x1489ee to 0x149205 (0x818 bytes) 000301[1] 1789049641.558281 mapwrite 0x3e2975 thru 0x3e47fb (0x1e87 bytes) 000302[1] 1789049641.699614 write 0x89232a thru 0x89256e (0x245 bytes) 000305[0] 1789049641.738814 mapwrite 0x70b180 thru 0x70cdc7 (0x1c48 bytes) 000307[1] 1789049641.958625 read 0x4677c9 thru 0x467997 (0x1cf bytes) 000308[1] 1789049641.989916 trunc from 0x89256f to 0x0fb402 (0x79716c bytes) 000309[1] 1789049642.014898 punch from 0x00af8f to 0x00cdf8 (0x1e6a bytes) 000310[0] 1789049642.019101 punch from 0x0ec64e to 0x0ec91e (0x2d1 bytes) 000313[1] 1789049642.073403 mapread 0x00bd91 thru 0x00c70b (0x97b bytes) 000314[0] 1789049642.100543 trunc from 0x0fb403 to 0x9604d2 (0x8650d0 bytes) 000315[1] 1789049642.156219 read 0x0ab5b6 thru 0x0acf28 (0x1973 bytes) 000317[0] 1789049642.192975 write 0x41fe43 thru 0x421a44 (0x1c02 bytes) 000318[0] 1789049642.201908 write 0x3722fa thru 0x373e86 (0x1b8d bytes) 000319[1] 1789049642.224001 mapread 0x78ec6d thru 0x7907d6 (0x1b6a bytes) 000320[0] 1789049642.291010 mapread 0x186b28 thru 0x18743a (0x913 bytes) 000325[1] 1789049642.319514 write 0x1bafca thru 0x1bb6d4 (0x70b bytes) 000326[1] 1789049642.371120 read 0x665c82 thru 0x667201 (0x1580 bytes) 000327[0] 1789049642.378659 write 0x868d84 thru 0x869032 (0x2af bytes) 000328[1] 1789049642.407649 punch from 0x534535 to 0x534cc7 (0x793 bytes) 000329[0] 1789049642.468506 write 0x51e45e thru 0x51f8df (0x1482 bytes) 000331[1] 1789049642.520256 mapwrite 0x46e807 thru 0x46f98a (0x1184 bytes) 000332[0] 1789049642.636851 write 0x31b73b thru 0x31d17b (0x1a41 bytes) 000336[1] 1789049642.743433 trunc from 0x9604d3 to 0x2883d7 (0x6d80fb bytes) 000337[0] 1789049642.844913 mapwrite 0x716e15 thru 0x718d8a (0x1f76 bytes) 000338[0] 1789049642.990177 write 0x61b3ce thru 0x61c97e (0x15b1 bytes) 000339[1] 1789049643.037276 punch from 0x110832 to 0x111b91 (0x1360 bytes) 000341[1] 1789049643.104336 read 0x51da45 thru 0x51ef45 (0x1501 bytes) 000342[0] 1789049643.161819 punch from 0x60bcc3 to 0x60dcba (0x1ff8 bytes) 000344[1] 1789049643.240115 punch from 0x1e87f8 to 0x1e91a6 (0x9af bytes) 000345[1] 1789049643.283257 write 0x3dd278 thru 0x3de08a (0xe13 bytes) 000347[0] 1789049643.363459 mapread 0x41a8ec thru 0x41c611 (0x1d26 bytes) 000348[1] 1789049643.407330 read 0x3698b9 thru 0x36b41b (0x1b63 bytes) 000349[1] 1789049643.432277 mapwrite 0x988ef7 thru 0x98a7d5 (0x18df bytes) 000350[1] 1789049643.547190 write 0x94ef04 thru 0x94eff3 (0xf0 bytes) 000351[1] 1789049643.559239 trunc from 0x98a7d6 to 0x6c17bf (0x2c9016 bytes) 000352[0] 1789049643.622218 mapread 0x0a414c thru 0x0a4a9b (0x950 bytes) 000353[1] 1789049643.668247 punch from 0x07d577 to 0x07e902 (0x138c bytes) 000354[1] 1789049643.725217 write 0x2c716c thru 0x2c829f (0x1134 bytes) 000355[1] 1789049643.747090 trunc from 0x6c17c0 to 0x015b56 (0x6abc69 bytes) 000356[0] 1789049643.805254 mapread 0x015a92 thru 0x015b56 (0xc5 bytes) 000357[1] 1789049643.876972 mapread 0x007bbf thru 0x008ad8 (0xf1a bytes) 000358[1] 1789049643.898213 trunc from 0x015b57 to 0x135e49 (0x1202f3 bytes) 000361[0] 1789049643.966094 punch from 0x0d7887 to 0x0d837e (0xaf8 bytes) 000362[0] 1789049644.026167 read 0x0744fa thru 0x076336 (0x1e3d bytes) 000363[1] 1789049644.082209 write 0x398923 thru 0x398c8c (0x36a bytes) 000364[1] 1789049644.117239 write 0x7b7e3c thru 0x7b918d (0x1352 bytes) 000366[0] 1789049644.188117 mapwrite 0x9cfc3c thru 0x9d0448 (0x80d bytes) 000367[1] 1789049644.732898 mapwrite 0x65a42a thru 0x65ac05 (0x7dc bytes) 000369[0] 1789049644.897625 write 0x829817 thru 0x82a015 (0x7ff bytes) 000371[1] 1789049644.955063 mapread 0x8bf3ad thru 0x8c0af5 (0x1749 bytes) 000372[1] 1789049645.039050 punch from 0x81407a to 0x815796 (0x171d bytes) 000375[1] 1789049645.108314 mapread 0x63852d thru 0x63a2de (0x1db2 bytes) 000376[1] 1789049645.152262 write 0x12878b thru 0x129005 (0x87b bytes) 000378[0] 1789049645.176695 mapwrite 0x2b3396 thru 0x2b34a9 (0x114 bytes) 000379[1] 1789049645.257290 mapread 0x7417fe thru 0x7433f1 (0x1bf4 bytes) 000386[1] 1789049645.492992 trunc from 0x9d0449 to 0x1d5712 (0x7fad36 bytes) 000389[1] 1789049645.596293 read 0x17efc6 thru 0x1803de (0x1419 bytes) 000391[1] 1789049645.682971 trunc from 0x1d5713 to 0x3a0145 (0x1caa33 bytes) 000392[0] 1789049645.752253 punch from 0x118550 to 0x11a018 (0x1ac9 bytes) 000394[0] 1789049645.813345 write 0x272bd2 thru 0x274358 (0x1787 bytes) 000395[0] 1789049645.844912 mapwrite 0x5e0a48 thru 0x5e244e (0x1a07 bytes) 000396[0] 1789049646.013603 punch from 0x561c78 to 0x561dcc (0x155 bytes) 000397[1] 1789049646.031119 mapread 0x251c20 thru 0x25385f (0x1c40 bytes) 000398[0] 1789049646.147371 write 0x368a46 thru 0x36a2db (0x1896 bytes) 000399[0] 1789049646.193205 trunc from 0x5e244f to 0x98aa99 (0x3a864b bytes) 000400[0] 1789049646.240713 punch from 0x104d33 to 0x105f51 (0x121f bytes) 000402[0] 1789049646.281370 write 0x9133bb thru 0x9140c6 (0xd0c bytes) 000404[0] 1789049646.370642 mapwrite 0x65e40b thru 0x65f37a (0xf70 bytes) 000405[0] 1789049646.470608 write 0x638c95 thru 0x63a7c9 (0x1b35 bytes) 000406[0] 1789049646.493651 write 0x2ceba6 thru 0x2d0082 (0x14dd bytes) 000407[0] 1789049646.566024 trunc from 0x98aa9a to 0x1132f5 (0x8777a4 bytes) 000408[0] 1789049646.613966 mapwrite 0x6f5dd7 thru 0x6f6f63 (0x118d bytes) 000410[0] 1789049646.820007 mapread 0x2159e2 thru 0x215bbf (0x1de bytes) 000411[1] 1789049646.875057 write 0x1c79a5 thru 0x1c8bd2 (0x122e bytes) 000412[1] 1789049646.944500 mapwrite 0x673e0f thru 0x6742cc (0x4be bytes) 000415[1] 1789049647.153031 punch from 0x333078 to 0x333c46 (0xbcf bytes) 000416[1] 1789049647.171556 mapread 0x4d4e19 thru 0x4d5e3d (0x1025 bytes) 000417[1] 1789049647.174935 punch from 0x5eb427 to 0x5ed1cf (0x1da9 bytes) 000418[1] 1789049647.227520 write 0x5106ea thru 0x511d8d (0x16a4 bytes) 000419[0] 1789049647.229496 read 0x5ecd13 thru 0x5ed3d1 (0x6bf bytes) 000420[1] 1789049647.290457 read 0x234cf6 thru 0x235a64 (0xd6f bytes) 000422[0] 1789049647.325455 read 0x6e0315 thru 0x6e0db5 (0xaa1 bytes) 000423[0] 1789049647.334483 write 0x0c64ab thru 0x0c739f (0xef5 bytes) 000425[0] 1789049647.365009 write 0x06d380 thru 0x06d90f (0x590 bytes) 000428[0] 1789049647.391054 mapread 0x66ef40 thru 0x67035d (0x141e bytes) 000429[0] 1789049647.451946 trunc from 0x6f6f64 to 0x297c21 (0x45f342 bytes) 000430[0] 1789049647.542333 trunc from 0x297c22 to 0x0dc212 (0x1bba0f bytes) 000431[1] 1789049647.576688 mapwrite 0x4c9617 thru 0x4cae6d (0x1857 bytes) 000432[0] 1789049647.806573 mapread 0x015829 thru 0x016f08 (0x16e0 bytes) 000433[1] 1789049647.885784 punch from 0x3bc908 to 0x3bdb24 (0x121d bytes) 000434[0] 1789049647.971658 punch from 0x0d754e to 0x0d9007 (0x1aba bytes) 000435[0] 1789049648.021424 mapread 0x26776c thru 0x2695af (0x1e44 bytes) 000436[1] 1789049648.120262 trunc from 0x4cae6e to 0x020636 (0x4aa837 bytes) 000437[0] 1789049648.178508 mapwrite 0x068e47 thru 0x069a32 (0xbec bytes) 000438[0] 1789049648.289575 read 0x00d83c thru 0x00edce (0x1593 bytes) 000439[1] 1789049648.340123 trunc from 0x069a33 to 0x3d85d4 (0x36eba2 bytes) 000440[0] 1789049648.391184 mapwrite 0x10f1a4 thru 0x110858 (0x16b5 bytes) 000441[0] 1789049648.651085 trunc from 0x3d85d5 to 0x7f3a4a (0x41b476 bytes) 000445[0] 1789049648.823203 punch from 0x375182 to 0x376715 (0x1594 bytes) 000446[1] 1789049648.831111 mapwrite 0x080450 thru 0x080f73 (0xb24 bytes) 000448[0] 1789049649.048315 punch from 0x1ee6e0 to 0x1ef03b (0x95c bytes) 000450[1] 1789049649.097121 punch from 0x724ef7 to 0x726ac5 (0x1bcf bytes) 000451[1] 1789049649.145946 punch from 0x637230 to 0x637e99 (0xc6a bytes) 000452[0] 1789049649.169165 punch from 0x447c7d to 0x448146 (0x4ca bytes) 000455[0] 1789049649.283808 trunc from 0x7f3a4b to 0x884250 (0x90806 bytes) 000456[1] 1789049649.320655 punch from 0x097c31 to 0x099290 (0x1660 bytes) 000458[1] 1789049649.408530 mapwrite 0x6c049e thru 0x6c234d (0x1eb0 bytes) 000459[0] 1789049649.540712 read 0x139fb5 thru 0x13bd47 (0x1d93 bytes) 000460[0] 1789049649.568049 punch from 0x86cc5d to 0x86db94 (0xf38 bytes) 000461[1] 1789049649.593299 trunc from 0x884251 to 0x5f97de (0x28aa72 bytes) 000462[0] 1789049649.657635 punch from 0x1dc6d3 to 0x1ddcb5 (0x15e3 bytes) 000463[0] 1789049649.759023 trunc from 0x5f97df to 0x2b1459 (0x348385 bytes) 000464[0] 1789049649.843544 read 0x1831df thru 0x185198 (0x1fba bytes) 000466[0] 1789049649.920002 mapwrite 0x019325 thru 0x01a6c3 (0x139f bytes) 000468[0] 1789049650.021986 write 0x461dfc thru 0x4630d5 (0x12da bytes) 000469[1] 1789049650.048221 write 0x71784e thru 0x718772 (0xf25 bytes) 000470[1] 1789049650.102082 punch from 0x003a98 to 0x003eec (0x455 bytes) 000471[0] 1789049650.146026 mapwrite 0x015dba thru 0x016ad3 (0xd1a bytes) 000472[1] 1789049650.314265 read 0x63f814 thru 0x640e50 (0x163d bytes) 000474[0] 1789049650.369806 mapread 0x243683 thru 0x244800 (0x117e bytes) 000477[1] 1789049650.472944 mapwrite 0x691d60 thru 0x693991 (0x1c32 bytes) 000478[1] 1789049650.586824 punch from 0x2e26b1 to 0x2e2746 (0x96 bytes) 000480[1] 1789049650.612181 write 0x86f9cf thru 0x870e3f (0x1471 bytes) 000487[0] 1789049650.711267 mapwrite 0x74e4e2 thru 0x74f649 (0x1168 bytes) 000489[0] 1789049650.838172 read 0x7fb234 thru 0x7fcbbd (0x198a bytes) 000490[0] 1789049650.864071 mapread 0x3c0e53 thru 0x3c1f91 (0x113f bytes) 000494[0] 1789049650.921058 read 0x250634 thru 0x25093d (0x30a bytes) 000495[1] 1789049650.945108 mapwrite 0x3ca100 thru 0x3cae2d (0xd2e bytes) 000499[1] 1789049651.125493 punch from 0x0e58fa to 0x0e695f (0x1066 bytes) 000500[1] 1789049651.139428 read 0x70bd47 thru 0x70c848 (0xb02 bytes) All operations completed A-OK! PASS 16b (29s) == sanityn test 16c: verify data consistency on ldiskfs with cache disabled (b=17397) ========================================================== 10:14:15 (1789049655) SKIP: sanityn test_16c dio on ldiskfs only SKIP 16c (3s) == sanityn test 16d: Verify DIO and buffer IO with two clients ========================================================== 10:14:19 (1789049659) 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.339197 s, 24.7 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.289155 s, 29.0 MB/s ldlm.namespaces.MGC192.168.203.141@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff9668d169d000.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff9668d8375800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff9668d169d000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff9668d8375800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff9668d169d000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff9668d8375800.lru_size=clear 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.66161 s, 12.1 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 2.70886 s, 38.7 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 2.61855 s, 40.0 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.61318 s, 18.7 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.43678 s, 14.1 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 2.86669 s, 36.6 MB/s PASS 16d (41s) == sanityn test 16e: Verify size consistency for O_DIRECT write ========================================================== 10:15:00 (1789049700) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.260049 s, 40.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141281 s, 7.4 MB/s PASS 16e (6s) == sanityn test 16f: rw sequential consistency vs drop_caches ========================================================== 10:15:06 (1789049706) TIMEOUT PASS 16f (27s) == sanityn test 16g: mmap rw sequential consistency vs drop_caches ========================================================== 10:15:33 (1789049733) TIMEOUT PASS 16g (26s) == sanityn test 16h: mmap read after truncate file ======= 10:16:00 (1789049760) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16h (6s) == sanityn test 16i: read after truncate file ============ 10:16:06 (1789049766) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16i (6s) == sanityn test 16j: race dio with buffered i/o ========== 10:16:12 (1789049772) 1+0 records in 1+0 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.13109 s, 23.2 MB/s bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.31347 s, 20.0 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.89788 s, 9.0 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.11041 s, 12.4 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.52585 s, 10.4 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.55651 s, 10.3 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.8116 s, 9.3 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.29923 s, 20.2 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.68319 s, 15.6 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.57051 s, 10.2 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.8729 s, 9.1 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.85508 s, 14.1 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.06578 s, 12.7 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.83484 s, 9.2 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.98513 s, 8.8 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.18963 s, 8.2 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.6446 s, 7.2 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.82225 s, 6.9 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 4.10303 s, 6.4 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK PASS 16j (38s) == sanityn test 16k: Parallel FSX and drop caches should not panic ========================================================== 10:16:51 (1789049811) + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.1 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.2 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.3 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.4 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.5 -l 67108864 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read truncating to largest ever: 0x1a7062c truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab All operations completed A-OK! + fsx 1 done: rc=0 All operations completed A-OK! Thu Sep 10 10:17:37 EDT 2026 All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! + fsx 2 done: rc=0 Thu Sep 10 10:17:38 EDT 2026 + fsx 3 done: rc=0 Thu Sep 10 10:17:38 EDT 2026 + fsx 4 done: rc=0 Thu Sep 10 10:17:38 EDT 2026 + fsx 5 done: rc=0 Thu Sep 10 10:17:38 EDT 2026 PASS 16k (56s) == sanityn test 17: resource creation/LVB creation race ========================================================================= 10:17:48 (1789049868) fail_loc=0x8000030a PASS 17 (11s) == sanityn test 18: mmap sanity check =========================================================================================== 10:17:59 (1789049879) mmap test1: basic mmap operation (PASS, 0.15949s) mmap test2: MAP_PRIVATE not write back (PASS, 0.14078s) mmap test3: concurrent mmap ops on two nodes (PASS, 7.3558s) mmap test4: c1 write to f1 from mmapped f2, c2 write to f1 from mmapped f1 (PASS, 4.4255s) mmap test5: read/write file to/from the buffer which mmapped to just this file (PASS, 0.20981s) mmap test6: check mmap write/read content on two nodes (PASS, 0.18208s) mmap test7: file i/o with an unmapped buffer (SKIPPED, 0s) mmap test8: SIGBUS for beyond file size (SKIPPED, 0s) mmap test9: SIGBUS for negative file offset (SKIPPED, 0s) mmap test10: multi-thread mmap access (PASS, 1.2929s) mmap test11: multi-thread shared mmap (PASS, 1.4393s) mmap test10: mtime not change for readonly mmap access (PASS, 1.0919s) PASS 18 (24s) == sanityn test 19: test concurrent uncached read races ========================================================================= 10:18:24 (1789049904) oleg341-server: error: get_param: param_path 'osd-*/lustre-OST*/read_cache_enable': No such file or directory pdsh@oleg341-client: oleg341-server: ssh exited with exit code 2 SKIP: sanityn test_19 not cache-capable obdfilter SKIP 19 (4s) == sanityn test 20: test extra readahead page left in cache ============================================================== 10:18:28 (1789049908) PASS 20 (6s) == sanityn test 21: Try to remove mountpoint on another dir ============================================================== 10:18:34 (1789049914) rmdir: removing directory, '/mnt/lustre/d21.sanityn' rmdir: failed to remove '/mnt/lustre/d21.sanityn': Device or resource busy rmdir: removing directory, '/mnt/lustre2/d21.sanityn' Removed mounted directory from another mountpoint, needs to be fixed PASS 21 (6s) == sanityn test 23: others should see updated atime while another read============================================================== 10:18:40 (1789049920) now is 1789049923 starting reads multiop /mnt/lustre/f23.sanityn vor20_c TMPPIPE=/tmp/multiop_open_wait_pipe.7620 new atime is 1789049984 PASS 23 (68s) == sanityn test 24a: lfs df [-ih] [path] test =================================================================================== 10:19:48 (1789049988) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 6144 3763200 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 6144 3763200 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre2 UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 115.3K 403 114.9K 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 114.5K 400 114.1K 1% /mnt/lustre[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.3K 403 114.9K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 399 114.1K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre2 UUID bytes Used Available Use% Mounted on lustre-MDT0000_UUID 2.1G 2.1M 2.1G 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3.6G 6.0M 3.6G 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3.6G 32.0M 3.6G 1% /mnt/lustre[OST:1] filesystem_summary: 7.2G 38.0M 7.2G 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118067 403 117664 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117231 399 116832 1% /mnt/lustre2[OST:1] filesystem_summary: 234882 386 234496 1% /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 6144 3763200 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.3K 403 114.9K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 399 114.1K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 118067 403 117664 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 117232 400 116832 1% /mnt/lustre[OST:1] filesystem_summary: 234882 386 234496 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118067 403 117664 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117231 399 116832 1% /mnt/lustre2[OST:1] filesystem_summary: 234882 386 234496 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 6144 3763200 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 6144 3763200 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre2 PASS 24a (6s) == sanityn test 24b: lfs df should show both filesystems ========================================================================= 10:19:54 (1789049994) PASS 24b (5s) == sanityn test 25a: change ACL on one mountpoint be seen on another ============================================================= 10:19:59 (1789049999) running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] PASS 25a (7s) == sanityn test 25b: change ACL under remote dir on one mountpoint be seen on another ========================================================== 10:20:06 (1789050006) SKIP: sanityn test_25b needs >= 2 MDTs SKIP 25b (3s) == sanityn test 26a: allow mtime to get older ============ 10:20:09 (1789050009) utime: good mknod times 1789050010 <= 1789050011 <= 1789050011 for /mnt/lustre/f26a utime: good mknod times 1789050010 <= 1789050011 <= 1789050011 for /mnt/lustre2/f26a utime: good utime mtimes 100000, atime 200000 utime: updated times for /mnt/lustre2/f26a PASS 26a (6s) == sanityn test 26b: sync mtime between ost and mds ====== 10:20:16 (1789050016) PASS 26b (8s) == sanityn test 26c: set-in-past on open file is not lost on close ========================================================== 10:20:24 (1789050024) multiop /mnt/lustre/f26c.sanityn vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7620 File: /mnt/lustre/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502398 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-09-10 10:20:25.778000000 -0400 Birth: 2026-09-10 10:20:25.792000000 -0400 File: /mnt/lustre2/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502398 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-09-10 10:20:25.778000000 -0400 Birth: 2026-09-10 10:20:25.792000000 -0400 PASS 26c (7s) SKIP: sanityn test_27 skipping excluded test 27 == sanityn test 30: recreate file race =================== 10:20:33 (1789050033) 0 PASS 30 (9s) == sanityn test 31a: voluntary cancel / blocking ast race======================================================================== 10:20:43 (1789050043) fail_loc=0x314 PASS 31a (8s) == sanityn test 31b: voluntary OST cancel / blocking ast race======================================================================== 10:20:52 (1789050052) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg341-server mds-ost sync done. sleep 5 for ZFS MDS Waiting for MDT destroys to complete fail_loc=0x314 fail_loc=0x316 fail_loc=0 fail_loc=0 Filesystem 1K-blocks Used Available Use% Mounted on 192.168.203.141@tcp:/lustre 7542784 39936 7498752 1% /mnt/lustre2 PASS 31b (24s) == sanityn test 31r: open-rename(replace) race =========== 10:21:15 (1789050075) -rw-r--r-- 1 root root 0 Sep 10 10:21 /mnt/lustre/f31r.sanityn.target fail_loc=0x1419 fail_val=3 PASS 31r (8s) == sanityn test 31s: open should not revalidate invalid dentry ========================================================== 10:21:23 (1789050083) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 1205: 43136 Terminated tail -f $tfile PASS 31s (7s) == sanityn test 31t: getattr should not revalidate invalid dentry ========================================================== 10:21:30 (1789050090) PASS 31t (7s) == sanityn test 32b: lockless i/o ======================== 10:21:37 (1789050097) SKIP: sanityn test_32b max_nolock_bytes is removed >= 2.17.53 SKIP 32b (3s) == sanityn test 32c: contention detection ================ 10:21:40 (1789050100) ldlm.namespaces.filter-lustre-OST0000_UUID.contended_locks=2 ldlm.namespaces.filter-lustre-OST0001_UUID.contended_locks=2 ldlm.namespaces.filter-lustre-OST0000_UUID.contention_events=0 ldlm.namespaces.filter-lustre-OST0001_UUID.contention_events=0 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.41933 s, 36.9 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.23015 s, 42.6 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.5189 s, 34.5 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.8883 s, 27.8 MB/s contention_seconds 4, contention_events 16 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.06919 s, 25.3 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.34973 s, 22.3 MB/s ldlm.namespaces.filter-lustre-OST0000_UUID.contention_events=0 ldlm.namespaces.filter-lustre-OST0001_UUID.contention_events=0 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.899334 s, 58.3 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.04317 s, 50.3 MB/s PASS 32c (25s) SKIP: sanityn test_33a skipping SLOW test 33a == sanityn test 33b: COS: cross create/delete, 2 clients, benchmark under remote dir ========================================================== 10:22:07 (1789050127) SKIP: sanityn test_33b Need two or more clients, have 1 SKIP 33b (3s) == sanityn test 33c: Cancel cross-MDT lock should trigger Sync-on-Lock-Cancel ========================================================== 10:22:10 (1789050130) SKIP: sanityn test_33c needs >= 2 MDTs SKIP 33c (3s) == sanityn test 33d: dependent transactions should trigger COS ========================================================== 10:22:13 (1789050133) SKIP: sanityn test_33d needs >= 2 MDTs SKIP 33d (3s) == sanityn test 33e: independent transactions shouldn't trigger COS ========================================================== 10:22:16 (1789050136) SKIP: sanityn test_33e needs >= 2 MDTs SKIP 33e (3s) == sanityn test 34: no lock timeout under IO ============= 10:22:19 (1789050139) fail_loc=0x512 fail_loc=0x512 lock should not expire writing on client1 reading on client2 success fail_loc=0x511 fail_loc=0x511 lock should expire writing on client1 reading on client2 success oleg341-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9668d169d000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9668d169d000.ost_server_uuid in FULL state after 0 sec oleg341-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9668d169d000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9668d169d000.ost_server_uuid in IDLE state after 0 sec oleg341-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9668d169d000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9668d169d000.ost_server_uuid in FULL state after 0 sec oleg341-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9668d169d000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9668d169d000.ost_server_uuid in IDLE state after 0 sec oleg341-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9668d169d000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9668d169d000.ost_server_uuid in FULL state after 0 sec oleg341-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9668d169d000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9668d169d000.ost_server_uuid in IDLE state after 0 sec PASS 34 (136s) == sanityn test 35: -EINTR cp_ast vs. bl_ast race does not evict client ========================================================== 10:24:35 (1789050275) fail_loc=0x80000317 Race attempt 0 Wait for 50438 50473 for 60 sec... - open/close 732 (time 1789050292.48 total 10.01 last 73.16) - open/close 1457 (time 1789050302.49 total 20.01 last 72.46) - open/close 2150 (time 1789050312.49 total 30.01 last 69.30) - open/close 2959 (time 1789050322.50 total 40.02 last 80.81) - open/close 3682 (time 1789050332.50 total 50.03 last 72.26) total: 4000 open/close in 54.13 seconds: 73.90 ops/second fail_loc=0x0 Filesystem Size Used Avail Use% Mounted on 192.168.203.141@tcp:/lustre 7.2G 39M 7.2G 1% /mnt/lustre 192.168.203.141@tcp:/lustre 7.2G 39M 7.2G 1% /mnt/lustre2 PASS 35 (72s) == sanityn test 36: handle ESTALE/open-unlink correctly == 10:25:47 (1789050347) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg341-server mds-ost sync done. Waiting for MDT destroys to complete 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.41844 s, 37.0 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7620 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(0) *** before(7498752) after_dd(7441408) after(7498752) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.33497 s, 39.3 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7620 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(1) *** before(7498752) after_dd(7440384) after(7491584) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.34357 s, 39.0 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7620 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(2) *** before(7491584) after_dd(7441408) after(7492608) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.3758 s, 38.1 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7620 sleep 5 for ZFS MDS Waiting for MDT destroys to complete *** cycle(3) *** before(7492608) after_dd(7442432) after(7493632) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.29731 s, 40.4 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7620 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(4) *** before(7493632) after_dd(7445504) after(7496704) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.26195 s, 41.5 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7620 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(5) *** before(7496704) after_dd(7441408) after(7492608) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.08584 s, 48.3 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7620 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(6) *** before(7492608) after_dd(7443456) after(7494656) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.05574 s, 49.7 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7620 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(7) *** before(7494656) after_dd(7442432) after(7498752) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.85811 s, 28.2 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7620 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(8) *** before(7498752) after_dd(7441408) after(7498752) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.07791 s, 48.6 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7620 sleep 5 for ZFS MDS Waiting for MDT destroys to complete *** cycle(9) *** before(7498752) after_dd(7447552) after(7498752) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.18669 s, 44.2 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7620 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(10) *** before(7498752) after_dd(7443456) after(7498752) PASS 36 (302s) == sanityn test 37: check i_size is not updated for directory on close (bug 18695) ======================================================================== 10:30:49 (1789050649) multiop /mnt/lustre/d37.sanityn vD_c TMPPIPE=/tmp/multiop_open_wait_pipe.7620 - create 1422 (time 1789050663.45 total 10.00 last 142.17) - create 2880 (time 1789050673.45 total 20.00 last 145.80) - create 4172 (time 1789050683.47 total 30.02 last 128.99) - create 5476 (time 1789050693.47 total 40.02 last 130.35) - create 6828 (time 1789050703.51 total 50.05 last 134.78) - create 8084 (time 1789050713.51 total 60.06 last 125.55) - create 9271 (time 1789050723.51 total 70.06 last 118.68) total: 10000 create in 75.52 seconds: 132.42 ops/second PASS 37 (86s) == sanityn test 39a: file mtime does not change after rename ========================================================== 10:32:15 (1789050735) repeat after cancel_lru_locks PASS 39a (7s) == sanityn test 39b: file mtime the same on clients with/out lock ========================================================== 10:32:22 (1789050742) repeat after cancel_lru_locks PASS 39b (7s) == sanityn test 39c: check truncate mtime update ================================================================================ 10:32:29 (1789050749) repeat after cancel_lru_locks PASS 39c (7s) == sanityn test 39d: sync write should update mtime ====== 10:32:36 (1789050756) fail_loc=0x411 fail_loc=0 PASS 39d (6s) SKIP: sanityn test_40a skipping ALWAYS excluded test 40a == sanityn test 40b: pdirops: open|create and others ======================================================================== 10:32:45 (1789050765) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40b (19s) == sanityn test 40c: pdirops: link and others ======================================================================== 10:33:03 (1789050783) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40c (15s) == sanityn test 40d: pdirops: unlink and others ======================================================================== 10:33:18 (1789050798) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40d (15s) == sanityn test 40e: pdirops: rename and others ======================================================================== 10:33:33 (1789050813) No conflict No conflict No conflict No conflict No conflict No conflict PASS 40e (17s) == sanityn test 41a: pdirops: create vs mkdir ======================================================================== 10:33:50 (1789050830) mkdir: cannot create directory '/mnt/lustre2/f41a.sanityn': File exists Conflict PASS 41a (13s) == sanityn test 41b: pdirops: create vs create ======================================================================== 10:34:03 (1789050843) open: File exists Conflict PASS 41b (14s) == sanityn test 41c: pdirops: create vs link ======================================================================== 10:34:17 (1789050857) link: cannot create link '/mnt/lustre2/f41c.sanityn' to '/mnt/lustre2/f41c.sanityn-2': File exists Conflict PASS 41c (13s) == sanityn test 41d: pdirops: create vs unlink ======================================================================== 10:34:30 (1789050870) Conflict PASS 41d (13s) == sanityn test 41e: pdirops: create and rename (tgt) ======================================================================== 10:34:43 (1789050883) Conflict PASS 41e (13s) == sanityn test 41f: pdirops: create and rename (src) ======================================================================== 10:34:56 (1789050896) Conflict PASS 41f (12s) == sanityn test 41g: pdirops: create vs getattr ======================================================================== 10:35:08 (1789050908) Conflict PASS 41g (12s) == sanityn test 41h: pdirops: create vs readdir ======================================================================== 10:35:21 (1789050921) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 2427: lctl set_param -n fail_loc=0 2>/dev/null || true: No such file or directory Conflict PASS 41h (12s) == sanityn test 41i: reint_open: create vs create ======== 10:35:33 (1789050933) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 41i (1050s) == sanityn test 42a: pdirops: mkdir vs mkdir ======================================================================== 10:53:03 (1789051983) mkdir: cannot create directory '/mnt/lustre2/f42a.sanityn': File exists Conflict PASS 42a (17s) == sanityn test 42b: pdirops: mkdir vs create ======================================================================== 10:53:20 (1789052000) open: File exists Conflict PASS 42b (12s) == sanityn test 42c: pdirops: mkdir vs link ======================================================================== 10:53:32 (1789052012) link: cannot create link '/mnt/lustre2/f42c.sanityn' to '/mnt/lustre2/f42c.sanityn-2': File exists Conflict PASS 42c (13s) == sanityn test 42d: pdirops: mkdir vs unlink ======================================================================== 10:53:45 (1789052025) Conflict PASS 42d (15s) == sanityn test 42e: pdirops: mkdir and rename (tgt) ======================================================================== 10:54:00 (1789052040) mv: cannot overwrite directory '/mnt/lustre2/f42e.sanityn' with non-directory Conflict PASS 42e (17s) == sanityn test 42f: pdirops: mkdir and rename (src) ======================================================================== 10:54:18 (1789052058) Conflict PASS 42f (15s) == sanityn test 42g: pdirops: mkdir vs getattr ======================================================================== 10:54:33 (1789052073) Conflict PASS 42g (14s) == sanityn test 42h: pdirops: mkdir vs readdir ======================================================================== 10:54:47 (1789052087) Conflict PASS 42h (12s) == sanityn test 43a: rmdir,mkdir doesn't return -EEXIST ======================================================================== 10:54:59 (1789052099) PASS 43a (71s) == sanityn test 43b: pdirops: unlink vs create ======================================================================== 10:56:11 (1789052171) Conflict PASS 43b (14s) == sanityn test 43c: pdirops: unlink vs link ======================================================================== 10:56:24 (1789052184) Conflict PASS 43c (13s) == sanityn test 43d: pdirops: unlink vs unlink ======================================================================== 10:56:37 (1789052197) rm: cannot remove '/mnt/lustre2/f43d.sanityn': No such file or directory Conflict PASS 43d (14s) == sanityn test 43e: pdirops: unlink and rename (tgt) ======================================================================== 10:56:51 (1789052211) Conflict PASS 43e (13s) == sanityn test 43f: pdirops: unlink and rename (src) ======================================================================== 10:57:05 (1789052225) mv: cannot stat '/mnt/lustre2/f43f.sanityn': No such file or directory Conflict PASS 43f (15s) == sanityn test 43g: pdirops: unlink vs getattr ======================================================================== 10:57:20 (1789052240) stat: cannot statx '/mnt/lustre2/f43g.sanityn': No such file or directory Conflict PASS 43g (14s) == sanityn test 43h: pdirops: unlink vs readdir ======================================================================== 10:57:35 (1789052255) Conflict PASS 43h (14s) == sanityn test 43i: pdirops: unlink vs remote mkdir ===== 10:57:49 (1789052269) SKIP: sanityn test_43i needs >= 2 MDTs SKIP 43i (4s) == sanityn test 43j: racy mkdir return EEXIST ======================================================================== 10:57:53 (1789052273) mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists PASS 43j (145s) == sanityn test 43k: unlink vs create ==================== 11:00:18 (1789052418) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 43k (1244s) == sanityn test 44a: pdirops: rename tgt vs mkdir ======================================================================== 11:21:02 (1789053662) mkdir: cannot create directory '/mnt/lustre2/f44a.sanityn': File exists Conflict Thu Sep 10 11:21:10 EDT 2026 PASS 44a (12s) == sanityn test 44b: pdirops: rename tgt vs create ======================================================================== 11:21:14 (1789053674) open: File exists Conflict PASS 44b (12s) == sanityn test 44c: pdirops: rename tgt vs link ======================================================================== 11:21:26 (1789053686) link: cannot create link '/mnt/lustre2/f44c.sanityn' to '/mnt/lustre2/f44c.sanityn-3': File exists Conflict PASS 44c (12s) == sanityn test 44d: pdirops: rename tgt vs unlink ======================================================================== 11:21:39 (1789053699) Conflict PASS 44d (11s) == sanityn test 44e: pdirops: rename tgt and rename (tgt) ======================================================================== 11:21:50 (1789053710) Conflict PASS 44e (12s) == sanityn test 44f: pdirops: rename tgt and rename (src) ======================================================================== 11:22:02 (1789053722) Conflict PASS 44f (13s) == sanityn test 44g: pdirops: rename tgt vs getattr ======================================================================== 11:22:15 (1789053735) Conflict PASS 44g (14s) == sanityn test 44h: pdirops: rename tgt vs readdir ======================================================================== 11:22:30 (1789053750) Conflict PASS 44h (15s) == sanityn test 44i: pdirops: rename tgt vs remote mkdir ========================================================== 11:22:44 (1789053764) SKIP: sanityn test_44i needs >= 2 MDTs SKIP 44i (4s) == sanityn test 45a: rename,mkdir doesn't return -EEXIST ======================================================================== 11:22:49 (1789053769) PASS 45a (142s) == sanityn test 45b: pdirops: rename src vs create ======================================================================== 11:25:10 (1789053910) Conflict PASS 45b (16s) == sanityn test 45c: pdirops: rename src vs link ======================================================================== 11:25:27 (1789053927) Conflict PASS 45c (12s) == sanityn test 45d: pdirops: rename src vs unlink ======================================================================== 11:25:38 (1789053938) rm: cannot remove '/mnt/lustre2/f45d.sanityn': No such file or directory Conflict PASS 45d (14s) == sanityn test 45e: pdirops: rename src and rename (tgt) ======================================================================== 11:25:52 (1789053952) Conflict PASS 45e (15s) == sanityn test 45f: pdirops: rename src and rename (src) ======================================================================== 11:26:07 (1789053967) mv: cannot stat '/mnt/lustre2/f45f.sanityn': No such file or directory Conflict PASS 45f (15s) == sanityn test 45g: pdirops: rename src vs getattr ======================================================================== 11:26:22 (1789053982) stat: cannot statx '/mnt/lustre2/f45g.sanityn': No such file or directory Conflict PASS 45g (15s) == sanityn test 45h: pdirops: unlink vs readdir ======================================================================== 11:26:37 (1789053997) Conflict PASS 45h (15s) == sanityn test 45i: pdirops: rename src vs remote mkdir ========================================================== 11:26:52 (1789054012) SKIP: sanityn test_45i needs >= 2 MDTs SKIP 45i (4s) == sanityn test 45j: read vs rename ====================== 11:26:56 (1789054016) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 45j (1187s) == sanityn test 46a: pdirops: link vs mkdir ======================================================================== 11:46:44 (1789055204) mkdir: cannot create directory '/mnt/lustre2/f46a.sanityn': File exists Conflict PASS 46a (12s) == sanityn test 46b: pdirops: link vs create ======================================================================== 11:46:55 (1789055215) open: File exists Conflict PASS 46b (13s) == sanityn test 46c: pdirops: link vs link ======================================================================== 11:47:08 (1789055228) link: cannot create link '/mnt/lustre2/f46c.sanityn' to '/mnt/lustre2/f46c.sanityn': File exists Conflict PASS 46c (12s) == sanityn test 46d: pdirops: link vs unlink ======================================================================== 11:47:20 (1789055240) Conflict PASS 46d (11s) == sanityn test 46e: pdirops: link and rename (tgt) ======================================================================== 11:47:32 (1789055252) Conflict PASS 46e (14s) == sanityn test 46f: pdirops: link and rename (src) ======================================================================== 11:47:45 (1789055265) Conflict PASS 46f (13s) == sanityn test 46g: pdirops: link vs getattr ======================================================================== 11:47:59 (1789055279) Conflict PASS 46g (13s) == sanityn test 46h: pdirops: link vs readdir ======================================================================== 11:48:11 (1789055291) Conflict PASS 46h (12s) == sanityn test 46i: pdirops: link vs remote mkdir ======= 11:48:23 (1789055303) SKIP: sanityn test_46i needs >= 2 MDTs SKIP 46i (3s) == sanityn test 47a: pdirops: remote mkdir vs mkdir ====== 11:48:26 (1789055306) SKIP: sanityn test_47a needs >= 2 MDTs SKIP 47a (3s) == sanityn test 47b: pdirops: remote mkdir vs create ===== 11:48:30 (1789055310) SKIP: sanityn test_47b needs >= 2 MDTs SKIP 47b (3s) == sanityn test 47c: pdirops: remote mkdir vs link ======= 11:48:33 (1789055313) SKIP: sanityn test_47c needs >= 2 MDTs SKIP 47c (3s) == sanityn test 47d: pdirops: remote mkdir vs unlink ===== 11:48:36 (1789055316) SKIP: sanityn test_47d needs >= 2 MDTs SKIP 47d (3s) == sanityn test 47e: pdirops: remote mkdir and rename (tgt) ========================================================== 11:48:40 (1789055320) SKIP: sanityn test_47e needs >= 2 MDTs SKIP 47e (2s) == sanityn test 47f: pdirops: remote mkdir and rename (src) ========================================================== 11:48:43 (1789055323) SKIP: sanityn test_47f needs >= 2 MDTs SKIP 47f (2s) == sanityn test 47g: pdirops: remote mkdir vs getattr ==== 11:48:46 (1789055326) SKIP: sanityn test_47g needs >= 2 MDTs SKIP 47g (3s) == sanityn test 50: osc lvb attrs: enqueue vs. CP AST ======================================================================== 11:48:49 (1789055329) 10+0 records in 10+0 records out 10240 bytes (10 kB, 10 KiB) copied, 0.164531 s, 62.2 kB/s fail_loc=0x410 fail_loc=0x0 PASS 50 (11s) == sanityn test 51a: layout lock: refresh layout should work ========================================================== 11:49:01 (1789055341) 0+1 records in 0+1 records out 158 bytes copied, 0.02377 s, 6.6 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 3690: kill: (218687) - No such process PASS 51a (8s) == sanityn test 51b: layout lock: glimpse should be able to restart if layout changed ========================================================== 11:49:09 (1789055349) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0131313 s, 78.0 kB/s fail_loc=0x1404 fail_val=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0760967 s, 13.8 MB/s 3145728 PASS 51b (19s) == sanityn test 51c: layout lock: IT_LAYOUT blocked and correct layout can be returned ========================================================== 11:49:28 (1789055368) fail_loc=0x172 Setting layout to have 2 stripes ... 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0403148 s, 25.4 kB/s PASS 51c (11s) == sanityn test 51d: layout lock: losing layout lock should clean up memory map region ========================================================== 11:49:39 (1789055379) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.059125 s, 17.7 MB/s Before revoking layout lock: 1024 KB mapped PASS 51d (8s) == sanityn test 51e: lfs getstripe does not break leases, part 2 ========================================================== 11:49:47 (1789055387) /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 2453 0x995 0x240000400 write lease(2) has applied. /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 2453 0x995 0x240000400 read lease(1) has applied. PASS 51e (9s) == sanityn test 54: rename locking ======================= 11:49:56 (1789055396) llite.lustre-ffff9668d169d000.xattr_cache=1 llite.lustre-ffff9668d8375800.xattr_cache=1 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505015 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-10 11:49:58.293287417 -0400 Modify: 2026-09-10 11:49:58.324976598 -0400 Change: 2026-09-10 11:49:58.324976598 -0400 Birth: 2026-09-10 11:49:58.390000000 -0400 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000154 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505018 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-10 11:50:04.908336690 -0400 Modify: 2026-09-10 11:50:06.346269064 -0400 Change: 2026-09-10 11:50:06.346269064 -0400 Birth: 2026-09-10 11:50:05.003000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505021 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-10 11:50:11.530635571 -0400 Modify: 2026-09-10 11:50:11.556459626 -0400 Change: 2026-09-10 11:50:11.556459626 -0400 Birth: 2026-09-10 11:50:11.624000000 -0400 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505020 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-10 11:50:11.514586916 -0400 Modify: 2026-09-10 11:50:12.606073779 -0400 Change: 2026-09-10 11:50:12.606073779 -0400 Birth: 2026-09-10 11:50:11.609000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000154 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505024 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-10 11:50:18.075555535 -0400 Modify: 2026-09-10 11:50:19.323032194 -0400 Change: 2026-09-10 11:50:19.323032194 -0400 Birth: 2026-09-10 11:50:18.169000000 -0400 File: /mnt/lustre/d1 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505023 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-10 11:50:18.046074572 -0400 Modify: 2026-09-10 11:50:19.323032194 -0400 Change: 2026-09-10 11:50:19.323032194 -0400 Birth: 2026-09-10 11:50:18.139000000 -0400 open(O_RDWR|O_CREAT): Is a directory PASS 54 (34s) == sanityn test 55a: rename vs unlink target dir ========= 11:50:30 (1789055430) fail_loc=0x80000156 mv: cannot move '/mnt/lustre/d55a.sanityn/d1/d2' to '/mnt/lustre/d55a.sanityn/d3/d2': No such file or directory PASS 55a (14s) == sanityn test 55b: rename vs unlink source dir ========= 11:50:44 (1789055444) fail_loc=0x80000156 mv: cannot stat '/mnt/lustre/d55b.sanityn/d1/d2': No such file or directory PASS 55b (14s) == sanityn test 55c: rename vs unlink orphan target dir == 11:50:58 (1789055458) fail_loc=0x156 mv: cannot move '/mnt/lustre/d55c.sanityn/d1/d2' to '/mnt/lustre/d55c.sanityn/d3/d2': No such file or directory PASS 55c (20s) == sanityn test 55d: rename file vs link ================= 11:51:18 (1789055478) fail_loc=0x155 mv: '/mnt/lustre/d55d.sanityn/f1' and '/mnt/lustre/d55d.sanityn/d55d.sanityn/f1' are the same file PASS 55d (15s) == sanityn test 55e: rename race AB/BA under the same parent dir ========================================================== 11:51:34 (1789055494) SKIP: sanityn test_55e needs >= 2 MDTs SKIP 55e (3s) == sanityn test 55f: rename: (P1/A -> P2/B) race with (P2/B -> P1/A) ========================================================== 11:51:36 (1789055496) fail_loc=0x155 We survived after P1A_P2B/P2B_P1A race test fail_loc=0x0 PASS 55f (20s) == sanityn test 55g: rename: race with trylock =========== 11:51:56 (1789055516) mdt.lustre-MDT0000.enable_rename_trylock=1 fail_loc=0x155 mv: cannot move '/mnt/lustre2/d55g.sanityn/b1/b2' to a subdirectory of itself, '/mnt/lustre2/d55g.sanityn/a1/a2/a3/a4/b2' We survived race with trylock test fail_loc=0x0 PASS 55g (27s) == sanityn test 56a: test llverdev with single large stripe ========================================================== 11:52:24 (1789055544) run_llverdev /mnt/lustre/f56a.sanityn -p -s 16000000 -c 4k llverdev: /mnt/lustre/f56a.sanityn is 16777216000000 bytes (15625 GB) in size timestamp: 1789055546 chunksize: 4096 size: 16777216000000 write offset: 16384000000kB 0.426847 MB/s read offset: 16384000000kB inf MB/s llverdev: data verified successfully PASS 56a (152s) == sanityn test 56b: test llverdev and partial verify of wide stripe file ========================================================== 11:54:56 (1789055696) run_llverdev /mnt/lustre/f56b.sanityn -p -s 32000000 -c 4k llverdev: /mnt/lustre/f56b.sanityn is 33554432000000 bytes (31250 GB) in size timestamp: 1789055702 chunksize: 4096 size: 33554432000000 write offset: 32768000000kB 0.98447 MB/s read offset: 32768000000kB inf MB/s llverdev: data verified successfully PASS 56b (138s) == sanityn test 60: Verify data_version behaviour ======== 11:57:14 (1789055834) PASS 60 (8s) == sanityn test 70a: cd directory && rm directory ======== 11:57:22 (1789055842) PASS 70a (7s) == sanityn test 70b: remove files after calling rm_entry ========================================================== 11:57:29 (1789055849) PASS 70b (9s) == sanityn test 71a: correct file map just after write operation is finished ========================================================== 11:57:38 (1789055858) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0165853 s, 4.0 MB/s SKIP: sanityn test_71a ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71a (5s) == sanityn test 71b: check fiemap support for stripecount > 1 ========================================================== 11:57:43 (1789055863) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0182703 s, 3.6 MB/s SKIP: sanityn test_71b ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71b (4s) == sanityn test 71c: check FIEMAP_EXTENT_LAST flag with different extents number ========================================================== 11:57:48 (1789055868) SKIP: sanityn test_71c support only ldiskfs ost SKIP 71c (3s) == sanityn test 71d: fiemap corruption test with fm_extent_count=0 ========================================================== 11:57:51 (1789055871) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0289807 s, 2.3 MB/s SKIP: sanityn test_71d ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71d (4s) == sanityn test 72: getxattr/setxattr cache should be consistent between nodes ========================================================== 11:57:55 (1789055875) llite.lustre-ffff9668d169d000.xattr_cache=1 llite.lustre-ffff9668d8375800.xattr_cache=1 getfattr: Removing leading '/' from absolute path names user.attr1="value1" getfattr: Removing leading '/' from absolute path names user.attr1="value2" getfattr: Removing leading '/' from absolute path names getfattr: Removing leading '/' from absolute path names b362740ad371eae580e7f85452e8830b - 870e072da1ebfc20ea8c0cd3d6bdf3b6 - PASS 72 (7s) == sanityn test 73: getxattr should not cause xattr lock cancellation ========================================================== 11:58:02 (1789055882) llite.lustre-ffff9668d169d000.xattr_cache=1 llite.lustre-ffff9668d8375800.xattr_cache=1 getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" PASS 73 (6s) == sanityn test 74: flock deadlock: different mounts ======================================================================== 11:58:09 (1789055889) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0220961 s, 46.3 kB/s 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0102907 s, 99.5 kB/s 231504: get lock2 231505: get lock1 231504: done 231504: get lock1 231505: done 231505: get lock2 fcntl cmd 7 failed: Resource deadlock avoided 231505: cannot get lock2: Resource deadlock avoided 231505: exit rc=0 231504: done 231504: exit rc=0 PASS 74 (11s) == sanityn test 75: osc: upcall after unuse lock============================================================================= 11:58:20 (1789055900) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0767581 s, 27.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0479757 s, 21.9 MB/s fail_loc=0x31d /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 4373: 232086 Killed stat -c %s $DIR1/$tfile fail_loc=0 PASS 75 (14s) == sanityn test 76: Verify MDT open_files listing ======== 11:58:34 (1789055914) opening files: ........ closing files: ........ PASS 76 (82s) == sanityn test 77a: check FIFO NRS policy =============== 11:59:56 (1789055996) ost.OSS.ost.nrs_policies=fifo ost.OSS.ost_create.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_out.nrs_policies=fifo ost.OSS.ost_seq.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.424015 s, 39.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0528711 s, 19.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0781178 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0989913 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136068 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0819222 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0969594 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.243543 s, 4.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188656 s, 5.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.262237 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.268379 s, 3.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205649 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198422 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21408 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202308 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.287852 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19299 s, 5.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9668d8375800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9668d8375800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0679243 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0488497 s, 21.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0460132 s, 22.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110524 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0507687 s, 20.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0580392 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0542323 s, 19.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101427 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.045776 s, 22.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110368 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0683577 s, 15.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0597824 s, 17.5 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0790242 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0743574 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0700552 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0842952 s, 12.4 MB/s removed '/mnt/lustre/d77a.sanityn/nrs_w_oleg341-client.virtnet' removed '/mnt/lustre/d77a.sanityn/nrs_r_oleg341-client.virtnet' removed directory '/mnt/lustre/d77a.sanityn' PASS 77a (9s) == sanityn test 77b: check CRR-N NRS policy ============== 12:00:05 (1789056005) ost.OSS.ost.nrs_policies=crrn ost.OSS.ost_create.nrs_policies=crrn ost.OSS.ost_io.nrs_policies=crrn ost.OSS.ost_out.nrs_policies=crrn ost.OSS.ost_seq.nrs_policies=crrn ost.OSS.ost.nrs_crrn_quantum=1 ost.OSS.ost_create.nrs_crrn_quantum=1 ost.OSS.ost_io.nrs_crrn_quantum=1 ost.OSS.ost_out.nrs_crrn_quantum=1 ost.OSS.ost_seq.nrs_crrn_quantum=1 policy: crr-n, crrn_quantum 1 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.531029 s, 31.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0385104 s, 27.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144789 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180605 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110596 s, 9.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116916 s, 9.0 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105359 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153592 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.238297 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212621 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198845 s, 5.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.269172 s, 3.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9668d8375800.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff9668d8375800.lock_unused_count=3 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.290849 s, 3.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.295213 s, 3.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.343419 s, 3.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.388845 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.317018 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0546953 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0861751 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132798 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00771668 s, 136 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00208146 s, 504 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0841296 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00258578 s, 406 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0441496 s, 23.8 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.06509 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188782 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0341393 s, 30.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0236102 s, 44.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.02992 s, 35.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153056 s, 68.5 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0230814 s, 45.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0234466 s, 44.7 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg341-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg341-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost.nrs_crrn_quantum=64 ost.OSS.ost_create.nrs_crrn_quantum=64 ost.OSS.ost_io.nrs_crrn_quantum=64 ost.OSS.ost_out.nrs_crrn_quantum=64 ost.OSS.ost_seq.nrs_crrn_quantum=64 policy: crr-n, crrn_quantum 64 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.397065 s, 42.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0453042 s, 23.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0595216 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0588699 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.091175 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105824 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221847 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106642 s, 9.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.241543 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.264257 s, 4.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211641 s, 5.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.258293 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237824 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.252818 s, 4.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178875 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211111 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.320597 s, 3.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9668d8375800.lock_unused_count=3 ldlm.namespaces.lustre-OST0001-osc-ffff9668d8375800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0201818 s, 52.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151395 s, 69.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0458231 s, 22.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0517489 s, 20.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0780591 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0628681 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0336886 s, 31.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0893742 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0862942 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0759216 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0401917 s, 26.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0777609 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0652049 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0744185 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0790659 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105475 s, 9.9 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg341-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg341-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77b (14s) == sanityn test 77c: check ORR NRS policy ================ 12:00:19 (1789056019) ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_orr_quantum=1 ost.OSS.ost_io.nrs_orr_offset_type=physical ost.OSS.ost_io.nrs_orr_supported=reads policy: orr, orr_quantum 1, orr_offset_type physical, orr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.431685 s, 38.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0321469 s, 32.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0778061 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0535975 s, 19.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122698 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139112 s, 7.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197835 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192945 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.240205 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.285001 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201273 s, 5.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19785 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191721 s, 5.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.238089 s, 4.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.290456 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.30712 s, 3.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9668d8375800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9668d8375800.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.344477 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0790819 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0664601 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0994888 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0998825 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102712 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113601 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0828014 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109046 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111171 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116338 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117176 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106884 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181542 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138073 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162171 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145615 s, 7.2 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg341-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg341-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=writes ost.OSS.ost_io.nrs_orr_quantum=64 policy: orr, orr_quantum 64, orr_offset_type physical, orr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.477431 s, 35.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0557434 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0918471 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.078388 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0698086 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143983 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.238442 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.310376 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203685 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177182 s, 5.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201448 s, 5.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135532 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183809 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.369023 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.325998 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.448807 s, 2.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9668d8375800.lock_unused_count=3 ldlm.namespaces.lustre-OST0001-osc-ffff9668d8375800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.529338 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0571706 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0611126 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0673683 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0692384 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0647472 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0562775 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0760739 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0662215 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0564481 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0809194 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0865045 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113521 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0954512 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0671759 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0803992 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0877902 s, 11.9 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg341-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg341-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=reads_and_writes ost.OSS.ost_io.nrs_orr_offset_type=logical policy: orr, orr_quantum 64, orr_offset_type logical, orr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.412932 s, 40.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.042329 s, 24.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0541124 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139041 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171245 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158129 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203168 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178179 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193143 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.232333 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170759 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152072 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181474 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168779 s, 6.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9668d8375800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9668d8375800.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1883 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237118 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.323349 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00292527 s, 358 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00879733 s, 119 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11952 s, 8.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104589 s, 10.0 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00252135 s, 416 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00794136 s, 132 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159459 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100226 s, 10.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0473227 s, 22.2 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0517417 s, 20.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0359132 s, 29.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0432754 s, 24.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0637042 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0321107 s, 32.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0271279 s, 38.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0493007 s, 21.3 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg341-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg341-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77c (15s) == sanityn test 77d: check TRR nrs policy ================ 12:00:34 (1789056034) ost.OSS.ost_io.nrs_policies=trr ost.OSS.ost_io.nrs_trr_quantum=1 ost.OSS.ost_io.nrs_trr_offset_type=physical ost.OSS.ost_io.nrs_trr_supported=reads policy: trr, trr_quantum 1, trr_offset_type physical, trr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.466443 s, 36.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0822516 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0952947 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101786 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17944 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213708 s, 4.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.239301 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.248998 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21546 s, 4.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.251264 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173227 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.274016 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.283075 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.247731 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.322087 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.340599 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.362217 s, 2.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9668d8375800.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff9668d8375800.lock_unused_count=5 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0617127 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131804 s, 79.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0821311 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0681279 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0412035 s, 25.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0666588 s, 15.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00811922 s, 129 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0104768 s, 100 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0693665 s, 15.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00693764 s, 151 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0741164 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.050523 s, 20.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120389 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.053801 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0577497 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0271968 s, 38.6 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg341-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg341-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=writes ost.OSS.ost_io.nrs_trr_quantum=64 policy: trr, trr_quantum 64, trr_offset_type physical, trr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.39244 s, 42.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0663882 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0682519 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0442063 s, 23.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155928 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0385739 s, 27.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0635796 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0555587 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198162 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.258024 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131167 s, 8.0 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9668d8375800.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190382 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.234178 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104909 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129024 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170747 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.273757 s, 3.8 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9668d8375800.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00814777 s, 129 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.075896 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127958 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114349 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109536 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162427 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0235233 s, 44.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163424 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122778 s, 8.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122316 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125211 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0632247 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123043 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133103 s, 7.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0616266 s, 17.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0563282 s, 18.6 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg341-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg341-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=reads_and_writes ost.OSS.ost_io.nrs_trr_offset_type=logical policy: trr, trr_quantum 64, trr_offset_type logical, trr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.319571 s, 52.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0475617 s, 22.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0461404 s, 22.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0562792 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0684387 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0191116 s, 54.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0867758 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102034 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0935754 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0953852 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0727779 s, 14.4 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9668d8375800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15441 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200735 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.218147 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196121 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212255 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.229543 s, 4.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9668d8375800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0639068 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0846363 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00410848 s, 255 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0287971 s, 36.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0773912 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10998 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0660474 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00255835 s, 410 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00523524 s, 200 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0232278 s, 45.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116916 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0074468 s, 141 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0371925 s, 28.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0375016 s, 28.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0307589 s, 34.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00806384 s, 130 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg341-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg341-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77d (15s) == sanityn test 77e: check TBF NID nrs policy ============ 12:00:49 (1789056049) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_tbf_rule=start localhost nid={0@lo} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start clients nid={192.168.203.41@tcp} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start others nid={*.*.*.*@tcp} rate=50 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.523796 s, 32.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122247 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141611 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152172 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129779 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.092296 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176185 s, 6.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.263207 s, 4.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211992 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19028 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191007 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.33081 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.282076 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199929 s, 5.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9668d8375800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.377269 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.426106 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.37577 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0653809 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.071567 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0437477 s, 24.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0851408 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0812362 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0218979 s, 47.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.081115 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0576593 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0953835 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0447133 s, 23.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0427939 s, 24.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0617065 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115956 s, 9.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0530782 s, 19.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0761969 s, 13.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0531141 s, 19.7 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg341-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg341-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change localhost rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change clients rate=101 ost.OSS.ost_io.nrs_tbf_rule=change others rate=51 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.764419 s, 21.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140157 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113436 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0938635 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196165 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146455 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104768 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0512575 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.273268 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.239609 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.283586 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.333491 s, 3.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.363628 s, 2.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.304681 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.437599 s, 2.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9668d8375800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9668d8375800.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.46803 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.434146 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132729 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103992 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128822 s, 8.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0808344 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0863826 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0884095 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0880871 s, 11.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121795 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.093476 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0964106 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101492 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101462 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106738 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107386 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117151 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123796 s, 8.5 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg341-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg341-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop localhost ost.OSS.ost_io.nrs_tbf_rule=stop clients ost.OSS.ost_io.nrs_tbf_rule=stop others 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.684217 s, 24.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0501996 s, 20.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0639727 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120517 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125437 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10872 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123508 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18293 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.247749 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.279792 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.336067 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.351512 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.219866 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.353347 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.362277 s, 2.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9668d8375800.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff9668d8375800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.257009 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.325578 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0439224 s, 23.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9668d8375800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00242055 s, 433 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00248696 s, 422 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00596485 s, 176 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0228744 s, 45.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00223424 s, 469 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208706 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00430742 s, 243 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196577 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117142 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0843929 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183621 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0860819 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101039 s, 10.4 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0521534 s, 20.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121602 s, 8.6 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg341-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg341-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.66395 s, 25.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0677753 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0935679 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.257483 s, 4.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.320444 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.338089 s, 3.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.343553 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.31882 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.325197 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.382478 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.343765 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.32734 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.324837 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.360772 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.407103 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.411075 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.508225 s, 2.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9668d8375800.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff9668d8375800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0545338 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0565946 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.040065 s, 26.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0458135 s, 22.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0455234 s, 23.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0723879 s, 14.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110082 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0649868 s, 16.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0583432 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0748529 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0751182 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0971885 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105952 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.218532 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116293 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120757 s, 8.7 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg341-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg341-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' PASS 77e (28s) == sanityn test 77f: check TBF JobID nrs policy ========== 12:01:17 (1789056077) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start runas jobid={iozone.500 dd.500 tiotest.500} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start iozone_runas jobid={iozone.500} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg341-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.349265 s, 48.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0556562 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0417629 s, 25.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111067 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107559 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.24022 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.260865 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.338058 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.296549 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.267046 s, 3.9 MB/s 1+0 records in 1+0 records out running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.309607 s, 3.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.403396 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.28476 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.392414 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.33045 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.449259 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.471395 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117987 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.064876 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122154 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0442495 s, 23.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0226885 s, 46.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00708145 s, 148 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.036506 s, 28.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00257525 s, 407 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00352519 s, 297 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00249167 s, 421 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0107965 s, 97.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00656918 s, 160 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00314124 s, 334 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189143 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0211129 s, 49.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014041 s, 74.7 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg341-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg341-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change runas rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change iozone_runas rate=101 ost.OSS.ost_io.nrs_tbf_rule=change dd_runas rate=51 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg341-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.704661 s, 23.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0562353 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121745 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139114 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175596 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.230115 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.345838 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.229017 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.273365 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.24089 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.425936 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.33216 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.411862 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.474417 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.352097 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.412272 s, 2.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied ldlm.namespaces.lustre-OST0000-osc-ffff9668d8375800.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff9668d8375800.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.40835 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132839 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112614 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110194 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0741854 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162373 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192098 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.091729 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177358 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174785 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016577 s, 63.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0675686 s, 15.5 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191671 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0620722 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0546501 s, 19.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181432 s, 5.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183304 s, 5.7 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg341-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg341-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop runas ost.OSS.ost_io.nrs_tbf_rule=stop iozone_runas ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg341-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.79601 s, 21.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105046 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107938 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169631 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0846143 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121287 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11138 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.284476 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.2547 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152208 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.372253 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.45362 s, 2.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9668d8375800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9668d8375800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.254474 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.336725 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.404987 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.383832 s, 2.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.546689 s, 1.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9668d8375800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00243789 s, 430 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151007 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0860563 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012192 s, 86.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0119162 s, 88.0 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0878879 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111529 s, 9.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166825 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0826412 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134389 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0680017 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0761652 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0619415 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0532546 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.048085 s, 21.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0427998 s, 24.5 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg341-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg341-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_policies=fifo running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg341-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.606853 s, 27.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0467602 s, 22.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10512 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0789424 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.218536 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160582 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205148 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.251497 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23975 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209031 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.280275 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220774 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.321564 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.251819 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.262232 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.293488 s, 3.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.317626 s, 3.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9668d8375800.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0022757 s, 461 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00209231 s, 501 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136831 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121257 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170871 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110399 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118341 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0966149 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0123688 s, 84.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0925992 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0838146 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0821364 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0571853 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0475715 s, 22.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0410099 s, 25.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0776381 s, 13.5 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg341-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg341-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' PASS 77f (27s) == sanityn test 77g: Change TBF type directly ============ 12:01:44 (1789056104) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 ost.OSS.ost_io.nrs_policies=fifo PASS 77g (11s) == sanityn test 77h: Wrong policy name should report error, not LBUG ========================================================== 12:01:55 (1789056115) oleg341-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=abc: No such file or directory oleg341-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='abc': No such file or directory pdsh@oleg341-client: oleg341-server: ssh exited with exit code 2 oleg341-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc: Invalid argument oleg341-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc': Invalid argument pdsh@oleg341-client: oleg341-server: ssh exited with exit code 22 oleg341-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf reg abc: Invalid argument oleg341-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf reg abc': Invalid argument pdsh@oleg341-client: oleg341-server: ssh exited with exit code 22 oleg341-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc efg: Invalid argument oleg341-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc efg': Invalid argument pdsh@oleg341-client: oleg341-server: ssh exited with exit code 22 PASS 77h (14s) == sanityn test 77i: Change rank of TBF rule ============= 12:02:09 (1789056129) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start before jobid={jobid} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start after jobid={jobid} rate=1000 rank=default ost.OSS.ost_io.nrs_tbf_rule=start target jobid={jobid} rate=1000 rank=after Move before itself ost.OSS.ost_io.nrs_tbf_rule=change target rank=target Move to higher rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=before Move to lower rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=after Move before default ost.OSS.ost_io.nrs_tbf_rule=change target rank=default ost.OSS.ost_io.nrs_policies=fifo PASS 77i (27s) == sanityn test 77j: check TBF-OPCode NRS policy ========= 12:02:36 (1789056156) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf opcode ost.OSS.ost_io.nrs_tbf_rule=start ost_r opcode={ost_read} rate=5 ost.OSS.ost_io.nrs_tbf_rule=start ost_w opcode={ost_write} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.61716 s, 27.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0752525 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127431 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144861 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111766 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0931909 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14456 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101416 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.258617 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.307696 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.234811 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.309922 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.380666 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.362661 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.40211 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.507515 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.545376 s, 1.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0670396 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0741445 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015844 s, 66.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0829667 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132784 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10988 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00804958 s, 130 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0100675 s, 104 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200087 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.240391 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.245232 s, 4.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0660218 s, 15.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15215 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153216 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155886 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158947 s, 6.6 MB/s removed '/mnt/lustre/d77j.sanityn/nrs_w_oleg341-client.virtnet' removed '/mnt/lustre/d77j.sanityn/nrs_r_oleg341-client.virtnet' removed directory '/mnt/lustre/d77j.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 5 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.0853 s, 9.5 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.6734 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77j.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_r ost.OSS.ost_io.nrs_tbf_rule=stop ost_w ost.OSS.ost_io.nrs_policies=fifo PASS 77j (74s) == sanityn test 77ja: check TBF-UID/GID NRS policy ======= 12:03:51 (1789056231) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg341-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg341-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg341-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg341-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg341-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.31747 s, 12.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127751 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120592 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131998 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176678 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203809 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216125 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22516 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211084 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19323 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.229106 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.326437 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.248697 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212521 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.295738 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.407336 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.454551 s, 2.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0795994 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0765808 s, 13.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.08083 s, 13.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.037964 s, 27.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0534345 s, 19.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.058032 s, 18.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0839342 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0604864 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0818999 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0985717 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0566911 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.051343 s, 20.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111001 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0868457 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0704014 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0865964 s, 12.1 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg341-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg341-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.958 s, 5.3 MB/s Write runtime is 22 s, speed is 4.545454 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9768 s, 5.2 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg341-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg341-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg341-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg341-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg341-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.27221 s, 13.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0866707 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127635 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140784 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124861 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142988 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170437 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1612 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0600532 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.056465 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215465 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.248887 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154635 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.247523 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.299638 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.302614 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.326191 s, 3.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0684729 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0325237 s, 32.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0843185 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0919419 s, 11.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0778293 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0453529 s, 23.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0645906 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0883834 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103646 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0920988 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.063303 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104502 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115859 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123717 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104159 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12063 s, 8.7 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg341-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg341-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9963 s, 5.2 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9948 s, 5.2 MB/s Read runtime is 22 s, speed is 4.545454 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77ja (154s) == sanityn test 77jb: check TBF-UID/GID NRS policy on files that don't belong to us ========================================================== 12:06:25 (1789056385) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg341-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg341-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg341-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg341-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg341-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.30587 s, 12.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104643 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131211 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130382 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161732 s, 6.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176451 s, 5.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167931 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157666 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197443 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192331 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.217425 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.233899 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23582 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.305308 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.341953 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.368224 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.452095 s, 2.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117887 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120374 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125058 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0502604 s, 20.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0755251 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110292 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141554 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157571 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0806314 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103478 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155325 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116949 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152337 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127898 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171001 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135994 s, 7.7 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg341-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg341-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.3626 s, 5.1 MB/s Write runtime is 22 s, speed is 4.545454 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.7799 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg341-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg341-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg341-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg341-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg341-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.29116 s, 13.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0871423 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101191 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130316 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188015 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190936 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106752 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0934528 s, 11.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0957369 s, 11.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0686309 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213506 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.343298 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.252372 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.26878 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.30246 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.430779 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.257882 s, 4.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0794615 s, 13.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0741432 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0835827 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0714682 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0418805 s, 25.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0570043 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0630795 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.074017 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0490465 s, 21.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107488 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0705688 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106987 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107175 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0641188 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10535 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0789139 s, 13.3 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg341-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg341-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.0547 s, 5.2 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.0061 s, 5.2 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77jb (158s) == sanityn test 77k: check TBF policy with UID/GID/JobID/OPCode expression ========================================================== 12:09:03 (1789056543) ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg341-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.786977 s, 21.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0776401 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128977 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0959235 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.246224 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.246777 s, 4.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.271564 s, 3.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.32643 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0809635 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181699 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.24714 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.317491 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.294412 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222457 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.259849 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.400059 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.295783 s, 3.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9668d8375800.lock_unused_count=1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114344 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156127 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147595 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12668 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116672 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135633 s, 7.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119871 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143888 s, 7.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196284 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19706 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171886 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134294 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177029 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208784 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203725 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.2744 s, 3.8 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg341-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg341-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.4382 s, 9.2 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.98685 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.203.41@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.203.41@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.606341 s, 27.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0747348 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0989509 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0873502 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178072 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.236329 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221638 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.31165 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.274249 s, 3.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215085 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.263388 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.319559 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.301716 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.30188 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.260209 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.283335 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.349381 s, 3.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9668d8375800.lock_unused_count=5 ldlm.namespaces.lustre-OST0001-osc-ffff9668d8375800.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13073 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113398 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166432 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179436 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205346 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112603 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135214 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147479 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180332 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158115 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123316 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165328 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141804 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194094 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130719 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.249717 s, 4.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9668d8375800.lock_unused_count=2 removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg341-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg341-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.62797 s, 13.7 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.89476 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.203.41@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg341-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.740263 s, 22.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0730728 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0688147 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.081657 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19764 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.258669 s, 4.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.230548 s, 4.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184709 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194195 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.319341 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.24628 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.314375 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.338479 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.314231 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.268986 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.289714 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.292522 s, 3.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied ldlm.namespaces.lustre-OST0000-osc-ffff9668d8375800.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff9668d8375800.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0955339 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102687 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0495082 s, 21.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0264161 s, 39.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0289671 s, 36.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0564369 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0459144 s, 22.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108294 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0724773 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0560433 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100874 s, 10.4 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14446 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0899547 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0676319 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116367 s, 9.0 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0770176 s, 13.6 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg341-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg341-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.40729 s, 12.5 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.22462 s, 20.1 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg341-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.896323 s, 18.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178119 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171056 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.26315 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.303314 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.284208 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.321288 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.274489 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.406006 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.450433 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.435046 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.384723 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.380703 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.516826 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.530683 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.381636 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.495492 s, 2.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174597 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163569 s, 64.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0788151 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0835503 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0788989 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0453975 s, 23.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171114 s, 6.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0641453 s, 16.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0996074 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178049 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0909948 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0947838 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0980779 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106555 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102254 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116044 s, 9.0 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg341-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg341-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 13.1616 s, 8.0 MB/s Write runtime is 14 s, speed is 7.142857 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.92596 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.945 s, 8.8 MB/s Write runtime is 13 s, speed is 7.692307 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.96846 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg341-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.04967 s, 16.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.067433 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0847267 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0811728 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216143 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.285927 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237197 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.323032 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.328399 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.362747 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.353165 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.387983 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.376389 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.390335 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.369444 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.347755 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.416306 s, 2.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114945 s, 9.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111151 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0481212 s, 21.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0645401 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0219693 s, 47.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0712249 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0973187 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0852238 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0908746 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0940254 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0924231 s, 11.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156324 s, 6.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101546 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108859 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109526 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139899 s, 7.5 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg341-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg341-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.7778 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.6525 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_ug ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg341-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.618548 s, 27.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0549625 s, 19.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0649343 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0667826 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0827514 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168845 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0744971 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149301 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115155 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.243571 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179445 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0768829 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.274681 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.267614 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.259153 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111666 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103428 s, 10.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9668d8375800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9668d8375800.lock_unused_count=4 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0900254 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0530299 s, 19.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0789637 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0946599 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0496963 s, 21.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0820386 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0674429 s, 15.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.074124 s, 14.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00222114 s, 472 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0745449 s, 14.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00887831 s, 118 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0526856 s, 19.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0776486 s, 13.5 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110338 s, 9.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0818007 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0878606 s, 11.9 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg341-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg341-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.6989 s, 12.1 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.91859 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg341-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.817976 s, 20.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0785068 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0518206 s, 20.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12785 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147431 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135488 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101927 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0853056 s, 12.3 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.265269 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0567457 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.331023 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.351089 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.297226 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.378238 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.331312 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.263584 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.4364 s, 2.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0303813 s, 34.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0402561 s, 26.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0841928 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0890649 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0357033 s, 29.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0437092 s, 24.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0560798 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0603797 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0748105 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0749053 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0598213 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0742474 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0727124 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0851612 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0929996 s, 11.3 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg341-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg341-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 12.0451 s, 8.7 MB/s Write runtime is 13 s, speed is 7.692307 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.98411 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.2862 s, 9.3 MB/s Write runtime is 13 s, speed is 7.692307 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.92011 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_policies=fifo PASS 77k (457s) == sanityn test 77kb: Check different granular TBF type combination: jobid+opcode ========================================================== 12:16:41 (1789057001) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/nrs_r_oleg341-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.817828 s, 20.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104174 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111992 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21097 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195538 s, 5.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.314157 s, 3.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.278978 s, 3.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.287366 s, 3.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.276053 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.306085 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.217872 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.285152 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.313968 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.251273 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.375243 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.289038 s, 3.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.276184 s, 3.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9668d8375800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00516198 s, 203 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116246 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112519 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.089115 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135931 s, 77.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0883108 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0710482 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0498378 s, 21.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0640868 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0484897 s, 21.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0432683 s, 24.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161247 s, 65.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0405084 s, 25.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0503455 s, 20.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0352391 s, 29.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0254714 s, 41.2 MB/s removed '/mnt/lustre/d77kb.sanityn/nrs_r_oleg341-client.virtnet' removed '/mnt/lustre/d77kb.sanityn/nrs_w_oleg341-client.virtnet' removed directory '/mnt/lustre/d77kb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.08417 s, 11.5 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.91359 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kb (50s) == sanityn test 77kc: Check different granular TBF type combination: nid+opcode ========================================================== 12:17:31 (1789057051) ost.OSS.ost_io.nrs_policies=tbf nid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.203.41@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.203.41@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.545598 s, 30.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0830745 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143359 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137111 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108698 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0638392 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0831819 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13111 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150927 s, 6.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139224 s, 7.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170062 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147606 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162188 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123918 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126306 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0748333 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124924 s, 8.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0490643 s, 21.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0518388 s, 20.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0420948 s, 24.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0517032 s, 20.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.046947 s, 22.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.033838 s, 31.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0846553 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0412424 s, 25.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0511583 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0355985 s, 29.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00601125 s, 174 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0452669 s, 23.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136002 s, 77.1 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0686506 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.06707 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00462286 s, 227 MB/s removed '/mnt/lustre/d77kc.sanityn/nrs_r_oleg341-client.virtnet' removed '/mnt/lustre/d77kc.sanityn/nrs_w_oleg341-client.virtnet' removed directory '/mnt/lustre/d77kc.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.60191 s, 13.8 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0143 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kc.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kc (50s) == sanityn test 77kd: Check different granular TBF type combination: nid+jobid ========================================================== 12:18:21 (1789057101) ost.OSS.ost_io.nrs_policies=tbf nid+jobid ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.203.41@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/nrs_r_oleg341-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.623058 s, 26.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0665397 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0684725 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182481 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213284 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221262 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.234027 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.264161 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.258313 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.315118 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.306271 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.328167 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.367989 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.393498 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.409269 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.448586 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.486516 s, 2.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9668d8375800.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff9668d8375800.lock_unused_count=2 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0887521 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0323781 s, 32.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0436381 s, 24.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0123814 s, 84.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0100307 s, 105 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00658795 s, 159 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00287495 s, 365 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113008 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00652406 s, 161 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00322277 s, 325 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109259 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166957 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116121 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0782229 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.031985 s, 32.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0340085 s, 30.8 MB/s removed '/mnt/lustre/d77kd.sanityn/nrs_w_oleg341-client.virtnet' removed '/mnt/lustre/d77kd.sanityn/nrs_r_oleg341-client.virtnet' removed directory '/mnt/lustre/d77kd.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.54739 s, 13.9 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kd.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.61643 s, 18.7 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kd.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kd (47s) == sanityn test 77ke: Check different granular TBF type combination: jobid+opcode ========================================================== 12:19:08 (1789057148) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/nrs_r_oleg341-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.729973 s, 23.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0490999 s, 21.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0482063 s, 21.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141975 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16706 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180339 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207579 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.238514 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.204773 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.249954 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.262076 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.276207 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.238409 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.271516 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.272173 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.352041 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.389756 s, 2.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0185901 s, 56.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0423873 s, 24.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0272157 s, 38.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.025567 s, 41.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0394133 s, 26.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0498314 s, 21.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0413168 s, 25.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0718857 s, 14.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0694903 s, 15.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0441071 s, 23.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0452381 s, 23.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.048109 s, 21.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0506177 s, 20.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0528722 s, 19.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0557154 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0575028 s, 18.2 MB/s removed '/mnt/lustre/d77ke.sanityn/nrs_r_oleg341-client.virtnet' removed '/mnt/lustre/d77ke.sanityn/nrs_w_oleg341-client.virtnet' removed directory '/mnt/lustre/d77ke.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.9131 s, 9.6 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ke.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.91955 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.5938 s, 13.8 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.86486 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77ke (98s) == sanityn test 77kf: Check different granular TBF type combination: uid+gid ========================================================== 12:20:46 (1789057246) ost.OSS.ost_io.nrs_policies=tbf uid+gid ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/nrs_r_oleg341-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.16916 s, 14.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0632327 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0864385 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147816 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174516 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.252833 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164008 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.301968 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138314 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173893 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0639404 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.315071 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.43943 s, 2.4 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.531175 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.506688 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.451497 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.485275 s, 2.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0476481 s, 22.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0391456 s, 26.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0560998 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0402346 s, 26.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0321585 s, 32.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0533599 s, 19.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0483435 s, 21.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0348022 s, 30.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0714799 s, 14.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0554229 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0793311 s, 13.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0448015 s, 23.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0452183 s, 23.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0473814 s, 22.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0660205 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0509815 s, 20.6 MB/s removed '/mnt/lustre/d77kf.sanityn/nrs_r_oleg341-client.virtnet' removed '/mnt/lustre/d77kf.sanityn/nrs_w_oleg341-client.virtnet' removed directory '/mnt/lustre/d77kf.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.0698 s, 5.2 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kf.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9912 s, 5.2 MB/s Read runtime is 22 s, speed is 4.545454 IOPS cleanup_tbf /mnt/lustre/d77kf.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kf (76s) == sanityn test 77kg: check different granular TBF type combination: uid+opcode ========================================================== 12:22:02 (1789057322) ost.OSS.ost_io.nrs_policies=tbf uid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg341-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.589783 s, 28.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0388992 s, 27.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0780261 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0834244 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0478985 s, 21.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174266 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21279 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.231368 s, 4.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223803 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.243791 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.255614 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.271122 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.262677 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.271349 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.317706 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.327805 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.334922 s, 3.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9668d8375800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9668d8375800.lock_unused_count=2 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0427045 s, 24.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0296693 s, 35.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0513664 s, 20.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0388779 s, 27.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0669828 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0275417 s, 38.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0349496 s, 30.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0541612 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0521217 s, 20.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111107 s, 9.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0595204 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0621123 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0495675 s, 21.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.069691 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0890392 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0641994 s, 16.3 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg341-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg341-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.49798 s, 11.0 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0006 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg341-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.949784 s, 17.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0773949 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153666 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187833 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136614 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193134 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.256698 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.338735 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.299088 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.291969 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.350848 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.267598 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.288769 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.306082 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.370038 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.26929 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.411261 s, 2.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0598159 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0913844 s, 11.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0702372 s, 14.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0967401 s, 10.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0821606 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0680624 s, 15.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115812 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129083 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0725869 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0908036 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0795055 s, 13.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0900898 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103824 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104628 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0935392 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14762 s, 7.1 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg341-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg341-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 12.4457 s, 8.4 MB/s Write runtime is 14 s, speed is 7.142857 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.94844 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 12.3463 s, 8.5 MB/s Write runtime is 13 s, speed is 7.692307 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.9698 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kg (149s) == sanityn test 77kh: Verify that Project ID support for NRS TBF rule ========================================================== 12:24:31 (1789057471) using SAVE_PROJECT_SUPPORTED=0 Stopping clients: oleg341-client.virtnet /mnt/lustre (opts:) Stopping client oleg341-client.virtnet /mnt/lustre opts: Stopping clients: oleg341-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg341-client.virtnet /mnt/lustre2 opts: Stopping /mnt/lustre-mds1 (opts:-f) on oleg341-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg341-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg341-server sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime) proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) devtmpfs on /dev type devtmpfs (rw,nosuid,mode=755) securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime) tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev) devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000) tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755) tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755) cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd) pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime) bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700) cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset) cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct) cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event) cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio) cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio) cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices) cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma) cgroup on /sys/fs/cgroup/misc type cgroup (rw,nosuid,nodev,noexec,relatime,misc) cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids) cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer) cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb) cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory) none on /sys/kernel/tracing type tracefs (rw,relatime) configfs on /sys/kernel/config type configfs (rw,relatime) /dev/nbd0 on / type ext4 (ro,relatime,stripe=32) rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M) debugfs on /sys/kernel/debug type debugfs (rw,relatime) mqueue on /dev/mqueue type mqueue (rw,relatime) none on /mnt type ramfs (rw,relatime) /dev/vda on /home/green/git/lustre-release type squashfs (ro,relatime) none on /var/lib/stateless/writable type tmpfs (rw,relatime) none on /var/cache/man type tmpfs (rw,relatime) none on /var/log type tmpfs (rw,relatime) none on /var/lib/dbus type tmpfs (rw,relatime) none on /tmp type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) none on /var/tmp type tmpfs (rw,relatime) none on /var/lib/NetworkManager type tmpfs (rw,relatime) none on /var/lib/systemd/random-seed type tmpfs (rw,relatime) none on /var/spool type tmpfs (rw,relatime) none on /var/lib/nfs type tmpfs (rw,relatime) none on /var/lib/gssproxy type tmpfs (rw,relatime) none on /var/lib/logrotate type tmpfs (rw,relatime) none on /etc type tmpfs (rw,relatime) none on /var/lib/rsyslog type tmpfs (rw,relatime) none on /root/.ssh type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) 192.168.200.253:/exports/state/oleg341-client.virtnet on /var/lib/stateless/state type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.203.41,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg341-client.virtnet/boot on /boot type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.203.41,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg341-client.virtnet/etc/kdump.conf on /etc/kdump.conf type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.203.41,local_lock=none,addr=192.168.200.253) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=378180k,mode=700) 192.168.200.253:/exports/testreports/70001/testresults/sanityn-zfs-rocky8.10_x86_64-rocky8.10_x86_64 on /tmp/testlogs type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.203.41,local_lock=none,addr=192.168.200.253) /dev/vda on /usr/sbin/mount.lustre type squashfs (ro,relatime) Checking servers environments Checking clients oleg341-client.virtnet environments Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg341-server' oleg341-server: oleg341-server.virtnet: executing load_modules_local oleg341-server: kmemleak disabled oleg341-server: Loading modules from /home/green/git/lustre-release/lustre oleg341-server: detected 4 online CPUs by sysfs oleg341-server: MODOPTS_LIBCFS= oleg341-server: Force libcfs to create 2 CPU partitions oleg341-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Setup mgs, mdt, osts Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg341-server: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg341-client: oleg341-server: ssh exited with exit code 1 Started lustre-MDT0000 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg341-server: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg341-client: oleg341-server: ssh exited with exit code 1 Started lustre-OST0000 Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg341-server: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg341-client: oleg341-server: ssh exited with exit code 1 Started lustre-OST0001 Starting client: oleg341-client.virtnet: -o user_xattr,flock 192.168.203.141@tcp:/lustre /mnt/lustre Starting client oleg341-client.virtnet: -o user_xattr,flock 192.168.203.141@tcp:/lustre /mnt/lustre Started clients oleg341-client.virtnet: 192.168.203.141@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Starting client: oleg341-client.virtnet: -o user_xattr,flock 192.168.203.141@tcp:/lustre /mnt/lustre2 Starting client oleg341-client.virtnet: -o user_xattr,flock 192.168.203.141@tcp:/lustre /mnt/lustre2 Started clients oleg341-client.virtnet: 192.168.203.141@tcp:/lustre on /mnt/lustre2 type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9668c3d5a800.idle_timeout=debug osc.lustre-OST0000-osc-ffff9668d03a8000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9668c3d5a800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9668d03a8000.idle_timeout=debug disable quota as required ost.OSS.ost_io.nrs_policies=tbf projid ost.OSS.ost_io.nrs_tbf_rule=start ext_proj_rw projid={100} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.541995 s, 31.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0720511 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110487 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133259 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194315 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.233139 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151981 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.272106 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.225211 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.299959 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222573 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.323318 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.304924 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.263974 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.333272 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.306091 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.353947 s, 3.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9668d03a8000.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffff9668d03a8000.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.224709 s, 4.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184842 s, 5.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194005 s, 5.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20314 s, 5.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.283596 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.25484 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209086 s, 5.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221005 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210805 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22441 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221879 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.225575 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.233406 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23349 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.247728 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.253972 s, 4.1 MB/s removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg341-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg341-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.3346 s, 10.1 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.2607 s, 19.9 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_proj_rw ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf projid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w projid={100}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r projid={100}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.750924 s, 22.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0898595 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119606 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.272992 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.344594 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.32042 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.291011 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.39587 s, 2.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.37376 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.399613 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.421873 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.355074 s, 3.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.340378 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.445593 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.471016 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.444748 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.407116 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0710746 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0870378 s, 12.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0853421 s, 12.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0957303 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114547 s, 9.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0574158 s, 18.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0966908 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126777 s, 8.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0730875 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0842346 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0516213 s, 20.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.054404 s, 19.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0773279 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0843298 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0801868 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0800683 s, 13.1 MB/s removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg341-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg341-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.54994 s, 13.9 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.89832 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77kh (210s) == sanityn test 77ki: Add rule with unsupported TBF type should fail for generic TBF ========================================================== 12:28:01 (1789057681) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 oleg341-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_fail projid={100}&opcode={ost_write} rate=20: Invalid argument oleg341-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_fail projid={100}&opcode={ost_write} rate=20': Invalid argument pdsh@oleg341-client: oleg341-server: ssh exited with exit code 22 ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77ki (19s) == sanityn test 77kj: Verify nodemap support for NRS TBF rule ========================================================== 12:28:21 (1789057701) On MGS 192.168.203.141, active = nodemap.active=1 On MGS 192.168.203.141, TBF = ost.OSS.ost_io.nrs_policies=tbf nodemap ost.OSS.ost_io.nrs_tbf_rule=start ext_nm_rw nodemap={TBF} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.73809 s, 22.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0855043 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0793055 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169628 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.225116 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206815 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.258208 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.278058 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.261157 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.301861 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.284459 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.325966 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.330956 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.309921 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.317144 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.334908 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.372323 s, 2.8 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9668d03a8000.lock_unused_count=3 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0474625 s, 22.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0772267 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113107 s, 9.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108467 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0803698 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0639113 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0742126 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0723906 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0935623 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0951781 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0777632 s, 13.5 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0864243 s, 12.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0895991 s, 11.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156965 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114985 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0962014 s, 10.9 MB/s removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg341-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg341-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.70127 s, 12.1 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.17467 s, 20.3 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete On MGS 192.168.203.141, TBF.id = On MGS 192.168.203.141, active = nodemap.active=0 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.683193 s, 24.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0854978 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104006 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.238019 s, 4.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220168 s, 4.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.219444 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.248365 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.291235 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.373336 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.339514 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.380282 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.359572 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.408684 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.355321 s, 3.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.446735 s, 2.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.375261 s, 2.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9668d03a8000.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffff9668d03a8000.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.454646 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00543146 s, 193 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00417412 s, 251 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0655737 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0415518 s, 25.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0680648 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0050779 s, 206 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00286731 s, 366 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00570365 s, 184 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.239296 s, 4.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162717 s, 6.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13198 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00805734 s, 130 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0858463 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187153 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168841 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0636565 s, 16.5 MB/s removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg341-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg341-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 1 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.653 s, 9.8 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 2.73432 s, 38.3 MB/s Read runtime is 4 s, speed is 25.000000 IOPS read rate (25.000000) should exceed limit (20 * 1) cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_nm_rw ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf nodemap+opcode oleg341-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_w nodemap={TBF}&opcode={ost_write} rate=20: Invalid argument oleg341-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_w nodemap={TBF}&opcode={ost_write} rate=20': Invalid argument oleg341-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_r nodemap={TBF}&opcode={ost_read} rate=10: Invalid argument oleg341-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_r nodemap={TBF}&opcode={ost_read} rate=10': Invalid argument pdsh@oleg341-client: oleg341-server: ssh exited with exit code 22 On MGS 192.168.203.141, active = nodemap.active=1 On MGS 192.168.203.141, TBF = ost.OSS.ost_io.nrs_tbf_rule=start ext_w nodemap={TBF}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nodemap={TBF}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.607575 s, 27.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0856469 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125947 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199058 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172648 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198461 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.219373 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.231024 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221423 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.262063 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.234513 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.284071 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.268956 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.320437 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.338074 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.332713 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.341038 s, 3.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9668d03a8000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9668d03a8000.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.064505 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0590589 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130005 s, 80.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0859697 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0769013 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146978 s, 71.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.056929 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166035 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120177 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16038 s, 6.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9668d03a8000.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131449 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127301 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0932747 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131614 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138305 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140006 s, 7.5 MB/s removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg341-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg341-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.1346 s, 9.4 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.96327 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo On MGS 192.168.203.141, TBF.id = On MGS 192.168.203.141, active = nodemap.active=0 PASS 77kj (190s) == sanityn test 77l: check the output of NRS policies for generic TBF ========================================================== 12:31:30 (1789057890) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_policies=fifo PASS 77l (10s) == sanityn test 77m: check NRS Delay slows write RPC processing ========================================================== 12:31:40 (1789057900) ost.OSS.ost_io.nrs_policies=delay ost.OSS.ost_io.nrs_delay_min=4 ost.OSS.ost_io.nrs_delay_max=4 ost.OSS.ost_io.nrs_delay_pct=100 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg341-client.virtnet] [bs=1M] [count=1] [oflag=direct] [conv=fdatasync] 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 3.23677 s, 324 kB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg341-client.virtnet] [bs=1M] [count=10] [oflag=direct] [conv=fdatasync] 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 40.0681 s, 262 kB/s ost.OSS.ost_io.nrs_policies=fifo PASS 77m (57s) == sanityn test 77n: check wildcard support for TBF JobID NRS policy ========================================================== 12:32:37 (1789057957) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={*.500} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.391171 s, 42.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0738615 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.071785 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133174 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0839509 s, 12.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13419 s, 7.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0824248 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191554 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174302 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129672 s, 8.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9668d03a8000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.264083 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.344732 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.36707 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.433646 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.427849 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.447561 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0120926 s, 86.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00370368 s, 283 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0097516 s, 108 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157566 s, 66.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.433592 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0241346 s, 43.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00225522 s, 465 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0024604 s, 426 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.228655 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168112 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127972 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123218 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00249375 s, 420 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167709 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0714276 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0783871 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173938 s, 60.3 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg341-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg341-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77n.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.17672 s, 11.4 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77n.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.14487 s, 20.4 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.*} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.695388 s, 24.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0662593 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0809487 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167056 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.238539 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.265301 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.26244 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.285339 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.303722 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.304299 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.330909 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.303292 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.352713 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.352792 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.329156 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.350941 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.372883 s, 2.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9668d03a8000.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff9668d03a8000.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.440169 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.420468 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.433918 s, 2.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.430877 s, 2.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.480862 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.423208 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.42967 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.433008 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.436712 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.44803 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.457398 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.441821 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.462948 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.450297 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.484833 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.457223 s, 2.3 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg341-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg341-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.64432 s, 13.7 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.11075 s, 20.5 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_policies=fifo PASS 77n (90s) == sanityn test 77o: Changing rank should not panic ====== 12:34:07 (1789058047) mds.MDS.mdt.nrs_policies=tbf nid mds.MDS.mdt.nrs_tbf_rule=start name nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=start name1 nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=change name1 rank=name mds.MDS.mdt.nrs_tbf_rule=stop name mds.MDS.mdt.nrs_policies=fifo PASS 77o (10s) == sanityn test 77q: Parallel TBF rule definitions should not panic ========================================================== 12:34:17 (1789058057) mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_policies=fifo PASS 77q (116s) == sanityn test 77p: Check validity of rule names for TBF policies ========================================================== 12:36:13 (1789058173) mds.MDS.mdt.nrs_policies=tbf oleg341-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_overflo uid={500} rate=500: File name too long oleg341-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_overflo uid={500} rate=500': File name too long pdsh@oleg341-client: oleg341-server: ssh exited with exit code 36 oleg341-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start uid={500} rate=500: Invalid argument oleg341-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start uid={500} rate=500': Invalid argument pdsh@oleg341-client: oleg341-server: ssh exited with exit code 22 oleg341-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_empty: Invalid argument oleg341-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_empty': Invalid argument pdsh@oleg341-client: oleg341-server: ssh exited with exit code 22 oleg341-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p@spec uid={500} rate=500: Invalid argument oleg341-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p@spec uid={500} rate=500': Invalid argument pdsh@oleg341-client: oleg341-server: ssh exited with exit code 22 oleg341-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p.spec uid={500} rate=500: Invalid argument oleg341-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p.spec uid={500} rate=500': Invalid argument pdsh@oleg341-client: oleg341-server: ssh exited with exit code 22 oleg341-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p~spec uid={500} rate=500: Invalid argument oleg341-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p~spec uid={500} rate=500': Invalid argument pdsh@oleg341-client: oleg341-server: ssh exited with exit code 22 oleg341-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p#spec uid={500} rate=500: Invalid argument oleg341-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p#spec uid={500} rate=500': Invalid argument pdsh@oleg341-client: oleg341-server: ssh exited with exit code 22 oleg341-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p/spec uid={500} rate=500: Invalid argument oleg341-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p/spec uid={500} rate=500': Invalid argument pdsh@oleg341-client: oleg341-server: ssh exited with exit code 22 oleg341-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p^spec uid={500} rate=500: Invalid argument oleg341-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p^spec uid={500} rate=500': Invalid argument pdsh@oleg341-client: oleg341-server: ssh exited with exit code 22 oleg341-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p%spec uid={500} rate=500: Invalid argument oleg341-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p%spec uid={500} rate=500': Invalid argument pdsh@oleg341-client: oleg341-server: ssh exited with exit code 22 oleg341-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p*spec uid={500} rate=500: Invalid argument oleg341-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p*spec uid={500} rate=500': Invalid argument pdsh@oleg341-client: oleg341-server: ssh exited with exit code 22 oleg341-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p;spec uid={500} rate=500: Invalid argument oleg341-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p;spec uid={500} rate=500': Invalid argument pdsh@oleg341-client: oleg341-server: ssh exited with exit code 22 oleg341-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p,spec uid={500} rate=500: Invalid argument oleg341-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p,spec uid={500} rate=500': Invalid argument pdsh@oleg341-client: oleg341-server: ssh exited with exit code 22 oleg341-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p?spec uid={500} rate=500: Invalid argument oleg341-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p?spec uid={500} rate=500': Invalid argument pdsh@oleg341-client: oleg341-server: ssh exited with exit code 22 oleg341-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77pspec uid={500} rate=500: Invalid argument oleg341-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p>spec uid={500} rate=500': Invalid argument pdsh@oleg341-client: oleg341-server: ssh exited with exit code 22 oleg341-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p:spec uid={500} rate=500: Invalid argument oleg341-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p:spec uid={500} rate=500': Invalid argument pdsh@oleg341-client: oleg341-server: ssh exited with exit code 22 oleg341-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p+spec uid={500} rate=500: Invalid argument oleg341-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p+spec uid={500} rate=500': Invalid argument pdsh@oleg341-client: oleg341-server: ssh exited with exit code 22 oleg341-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p=spec uid={500} rate=500: Invalid argument oleg341-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p=spec uid={500} rate=500': Invalid argument pdsh@oleg341-client: oleg341-server: ssh exited with exit code 22 oleg341-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p)spec uid={500} rate=500: Invalid argument oleg341-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p)spec uid={500} rate=500': Invalid argument pdsh@oleg341-client: oleg341-server: ssh exited with exit code 22 oleg341-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p(spec uid={500} rate=500: Invalid argument oleg341-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p(spec uid={500} rate=500': Invalid argument pdsh@oleg341-client: oleg341-server: ssh exited with exit code 22 oleg341-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p{spec uid={500} rate=500: Invalid argument oleg341-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p{spec uid={500} rate=500': Invalid argument pdsh@oleg341-client: oleg341-server: ssh exited with exit code 22 oleg341-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p}spec uid={500} rate=500: Invalid argument oleg341-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p}spec uid={500} rate=500': Invalid argument pdsh@oleg341-client: oleg341-server: ssh exited with exit code 22 oleg341-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p|spec uid={500} rate=500: Invalid argument oleg341-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p|spec uid={500} rate=500': Invalid argument pdsh@oleg341-client: oleg341-server: ssh exited with exit code 22 oleg341-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p[spec uid={500} rate=500: Invalid argument oleg341-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p[spec uid={500} rate=500': Invalid argument pdsh@oleg341-client: oleg341-server: ssh exited with exit code 22 oleg341-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p]spec uid={500} rate=500: Invalid argument oleg341-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p]spec uid={500} rate=500': Invalid argument pdsh@oleg341-client: oleg341-server: ssh exited with exit code 22 oleg341-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p!spec uid={500} rate=500: Invalid argument oleg341-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p!spec uid={500} rate=500': Invalid argument pdsh@oleg341-client: oleg341-server: ssh exited with exit code 22 oleg341-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p&spec uid={500} rate=500: Invalid argument oleg341-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p&spec uid={500} rate=500': Invalid argument pdsh@oleg341-client: oleg341-server: ssh exited with exit code 22 oleg341-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p$spec uid={500} rate=500: Invalid argument oleg341-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p$spec uid={500} rate=500': Invalid argument pdsh@oleg341-client: oleg341-server: ssh exited with exit code 22 oleg341-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p`spec uid={500} rate=500: Invalid argument oleg341-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p`spec uid={500} rate=500': Invalid argument pdsh@oleg341-client: oleg341-server: ssh exited with exit code 22 oleg341-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p\spec uid={500} rate=500: Invalid argument oleg341-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p\spec uid={500} rate=500': Invalid argument pdsh@oleg341-client: oleg341-server: ssh exited with exit code 22 mds.MDS.mdt.nrs_policies=fifo PASS 77p (32s) == sanityn test 77r: Change type of tbf policy at run time ========================================================== 12:36:45 (1789058205) mds.MDS.mdt.nrs_policies=tbf uid mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf gid mds.MDS.mdt.nrs_tbf_rule=start md_rule gid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start md_rule jobid={*.500} rate=1 mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=fifo check the number of requests in queue: fifo: 0 crrn: 0 tbf: 0 delay: 0 fifo: 0 crrn: 0 tbf: 0 delay: 0 cleanup 77r 273376 kill md_thread xargs PASS 77r (53s) == sanityn test 77s: Check TBF LRU shrinker ============== 12:37:38 (1789058258) jobid_name=%e.%p jobid_var=TEST_77s_JOBVAR mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start touch jobid={touch.*} rate=1000 mds.MDS.mdt.nrs_tbf_rule=start unlink jobid={unlink.*} rate=1000 touch jobid={touch.*} 1000, ref 9 touch jobid={touch.*} 1000, ref 0 unlink jobid={unlink.*} 1000, ref 9 unlink jobid={unlink.*} 1000, ref 0 mds.MDS.mdt.nrs_policies=fifo jobid_var=procname_uid jobid_name=%e.%u PASS 77s (63s) == sanityn test 78: Enable policy and specify tunings right away ========================================================== 12:38:42 (1789058322) ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_orr_quantum=1 ost.OSS.ost_io.nrs_policies=fifo PASS 78 (10s) == sanityn test 79: xattr: intent error ================== 12:38:52 (1789058332) File: /mnt/lustre/d79.sanityn Size: 16384 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115238826935667 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-10 12:38:54.519444044 -0400 Modify: 2026-09-10 12:38:54.519444044 -0400 Change: 2026-09-10 12:38:54.602978244 -0400 Birth: 2026-09-10 12:38:54.632000000 -0400 fail_val=0 fail_loc=0x80000160 fail_val=0 fail_loc=0x80000131 # file: mnt/lustre/d79.sanityn trusted.name1="value1" PASS 79 (19s) == sanityn test 80a: migrate directory when some children is being opened ========================================================== 12:39:11 (1789058351) SKIP: sanityn test_80a needs >= 2 MDTs SKIP 80a (3s) == sanityn test 80b: Accessing directory during migration ========================================================== 12:39:14 (1789058354) SKIP: sanityn test_80b needs >= 2 MDTs SKIP 80b (3s) == sanityn test 81a: rename and stat under striped directory ========================================================== 12:39:17 (1789058357) SKIP: sanityn test_81a needs >= 2 MDTs SKIP 81a (3s) == sanityn test 81b: rename under striped directory doesn't deadlock ========================================================== 12:39:20 (1789058360) SKIP: sanityn test_81b We need at least 2 MDTs for this test SKIP 81b (3s) == sanityn test 81c: rename revoke LOOKUP lock for remote object ========================================================== 12:39:23 (1789058363) SKIP: sanityn test_81c needs >= 4 MDTs SKIP 81c (3s) == sanityn test 81d: parallel rename file cross-dir on same MDT ========================================================== 12:39:26 (1789058366) - open/close 741 (time 1789058381.12 total 10.00 last 74.08) - open/close 1514 (time 1789058391.13 total 20.01 last 77.22) - open/close 2333 (time 1789058401.13 total 30.02 last 81.89) total: 2500 open/close in 32.00 seconds: 78.12 ops/second snapshot_time 1789058495.277310047 secs.nsecs start_time 1789058369.054774205 secs.nsecs elapsed_time 126.222535842 secs.nsecs open 2505 samples [usecs] 118 19145 3513637 8376758991 close 2504 samples [usecs] 152 6681 840331 578854533 mknod 2501 samples [usecs] 533 18919 3076340 6723847890 unlink 2500 samples [usecs] 758 62433 4876236 23666126042 mkdir 2 samples [usecs] 942 5558 6500 31778728 rmdir 2 samples [usecs] 13726 17552 31278 496475780 rename 2500 samples [usecs] 477 42326 16486913 227707351851 getattr 3760 samples [usecs] 9 9708 272689 437956851 parallel_rename_file 2500 samples [usecs] 477 42326 16486913 227707351851 crossdir_rename 2500 samples [usecs] 477 42326 16486913 227707351851 readpage 6 samples [usecs] 4600 34480 75344 1579571918 PASS 81d (133s) == sanityn test 82: fsetxattr and fgetxattr on orphan files ========================================================== 12:41:39 (1789058499) multiop /mnt/lustre/f82.sanityn vO_ac TMPPIPE=/tmp/multiop_open_wait_pipe.7620 multiop /mnt/lustre2/f82.sanityn vo_Ac TMPPIPE=/tmp/multiop_open_wait_pipe.7620 PASS 82 (6s) == sanityn test 83: access striped directory while it is being created/unlinked ========================================================== 12:41:45 (1789058505) SKIP: sanityn test_83 needs >= 2 MDTs SKIP 83 (3s) == sanityn test 84: 0-nlink race in lu_object_find() ===== 12:41:48 (1789058508) fail_loc=0x8000060b PASS 84 (12s) == sanityn test 85: Lustre API root cache race =========== 12:42:00 (1789058520) Starting: llapi_root_test /mnt/lustre/f85.sanityn /mnt/lustre2/f85.sanityn Starting test test1 at 1789058522 Finishing test test1 at 1789058526 PASS 85 (11s) == sanityn test 90: open/create and unlink striped directory ========================================================== 12:42:12 (1789058532) SKIP: sanityn test_90 needs >= 2 MDTs SKIP 90 (3s) == sanityn test 91: chmod and unlink striped directory === 12:42:15 (1789058535) SKIP: sanityn test_91 needs >= 2 MDTs SKIP 91 (4s) == sanityn test 92: create remote directory under orphan directory ========================================================== 12:42:19 (1789058539) SKIP: sanityn test_92 needs >= 2 MDTs SKIP 92 (4s) == sanityn test 93: alloc_rr should not allocate on same ost ========================================================== 12:42:23 (1789058543) fail_loc=0x00000163 fail_loc=0x0 /mnt/lustre/f93.sanityn-1/file1 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5794 0x16a2 0x240000400 1 5753 0x1679 0x280000400 /mnt/lustre2/f93.sanityn-2/file2 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 5754 0x167a 0x280000400 0 5795 0x16a3 0x240000400 rc1=0 and rc2=0 PASS 93 (22s) == sanityn test 94: signal vs CP callback race =========== 12:42:45 (1789058565) osc.lustre-OST0000-osc-ffff9668c3d5a800.idle_timeout=0 osc.lustre-OST0000-osc-ffff9668d03a8000.idle_timeout=0 osc.lustre-OST0001-osc-ffff9668c3d5a800.idle_timeout=0 osc.lustre-OST0001-osc-ffff9668d03a8000.idle_timeout=0 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.231753 s, 35.3 kB/s fail_val=5 fail_loc=0x80000312 fail_val=6 fail_loc=0x80000329 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 3.06693 s, 1.3 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 6389: 290455 Alarm clock dd if=/dev/zero of=$DIR/$tfile conv=notrunc oflag=append bs=4k count=1 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0827715 s, 49.5 kB/s osc.lustre-OST0000-osc-ffff9668c3d5a800.idle_timeout=debug osc.lustre-OST0000-osc-ffff9668d03a8000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9668c3d5a800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9668d03a8000.idle_timeout=debug PASS 94 (22s) == sanityn test 95a: Check readpage() on a page that was removed from page cache ========================================================== 12:43:07 (1789058587) llite.lustre-ffff9668c3d5a800.fast_read=0 llite.lustre-ffff9668d03a8000.fast_read=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.133412 s, 15.7 MB/s fail_loc=0x80001421 fail_val=7 fail_loc=0x80001422 fail_val=10 llite.lustre-ffff9668c3d5a800.fast_read=1 llite.lustre-ffff9668d03a8000.fast_read=1 PASS 95a (22s) == sanityn test 95b: Check readpage() on a page that is no longer uptodate ========================================================== 12:43:29 (1789058609) llite.lustre-ffff9668c3d5a800.fast_read=0 llite.lustre-ffff9668d03a8000.fast_read=0 1+0 records in 1+0 records out 12288 bytes (12 kB, 12 KiB) copied, 0.0117864 s, 1.0 MB/s fail_loc=0x80001424 fail_val=5 fail_loc=0 llite.lustre-ffff9668c3d5a800.fast_read=1 llite.lustre-ffff9668d03a8000.fast_read=1 PASS 95b (15s) == sanityn test 100a: DoM: glimpse RPCs for stat without IO lock (DoM only file) ========================================================== 12:43:44 (1789058624) SKIP: sanityn test_100a Reserved for glimpse-ahead SKIP 100a (3s) == sanityn test 100b: DoM: no glimpse RPC for stat with IO lock (DoM only file) ========================================================== 12:43:47 (1789058627) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00469814 s, 872 kB/s /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK PASS 100b (8s) == sanityn test 100c: DoM: write vs stat without IO lock (combined file) ========================================================== 12:43:56 (1789058636) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.293977 s, 7.1 MB/s /mnt/lustre/d100c.sanityn/dom has type file OK /mnt/lustre/d100c.sanityn/dom has size 2097152 OK PASS 100c (8s) == sanityn test 100d: DoM: write+truncate vs stat without IO lock (combined file) ========================================================== 12:44:04 (1789058644) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.178091 s, 11.8 MB/s /mnt/lustre/d100d.sanityn/dom has type file OK /mnt/lustre/d100d.sanityn/dom has size 4096 OK PASS 100d (7s) == sanityn test 100e: DoM: read on open and file size ==== 12:44:11 (1789058651) 1+0 records in 1+0 records out 12000 bytes (12 kB, 12 KiB) copied, 0.00902906 s, 1.3 MB/s PASS 100e (7s) == sanityn test 101a: Discard DoM data on unlink ========= 12:44:18 (1789058658) /mnt/lustre/f101a.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0281094 s, 146 kB/s PASS 101a (8s) == sanityn test 101b: Discard DoM data on rename ========= 12:44:26 (1789058666) /mnt/lustre/f101b.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0100153 s, 409 kB/s PASS 101b (8s) == sanityn test 101c: Discard DoM data on close-unlink === 12:44:34 (1789058674) /mnt/lustre/f101c.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0183653 s, 223 kB/s PASS 101c (8s) == sanityn test 102: Test open by handle of unlinked file ========================================================== 12:44:42 (1789058682) file: /mnt/lustre/f102.sanityn fh_bytes: 32 fh_type: 151 fh_data: d1 0b 00 00 02 00 00 00 54 0f 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 file's parent FID is [0:0x0:0x0] file FID is [0x200000bd1:0xf54:0x0] access via mount point '/mnt/lustre2' - OK access after unlink - OK check_fhandle_syscalls test Passed! PASS 102 (7s) == sanityn test 103: Test size correctness with lockahead ========================================================== 12:44:49 (1789058689) fail_loc=0x415 fail_loc=0x415 fail_val=2 Incorrect size expected (no glimpse fix): Starting test test23 at 1789058692 lockahead_test: lockahead_test.c:1171: test23: assertion 'sb.st_size == sb2.st_size' failed: size on /mnt/lustre/d103.sanityn/f103.sanityn and /mnt/lustre2/d103.sanityn/f103.sanityn differs: 0 vs 1048576 fail_loc=0 fail_loc=0x214 fail_val=2 Starting test test23 at 1789058696 Finishing test test23 at 1789058696 PASS 103 (13s) == sanityn test 104: Verify that MDS stores atime/mtime/ctime during close ========================================================== 12:45:03 (1789058703) SKIP: sanityn test_104 ldiskfs only test SKIP 104 (3s) == sanityn test 105: Glimpse and lock cancel race ======== 12:45:06 (1789058706) fail_loc=0x416 PASS 105 (28s) == sanityn test 106a: Verify the btime via statx() ======= 12:45:34 (1789058734) SKIP: sanityn test_106a Test only for ldiskfs and statx() supported SKIP 106a (4s) == sanityn test 106b: Glimpse RPCs test for statx ======== 12:45:38 (1789058738) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00864064 s, 119 kB/s File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238826938203 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-10 12:45:40.956833729 -0400 Modify: 2026-09-10 12:45:41.057810927 -0400 Change: 2026-09-10 12:45:41.057810927 -0400 Birth: 2026-09-10 12:45:41.066000000 -0400 /mnt/lustre/f106b.sanityn 144115238826938203 -rw-r--r-- 1 0 0 1789058741 1789058740 1789058741 File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238826938203 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-10 12:45:40.956833729 -0400 Modify: 2026-09-10 12:45:41.057810927 -0400 Change: 2026-09-10 12:45:41.057810927 -0400 Birth: 2026-09-10 12:45:41.066000000 -0400 1789058741 1024 1 PASS 106b (9s) == sanityn test 106c: Verify statx attributes mask ======= 12:45:47 (1789058747) PASS 106c (7s) == sanityn test 107a: Basic grouplock conflict =========== 12:45:54 (1789058754) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.22411 s, 46.8 MB/s /mnt/lustre/f107a.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 5761 0x1681 0x280000400 multiop /mnt/lustre/f107a.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7620 multiop /mnt/lustre2/f107a.sanityn vO_G16022000r10g16022000c TMPPIPE=/tmp/multiop_open_wait_pipe.7620 First grouplock blocks second one PASS 107a (10s) == sanityn test 107b: Grouplock is added to the head of waiting list ========================================================== 12:46:04 (1789058764) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.267658 s, 39.2 MB/s /mnt/lustre/f107b.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5804 0x16ac 0x240000400 multiop /mnt/lustre/f107b.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7620 Grouplock blocks IO First grouplock blocks second one Second grouplock blocks IO PASS 107b (14s) == sanityn test 108a: lseek: parallel updates ============ 12:46:18 (1789058778) 1+0 records in 1+0 records out 10000 bytes (10 kB, 9.8 KiB) copied, 0.00494042 s, 2.0 MB/s llite.lustre-ffff9668c3d5a800.hybrid_io=0 llite.lustre-ffff9668d03a8000.hybrid_io=0 fail_val=4 fail_loc=0x80000414 1+0 records in 1+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 4.86971 s, 1.7 MB/s PASS 108a (14s) == sanityn test 109: Race with several mount instances on 1 node ========================================================== 12:46:32 (1789058792) 192.168.203.141@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg341-client.virtnet /mnt/lustre (opts:) 192.168.203.141@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg341-client.virtnet /mnt/lustre2 (opts:) Starting race between client mounts (duration 30) Iteration 0 Starting client: oleg341-client.virtnet: -o user_xattr,flock 192.168.203.141@tcp:/lustre /mnt/lustre Starting client: oleg341-client.virtnet: -o user_xattr,flock 192.168.203.141@tcp:/lustre /mnt/lustre2 192.168.203.141@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.141@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg341-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg341-client.virtnet /mnt/lustre (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg341-server' oleg341-server: oleg341-server.virtnet: executing load_modules_local oleg341-server: kmemleak disabled oleg341-server: Loading modules from /home/green/git/lustre-release/lustre oleg341-server: detected 4 online CPUs by sysfs oleg341-server: MODOPTS_LIBCFS= oleg341-server: Force libcfs to create 2 CPU partitions oleg341-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Starting client: oleg341-client.virtnet: -o user_xattr,flock 192.168.203.141@tcp:/lustre /mnt/lustre Starting client: oleg341-client.virtnet: -o user_xattr,flock 192.168.203.141@tcp:/lustre /mnt/lustre2 PASS 109 (47s) == sanityn test 110: do not grant another lock on resend ========================================================== 12:47:19 (1789058839) fail_loc=0x525 fail_val=3 fail_loc=0x534 File: /mnt/lustre/f110.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238860488707 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-10 12:47:21.618000000 -0400 Modify: 2026-09-10 12:47:21.618000000 -0400 Change: 2026-09-10 12:47:21.618000000 -0400 Birth: 2026-09-10 12:47:21.707000000 -0400 fail_loc=0 fail_val=0 Connected clients: oleg341-client.virtnet oleg341-client.virtnet PASS 110 (168s) == sanityn test 111: A racy rename/link an open file should not cause fs corruption ========================================================== 12:50:07 (1789059007) SKIP: sanityn test_111 needs >= 2 MDTs SKIP 111 (4s) == sanityn test 112: update max-inherit in default LMV === 12:50:11 (1789059011) SKIP: sanityn test_112 We need at least 2 MDTs for this test SKIP 112 (4s) == sanityn test 113: check servers of specified fs ======= 12:50:15 (1789059015) lustre-OST0000-osc-ffff9668d1cd5800 active. lustre-OST0001-osc-ffff9668d1cd5800 active. lustre-OST0000-osc-ffff9668d1cd5800 active. lustre-OST0001-osc-ffff9668d1cd5800 active. MGC192.168.203.141@tcp active. PASS 113 (7s) == sanityn test 114: implicit default LMV inherit ======== 12:50:22 (1789059022) SKIP: sanityn test_114 We need at least 2 MDTs for this test SKIP 114 (3s) == sanityn test 115: ldiskfs doesn't check direntry for uniqueness ========================================================== 12:50:25 (1789059025) SKIP: sanityn test_115 ldiskfs only test SKIP 115 (3s) == sanityn test 116: DNE: Set default LMV layout from a remote client ========================================================== 12:50:28 (1789059028) SKIP: sanityn test_116 needs >= 2 MDTs SKIP 116 (2s) == sanityn test 121: trunc append race =================== 12:50:31 (1789059031) 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0186222 s, 1.1 MB/s fail_loc=0x80001436 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0549447 s, 373 kB/s 0000000 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0005000 0000 0000 0000 0000 0000 0000 0000 0000 * 000c800 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0011800 PASS 121 (8s) == sanityn test 122: directory size is consistent across mounts ========================================================== 12:50:40 (1789059040) PASS 122 (9s) == sanityn test 200: service remains healthy while able to process request ========================================================== 12:50:49 (1789059049) at_max=10 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=9 fail_loc=0x1e1 fail_val=20 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1461 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.2041 s, 0.2 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1709 s, 0.2 kB/s 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.218 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1427 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1898 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1308 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.2526 s, 0.1 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.3289 s, 0.1 kB/s4096 bytes (4.1 kB, 4.0 KiB) copied, 40.3657 s, 0.1 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.4059 s, 0.1 kB/s 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.3299 s, 0.1 kB/s 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.3471 s, 0.1 kB/s 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.3216 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.4235 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.3627 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.5069 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.3464 s, 0.1 kB/s at_max=600 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=116 oleg341-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9668d0c85800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9668d0c85800.ost_server_uuid in IDLE state after 0 sec PASS 200 (82s) cleanup: ====================================================== == sanityn test complete, duration 9964 sec ============== 12:52:12 (1789059132) === sanityn: start cleanup 12:52:14 (1789059134) === Stopping clients: oleg341-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg341-client.virtnet /mnt/lustre2 opts: === sanityn: finish cleanup 12:56:36 (1789059396) ===