-----============= acceptance-small: sanityn ============----- Wed Sep 9 01:37:32 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg645-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanityn.*ex': No such file or directory excepting tests: 27 40a skipping tests SLOW=no: 33a === sanityn: start setup 01:37:53 (1788932273) === Starting client oleg645-client.virtnet: -o user_xattr,flock 192.168.206.145@tcp:/lustre /mnt/lustre2 Mount client oleg645-client.virtnet: mount -t lustre -o user_xattr,flock 192.168.206.145@tcp:/lustre /mnt/lustre2 Started clients oleg645-client.virtnet: 192.168.206.145@tcp:/lustre on /mnt/lustre2 type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) oleg645-client.virtnet: executing check_config_client /mnt/lustre oleg645-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg645-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9fbb42a51000.idle_timeout=debug osc.lustre-OST0000-osc-ffff9fbb50d9c000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9fbb42a51000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9fbb50d9c000.idle_timeout=debug disable quota as required oleg645-server: oleg645-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanityn: finish setup 01:38:24 (1788932304) === 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.025415 s, 41.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7610] == sanityn test 0: do_node_vp() and do_facet_vp() do the right thing ========================================================== 01:38:26 (1788932306) PASS 0 (7s) == sanityn test 1: Check attribute updates on 2 mount points ========================================================== 01:38:33 (1788932313) /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0777 OK /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0666 OK /mnt/lustre/f1.sanityn: absent OK PASS 1 (7s) == sanityn test 2a: check cached attribute updates on 2 mtpt's ================================================================== 01:38:40 (1788932320) -rw-r--r-- 1 root root 0 Sep 9 01:38 /mnt/lustre2/f2a /mnt/lustre/f2a has type file OK /mnt/lustre/f2a has perms 0777 OK PASS 2a (7s) == sanityn test 2b: check cached attribute updates on 2 mtpt's ================================================================== 01:38:47 (1788932327) -rw-r--r-- 1 root root 0 Sep 9 01:38 /mnt/lustre2/f2b /mnt/lustre2/f2b has type file OK /mnt/lustre2/f2b has perms 0777 OK PASS 2b (6s) == sanityn test 2c: check cached attribute updates on 2 mtpt's root ============================================================= 01:38:53 (1788932333) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0777 OK PASS 2c (7s) == sanityn test 2d: check cached attribute updates on 2 mtpt's root ============================================================= 01:39:00 (1788932340) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0755 OK PASS 2d (6s) == sanityn test 2e: check chmod on root is propagated to others ========================================================== 01:39:07 (1788932347) total 1 -rwxrwxrwx 1 root root 0 Sep 9 01:38 f2a -rwxrwxrwx 1 root root 0 Sep 9 01:38 f2b total 1 -rwxrwxrwx 1 root root 0 Sep 9 01:38 f2a -rwxrwxrwx 1 root root 0 Sep 9 01:38 f2b running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre2/f2e.sanityn] [count=1] 1+0 records in 1+0 records out 512 bytes copied, 0.0109523 s, 46.7 kB/s PASS 2e (6s) == sanityn test 2f: check attr/owner updates on DNE with 2 mtpt's ========================================================== 01:39:12 (1788932352) SKIP: sanityn test_2f needs >= 2 MDTs SKIP 2f (3s) == sanityn test 2g: check blocks update on sync write ==== 01:39:16 (1788932356) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.374138 s, 5.6 MB/s /mnt/lustre/f2g.sanityn has 4103 blocks /mnt/lustre2/f2g.sanityn has 4103 blocks PASS 2g (6s) == sanityn test 3: symlink on one mtpt, readlink on another ===================================================================== 01:39:22 (1788932362) PASS 3 (6s) == sanityn test 4: fstat validation on multiple mount points ==================================================================== 01:39:28 (1788932368) PASS 4 (7s) == sanityn test 5: create a file on one mount, truncate it on the other ========================================================== 01:39:36 (1788932376) /mnt/lustre/f5 has type file OK /mnt/lustre/f5 has size 100 OK PASS 5 (6s) == sanityn test 6: remove of open file on other node ============================================================================ 01:39:42 (1788932382) opening writing unlinking /mnt/lustre2/f6.sanityn accessing (1) seeking (1) accessing (2) fstat... reading comparing data truncating seeking (2) writing again seeking (3) reading again comparing data again closing SUCCESS - goto beer PASS 6 (7s) == sanityn test 7: remove of open directory on other node ======================================================================= 01:39:49 (1788932389) creating directory /mnt/lustre/d7 opening directory unlinking /mnt/lustre/d7 Ok, everything goes well. PASS 7 (8s) == sanityn test 8: remove of open special file on other node ==================================================================== 01:39:57 (1788932397) creating special file /mnt/lustre/f8.sanityn opening file unlinking /mnt/lustre/f8.sanityn Ok, everything goes well. PASS 8 (9s) == sanityn test 9a: append of file with sub-page size on multiple mounts ========================================================== 01:40:06 (1788932406) PASS 9a (8s) == sanityn test 9b: append to striped sparse file ======== 01:40:14 (1788932414) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0516654 s, 20.3 MB/s 3+0 records in 3+0 records out 3 bytes copied, 0.0770414 s, 0.0 kB/s Data read (expecting 'foo'): foo PASS 9b (7s) == sanityn test 10a: write of file with sub-page size on multiple mounts ========================================================== 01:40:21 (1788932421) 1+0 records in 1+0 records out 1 byte copied, 0.00840212 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0139131 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0121282 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.010245 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00940843 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0118843 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0147226 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0113265 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0117503 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0117467 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0077455 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0219136 s, 0.0 kB/s PASS 10a (8s) == sanityn test 10b: write of file with sub-page size on multiple mounts ========================================================== 01:40:29 (1788932429) 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.0274997 s, 112 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0518716 s, 79.0 kB/s 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.000210472 s, 14.6 MB/s PASS 10b (6s) == sanityn test 11: execution of file opened for write should return error ============================================================== 01:40:35 (1788932435) multiop /mnt/lustre/d11/f vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7610 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 298: /mnt/lustre2/d11/f: Text file busy PASS 11 (7s) == sanityn test 12: test lock ordering (link, stat, unlink) ========================================================== 01:40:42 (1788932442) warning: '-runtime' deprecated, use '-t runtime' instead - link 389 (time 1788932454.88 total 10.01 last 38.85) - link 758 (time 1788932464.92 total 20.05 last 36.78) - link 1136 (time 1788932474.94 total 30.07 last 37.70) - link 1470 (time 1788932484.96 total 40.09 last 33.33) - link 1782 (time 1788932494.98 total 50.11 last 31.16) - link 2166 (time 1788932504.99 total 60.12 last 38.35) - link 2512 (time 1788932515.02 total 70.15 last 34.51) - link 2906 (time 1788932525.02 total 80.15 last 39.37) - link 3314 (time 1788932535.04 total 90.17 last 40.75) total: 3677 link in 100.01 seconds: 36.77 ops/second using seed 2699722591 running for 100 seconds - stat 5426 (time 1788932456 ; total 11 ; last 11) - stat 10000 (time 1788932464 ; total 19 ; last 8) - stat 15642 (time 1788932475 ; total 30 ; last 11) - stat 20000 (time 1788932483 ; total 38 ; last 8) - stat 24623 (time 1788932494 ; total 49 ; last 11) - stat 27986 (time 1788932505 ; total 60 ; last 11) - stat 30000 (time 1788932512 ; total 67 ; last 7) - stat 32925 (time 1788932523 ; total 78 ; last 11) - stat 36454 (time 1788932534 ; total 89 ; last 11) - stat 38901 (time 1788932545 ; total 100 ; last 11) total: 38902 stats in 100 seconds: 389.019989 stats/second - unlinked 0 (time 1788932495 ; total 0 ; last 0) unlink(/mnt/lustre2/lockdir/lockfile3677) error: No such file or directory total: 3676 unlinks in 91 seconds: 40.395603 unlinks/second /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 77: kill: (21416) - No such process /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 78: kill: (21418) - No such process PASS 12 (149s) == sanityn test 13: test directory page revocation ======= 01:43:11 (1788932591) f13 f13 PASS 13 (7s) == sanityn test 14aa: execution of file open for write returns -ETXTBSY ========================================================== 01:43:19 (1788932599) multiop /mnt/lustre/d14aa.sanityn/f14aa.sanityn vOw_c TMPPIPE=/tmp/multiop_open_wait_pipe.7610 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 333: /mnt/lustre2/d14aa.sanityn/f14aa.sanityn: Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5116: 22605 Terminated $MULTIOP_PROG $FILE v$ARGS > $TMPPIPE (wd: ~) PASS 14aa (6s) == sanityn test 14ab: open(RDWR) of executing file returns -ETXTBSY ========================================================== 01:43:25 (1788932605) open(O_RDWR|O_CREAT): Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5116: 23184 Terminated $DIR1/$tdir/sleep 60 (wd: ~) PASS 14ab (6s) == sanityn test 14b: truncate of executing file returns -ETXTBSY ================================================================ 01:43:31 (1788932611) truncate: cannot truncate '/mnt/lustre2/d14b.sanityn/sleep' to length 60: Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 348: 23762 Terminated $DIR1/$tdir/sleep 60 PASS 14b (5s) == sanityn test 14c: open(O_TRUNC) of executing file return -ETXTBSY ============================================================ 01:43:36 (1788932616) cp: cannot create regular file '/mnt/lustre2/d14c.sanityn/sleep': Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 360: 24345 Terminated $DIR1/$tdir/sleep 60 PASS 14c (6s) == sanityn test 14d: chmod of executing file is still possible ================================================================== 01:43:42 (1788932622) chmod /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 371: 24928 Terminated $DIR1/$tdir/sleep 60 PASS 14d (8s) == sanityn test 15: test out-of-space with multiple writers ===================================================================== 01:43:50 (1788932630) PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin: MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky sleep 5 for ZFS MDS Waiting for MDT destroys to complete fallocate on zfs doesn't consume space STRIPECOUNT=2 ORIGFREE=7531520 MAXFREE=800000 SKIP: oos2 test_15 oos2.sh: 7531520KB free > 800000KB, increase MAXFREE (or reduce fs size) sleep 5 for ZFS MDS Waiting for MDT destroys to complete SKIP 15 (52s) == sanityn test 16a: 500 iterations of dual-mount fsx ==== 01:44:42 (1788932682) keep default fallocate mode: 2 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.327176 s, 25.6 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.321778 s, 26.1 MB/s Chance of close/open is 1 in 50 Seed set to 2687 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000001[1] 1788932687.889378 mapwrite 0x65a11f thru 0x667820 (0xd702 bytes) 000002[1] 1788932688.074749 punch from 0x52ac1c to 0x52b927 (0xd0c bytes) truncating to largest ever: 0x437e28 000003[0] 1788932688.107888 trunc from 0x667821 to 0x437e27 (0x22f9f9 bytes) 000004[0] 1788932688.150797 mapread 0x18f85a thru 0x1961e3 (0x698a bytes) truncating to largest ever: 0x970022 000005[0] 1788932688.196421 trunc from 0x437e28 to 0x970021 (0x5381fa bytes) 000006[0] 1788932688.255379 read 0x3d7c21 thru 0x3e47ac (0xcb8c bytes) 000007[1] 1788932688.303208 write 0x8681e7 thru 0x86f27f (0x7099 bytes) 000008[1] 1788932688.344188 read 0x939064 thru 0x941a6f (0x8a0c bytes) 000009[1] 1788932688.371381 read 0x69bd7d thru 0x6a3e85 (0x8109 bytes) 000012[0] 1788932688.421398 mapwrite 0x2e107b thru 0x2eb6ec (0xa672 bytes) 000014[0] 1788932688.739439 punch from 0x30afb3 to 0x31078a (0x57d8 bytes) 000016[0] 1788932688.763970 punch from 0x23391c to 0x240102 (0xc7e7 bytes) 000018[0] 1788932688.774628 punch from 0x80bbc5 to 0x81a623 (0xea5f bytes) 000019[0] 1788932688.793083 read 0x3d4618 thru 0x3de0a2 (0x9a8b bytes) 000020[1] 1788932688.904499 trunc from 0x970022 to 0x2ba681 (0x6b59a0 bytes) 000022[1] 1788932689.030440 mapwrite 0x5cfa1a thru 0x5dcaa9 (0xd090 bytes) 000027[0] 1788932689.276529 write 0x624087 thru 0x62d5ac (0x9526 bytes) 000028[1] 1788932689.313028 punch from 0x1e9d15 to 0x1f3223 (0x950f bytes) 000029[0] 1788932689.339458 write 0x444915 thru 0x45014d (0xb839 bytes) 000033[0] 1788932689.415909 trunc from 0x62d5ad to 0x0fc36a (0x531242 bytes) 000034[1] 1788932689.465055 read 0x017c36 thru 0x025d85 (0xe150 bytes) 000036[0] 1788932689.569769 punch from 0x0102e6 to 0x018dae (0x8ac9 bytes) 000037[1] 1788932689.586904 trunc from 0x0fc36b to 0x6d1496 (0x5d512c bytes) 000038[1] 1788932689.641928 write 0x0f1a2c thru 0x0fcfa7 (0xb57c bytes) 000040[0] 1788932689.758723 mapread 0x41c9de thru 0x42b05d (0xe680 bytes) 000041[0] 1788932689.810666 read 0x564dfa thru 0x571b17 (0xcd1e bytes) 000042[0] 1788932689.819827 punch from 0x05c889 to 0x05ca7b (0x1f3 bytes) 000043[1] 1788932689.847364 write 0x9ca035 thru 0x9cbd9c (0x1d68 bytes) 000046[0] 1788932689.878129 mapwrite 0x635f7f thru 0x63ea83 (0x8b05 bytes) 000047[1] 1788932690.013210 punch from 0x56818a to 0x5736f6 (0xb56d bytes) truncating to largest ever: 0x9dbd0f 000048[0] 1788932690.076017 trunc from 0x9cbd9d to 0x9dbd0e (0xff72 bytes) 000050[1] 1788932690.224123 write 0x80ffad thru 0x81f893 (0xf8e7 bytes) 000051[0] 1788932690.262585 trunc from 0x9dbd0f to 0x3b6cff (0x62500f bytes) 000055[0] 1788932690.355314 trunc from 0x3b6d00 to 0x684036 (0x2cd337 bytes) 000056[0] 1788932690.378638 mapwrite 0x5b6034 thru 0x5c5a85 (0xfa52 bytes) 000057[0] 1788932690.477106 punch from 0x1b1380 to 0x1b9233 (0x7eb4 bytes) 000058[0] 1788932690.482398 trunc from 0x684037 to 0x1f0e26 (0x493210 bytes) 000059[0] 1788932690.509059 mapwrite 0x7ab3a1 thru 0x7b429d (0x8efd bytes) 000063[0] 1788932690.626462 write 0x9b851a thru 0x9bb465 (0x2f4c bytes) 000064[1] 1788932690.662721 mapread 0x31f85a thru 0x32d087 (0xd82e bytes) 000065[1] 1788932690.778382 trunc from 0x9bb466 to 0x930013 (0x8b452 bytes) 000066[0] 1788932690.824877 read 0x9011e1 thru 0x910003 (0xee23 bytes) 000067[0] 1788932690.866856 mapwrite 0x218148 thru 0x21c541 (0x43fa bytes) 000068[0] 1788932690.956073 write 0x06ae74 thru 0x0751a3 (0xa330 bytes) 000074[1] 1788932691.116976 write 0x12c765 thru 0x137a83 (0xb31f bytes) 000075[0] 1788932691.170600 read 0x01e510 thru 0x020948 (0x2439 bytes) 000077[0] 1788932691.219513 punch from 0x12c6c6 to 0x12d595 (0xed0 bytes) 000078[1] 1788932691.240369 write 0x64eba7 thru 0x655265 (0x66bf bytes) 000080[1] 1788932691.326966 trunc from 0x930014 to 0x3ec1b8 (0x543e5b bytes) 000082[1] 1788932691.431932 write 0x44816b thru 0x44a0c0 (0x1f56 bytes) 000083[1] 1788932691.439873 punch from 0x025842 to 0x0280f9 (0x28b8 bytes) 000084[1] 1788932691.480826 mapwrite 0x3d6aa7 thru 0x3d8113 (0x166d bytes) 000085[0] 1788932691.608520 trunc from 0x44a0c1 to 0x30b322 (0x13ed9e bytes) 000086[0] 1788932691.700895 trunc from 0x30b323 to 0x119f61 (0x1f13c1 bytes) 000087[1] 1788932691.776800 write 0x92bebe thru 0x936401 (0xa544 bytes) 000088[1] 1788932691.816037 trunc from 0x936402 to 0x02dc06 (0x9087fb bytes) 000089[1] 1788932691.852962 read 0x02d4ff thru 0x02dc06 (0x708 bytes) 000090[0] 1788932691.873427 mapwrite 0x95273a thru 0x95bea1 (0x9768 bytes) 000094[1] 1788932692.089488 mapread 0x64c2d2 thru 0x65c0eb (0xfe1a bytes) 000095[0] 1788932692.213303 mapwrite 0x4ad30b thru 0x4b147d (0x4173 bytes) 000096[1] 1788932692.314225 write 0x590fa4 thru 0x59a3b7 (0x9414 bytes) 000098[0] 1788932692.367303 write 0x5b4236 thru 0x5c1de4 (0xdbaf bytes) 000099[0] 1788932692.453870 trunc from 0x95bea2 to 0x3fb90f (0x560592 bytes) 000102[1] 1788932692.534406 punch from 0x25ff8a to 0x262fca (0x3041 bytes) 000104[1] 1788932692.622662 trunc from 0x3fb910 to 0x02d9bb (0x3cdf54 bytes) 000105[0] 1788932692.662951 read 0x006719 thru 0x010010 (0x98f8 bytes) 000106[1] 1788932692.695616 trunc from 0x02d9bc to 0x430405 (0x402a4a bytes) 000107[0] 1788932692.728797 mapread 0x1b1d16 thru 0x1c013e (0xe429 bytes) 000109[0] 1788932692.853453 punch from 0x38d9b2 to 0x3976cd (0x9d1c bytes) 000110[1] 1788932692.918211 punch from 0x02c9ce to 0x031479 (0x4aac bytes) 000111[1] 1788932692.946345 mapwrite 0x09b775 thru 0x0a2d83 (0x760f bytes) 000112[0] 1788932693.074901 mapwrite 0x102efb thru 0x110336 (0xd43c bytes) 000113[1] 1788932693.312859 mapwrite 0x3fbd2f thru 0x4090b1 (0xd383 bytes) 000114[0] 1788932693.433352 read 0x315fe2 thru 0x31785e (0x187d bytes) 000115[0] 1788932693.468364 trunc from 0x430406 to 0x2e3851 (0x14cbb4 bytes) 000116[0] 1788932693.589053 mapwrite 0x928e16 thru 0x92f759 (0x6944 bytes) 000117[0] 1788932693.716840 mapwrite 0x345664 thru 0x347368 (0x1d05 bytes) 000118[1] 1788932693.911170 mapread 0x6e4307 thru 0x6eb536 (0x7230 bytes) 000120[0] 1788932694.023730 read 0x4e6a76 thru 0x4ef976 (0x8f01 bytes) 000121[1] 1788932694.093116 punch from 0x16d16e to 0x175180 (0x8013 bytes) 000122[1] 1788932694.126929 mapread 0x6978a5 thru 0x69b4fd (0x3c59 bytes) 000123[0] 1788932694.127744 mapread 0x6d9b56 thru 0x6e5be1 (0xc08c bytes) 000125[1] 1788932694.281869 write 0x57745d thru 0x57d8c9 (0x646d bytes) 000127[1] 1788932694.361351 read 0x089e2e thru 0x091d7a (0x7f4d bytes) 000128[1] 1788932694.373105 write 0x19e1a7 thru 0x1a465d (0x64b7 bytes) 000130[1] 1788932694.421407 mapwrite 0x9210e1 thru 0x92b585 (0xa4a5 bytes) 000131[1] 1788932694.549237 read 0x59225c thru 0x59bc54 (0x99f9 bytes) 000132[1] 1788932694.569389 punch from 0x6967b5 to 0x6980f9 (0x1945 bytes) 000133[1] 1788932694.576136 trunc from 0x92f75a to 0x00408e (0x92b6cb bytes) 000134[1] 1788932694.603978 mapwrite 0x6903e9 thru 0x69809b (0x7cb3 bytes) 000135[0] 1788932694.749767 mapwrite 0x8b87a2 thru 0x8c23b2 (0x9c11 bytes) 000136[1] 1788932694.998119 trunc from 0x8c23b3 to 0x571190 (0x351222 bytes) 000138[0] 1788932695.031391 punch from 0x4a89f2 to 0x4b7592 (0xeba1 bytes) 000139[0] 1788932695.061849 mapread 0x085380 thru 0x08ec55 (0x98d6 bytes) 000140[1] 1788932695.094749 write 0x3638fe thru 0x363bd6 (0x2d9 bytes) 000141[0] 1788932695.125197 trunc from 0x571191 to 0x96e249 (0x3fd0b9 bytes) 000143[1] 1788932695.225223 trunc from 0x96e24a to 0x4ae3bc (0x4bfe8d bytes) 000144[1] 1788932695.268683 punch from 0x3418ab to 0x343cc5 (0x241b bytes) 000145[1] 1788932695.302123 mapwrite 0x1c9576 thru 0x1cb1bd (0x1c48 bytes) 000146[1] 1788932695.413163 write 0x567f98 thru 0x56acf9 (0x2d62 bytes) 000147[0] 1788932695.446857 write 0x98a7c6 thru 0x993a5c (0x9297 bytes) 000150[0] 1788932695.584522 read 0x6b84e8 thru 0x6c54bd (0xcfd6 bytes) 000151[1] 1788932695.669793 read 0x1ad7f0 thru 0x1ba8f1 (0xd102 bytes) 000153[0] 1788932695.760830 mapread 0x8a098b thru 0x8a6abd (0x6133 bytes) 000154[0] 1788932695.809022 mapread 0x3dec0c thru 0x3e0b71 (0x1f66 bytes) 000156[0] 1788932695.904106 mapwrite 0x12a8c1 thru 0x131886 (0x6fc6 bytes) 000158[0] 1788932696.026359 mapwrite 0x662ac3 thru 0x66e016 (0xb554 bytes) 000159[1] 1788932696.353530 trunc from 0x993a5d to 0x32a149 (0x669913 bytes) 000160[1] 1788932696.489023 mapread 0x1fae24 thru 0x205660 (0xa83d bytes) 000162[1] 1788932696.647345 mapwrite 0x61b5c3 thru 0x61e897 (0x32d5 bytes) 000163[1] 1788932696.852702 punch from 0x05b8a7 to 0x06327d (0x79d7 bytes) 000164[1] 1788932696.863319 punch from 0x5629f8 to 0x56accb (0x82d4 bytes) 000167[1] 1788932696.917971 mapread 0x5b2abe thru 0x5bf1fc (0xc73f bytes) 000170[1] 1788932696.981632 read 0x610f38 thru 0x61d7ed (0xc8b6 bytes) 000173[0] 1788932697.065094 write 0x571d73 thru 0x576446 (0x46d4 bytes) 000174[0] 1788932697.107522 punch from 0x1269e5 to 0x133c6f (0xd28b bytes) 000176[1] 1788932697.161975 read 0x465707 thru 0x471900 (0xc1fa bytes) 000177[1] 1788932697.197660 mapread 0x209cf3 thru 0x20d454 (0x3762 bytes) 000178[1] 1788932697.270112 mapwrite 0x793479 thru 0x7a132c (0xdeb4 bytes) 000179[0] 1788932697.428413 trunc from 0x7a132d to 0x0dfd8c (0x6c15a0 bytes) 000180[0] 1788932697.468783 punch from 0x0674cd to 0x06f5cc (0x8100 bytes) 000182[1] 1788932697.505346 punch from 0x0a917e to 0x0b4a84 (0xb907 bytes) 000184[0] 1788932697.545760 read 0x0cca6c thru 0x0cd9fd (0xf92 bytes) 000185[0] 1788932697.578365 write 0x852948 thru 0x8568ea (0x3fa3 bytes) 000186[0] 1788932697.611854 read 0x324e6b thru 0x32f4ac (0xa642 bytes) 000187[0] 1788932697.635365 mapwrite 0x90d9ac thru 0x9149f1 (0x7046 bytes) 000189[0] 1788932697.788947 read 0x7a2835 thru 0x7b225b (0xfa27 bytes) 000193[1] 1788932697.862617 mapread 0x449b97 thru 0x453125 (0x958f bytes) 000196[1] 1788932697.950520 trunc from 0x9149f2 to 0x040b69 (0x8d3e88 bytes) 000197[1] 1788932697.995081 write 0x5e2705 thru 0x5eb6cd (0x8fc9 bytes) 000198[1] 1788932698.040355 punch from 0x1c2798 to 0x1d155a (0xedc3 bytes) 000199[0] 1788932698.085067 read 0x366e61 thru 0x36a93c (0x3adc bytes) 000200[0] 1788932698.153797 mapwrite 0x02c3c1 thru 0x02eebe (0x2afe bytes) 000201[0] 1788932698.322933 punch from 0x383550 to 0x38593d (0x23ee bytes) 000202[0] 1788932698.360733 mapwrite 0x1c59a0 thru 0x1c8aec (0x314d bytes) 000203[0] 1788932698.501595 punch from 0x53fb0a to 0x53fb81 (0x78 bytes) 000204[0] 1788932698.527250 trunc from 0x5eb6ce to 0x76efb5 (0x1838e8 bytes) 000205[1] 1788932698.569812 read 0x5e81f9 thru 0x5f660a (0xe412 bytes) 000206[0] 1788932698.601592 read 0x4950d9 thru 0x4a44d6 (0xf3fe bytes) 000207[1] 1788932698.615457 write 0x3fade7 thru 0x3fdf23 (0x313d bytes) 000208[1] 1788932698.636935 mapwrite 0x5dc275 thru 0x5e2e70 (0x6bfc bytes) 000209[1] 1788932698.810539 write 0x13ecf0 thru 0x13f287 (0x598 bytes) 000210[0] 1788932698.860281 mapwrite 0x6168e7 thru 0x6194f3 (0x2c0d bytes) 000211[1] 1788932698.980059 mapread 0x0ab6f6 thru 0x0b9743 (0xe04e bytes) 000214[0] 1788932699.165891 read 0x24996d thru 0x24c211 (0x28a5 bytes) 000215[0] 1788932699.199499 punch from 0x077d64 to 0x085839 (0xdad6 bytes) 000217[0] 1788932699.215432 mapread 0x4ce5b4 thru 0x4db6ae (0xd0fb bytes) 000218[1] 1788932699.262585 write 0x84948d thru 0x858777 (0xf2eb bytes) 000221[1] 1788932699.361158 mapread 0x56159b thru 0x5696a3 (0x8109 bytes) 000222[0] 1788932699.407672 trunc from 0x858778 to 0x1f318a (0x6655ed bytes) 000223[1] 1788932699.476239 read 0x016dc7 thru 0x022360 (0xb59a bytes) 000225[0] 1788932699.601678 punch from 0x1e3e59 to 0x1e874f (0x48f7 bytes) 000227[1] 1788932699.674262 trunc from 0x1f318b to 0x2e881e (0xf5694 bytes) 000228[0] 1788932699.740404 mapwrite 0x2ef749 thru 0x2fca7e (0xd336 bytes) 000229[0] 1788932699.921212 trunc from 0x2fca7f to 0x8a1fd7 (0x5a5559 bytes) 000232[1] 1788932699.945510 read 0x07e535 thru 0x088407 (0x9ed3 bytes) 000234[1] 1788932700.078520 mapwrite 0x7e18ce thru 0x7f0526 (0xec59 bytes) 000236[1] 1788932700.248716 write 0x30c25c thru 0x30d328 (0x10cd bytes) 000238[1] 1788932700.312140 punch from 0x861a85 to 0x861bf4 (0x170 bytes) 000241[0] 1788932700.397121 mapread 0x4fec38 thru 0x503f46 (0x530f bytes) 000242[0] 1788932700.487232 trunc from 0x8a1fd8 to 0x126e49 (0x77b18e bytes) 000244[1] 1788932700.572117 mapwrite 0x5d62bb thru 0x5d9a4a (0x3790 bytes) 000245[1] 1788932700.687166 punch from 0x382593 to 0x38c4c5 (0x9f33 bytes) 000247[0] 1788932700.701408 mapwrite 0x1ab32c thru 0x1ab33d (0x12 bytes) 000248[1] 1788932700.894473 read 0x457028 thru 0x45fa94 (0x8a6d bytes) 000249[1] 1788932700.914401 punch from 0x1b6f23 to 0x1b959a (0x2678 bytes) 000251[1] 1788932701.009332 write 0x968805 thru 0x9703be (0x7bba bytes) 000252[1] 1788932701.078687 write 0x608d6d thru 0x610a7b (0x7d0f bytes) 000254[1] 1788932701.146389 punch from 0x23df57 to 0x249bb9 (0xbc63 bytes) 000255[0] 1788932701.193514 write 0x89fba0 thru 0x8a7ba2 (0x8003 bytes) 000256[0] 1788932701.272466 trunc from 0x9703bf to 0x4ffe3c (0x470582 bytes) 000258[0] 1788932701.478607 mapwrite 0x43abfc thru 0x43d2e0 (0x26e5 bytes) 000259[1] 1788932701.691346 punch from 0x225ec5 to 0x231fb0 (0xc0ec bytes) 000260[0] 1788932701.739491 mapwrite 0x9ca7ef thru 0x9d3c93 (0x94a5 bytes) 000262[0] 1788932701.942110 read 0x1c1e80 thru 0x1c70f4 (0x5275 bytes) 000265[0] 1788932702.024495 mapwrite 0x2b11c3 thru 0x2bc077 (0xaeb5 bytes) 000266[0] 1788932702.155289 trunc from 0x9d3c94 to 0x76188d (0x272406 bytes) 000267[1] 1788932702.184323 trunc from 0x76188e to 0x146ace (0x61adbf bytes) 000268[0] 1788932702.297514 read 0x0c7028 thru 0x0cd794 (0x676d bytes) 000269[1] 1788932702.414894 write 0x29976a thru 0x29d517 (0x3dae bytes) 000270[1] 1788932702.453685 mapwrite 0x6fe7b9 thru 0x702b4c (0x4394 bytes) 000272[0] 1788932702.543647 trunc from 0x702b4d to 0x6eeb21 (0x1402b bytes) 000276[0] 1788932702.700106 mapread 0x0bb519 thru 0x0c7749 (0xc231 bytes) 000277[1] 1788932702.825441 trunc from 0x6eeb22 to 0x33f3a1 (0x3af780 bytes) 000279[1] 1788932702.916202 trunc from 0x33f3a2 to 0x7c3406 (0x484065 bytes) 000280[0] 1788932702.978508 mapwrite 0x6287f2 thru 0x634c9c (0xc4ab bytes) 000281[1] 1788932703.223180 write 0x0aaab4 thru 0x0b690a (0xbe57 bytes) 000282[1] 1788932703.270176 punch from 0x115ace to 0x123346 (0xd879 bytes) 000283[0] 1788932703.300865 mapwrite 0x68b2c6 thru 0x68e856 (0x3591 bytes) 000284[1] 1788932703.402716 read 0x31d44b thru 0x326029 (0x8bdf bytes) 000287[0] 1788932703.481316 mapwrite 0x38ab63 thru 0x38b65f (0xafd bytes) 000289[0] 1788932703.598212 read 0x142e7f thru 0x148e87 (0x6009 bytes) 000290[1] 1788932703.661957 mapwrite 0x2dd4c3 thru 0x2e9f20 (0xca5e bytes) 000292[0] 1788932703.886440 mapwrite 0x0b2583 thru 0x0b79e0 (0x545e bytes) 000295[1] 1788932704.040190 mapread 0x324eec thru 0x33174c (0xc861 bytes) 000296[0] 1788932704.084467 read 0x1d772d thru 0x1dad7f (0x3653 bytes) 000298[0] 1788932704.117772 read 0x4614ff thru 0x4628a5 (0x13a7 bytes) 000300[0] 1788932704.174615 trunc from 0x7c3407 to 0x9616d9 (0x19e2d3 bytes) 000302[1] 1788932704.280127 trunc from 0x9616da to 0x506b2b (0x45abae bytes) 000303[1] 1788932704.338724 read 0x3be909 thru 0x3cbf6d (0xd665 bytes) 000305[0] 1788932704.445328 trunc from 0x506b2c to 0x5877f3 (0x80cc8 bytes) 000306[1] 1788932704.524448 mapwrite 0x86374d thru 0x86a67f (0x6f33 bytes) 000310[0] 1788932704.823280 mapread 0x5b959d thru 0x5c09e0 (0x7444 bytes) 000313[1] 1788932704.888060 trunc from 0x86a680 to 0x8dfe34 (0x757b5 bytes) 000316[1] 1788932705.012595 read 0x7c1489 thru 0x7c700f (0x5b87 bytes) 000318[0] 1788932705.026874 mapread 0x503cf0 thru 0x507773 (0x3a84 bytes) 000320[1] 1788932705.136122 read 0x3feed0 thru 0x4044e1 (0x5612 bytes) 000321[0] 1788932705.179338 trunc from 0x8dfe35 to 0x2daa11 (0x605423 bytes) 000322[0] 1788932705.268986 write 0x154339 thru 0x15809c (0x3d64 bytes) 000323[1] 1788932705.311937 mapwrite 0x260e21 thru 0x26cc2a (0xbe0a bytes) 000324[1] 1788932705.533857 trunc from 0x2daa12 to 0x983f0d (0x6a94fc bytes) 000326[0] 1788932705.643269 punch from 0x60f9d7 to 0x61de03 (0xe42d bytes) 000327[1] 1788932705.696602 punch from 0x27ece8 to 0x2837ce (0x4ae7 bytes) 000328[0] 1788932705.722909 write 0x54f669 thru 0x554e80 (0x5818 bytes) 000331[0] 1788932705.787872 read 0x808840 thru 0x8095a5 (0xd66 bytes) 000332[1] 1788932705.804276 punch from 0x59a62c to 0x59e267 (0x3c3c bytes) 000333[0] 1788932705.841854 mapwrite 0x7a77d6 thru 0x7aea15 (0x7240 bytes) 000337[0] 1788932706.043618 mapread 0x420a4a thru 0x422e7e (0x2435 bytes) 000339[0] 1788932706.100107 trunc from 0x983f0e to 0x343783 (0x64078a bytes) 000340[1] 1788932706.135437 write 0x12a3dd thru 0x1343ef (0xa013 bytes) truncating to largest ever: 0x9e9753 000341[1] 1788932706.179770 trunc from 0x343784 to 0x9e9752 (0x6a5fcf bytes) 000343[0] 1788932706.216962 mapread 0x8b6c2b thru 0x8b7902 (0xcd8 bytes) 000344[0] 1788932706.296907 punch from 0x980496 to 0x980c37 (0x7a2 bytes) 000345[1] 1788932706.361654 trunc from 0x9e9753 to 0x1bcc5f (0x82caf3 bytes) 000349[0] 1788932706.440170 mapread 0x09915b thru 0x09d145 (0x3feb bytes) 000350[1] 1788932706.481812 mapwrite 0x483697 thru 0x48565a (0x1fc4 bytes) 000352[0] 1788932706.592229 read 0x3a5fea thru 0x3a9f0e (0x3f25 bytes) 000353[1] 1788932706.727414 write 0x8da026 thru 0x8ddf28 (0x3f03 bytes) 000354[0] 1788932706.825316 punch from 0x42f047 to 0x4315c5 (0x257f bytes) 000356[0] 1788932706.893138 mapwrite 0x2dbdfb thru 0x2e1138 (0x533e bytes) 000357[1] 1788932707.083040 trunc from 0x8ddf29 to 0x3ad294 (0x530c94 bytes) 000359[0] 1788932707.178840 read 0x2a1d0c thru 0x2b0bb6 (0xeeab bytes) 000361[1] 1788932707.298592 trunc from 0x3ad295 to 0x6f69af (0x34971b bytes) 000362[0] 1788932707.365856 trunc from 0x6f69b0 to 0x568c2c (0x18dd83 bytes) 000364[1] 1788932707.434818 write 0x650ce2 thru 0x653af2 (0x2e11 bytes) 000365[0] 1788932707.459908 read 0x0b8e8d thru 0x0c4188 (0xb2fc bytes) 000366[0] 1788932707.486743 write 0x958942 thru 0x960124 (0x77e3 bytes) 000367[0] 1788932707.506240 mapread 0x27d53c thru 0x286216 (0x8cdb bytes) 000368[1] 1788932707.543925 trunc from 0x960125 to 0x31a903 (0x645821 bytes) 000370[1] 1788932707.623766 punch from 0x1d546b to 0x1dc677 (0x720d bytes) 000372[1] 1788932707.644791 write 0x0d88b1 thru 0x0ddb81 (0x52d1 bytes) 000373[0] 1788932707.678929 punch from 0x03c159 to 0x04a87f (0xe727 bytes) 000374[1] 1788932707.715049 mapread 0x0b6014 thru 0x0be142 (0x812f bytes) 000375[1] 1788932707.748591 punch from 0x18fbbb to 0x19b892 (0xbcd8 bytes) 000376[0] 1788932707.762407 mapwrite 0x035730 thru 0x0425fa (0xcecb bytes) 000377[1] 1788932707.895922 mapread 0x2c3a07 thru 0x2d39bc (0xffb6 bytes) 000378[1] 1788932707.974867 trunc from 0x31a904 to 0x532d19 (0x218416 bytes) 000382[1] 1788932708.090206 read 0x20c43e thru 0x211fd8 (0x5b9b bytes) 000383[1] 1788932708.097425 punch from 0x0bc426 to 0x0c99e2 (0xd5bd bytes) 000384[0] 1788932708.137740 read 0x46e56d thru 0x47b445 (0xced9 bytes) 000386[0] 1788932708.205915 mapread 0x4a4fc0 thru 0x4aa5bc (0x55fd bytes) 000389[1] 1788932708.346662 write 0x7f8239 thru 0x803b22 (0xb8ea bytes) 000390[1] 1788932708.383347 mapread 0x6cacde thru 0x6cd35e (0x2681 bytes) 000392[0] 1788932708.425926 write 0x8e0178 thru 0x8e0b69 (0x9f2 bytes) 000393[1] 1788932708.456830 punch from 0x207b81 to 0x214dea (0xd26a bytes) 000394[0] 1788932708.481075 read 0x409e03 thru 0x419a97 (0xfc95 bytes) 000395[1] 1788932708.521305 punch from 0x433a71 to 0x43bd72 (0x8302 bytes) 000396[1] 1788932708.538527 mapread 0x05e39d thru 0x06d0ea (0xed4e bytes) 000397[0] 1788932708.590509 write 0x975b4d thru 0x9812ff (0xb7b3 bytes) 000398[0] 1788932708.617771 trunc from 0x981300 to 0x4ae971 (0x4d298e bytes) 000399[0] 1788932708.678164 write 0x12b53a thru 0x13a11e (0xebe5 bytes) 000401[0] 1788932708.717943 mapread 0x1f89e2 thru 0x20558f (0xcbae bytes) 000403[1] 1788932708.775100 read 0x44e85c thru 0x45c2ed (0xda92 bytes) 000404[1] 1788932708.805490 mapwrite 0x212c4e thru 0x222b1f (0xfed2 bytes) 000405[0] 1788932708.975104 read 0x05ebdd thru 0x06ccf1 (0xe115 bytes) 000406[1] 1788932709.003762 punch from 0x48a2ed to 0x491a51 (0x7765 bytes) 000408[0] 1788932709.059294 trunc from 0x4ae972 to 0x9d99e0 (0x52b06f bytes) 000409[1] 1788932709.122827 mapwrite 0x567ba2 thru 0x573c86 (0xc0e5 bytes) 000412[0] 1788932709.344761 write 0x3be585 thru 0x3c0ffe (0x2a7a bytes) 000413[0] 1788932709.369644 write 0x153800 thru 0x15f179 (0xb97a bytes) 000414[0] 1788932709.393001 punch from 0x07a6e3 to 0x0851cc (0xaaea bytes) 000415[0] 1788932709.421327 mapread 0x6eb0ec thru 0x6ebbf5 (0xb0a bytes) 000421[1] 1788932709.467869 punch from 0x804fe0 to 0x80c8bf (0x78e0 bytes) 000422[0] 1788932709.492644 read 0x3eab6f thru 0x3f43c1 (0x9853 bytes) 000423[0] 1788932709.520051 read 0x1cf5bf thru 0x1d1b71 (0x25b3 bytes) 000424[1] 1788932709.525104 read 0x83883e thru 0x83c0be (0x3881 bytes) 000425[1] 1788932709.533336 mapwrite 0x7c532e thru 0x7cc696 (0x7369 bytes) 000427[1] 1788932709.626844 write 0x7e9497 thru 0x7f079f (0x7309 bytes) 000429[0] 1788932709.654068 read 0x070afc thru 0x07b890 (0xad95 bytes) 000430[0] 1788932709.676827 mapread 0x7cd055 thru 0x7d3b1c (0x6ac8 bytes) 000433[1] 1788932709.720524 read 0x3f0e58 thru 0x3f5a8b (0x4c34 bytes) 000434[1] 1788932709.730451 punch from 0x5c250b to 0x5c589d (0x3393 bytes) 000437[1] 1788932709.746841 mapread 0x26df8e thru 0x26e80e (0x881 bytes) 000440[0] 1788932709.768388 punch from 0x011975 to 0x019faf (0x863b bytes) 000443[0] 1788932709.833122 mapread 0x0a8c13 thru 0x0ae4c5 (0x58b3 bytes) 000444[0] 1788932709.874382 read 0x20d958 thru 0x21041f (0x2ac8 bytes) 000445[1] 1788932709.894429 punch from 0x0b53ce to 0x0bd608 (0x823b bytes) 000446[0] 1788932709.914352 punch from 0x823b42 to 0x8255c0 (0x1a7f bytes) 000447[1] 1788932709.947294 punch from 0x7adfa1 to 0x7b5f20 (0x7f80 bytes) 000450[0] 1788932709.990110 mapwrite 0x13099f thru 0x13e65c (0xdcbe bytes) 000452[1] 1788932710.121995 mapwrite 0x094a6b thru 0x0a3950 (0xeee6 bytes) 000454[0] 1788932710.250715 mapwrite 0x89ddfe thru 0x8a4933 (0x6b36 bytes) 000458[0] 1788932710.461072 mapwrite 0x96b250 thru 0x974a11 (0x97c2 bytes) 000459[1] 1788932710.559246 mapwrite 0x9977c6 thru 0x99c2ec (0x4b27 bytes) 000462[0] 1788932710.684218 mapwrite 0x5f1015 thru 0x5fe446 (0xd432 bytes) 000463[0] 1788932710.865457 trunc from 0x9d99e1 to 0x6bc973 (0x31d06d bytes) 000464[1] 1788932710.946354 mapwrite 0x64431e thru 0x65318a (0xee6d bytes) 000466[1] 1788932711.243283 trunc from 0x6bc974 to 0x1b4135 (0x50883e bytes) 000468[1] 1788932711.360317 mapwrite 0x0f0c7d thru 0x0f438a (0x370e bytes) 000470[1] 1788932711.559144 mapread 0x17c2c0 thru 0x185381 (0x90c2 bytes) 000471[1] 1788932711.601363 write 0x7a05c2 thru 0x7a4383 (0x3dc2 bytes) 000473[0] 1788932711.606451 punch from 0x66b69c to 0x67913b (0xdaa0 bytes) 000474[0] 1788932711.665963 trunc from 0x7a4384 to 0x11c641 (0x687d42 bytes) 000476[0] 1788932711.747806 read 0x07778a thru 0x082a7e (0xb2f5 bytes) 000478[0] 1788932711.865746 mapwrite 0x3360fd thru 0x341958 (0xb85c bytes) 000480[1] 1788932712.088544 read 0x03fe22 thru 0x048add (0x8cbc bytes) 000483[0] 1788932712.257967 mapwrite 0x3017b4 thru 0x307af3 (0x6340 bytes) 000484[0] 1788932712.406915 write 0x08cc39 thru 0x0949c8 (0x7d90 bytes) 000486[0] 1788932712.480416 read 0x12fd7f thru 0x1373a3 (0x7625 bytes) truncating to largest ever: 0x9fdbd2 000487[1] 1788932712.570797 trunc from 0x341959 to 0x9fdbd1 (0x6bc279 bytes) 000488[1] 1788932712.670833 write 0x0a8ddd thru 0x0aa759 (0x197d bytes) 000490[0] 1788932712.712875 mapread 0x73681a thru 0x744f4a (0xe731 bytes) 000491[1] 1788932712.889502 mapwrite 0x6cd896 thru 0x6cd9c1 (0x12c bytes) 000492[1] 1788932713.018305 read 0x6f0d6a thru 0x6f296e (0x1c05 bytes) 000493[0] 1788932713.043123 punch from 0x2af3b6 to 0x2b2363 (0x2fae bytes) 000494[1] 1788932713.077777 mapread 0x8f7175 thru 0x904299 (0xd125 bytes) 000495[0] 1788932713.141924 mapread 0x9d1242 thru 0x9da68f (0x944e bytes) 000497[1] 1788932713.211980 mapwrite 0x62a496 thru 0x635219 (0xad84 bytes) 000498[1] 1788932713.273665 punch from 0x0b27fe to 0x0b49df (0x21e2 bytes) 000500[0] 1788932713.322459 mapwrite 0x935634 thru 0x93b60d (0x5fda bytes) All operations completed A-OK! Chance of close/open is 1 in 50 Seed set to 2713 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read skipping zero length punch hole 000008[0] 1788932714.639913 write_OD 0x3de000 thru 0x3e4fff (0x7000 bytes) truncating to largest ever: 0x166c2a 000009[0] 1788932714.761506 trunc from 0x3e5000 to 0x166c29 (0x27e3d6 bytes) 000010[1] 1788932714.820432 read 0x051000 thru 0x05e6fe (0xd6ff bytes) 000011[1] 1788932714.855691 punch from 0x0573d6 to 0x057b11 (0x73c bytes) 000012[0] 1788932714.882287 mapread 0x116000 thru 0x118bb0 (0x2bb1 bytes) 000013[0] 1788932714.929453 punch from 0x093de6 to 0x099236 (0x5451 bytes) 000014[1] 1788932714.946594 mapwrite 0x76c000 thru 0x76f6aa (0x36ab bytes) 000015[0] 1788932715.069403 trunc from 0x76f6ab to 0x04ec6c (0x720a3e bytes) 000016[0] 1788932715.109440 read_OD 0x03a000 thru 0x03afff (0x1000 bytes) 000018[1] 1788932715.131705 punch from 0x04116b to 0x04a1a9 (0x903f bytes) truncating to largest ever: 0x25136d 000022[1] 1788932715.186649 trunc from 0x04ec6d to 0x25136c (0x202700 bytes) 000023[0] 1788932715.238364 write_OD 0x713000 thru 0x720fff (0xe000 bytes) truncating to largest ever: 0x2fd4b0 000025[1] 1788932715.313705 trunc from 0x721000 to 0x2fd4af (0x423b50 bytes) 000026[1] 1788932715.362424 punch from 0x0f7490 to 0x0fe1b9 (0x6d2a bytes) 000027[1] 1788932715.373637 punch from 0x0b79b3 to 0x0be836 (0x6e84 bytes) truncating to largest ever: 0x5cb698 000028[0] 1788932715.391738 trunc from 0x2fd4b0 to 0x5cb697 (0x2ce1e8 bytes) 000030[0] 1788932715.440865 write_OD 0x581000 thru 0x58dfff (0xd000 bytes) 000031[0] 1788932715.498118 mapread 0x0e7000 thru 0x0f4d1c (0xdd1d bytes) 000033[0] 1788932715.524739 read_OD 0x529000 thru 0x533fff (0xb000 bytes) 000035[0] 1788932715.540137 mapwrite 0x4b9000 thru 0x4bc68e (0x368f bytes) 000036[1] 1788932715.652041 mapread 0x164000 thru 0x16aba4 (0x6ba5 bytes) 000037[1] 1788932715.703887 write 0x182000 thru 0x184b0f (0x2b10 bytes) 000038[1] 1788932715.723434 mapwrite 0x18c000 thru 0x1937a3 (0x77a4 bytes) 000039[1] 1788932715.791883 trunc from 0x5cb698 to 0x341b81 (0x289b16 bytes) 000043[0] 1788932715.881835 mapwrite 0x073000 thru 0x075d19 (0x2d1a bytes) 000047[0] 1788932715.965078 trunc from 0x341b82 to 0x586df8 (0x245277 bytes) 000048[1] 1788932715.982480 mapwrite 0x62e000 thru 0x6315e3 (0x35e4 bytes) 000049[1] 1788932716.121852 trunc from 0x6315e4 to 0x496860 (0x19ad83 bytes) 000051[0] 1788932716.147773 read_OD 0x0d0000 thru 0x0d1fff (0x2000 bytes) 000052[1] 1788932716.172705 mapwrite 0x67c000 thru 0x67c54f (0x550 bytes) 000053[0] 1788932716.306426 mapwrite 0x5ac000 thru 0x5b26f9 (0x66fa bytes) 000055[1] 1788932716.434679 mapwrite 0x511000 thru 0x514458 (0x3459 bytes) 000057[0] 1788932716.591959 write_OD 0x99f000 thru 0x9a9fff (0xb000 bytes) 000058[0] 1788932716.688374 mapwrite 0x06a000 thru 0x06c29e (0x229f bytes) 000060[1] 1788932716.802725 mapread 0x296000 thru 0x2a44db (0xe4dc bytes) truncating to largest ever: 0x9236ec 000062[1] 1788932716.859244 trunc from 0x9aa000 to 0x9236eb (0x86914 bytes) 000063[0] 1788932716.906078 mapwrite 0x908000 thru 0x90fb07 (0x7b08 bytes) 000065[0] 1788932717.012737 trunc from 0x9236ec to 0x501a42 (0x421ca9 bytes) 000066[0] 1788932717.027729 write_OD 0x07a000 thru 0x07afff (0x1000 bytes) 000067[1] 1788932717.084531 read 0x163000 thru 0x168d39 (0x5d3a bytes) 000068[1] 1788932717.109674 trunc from 0x501a43 to 0x314a7f (0x1ecfc3 bytes) 000069[0] 1788932717.139402 write_OD 0x983000 thru 0x98cfff (0xa000 bytes) 000071[0] 1788932717.197105 read_OD 0x4f5000 thru 0x4f9fff (0x5000 bytes) 000076[0] 1788932717.228166 trunc from 0x98d000 to 0x286e9d (0x706162 bytes) 000080[0] 1788932717.295750 punch from 0x06dd04 to 0x0706c2 (0x29bf bytes) 000081[1] 1788932717.299503 mapwrite 0x191000 thru 0x19df36 (0xcf37 bytes) 000082[1] 1788932717.466315 trunc from 0x286e9e to 0x27463d (0x12860 bytes) 000084[1] 1788932717.503849 punch from 0x0ce6e9 to 0x0ddcbd (0xf5d5 bytes) 000085[0] 1788932717.509465 mapwrite 0x29f000 thru 0x2ae04b (0xf04c bytes) 000086[1] 1788932717.667288 mapwrite 0x263000 thru 0x26702e (0x402f bytes) 000087[1] 1788932717.738373 mapread 0x097000 thru 0x0a3222 (0xc223 bytes) 000089[1] 1788932717.772457 punch from 0x190a82 to 0x19e43a (0xd9b9 bytes) 000091[1] 1788932717.780285 trunc from 0x2ae04c to 0x278de1 (0x3526a bytes) 000096[1] 1788932717.813879 mapread 0x043000 thru 0x047916 (0x4917 bytes) 000097[1] 1788932717.831505 mapwrite 0x09b000 thru 0x0a0146 (0x5147 bytes) 000098[1] 1788932717.889974 write_OD 0x8df000 thru 0x8e2fff (0x4000 bytes) 000099[1] 1788932717.951329 mapread 0x38c000 thru 0x392ad0 (0x6ad1 bytes) 000102[0] 1788932718.008817 read_OD 0x04d000 thru 0x055fff (0x9000 bytes) 000103[0] 1788932718.021102 mapwrite 0x41f000 thru 0x427510 (0x8511 bytes) 000104[0] 1788932718.111353 write_OD 0x56c000 thru 0x574fff (0x9000 bytes) 000106[1] 1788932718.169354 punch from 0x2576be to 0x25a251 (0x2b94 bytes) 000107[1] 1788932718.181743 mapwrite 0x7d0000 thru 0x7dbe93 (0xbe94 bytes) 000111[1] 1788932718.328141 mapwrite 0x144000 thru 0x152883 (0xe884 bytes) 000113[1] 1788932718.484537 mapread 0x6b8000 thru 0x6c486c (0xc86d bytes) 000114[0] 1788932718.528173 punch from 0x52b450 to 0x534781 (0x9332 bytes) 000115[0] 1788932718.546086 write_OD 0x67a000 thru 0x67dfff (0x4000 bytes) 000116[1] 1788932718.604479 mapwrite 0x2fa000 thru 0x302528 (0x8529 bytes) 000119[0] 1788932718.703875 trunc from 0x8e3000 to 0x740561 (0x1a2a9e bytes) truncating to largest ever: 0x982c0e 000122[0] 1788932718.742155 trunc from 0x740562 to 0x982c0d (0x2426ac bytes) 000123[0] 1788932718.773953 punch from 0x90c1c8 to 0x912a7c (0x68b5 bytes) 000124[1] 1788932718.794335 mapread 0x24b000 thru 0x24e719 (0x371a bytes) 000125[1] 1788932718.839223 mapread 0x240000 thru 0x24df1b (0xdf1c bytes) 000126[1] 1788932718.850074 write_OD 0x1a3000 thru 0x1adfff (0xb000 bytes) 000127[1] 1788932718.934033 mapwrite 0x113000 thru 0x120e6a (0xde6b bytes) 000129[0] 1788932719.102227 mapwrite 0x3b5000 thru 0x3c39d3 (0xe9d4 bytes) 000130[1] 1788932719.355297 trunc from 0x982c0e to 0x39fdcd (0x5e2e40 bytes) 000133[1] 1788932719.481098 write_OD 0x1c1000 thru 0x1cffff (0xf000 bytes) 000134[1] 1788932719.537052 punch from 0x28b093 to 0x297d83 (0xccf1 bytes) 000135[1] 1788932719.583369 write_OD 0x3c5000 thru 0x3cafff (0x6000 bytes) 000136[0] 1788932719.663463 read_OD 0x21e000 thru 0x22bfff (0xe000 bytes) 000138[0] 1788932719.697313 write_OD 0x2eb000 thru 0x2f1fff (0x7000 bytes) 000139[0] 1788932719.780171 mapwrite 0x0a3000 thru 0x0b23db (0xf3dc bytes) 000140[1] 1788932719.932369 mapwrite 0x204000 thru 0x213242 (0xf243 bytes) 000141[1] 1788932720.125433 mapread 0x2c3000 thru 0x2d29b2 (0xf9b3 bytes) 000143[1] 1788932720.235026 punch from 0x33c416 to 0x342978 (0x6563 bytes) 000144[0] 1788932720.240963 read_OD 0x0c7000 thru 0x0cbfff (0x5000 bytes) 000145[1] 1788932720.274469 punch from 0x27f635 to 0x280857 (0x1223 bytes) 000146[0] 1788932720.299482 read_OD 0x245000 thru 0x24afff (0x6000 bytes) 000151[1] 1788932720.333085 punch from 0x027e2d to 0x037aa4 (0xfc78 bytes) 000153[1] 1788932720.387724 write_OD 0x686000 thru 0x686fff (0x1000 bytes) 000154[0] 1788932720.470508 mapwrite 0x162000 thru 0x166728 (0x4729 bytes) 000155[1] 1788932720.708990 mapwrite 0x93e000 thru 0x944e17 (0x6e18 bytes) 000156[0] 1788932720.974667 write_OD 0x317000 thru 0x31bfff (0x5000 bytes) 000157[0] 1788932721.118107 punch from 0x43d6af to 0x43ee95 (0x17e7 bytes) 000158[1] 1788932721.197509 write_OD 0x19d000 thru 0x1a7fff (0xb000 bytes) 000160[1] 1788932721.344824 read_OD 0x174000 thru 0x175fff (0x2000 bytes) 000162[1] 1788932721.360351 read_OD 0x543000 thru 0x54ffff (0xd000 bytes) 000165[0] 1788932721.387843 write_OD 00000000 thru 0x006fff (0x7000 bytes) 000166[1] 1788932721.506661 read_OD 0x103000 thru 0x105fff (0x3000 bytes) 000168[0] 1788932721.522863 write_OD 0x48a000 thru 0x490fff (0x7000 bytes) 000169[0] 1788932721.620930 mapwrite 0x48f000 thru 0x495538 (0x6539 bytes) 000170[1] 1788932721.766792 mapread 0x3ac000 thru 0x3bb927 (0xf928 bytes) 000172[0] 1788932721.826323 write_OD 0x383000 thru 0x38ffff (0xd000 bytes) 000173[0] 1788932721.893510 trunc from 0x944e18 to 0x7d2663 (0x1727b4 bytes) 000176[1] 1788932721.951331 punch from 0x565e87 to 0x56e0a6 (0x8220 bytes) 000177[0] 1788932721.979414 write_OD 0x4c2000 thru 0x4c4fff (0x3000 bytes) 000180[0] 1788932722.035366 punch from 0x22628a to 0x22fb5f (0x98d6 bytes) 000182[0] 1788932722.063370 read_OD 0x6fd000 thru 0x70bfff (0xf000 bytes) 000183[0] 1788932722.081539 mapread 0x206000 thru 0x20f8bc (0x98bd bytes) 000187[1] 1788932722.119907 punch from 0x05fd61 to 0x069d58 (0x9ff8 bytes) 000190[0] 1788932722.208835 mapwrite 0x283000 thru 0x28a560 (0x7561 bytes) 000191[0] 1788932722.291550 punch from 0x45b2cd to 0x468f56 (0xdc8a bytes) 000193[1] 1788932722.308839 trunc from 0x7d2664 to 0x50530f (0x2cd354 bytes) 000194[0] 1788932722.347284 write_OD 0x4f1000 thru 0x4f7fff (0x7000 bytes) 000195[0] 1788932722.428071 trunc from 0x505310 to 0x1f3716 (0x311bf9 bytes) 000196[0] 1788932722.494658 trunc from 0x1f3717 to 0x5da148 (0x3e6a32 bytes) 000198[0] 1788932722.531732 punch from 0x323d43 to 0x32fd23 (0xbfe1 bytes) 000199[0] 1788932722.536398 mapread 0x5cc000 thru 0x5d9bad (0xdbae bytes) 000200[1] 1788932722.556074 mapread 0x30b000 thru 0x3102bd (0x52be bytes) 000203[1] 1788932722.588839 read_OD 0x4ef000 thru 0x4f7fff (0x9000 bytes) 000205[0] 1788932722.597393 mapwrite 0x5d2000 thru 0x5db1ad (0x91ae bytes) 000207[0] 1788932722.730380 read_OD 0x55a000 thru 0x561fff (0x8000 bytes) 000210[0] 1788932722.755263 write_OD 0x0cb000 thru 0x0d7fff (0xd000 bytes) 000211[1] 1788932722.810983 write_OD 0x754000 thru 0x75ffff (0xc000 bytes) 000216[1] 1788932722.881824 write_OD 0x2de000 thru 0x2defff (0x1000 bytes) 000218[1] 1788932722.948699 mapread 0x61c000 thru 0x627450 (0xb451 bytes) 000219[0] 1788932722.996405 write_OD 0x410000 thru 0x419fff (0xa000 bytes) 000220[1] 1788932723.071731 mapread 0x517000 thru 0x51976a (0x276b bytes) 000222[0] 1788932723.091506 mapread 0x5c2000 thru 0x5ca25c (0x825d bytes) 000223[1] 1788932723.105754 mapwrite 0x4bf000 thru 0x4c88b6 (0x98b7 bytes) 000227[0] 1788932723.264673 mapread 0x495000 thru 0x49aa2a (0x5a2b bytes) 000229[0] 1788932723.343521 mapwrite 0x439000 thru 0x43d923 (0x4924 bytes) 000230[0] 1788932723.509186 mapread 0x381000 thru 0x385024 (0x4025 bytes) 000234[0] 1788932723.602143 mapread 0x3a5000 thru 0x3ae709 (0x970a bytes) 000236[1] 1788932723.618974 trunc from 0x760000 to 0x0c43c0 (0x69bc3f bytes) 000237[0] 1788932723.649418 punch from 0x031ba2 to 0x03858e (0x69ed bytes) 000238[1] 1788932723.667223 mapwrite 0x669000 thru 0x675c90 (0xcc91 bytes) 000239[1] 1788932723.833160 write_OD 0x7d0000 thru 0x7d4fff (0x5000 bytes) 000240[1] 1788932723.911280 punch from 0x72bfd6 to 0x730f00 (0x4f2b bytes) 000241[0] 1788932723.936637 mapread 0x3fc000 thru 0x40ab98 (0xeb99 bytes) 000242[1] 1788932723.977846 punch from 0x490c64 to 0x4991c0 (0x855d bytes) 000244[0] 1788932724.013453 trunc from 0x7d5000 to 0x110324 (0x6c4cdb bytes) 000245[1] 1788932724.094474 read_OD 0x072000 thru 0x072fff (0x1000 bytes) 000247[1] 1788932724.156769 write_OD 0x9d3000 thru 0x9d5fff (0x3000 bytes) 000248[0] 1788932724.240913 write_OD 0x2a6000 thru 0x2b4fff (0xf000 bytes) 000249[0] 1788932724.331419 mapread 0x29d000 thru 0x2a53b3 (0x83b4 bytes) 000250[1] 1788932724.368415 write_OD 0x7f1000 thru 0x7fcfff (0xc000 bytes) 000251[1] 1788932724.456335 mapwrite 0x55f000 thru 0x5628a7 (0x38a8 bytes) 000252[1] 1788932724.766856 read_OD 0x2ce000 thru 0x2d1fff (0x4000 bytes) 000253[1] 1788932724.807741 mapread 0x605000 thru 0x60cb84 (0x7b85 bytes) 000255[1] 1788932724.834321 write_OD 0x323000 thru 0x32cfff (0xa000 bytes) 000256[0] 1788932724.905189 trunc from 0x9d6000 to 0x404eb4 (0x5d114b bytes) 000257[0] 1788932724.948818 read_OD 0x2d0000 thru 0x2defff (0xf000 bytes) 000258[1] 1788932724.972362 trunc from 0x404eb5 to 0x61e5ca (0x219716 bytes) 000259[1] 1788932725.024109 mapwrite 0x07a000 thru 0x07c4c0 (0x24c1 bytes) 000262[0] 1788932725.143134 mapread 0x1bf000 thru 0x1c3901 (0x4902 bytes) 000263[1] 1788932725.205349 mapwrite 0x86a000 thru 0x879f85 (0xff86 bytes) 000264[0] 1788932725.349987 mapwrite 0x7fc000 thru 0x7ff3e2 (0x33e3 bytes) 000265[0] 1788932725.433424 write_OD 0x67f000 thru 0x685fff (0x7000 bytes) 000266[1] 1788932725.504775 write_OD 0x437000 thru 0x43ffff (0x9000 bytes) 000268[1] 1788932725.577291 write_OD 0x11e000 thru 0x11ffff (0x2000 bytes) 000273[1] 1788932725.658191 write_OD 0x03d000 thru 0x04bfff (0xf000 bytes) 000276[0] 1788932725.719052 read_OD 0x055000 thru 0x058fff (0x4000 bytes) 000277[1] 1788932725.728825 punch from 0x3a32ad to 0x3aee4d (0xbba1 bytes) 000281[1] 1788932725.879512 read_OD 0x39b000 thru 0x3a5fff (0xb000 bytes) 000282[1] 1788932725.903639 trunc from 0x879f86 to 0x26e1d3 (0x60bdb2 bytes) 000283[0] 1788932725.940472 write_OD 0x994000 thru 0x9a2fff (0xf000 bytes) 000285[1] 1788932726.005433 punch from 0x417dfa to 0x41b7b5 (0x39bc bytes) 000287[1] 1788932726.036857 read_OD 0x8a5000 thru 0x8adfff (0x9000 bytes) 000291[0] 1788932726.077283 trunc from 0x9a3000 to 0x5fd039 (0x3a5fc6 bytes) 000294[0] 1788932726.177693 trunc from 0x5fd03a to 0x12c4b2 (0x4d0b87 bytes) 000295[0] 1788932726.208507 read_OD 0x0c5000 thru 0x0cdfff (0x9000 bytes) 000296[0] 1788932726.241363 read_OD 0x073000 thru 0x079fff (0x7000 bytes) 000298[1] 1788932726.259471 mapread 0x029000 thru 0x031059 (0x805a bytes) 000299[1] 1788932726.282968 mapread 0x086000 thru 0x0934cb (0xd4cc bytes) 000306[1] 1788932726.315069 punch from 0x0133e8 to 0x020aa7 (0xd6c0 bytes) 000307[1] 1788932726.330717 trunc from 0x12c4b3 to 0x71a56f (0x5ee0bd bytes) 000310[0] 1788932726.397932 mapwrite 0x713000 thru 0x720d95 (0xdd96 bytes) 000313[1] 1788932726.620836 mapwrite 0x528000 thru 0x52dc80 (0x5c81 bytes) 000314[1] 1788932726.724467 mapwrite 0x0ab000 thru 0x0ab699 (0x69a bytes) 000315[0] 1788932726.786145 trunc from 0x720d96 to 0x7fa5d8 (0xd9843 bytes) 000321[1] 1788932726.917513 write_OD 0x639000 thru 0x645fff (0xd000 bytes) 000323[1] 1788932726.989805 mapread 0x1b5000 thru 0x1c474f (0xf750 bytes) 000324[0] 1788932727.024630 read_OD 0x42a000 thru 0x42efff (0x5000 bytes) 000327[1] 1788932727.032436 mapread 0x038000 thru 0x04615b (0xe15c bytes) 000328[0] 1788932727.061441 read_OD 0x5b2000 thru 0x5befff (0xd000 bytes) 000330[1] 1788932727.070742 mapread 0x7a3000 thru 0x7a7c34 (0x4c35 bytes) 000332[1] 1788932727.103930 read_OD 0x74e000 thru 0x75cfff (0xf000 bytes) 000334[1] 1788932727.123534 trunc from 0x7fa5d9 to 0x40ff74 (0x3ea664 bytes) 000335[1] 1788932727.167759 trunc from 0x40ff75 to 0x8ad793 (0x49d81f bytes) 000336[1] 1788932727.185877 mapwrite 0x7a7000 thru 0x7ae361 (0x7362 bytes) 000337[0] 1788932727.347414 trunc from 0x8ad794 to 0x6f6aef (0x1b6ca4 bytes) 000340[1] 1788932727.392955 mapwrite 0x2cd000 thru 0x2d34b4 (0x64b5 bytes) 000341[1] 1788932727.497228 punch from 0x0fb1f0 to 0x1019da (0x67eb bytes) 000343[0] 1788932727.532888 write_OD 0x64f000 thru 0x651fff (0x3000 bytes) 000344[0] 1788932727.574128 read_OD 0x6b9000 thru 0x6c6fff (0xe000 bytes) 000345[0] 1788932727.588985 write_OD 0x746000 thru 0x74bfff (0x6000 bytes) 000347[0] 1788932727.633265 write_OD 0x5b8000 thru 0x5bafff (0x3000 bytes) 000348[1] 1788932727.675838 punch from 0x3e330c to 0x3ee1a5 (0xae9a bytes) 000350[0] 1788932727.743406 mapwrite 0x986000 thru 0x98966c (0x366d bytes) 000354[1] 1788932728.176461 read_OD 0x85a000 thru 0x865fff (0xc000 bytes) 000357[0] 1788932728.245736 mapwrite 0x2a3000 thru 0x2a40cf (0x10d0 bytes) 000361[0] 1788932728.397404 punch from 0x521038 to 0x526927 (0x58f0 bytes) 000362[0] 1788932728.407409 write_OD 0x8a7000 thru 0x8b0fff (0xa000 bytes) 000364[1] 1788932728.504745 mapwrite 0x469000 thru 0x475d48 (0xcd49 bytes) 000365[0] 1788932728.608769 punch from 0x847abc to 0x854bbc (0xd101 bytes) 000367[1] 1788932728.646290 punch from 0x6004d9 to 0x60d875 (0xd39d bytes) 000368[0] 1788932728.668668 mapwrite 0x4a8000 thru 0x4ab8a2 (0x38a3 bytes) 000371[0] 1788932728.810064 write_OD 0x641000 thru 0x643fff (0x3000 bytes) 000372[1] 1788932728.861988 trunc from 0x98966d to 0x66fc8a (0x3199e2 bytes) 000373[1] 1788932728.911742 mapwrite 0x235000 thru 0x2351cc (0x1cd bytes) 000374[0] 1788932729.015440 mapwrite 0x001000 thru 0x00b387 (0xa388 bytes) 000375[0] 1788932729.131430 trunc from 0x66fc8b to 0x85f6af (0x1efa25 bytes) 000376[0] 1788932729.218502 punch from 0x307a3a to 0x314eb3 (0xd47a bytes) 000377[0] 1788932729.241668 trunc from 0x85f6b0 to 0x083ad3 (0x7dbbdc bytes) 000378[0] 1788932729.299969 punch from 0x0421c6 to 0x046a3b (0x4876 bytes) 000379[1] 1788932729.304814 punch from 0x03e5e1 to 0x042f49 (0x4969 bytes) 000380[0] 1788932729.353847 mapread 0x045000 thru 0x04bc86 (0x6c87 bytes) 000381[0] 1788932729.422926 read_OD 0x002000 thru 0x008fff (0x7000 bytes) 000382[1] 1788932729.440207 mapwrite 0x071000 thru 0x071f72 (0xf73 bytes) 000383[1] 1788932729.559383 trunc from 0x083ad4 to 0x4d22c0 (0x44e7ed bytes) 000386[0] 1788932729.606835 write_OD 0x5c6000 thru 0x5c9fff (0x4000 bytes) 000387[0] 1788932729.649201 write_OD 0x963000 thru 0x96bfff (0x9000 bytes) 000388[0] 1788932729.702634 read_OD 0x501000 thru 0x50dfff (0xd000 bytes) 000391[1] 1788932729.718131 read_OD 0x461000 thru 0x461fff (0x1000 bytes) truncating to largest ever: 0x9c4ef4 000394[1] 1788932729.724059 trunc from 0x96c000 to 0x9c4ef3 (0x58ef4 bytes) 000395[0] 1788932729.768510 read_OD 0x578000 thru 0x57dfff (0x6000 bytes) 000396[1] 1788932729.796244 mapwrite 0x32c000 thru 0x338b4d (0xcb4e bytes) 000397[1] 1788932729.955159 trunc from 0x9c4ef4 to 0x410232 (0x5b4cc1 bytes) 000398[1] 1788932729.995806 mapwrite 0x350000 thru 0x3550b9 (0x50ba bytes) 000399[0] 1788932730.094751 mapread 0x3fc000 thru 0x3fe44a (0x244b bytes) 000400[0] 1788932730.150766 read_OD 0x37c000 thru 0x384fff (0x9000 bytes) 000401[1] 1788932730.165214 read_OD 0x162000 thru 0x163fff (0x2000 bytes) 000404[1] 1788932730.171620 read_OD 0x3d8000 thru 0x3d9fff (0x2000 bytes) 000406[0] 1788932730.178751 mapread 0x22c000 thru 0x235ff6 (0x9ff7 bytes) 000407[1] 1788932730.197446 mapwrite 0x1dc000 thru 0x1e2833 (0x6834 bytes) 000408[0] 1788932730.304195 write_OD 0x61c000 thru 0x61dfff (0x2000 bytes) 000409[1] 1788932730.366354 mapwrite 0x76d000 thru 0x7735be (0x65bf bytes) 000410[0] 1788932730.464489 read_OD 0x735000 thru 0x73cfff (0x8000 bytes) 000411[0] 1788932730.494834 read_OD 0x5aa000 thru 0x5b8fff (0xf000 bytes) 000415[1] 1788932730.513097 mapread 0x3c2000 thru 0x3c83e8 (0x63e9 bytes) 000416[0] 1788932730.536145 punch from 0x66cb02 to 0x675c7e (0x917d bytes) 000417[1] 1788932730.565003 write_OD 0x771000 thru 0x77ffff (0xf000 bytes) 000420[0] 1788932730.617753 write_OD 0x86f000 thru 0x86ffff (0x1000 bytes) 000425[1] 1788932730.710738 read_OD 0x4b4000 thru 0x4c2fff (0xf000 bytes) 000426[1] 1788932730.720215 trunc from 0x870000 to 0x08be6b (0x7e4194 bytes) 000427[1] 1788932730.798307 trunc from 0x08be6c to 0x7f1e03 (0x765f98 bytes) 000428[0] 1788932730.858130 punch from 0x6b3664 to 0x6c0c79 (0xd616 bytes) 000430[1] 1788932730.882764 trunc from 0x7f1e04 to 0x05d6f6 (0x79470d bytes) 000432[1] 1788932730.962154 mapwrite 0x23b000 thru 0x23f19b (0x419c bytes) 000434[1] 1788932731.154390 mapread 0x004000 thru 0x00e47b (0xa47c bytes) 000435[1] 1788932731.195449 punch from 0x0af453 to 0x0b4a91 (0x563f bytes) 000436[1] 1788932731.200502 trunc from 0x23f19c to 0x85adb6 (0x61bc1b bytes) 000437[1] 1788932731.261389 punch from 0x233942 to 0x23662e (0x2ced bytes) 000438[1] 1788932731.276362 read_OD 0x6c5000 thru 0x6d1fff (0xd000 bytes) 000439[1] 1788932731.301796 read_OD 0x730000 thru 0x732fff (0x3000 bytes) 000440[0] 1788932731.314487 mapread 0x426000 thru 0x432e8d (0xce8e bytes) 000442[1] 1788932731.362229 trunc from 0x85adb7 to 0x424e65 (0x435f51 bytes) 000446[1] 1788932731.460289 read_OD 0x1bb000 thru 0x1c8fff (0xe000 bytes) 000449[0] 1788932731.483173 mapwrite 0x87b000 thru 0x87da8e (0x2a8f bytes) 000452[0] 1788932731.602030 mapwrite 0x3c5000 thru 0x3cc854 (0x7855 bytes) 000454[0] 1788932731.724482 read_OD 0x65f000 thru 0x66dfff (0xf000 bytes) 000456[0] 1788932731.740997 mapwrite 0x9aa000 thru 0x9aaf6c (0xf6d bytes) 000460[0] 1788932731.935038 mapread 0x453000 thru 0x453190 (0x191 bytes) 000462[0] 1788932731.965628 punch from 0x2d2276 to 0x2d62b7 (0x4042 bytes) 000463[1] 1788932731.973341 mapwrite 0x8f1000 thru 0x900503 (0xf504 bytes) 000466[0] 1788932732.130310 read_OD 0x6eb000 thru 0x6f0fff (0x6000 bytes) 000467[1] 1788932732.162499 mapwrite 0x4e6000 thru 0x4f4de7 (0xede8 bytes) 000471[1] 1788932732.288046 read_OD 0x5ad000 thru 0x5b0fff (0x4000 bytes) 000472[0] 1788932732.321796 trunc from 0x9aaf6d to 0x2c791d (0x6e364f bytes) 000473[0] 1788932732.353958 trunc from 0x2c791e to 0x40d8fb (0x145fde bytes) 000476[1] 1788932732.400187 mapread 0x0e8000 thru 0x0e9896 (0x1897 bytes) 000480[0] 1788932732.443186 write_OD 0x041000 thru 0x042fff (0x2000 bytes) 000484[1] 1788932732.505206 mapread 0x138000 thru 0x142dfa (0xadfb bytes) 000485[0] 1788932732.539023 punch from 0x280712 to 0x2894d6 (0x8dc5 bytes) 000486[0] 1788932732.562667 mapwrite 0x6d8000 thru 0x6e0dd6 (0x8dd7 bytes) 000488[1] 1788932732.716927 mapwrite 0x1a6000 thru 0x1abd0e (0x5d0f bytes) 000490[0] 1788932732.814614 punch from 0x13d3c3 to 0x13f0c8 (0x1d06 bytes) 000491[0] 1788932732.843374 trunc from 0x6e0dd7 to 0x720afa (0x3fd24 bytes) 000492[0] 1788932732.907680 punch from 0x5ca7b2 to 0x5d6839 (0xc088 bytes) 000493[1] 1788932732.924913 write_OD 0x0ca000 thru 0x0d2fff (0x9000 bytes) 000495[0] 1788932732.974951 mapread 0x113000 thru 0x115064 (0x2065 bytes) 000496[1] 1788932732.998422 mapwrite 0x581000 thru 0x585e29 (0x4e2a bytes) 000497[0] 1788932733.136382 write_OD 0x02f000 thru 0x039fff (0xb000 bytes) 000498[0] 1788932733.180254 read_OD 0x23f000 thru 0x247fff (0x9000 bytes) 000499[0] 1788932733.191006 trunc from 0x720afb to 0x276bab (0x4a9f4f bytes) 000500[0] 1788932733.229356 trunc from 0x276bac to 0x9561d6 (0x6df62b bytes) All operations completed A-OK! PASS 16a (57s) == sanityn test 16b: 500 iterations of dual-mount fsx at small size ========================================================== 01:45:39 (1788932739) keep default fallocate mode: 2 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.711497 s, 11.8 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.624282 s, 13.4 MB/s Chance of close/open is 1 in 50 Seed set to 2744 fd 0: /mnt/lustre/f16b.sanityn fd 1: /mnt/lustre2/f16b.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read truncating to largest ever: 0xcd860 000004[1] 1788932745.436085 trunc from 00000000 to 0x0cd85f (0xcd860 bytes) 000005[1] 1788932745.458550 punch from 0x08e5d2 to 0x08fd0c (0x173b bytes) 000007[0] 1788932745.541179 read 0x014e2e thru 0x01612d (0x1300 bytes) 000008[0] 1788932745.563042 mapread 0x078818 thru 0x079f16 (0x16ff bytes) 000011[0] 1788932745.616213 write 0x5f4c89 thru 0x5f62eb (0x1663 bytes) 000012[1] 1788932745.635759 mapwrite 0x06db70 thru 0x06de65 (0x2f6 bytes) 000013[1] 1788932745.740401 mapread 0x056223 thru 0x0570e3 (0xec1 bytes) truncating to largest ever: 0x74ba2b 000015[0] 1788932745.799364 trunc from 0x5f62ec to 0x74ba2a (0x15573f bytes) 000016[0] 1788932745.872617 punch from 0x345ed0 to 0x347bc7 (0x1cf8 bytes) 000017[1] 1788932745.889291 read 0x365621 thru 0x365bbd (0x59d bytes) 000019[1] 1788932745.927949 punch from 0x62ca4d to 0x62d3bc (0x970 bytes) 000020[0] 1788932745.972213 punch from 0x3807b8 to 0x38098e (0x1d7 bytes) 000021[0] 1788932746.018642 mapread 0x491bb8 thru 0x493420 (0x1869 bytes) 000022[1] 1788932746.070801 trunc from 0x74ba2b to 0x0815ac (0x6ca47e bytes) 000023[0] 1788932746.135394 read 0x074d4f thru 0x075591 (0x843 bytes) 000026[1] 1788932746.188418 read 0x01f108 thru 0x020e7d (0x1d76 bytes) 000027[1] 1788932746.211040 read 0x006801 thru 0x0068df (0xdf bytes) 000028[1] 1788932746.222233 punch from 0x061e54 to 0x061eff (0xac bytes) 000029[1] 1788932746.250081 punch from 0x041fc2 to 0x042136 (0x175 bytes) 000030[0] 1788932746.289918 punch from 0x04d990 to 0x04f50d (0x1b7e bytes) 000033[0] 1788932746.350226 mapwrite 0x7018a4 thru 0x702580 (0xcdd bytes) 000034[1] 1788932746.502364 mapwrite 0x21937f thru 0x21a6ac (0x132e bytes) 000036[0] 1788932746.728481 punch from 0x01ae2c to 0x01aef1 (0xc6 bytes) 000037[0] 1788932746.768089 mapwrite 0x1a72f4 thru 0x1a7d9f (0xaac bytes) 000038[1] 1788932747.005780 mapread 0x25bd4a thru 0x25c123 (0x3da bytes) 000039[0] 1788932747.123318 punch from 0x15ec4b to 0x15f08d (0x443 bytes) 000041[1] 1788932747.151758 mapwrite 0x97065d thru 0x971b7c (0x1520 bytes) 000042[1] 1788932747.555424 mapwrite 0x87c0a4 thru 0x87c599 (0x4f6 bytes) 000043[0] 1788932747.693663 write 0x9de139 thru 0x9df027 (0xeef bytes) 000044[0] 1788932747.744783 read 0x968403 thru 0x96958a (0x1188 bytes) 000045[1] 1788932747.772412 write 0x908a4e thru 0x908db6 (0x369 bytes) 000046[1] 1788932747.868868 mapread 0x985454 thru 0x9859f7 (0x5a4 bytes) 000047[0] 1788932747.969875 mapread 0x7d6493 thru 0x7d669b (0x209 bytes) 000049[1] 1788932748.083024 read 0x6c97fa thru 0x6c9904 (0x10b bytes) 000051[0] 1788932748.116794 mapwrite 0x0acf2b thru 0x0ae20e (0x12e4 bytes) 000053[0] 1788932748.403072 trunc from 0x9df028 to 0x218e81 (0x7c61a6 bytes) 000055[0] 1788932748.465418 punch from 0x0f0c30 to 0x0f2505 (0x18d6 bytes) 000056[0] 1788932748.538513 write 0x813c83 thru 0x81585b (0x1bd9 bytes) 000058[0] 1788932748.602420 mapwrite 0x3b8a5d thru 0x3b9b70 (0x1114 bytes) 000060[0] 1788932748.750199 read 0x46ced9 thru 0x46d4a7 (0x5cf bytes) 000062[0] 1788932748.820932 write 0x6ba19e thru 0x6ba5a4 (0x407 bytes) 000065[0] 1788932748.840541 write 0x3745a1 thru 0x374809 (0x269 bytes) 000066[0] 1788932748.939673 punch from 0x214d8e to 0x21627a (0x14ed bytes) 000067[0] 1788932749.021977 trunc from 0x81585c to 0x72d968 (0xe7ef3 bytes) 000069[0] 1788932749.116376 mapread 0x6df4b3 thru 0x6e07ab (0x12f9 bytes) 000070[0] 1788932749.149822 mapwrite 0x41b7b2 thru 0x41d0ea (0x1939 bytes) 000071[1] 1788932749.302460 mapread 0x2d55a2 thru 0x2d69a5 (0x1404 bytes) 000072[1] 1788932749.400194 read 0x43d384 thru 0x43e8af (0x152c bytes) 000074[1] 1788932749.459617 trunc from 0x72d969 to 0x04e153 (0x6df815 bytes) 000075[1] 1788932749.534207 punch from 0x03e7d0 to 0x040405 (0x1c36 bytes) 000078[0] 1788932749.629398 read 0x04ce13 thru 0x04dbe2 (0xdd0 bytes) 000079[1] 1788932749.748171 punch from 0x013222 to 0x014c6d (0x1a4c bytes) 000080[1] 1788932749.841706 punch from 0x01798f to 0x018555 (0xbc7 bytes) 000082[0] 1788932750.001446 mapread 0x01179b thru 0x012ac1 (0x1327 bytes) 000083[0] 1788932750.107073 mapwrite 0x19f6b5 thru 0x1a0d53 (0x169f bytes) 000084[0] 1788932750.460062 write 0x4c0456 thru 0x4c1cb6 (0x1861 bytes) 000086[1] 1788932750.512046 mapwrite 0x51b903 thru 0x51c1d6 (0x8d4 bytes) 000087[0] 1788932750.699209 trunc from 0x51c1d7 to 0x046759 (0x4d5a7d bytes) 000088[1] 1788932750.729357 mapwrite 0x424668 thru 0x424efb (0x894 bytes) 000089[0] 1788932750.941313 write 0x36edb0 thru 0x370b76 (0x1dc7 bytes) 000090[0] 1788932751.094266 mapwrite 0x7e5615 thru 0x7e6444 (0xe30 bytes) 000091[1] 1788932751.368399 write 0x3d2cf1 thru 0x3d371d (0xa2d bytes) 000092[0] 1788932751.439869 trunc from 0x7e6445 to 0x540515 (0x2a5f2f bytes) 000093[1] 1788932751.490176 read 0x1a4152 thru 0x1a5b15 (0x19c4 bytes) 000094[0] 1788932751.544310 read 0x4b83ac thru 0x4b8a6a (0x6bf bytes) 000096[0] 1788932751.595265 punch from 0x2f45c6 to 0x2f50fc (0xb37 bytes) 000098[1] 1788932751.623504 write 0x962165 thru 0x962a74 (0x910 bytes) 000100[0] 1788932751.687935 mapread 0x082c82 thru 0x083cdb (0x105a bytes) 000101[1] 1788932751.762323 mapwrite 0x6423ad thru 0x643ac3 (0x1717 bytes) truncating to largest ever: 0x8301a1 000102[0] 1788932751.894212 trunc from 0x962a75 to 0x8301a0 (0x1328d4 bytes) 000103[1] 1788932751.975216 trunc from 0x8301a1 to 0x00ecc1 (0x8214df bytes) 000105[1] 1788932752.073064 mapread 0x00351e thru 0x003a07 (0x4ea bytes) 000107[0] 1788932752.082306 mapread 0x009814 thru 0x00a4a4 (0xc91 bytes) 000108[0] 1788932752.151844 read 0x00a94c thru 0x00c3fb (0x1ab0 bytes) 000110[1] 1788932752.155642 write 0x76ce52 thru 0x76edd3 (0x1f82 bytes) 000111[1] 1788932752.160171 mapread 0x06a519 thru 0x06b80e (0x12f6 bytes) 000114[1] 1788932752.226883 mapread 0x52c07a thru 0x52de27 (0x1dae bytes) 000116[1] 1788932752.274427 write 0x6d7385 thru 0x6d7cbb (0x937 bytes) truncating to largest ever: 0x8b56af 000118[0] 1788932752.309998 trunc from 0x76edd4 to 0x8b56ae (0x1468db bytes) 000120[1] 1788932752.442430 trunc from 0x8b56af to 0x235ef5 (0x67f7b9 bytes) 000121[0] 1788932752.562171 mapread 0x2022fc thru 0x203967 (0x166c bytes) 000122[0] 1788932752.660256 read 0x21e18f thru 0x21e61f (0x491 bytes) 000123[1] 1788932752.671586 punch from 0x122fa9 to 0x124a94 (0x1aec bytes) 000126[0] 1788932752.833192 mapwrite 0x0eeaaa thru 0x0ef902 (0xe59 bytes) 000129[1] 1788932753.377539 write 0x903967 thru 0x90556a (0x1c04 bytes) 000130[1] 1788932753.489706 trunc from 0x90556b to 0x77ed7d (0x1867ed bytes) 000131[0] 1788932753.519185 write 0x6afa20 thru 0x6b0ed2 (0x14b3 bytes) 000132[0] 1788932753.567019 mapread 0x3e0325 thru 0x3e0dbf (0xa9b bytes) 000135[1] 1788932753.734483 read 0x631343 thru 0x63168d (0x34b bytes) 000140[0] 1788932753.763114 mapread 0x1b35dc thru 0x1b49b2 (0x13d7 bytes) 000142[1] 1788932753.773976 mapwrite 0x53ba73 thru 0x53c521 (0xaaf bytes) 000145[1] 1788932754.070074 mapwrite 0x2c86b5 thru 0x2c909a (0x9e6 bytes) 000146[1] 1788932754.176799 write 0x82423d thru 0x825f0e (0x1cd2 bytes) 000147[0] 1788932754.180889 write 0x6101b0 thru 0x610212 (0x63 bytes) 000148[0] 1788932754.244450 trunc from 0x825f0f to 0x4f95cd (0x32c941 bytes) 000149[0] 1788932754.381304 mapread 0x36b587 thru 0x36cb7d (0x15f7 bytes) 000152[1] 1788932754.433405 punch from 0x4b4c5d to 0x4b5251 (0x5f5 bytes) truncating to largest ever: 0x94f209 000154[0] 1788932754.518204 trunc from 0x4f95ce to 0x94f208 (0x455c3b bytes) 000155[0] 1788932754.622026 mapread 0x75d368 thru 0x75dc58 (0x8f1 bytes) 000156[0] 1788932754.788232 read 0x926f96 thru 0x928020 (0x108b bytes) 000159[1] 1788932754.819503 read 0x5412cd thru 0x5414bd (0x1f1 bytes) 000161[1] 1788932754.897189 mapwrite 0x1762ee thru 0x176325 (0x38 bytes) 000164[1] 1788932755.191898 write 0x94d560 thru 0x94e107 (0xba8 bytes) 000165[0] 1788932755.202188 read 0x081c6e thru 0x0831fe (0x1591 bytes) 000167[1] 1788932755.227371 read 0x0000fe thru 0x001790 (0x1693 bytes) 000168[0] 1788932755.268459 mapread 0x42fa5d thru 0x430d3f (0x12e3 bytes) 000170[0] 1788932755.285899 trunc from 0x94f209 to 0x8add43 (0xa14c5 bytes) 000171[1] 1788932755.370284 mapwrite 0x8711f1 thru 0x871633 (0x443 bytes) 000175[0] 1788932755.525449 trunc from 0x8add44 to 0x1b9ac2 (0x6f4281 bytes) 000176[0] 1788932755.570295 read 0x01a1d4 thru 0x01b4e3 (0x1310 bytes) 000177[1] 1788932755.612950 mapread 0x15a781 thru 0x15a946 (0x1c6 bytes) truncating to largest ever: 0x99a518 000178[0] 1788932755.664981 trunc from 0x1b9ac3 to 0x99a517 (0x7e0a55 bytes) 000179[1] 1788932755.720504 read 0x97f412 thru 0x97ff34 (0xb23 bytes) 000180[1] 1788932755.857832 write 0x51fa56 thru 0x52075a (0xd05 bytes) 000181[0] 1788932755.960034 mapwrite 0x6ad434 thru 0x6ae759 (0x1326 bytes) 000182[0] 1788932756.246181 mapwrite 0x46f7ea thru 0x470454 (0xc6b bytes) 000183[1] 1788932756.328485 read 0x3a5d3c thru 0x3a6134 (0x3f9 bytes) 000185[0] 1788932756.365641 mapwrite 0x978013 thru 0x978118 (0x106 bytes) 000187[1] 1788932756.606815 write 0x854ea0 thru 0x855b46 (0xca7 bytes) 000189[1] 1788932756.701609 write 0x535974 thru 0x537767 (0x1df4 bytes) 000191[0] 1788932756.774358 punch from 0x1420d4 to 0x143408 (0x1335 bytes) 000195[0] 1788932756.881844 trunc from 0x99a518 to 0x5ae4fd (0x3ec01a bytes) 000197[1] 1788932756.986438 write 0x764661 thru 0x76650b (0x1eab bytes) 000198[0] 1788932757.026770 mapwrite 0x08618e thru 0x086cc7 (0xb3a bytes) 000199[1] 1788932757.143056 punch from 0x32d62e to 0x32de27 (0x7fa bytes) 000200[1] 1788932757.225531 mapread 0x0e7a77 thru 0x0e7ca8 (0x232 bytes) 000201[1] 1788932757.241963 mapread 0x66efa6 thru 0x66f317 (0x372 bytes) 000203[0] 1788932757.308092 mapwrite 0x3b5f6e thru 0x3b70dd (0x1170 bytes) 000204[1] 1788932757.429842 mapread 0x44d66c thru 0x44dbd7 (0x56c bytes) 000207[0] 1788932757.481304 mapwrite 0x3ce8c4 thru 0x3cf84e (0xf8b bytes) 000208[0] 1788932757.559329 mapwrite 0x82f859 thru 0x830730 (0xed8 bytes) 000209[1] 1788932757.759781 write 0x044be8 thru 0x04505c (0x475 bytes) 000211[0] 1788932757.829136 read 0x7c2688 thru 0x7c4630 (0x1fa9 bytes) 000215[0] 1788932757.922152 trunc from 0x830731 to 0x36fc5b (0x4c0ad5 bytes) 000217[0] 1788932758.005319 mapwrite 0x42acf2 thru 0x42c4d6 (0x17e5 bytes) 000219[1] 1788932758.213962 mapwrite 0x002fe7 thru 0x004de7 (0x1e01 bytes) 000221[0] 1788932758.452202 mapwrite 0x7eea1f thru 0x7f0771 (0x1d53 bytes) 000222[0] 1788932758.664755 punch from 0x003ed8 to 0x00523a (0x1363 bytes) 000224[0] 1788932758.745236 read 0x26271d thru 0x262cec (0x5d0 bytes) 000225[0] 1788932758.752342 read 0x676d49 thru 0x6785d7 (0x188f bytes) 000226[0] 1788932758.801622 mapwrite 0x549072 thru 0x5494a7 (0x436 bytes) 000228[0] 1788932758.897623 trunc from 0x7f0772 to 0x8002b7 (0xfb46 bytes) 000229[1] 1788932758.933509 trunc from 0x8002b8 to 0x5bff06 (0x2403b1 bytes) 000230[0] 1788932758.984520 read 0x3906bf thru 0x3923bf (0x1d01 bytes) 000231[0] 1788932759.021188 mapwrite 0x73b5d7 thru 0x73b892 (0x2bc bytes) 000232[1] 1788932759.187283 trunc from 0x73b893 to 0x0fcf4d (0x63e945 bytes) 000233[0] 1788932759.275427 punch from 0x072e19 to 0x072ee7 (0xcf bytes) 000234[1] 1788932759.317822 trunc from 0x0fcf4e to 0x07b947 (0x81606 bytes) 000235[0] 1788932759.368632 punch from 0x00a9c9 to 0x00bf6a (0x15a2 bytes) 000236[0] 1788932759.437517 trunc from 0x07b948 to 0x05336f (0x285d8 bytes) 000237[1] 1788932759.512164 punch from 0x021d6f to 0x0222f2 (0x584 bytes) 000238[0] 1788932759.563267 write 0x107f9e thru 0x108a47 (0xaaa bytes) 000239[0] 1788932759.621466 mapwrite 0x7261eb thru 0x726553 (0x369 bytes) 000240[1] 1788932759.775892 read 0x70c040 thru 0x70c838 (0x7f9 bytes) 000242[0] 1788932759.846152 write 0x61cfbb thru 0x61d27c (0x2c2 bytes) 000245[1] 1788932759.952477 punch from 0x2af230 to 0x2afb95 (0x966 bytes) 000246[1] 1788932760.046409 trunc from 0x726554 to 0x408de0 (0x31d773 bytes) 000249[1] 1788932760.193848 mapread 0x0bbaea thru 0x0bd8f0 (0x1e07 bytes) 000250[0] 1788932760.324792 punch from 0x315108 to 0x315af6 (0x9ef bytes) 000251[0] 1788932760.386029 mapread 0x070f63 thru 0x072dd0 (0x1e6e bytes) 000252[0] 1788932760.447361 punch from 0x3bc276 to 0x3bd984 (0x170f bytes) 000253[1] 1788932760.465242 mapwrite 0x105d61 thru 0x1073e6 (0x1686 bytes) 000256[0] 1788932760.586109 punch from 0x272b21 to 0x2732be (0x79e bytes) 000258[1] 1788932760.640534 mapread 0x3dfd65 thru 0x3e073e (0x9da bytes) 000259[1] 1788932760.679422 mapread 0x345070 thru 0x3465c0 (0x1551 bytes) 000263[1] 1788932760.739952 trunc from 0x408de1 to 0x211279 (0x1f7b67 bytes) 000264[0] 1788932760.801250 write 0x450e7b thru 0x4513ff (0x585 bytes) 000265[0] 1788932760.835110 write 0x899753 thru 0x89b160 (0x1a0e bytes) 000266[1] 1788932760.867482 punch from 0x2dc318 to 0x2ddc6b (0x1954 bytes) 000267[1] 1788932760.908537 read 0x39efa9 thru 0x39fc0b (0xc63 bytes) 000271[0] 1788932760.990267 mapread 0x0f7e2e thru 0x0f937c (0x154f bytes) 000274[0] 1788932761.050016 mapwrite 0x0386a2 thru 0x03917a (0xad9 bytes) 000276[1] 1788932761.111103 trunc from 0x89b161 to 0x0be0b3 (0x7dd0ad bytes) 000278[1] 1788932761.208301 read 0x05e86e thru 0x05ffd2 (0x1765 bytes) 000280[0] 1788932761.288535 punch from 0x0a65bb to 0x0a6a59 (0x49f bytes) 000281[1] 1788932761.322099 punch from 0x014af8 to 0x0158ea (0xdf3 bytes) 000285[1] 1788932761.428381 punch from 0x0aca16 to 0x0ae930 (0x1f1b bytes) 000288[1] 1788932761.480836 read 0x02e2b0 thru 0x02f3a0 (0x10f1 bytes) 000289[1] 1788932761.501550 mapwrite 0x1fb5a4 thru 0x1fc74d (0x11aa bytes) 000293[0] 1788932761.642444 trunc from 0x1fc74e to 0x244fea (0x4889d bytes) 000294[0] 1788932761.701088 trunc from 0x244feb to 0x502c04 (0x2bdc1a bytes) 000296[0] 1788932761.727385 write 0x721b09 thru 0x722f74 (0x146c bytes) 000297[0] 1788932761.765267 mapread 0x03800d thru 0x03848a (0x47e bytes) 000298[0] 1788932761.807267 write 0x2dffc3 thru 0x2e01ee (0x22c bytes) 000299[0] 1788932761.839993 punch from 0x47649b to 0x4780d2 (0x1c38 bytes) 000303[0] 1788932761.890444 mapread 0x6ace16 thru 0x6ae685 (0x1870 bytes) 000304[0] 1788932761.936637 mapread 0x28d005 thru 0x28e6a9 (0x16a5 bytes) 000308[0] 1788932761.967195 trunc from 0x722f75 to 0x953877 (0x230903 bytes) 000311[1] 1788932762.024970 mapwrite 0x037a07 thru 0x039370 (0x196a bytes) 000312[1] 1788932762.183224 write 0x8c22a4 thru 0x8c3315 (0x1072 bytes) 000313[0] 1788932762.207463 read 0x5982b6 thru 0x599e5a (0x1ba5 bytes) 000314[0] 1788932762.231819 write 0x6e56dc thru 0x6e697b (0x12a0 bytes) 000317[0] 1788932762.313153 read 0x24760f thru 0x247d39 (0x72b bytes) 000318[0] 1788932762.362511 mapread 0x2758af thru 0x276bbe (0x1310 bytes) 000322[0] 1788932762.435610 mapread 0x042053 thru 0x0428ea (0x898 bytes) 000323[1] 1788932762.454527 punch from 0x53aac3 to 0x53be15 (0x1353 bytes) 000326[1] 1788932762.548013 write 0x0e009c thru 0x0e0120 (0x85 bytes) 000327[0] 1788932762.588375 write 0x8fa1f6 thru 0x8fa805 (0x610 bytes) 000328[1] 1788932762.649614 read 0x8d89f0 thru 0x8da684 (0x1c95 bytes) 000330[0] 1788932762.707463 punch from 0x6bd3ad to 0x6be3b6 (0x100a bytes) 000333[0] 1788932762.740036 mapwrite 0x3704f9 thru 0x371070 (0xb78 bytes) 000334[1] 1788932762.864059 mapwrite 0x178337 thru 0x17a093 (0x1d5d bytes) 000336[1] 1788932763.097900 trunc from 0x953878 to 0x97c9eb (0x29174 bytes) 000341[1] 1788932763.193183 read 0x430244 thru 0x431a9d (0x185a bytes) 000343[0] 1788932763.249506 mapwrite 0x620662 thru 0x621eed (0x188c bytes) 000345[1] 1788932763.365427 punch from 0x376396 to 0x3766e7 (0x352 bytes) 000346[1] 1788932763.381246 read 0x14b366 thru 0x14b48b (0x126 bytes) 000348[1] 1788932763.417107 mapwrite 0x5a4365 thru 0x5a453f (0x1db bytes) 000353[1] 1788932763.558334 mapwrite 0x8f3ff6 thru 0x8f54b3 (0x14be bytes) 000355[0] 1788932763.630287 punch from 0x92db0f to 0x92eeb9 (0x13ab bytes) 000356[0] 1788932763.656387 write 0x1f2e60 thru 0x1f3884 (0xa25 bytes) 000357[0] 1788932763.681955 read 0x646f01 thru 0x647b38 (0xc38 bytes) 000359[0] 1788932763.713538 mapwrite 0x3b8b1f thru 0x3b9fb0 (0x1492 bytes) 000360[0] 1788932763.765274 punch from 0x356a63 to 0x357fb6 (0x1554 bytes) 000361[1] 1788932763.782106 mapread 0x723156 thru 0x7231b4 (0x5f bytes) 000362[0] 1788932763.808530 punch from 0x4366bd to 0x4384d1 (0x1e15 bytes) 000363[1] 1788932763.860097 mapwrite 0x71c34f thru 0x71de02 (0x1ab4 bytes) 000364[0] 1788932764.052847 punch from 0x18ee08 to 0x18f900 (0xaf9 bytes) 000365[1] 1788932764.096792 punch from 0x238e71 to 0x239348 (0x4d8 bytes) 000367[1] 1788932764.204170 mapwrite 0x295d4f thru 0x296f51 (0x1203 bytes) 000371[1] 1788932764.334995 mapwrite 0x635024 thru 0x63607b (0x1058 bytes) 000376[0] 1788932764.510187 trunc from 0x97c9ec to 0x572439 (0x40a5b2 bytes) 000377[1] 1788932764.589165 trunc from 0x57243a to 0x3f50ec (0x17d34d bytes) 000378[1] 1788932764.634306 punch from 0x0ea65d to 0x0ea79c (0x140 bytes) 000379[1] 1788932764.640423 mapread 0x185b15 thru 0x1876c5 (0x1bb1 bytes) 000380[0] 1788932764.704050 punch from 0x3337cf to 0x334279 (0xaab bytes) 000384[1] 1788932764.805982 mapread 0x29a863 thru 0x29c589 (0x1d27 bytes) 000385[0] 1788932764.851520 write 0x03c98d thru 0x03e8ab (0x1f1f bytes) 000387[0] 1788932764.925695 trunc from 0x3f50ed to 0x62d97a (0x23888e bytes) 000389[0] 1788932765.001060 punch from 0x39fdef to 0x3a1c64 (0x1e76 bytes) 000390[0] 1788932765.006397 write 0x30ed37 thru 0x310a4d (0x1d17 bytes) 000391[1] 1788932765.016756 mapwrite 0x8ef623 thru 0x8f0ef2 (0x18d0 bytes) 000392[1] 1788932765.195770 trunc from 0x8f0ef3 to 0x106af6 (0x7ea3fc bytes) 000395[1] 1788932765.290365 write 0x903b1c thru 0x904cfe (0x11e3 bytes) 000398[1] 1788932765.350528 mapwrite 0x9b62e7 thru 0x9b7988 (0x16a2 bytes) 000400[1] 1788932765.499498 write 0x445eee thru 0x446211 (0x324 bytes) 000402[0] 1788932765.512977 mapwrite 0x60c8e3 thru 0x60e69e (0x1dbc bytes) 000403[0] 1788932765.667108 trunc from 0x9b7989 to 0x40b7a2 (0x5ac1e6 bytes) 000408[0] 1788932765.801108 write 0x4dac36 thru 0x4dae57 (0x222 bytes) 000413[0] 1788932765.865900 trunc from 0x4dae58 to 0x78dc53 (0x2b2dfc bytes) 000416[0] 1788932765.897505 mapread 0x4d1019 thru 0x4d21df (0x11c7 bytes) 000418[0] 1788932765.908260 mapwrite 0x0790aa thru 0x079d94 (0xceb bytes) 000419[0] 1788932766.017439 write 0x11302f thru 0x1136df (0x6b1 bytes) 000420[0] 1788932766.055850 punch from 0x298a5a to 0x299bd6 (0x117d bytes) 000422[1] 1788932766.093500 mapwrite 0x55d779 thru 0x55da4f (0x2d7 bytes) 000423[0] 1788932766.200658 write 0x9c22d2 thru 0x9c300f (0xd3e bytes) 000425[1] 1788932766.234251 trunc from 0x9c3010 to 0x75e2c7 (0x264d48 bytes) 000426[0] 1788932766.303056 write 0x057fe1 thru 0x059dd4 (0x1df4 bytes) 000427[0] 1788932766.362120 mapwrite 0x116111 thru 0x1174bf (0x13af bytes) 000428[0] 1788932766.498337 trunc from 0x75e2c8 to 0x7b0809 (0x52542 bytes) 000429[0] 1788932766.552094 mapread 0x4590ba thru 0x45ae30 (0x1d77 bytes) 000430[1] 1788932766.602932 mapread 0x194231 thru 0x194689 (0x459 bytes) 000431[0] 1788932766.674089 trunc from 0x7b080a to 0x7e655c (0x35d53 bytes) 000432[1] 1788932766.731489 punch from 0x756ebc to 0x757bda (0xd1f bytes) 000434[0] 1788932766.857447 read 0x4c2633 thru 0x4c374d (0x111b bytes) 000435[1] 1788932766.919289 trunc from 0x7e655d to 0x01be4a (0x7ca712 bytes) 000436[1] 1788932767.030152 write 0x646807 thru 0x647279 (0xa73 bytes) 000437[0] 1788932767.086663 mapread 0x132b8e thru 0x13419d (0x1610 bytes) 000438[1] 1788932767.167426 mapread 0x41f075 thru 0x42013c (0x10c8 bytes) 000439[0] 1788932767.204069 read 0x0f7b79 thru 0x0f8730 (0xbb8 bytes) 000440[1] 1788932767.221900 punch from 0x51b68e to 0x51d0bd (0x1a30 bytes) 000441[1] 1788932767.226207 trunc from 0x64727a to 0x15733e (0x4eff3b bytes) 000442[0] 1788932767.279035 write 0x506859 thru 0x506d0f (0x4b7 bytes) 000443[0] 1788932767.343399 trunc from 0x506d10 to 0x552272 (0x4b563 bytes) 000444[1] 1788932767.431155 mapwrite 0x514f96 thru 0x5157e5 (0x850 bytes) 000445[0] 1788932767.614195 write 0x693915 thru 0x6945ff (0xceb bytes) 000446[0] 1788932767.668804 trunc from 0x694600 to 0x94922c (0x2b4c2d bytes) 000447[1] 1788932767.696092 trunc from 0x94922d to 0x78bfc2 (0x1bd26a bytes) 000449[0] 1788932767.768405 mapwrite 0x4dde2d thru 0x4df10d (0x12e1 bytes) 000450[0] 1788932767.884688 trunc from 0x78bfc3 to 0x85d710 (0xd174e bytes) 000451[1] 1788932767.923402 write 0x765d3d thru 0x767535 (0x17f9 bytes) 000452[1] 1788932767.972083 read 0x7fa340 thru 0x7fc159 (0x1e1a bytes) 000454[1] 1788932768.033826 trunc from 0x85d711 to 0x423ec6 (0x43984a bytes) 000455[1] 1788932768.124032 mapread 0x04a203 thru 0x04adeb (0xbe9 bytes) 000459[1] 1788932768.262872 read 0x05f41b thru 0x061320 (0x1f06 bytes) 000460[0] 1788932768.304764 read 0x3a1af7 thru 0x3a3331 (0x183b bytes) 000462[1] 1788932768.370967 mapwrite 0x336006 thru 0x337646 (0x1641 bytes) 000463[0] 1788932768.463086 read 0x0e939d thru 0x0ead4d (0x19b1 bytes) 000464[1] 1788932768.497057 mapread 0x27385d thru 0x2745f0 (0xd94 bytes) 000466[1] 1788932768.532966 write 0x329f58 thru 0x32ac09 (0xcb2 bytes) 000472[1] 1788932768.597444 punch from 0x163240 to 0x16354e (0x30f bytes) 000473[0] 1788932768.602049 read 0x032b79 thru 0x03414a (0x15d2 bytes) 000474[0] 1788932768.633211 punch from 0x055f33 to 0x0568c8 (0x996 bytes) 000475[0] 1788932768.659829 mapwrite 0x11cd89 thru 0x11d75b (0x9d3 bytes) 000476[0] 1788932768.761979 mapwrite 0x1f5a90 thru 0x1f7923 (0x1e94 bytes) 000478[0] 1788932768.893230 mapread 0x249064 thru 0x24944e (0x3eb bytes) 000479[0] 1788932768.920551 read 0x37a403 thru 0x37aa39 (0x637 bytes) 000480[1] 1788932769.017639 punch from 0x343460 to 0x343905 (0x4a6 bytes) 000481[0] 1788932769.050865 punch from 0x04727b to 0x047fad (0xd33 bytes) 000482[0] 1788932769.085118 write 0x354e64 thru 0x3550f6 (0x293 bytes) 000484[0] 1788932769.166039 mapwrite 0x99584a thru 0x99608f (0x846 bytes) 000488[0] 1788932769.418215 trunc from 0x996090 to 0x0cc884 (0x8c980b bytes) 000489[0] 1788932769.468388 write 0x7cd586 thru 0x7ce6aa (0x1125 bytes) 000490[0] 1788932769.477966 read 0x73e358 thru 0x73ebf6 (0x89f bytes) 000491[1] 1788932769.483279 mapread 0x55933b thru 0x5595bf (0x285 bytes) 000493[0] 1788932769.544220 mapwrite 0x0bb982 thru 0x0bd424 (0x1aa3 bytes) 000495[0] 1788932769.601503 mapread 0x3ae2e2 thru 0x3af946 (0x1665 bytes) truncating to largest ever: 0x9bfb14 000496[0] 1788932769.624032 trunc from 0x7ce6ab to 0x9bfb13 (0x1f1469 bytes) 000497[1] 1788932769.663492 punch from 0x1dfe84 to 0x1e0416 (0x593 bytes) 000499[0] 1788932769.707072 punch from 0x6c778b to 0x6c935a (0x1bd0 bytes) All operations completed A-OK! PASS 16b (36s) == sanityn test 16c: verify data consistency on ldiskfs with cache disabled (b=17397) ========================================================== 01:46:15 (1788932775) SKIP: sanityn test_16c dio on ldiskfs only SKIP 16c (3s) == sanityn test 16d: Verify DIO and buffer IO with two clients ========================================================== 01:46:19 (1788932779) 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.408961 s, 20.5 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.272786 s, 30.8 MB/s ldlm.namespaces.MGC192.168.206.145@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff9fbb42a51000.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff9fbb50d9c000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff9fbb42a51000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff9fbb50d9c000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff9fbb42a51000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff9fbb50d9c000.lru_size=clear 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.4266 s, 10.1 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 3.02107 s, 34.7 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 2.90331 s, 36.1 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.85257 s, 17.9 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.87564 s, 13.3 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 3.83668 s, 27.3 MB/s PASS 16d (47s) == sanityn test 16e: Verify size consistency for O_DIRECT write ========================================================== 01:47:05 (1788932825) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.214022 s, 49.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0998624 s, 10.5 MB/s PASS 16e (6s) == sanityn test 16f: rw sequential consistency vs drop_caches ========================================================== 01:47:11 (1788932831) TIMEOUT PASS 16f (29s) == sanityn test 16g: mmap rw sequential consistency vs drop_caches ========================================================== 01:47:40 (1788932860) TIMEOUT PASS 16g (28s) == sanityn test 16h: mmap read after truncate file ======= 01:48:09 (1788932889) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16h (7s) == sanityn test 16i: read after truncate file ============ 01:48:17 (1788932897) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16i (6s) == sanityn test 16j: race dio with buffered i/o ========== 01:48:23 (1788932903) 1+0 records in 1+0 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.53921 s, 17.0 MB/s bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.42219 s, 18.4 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.01325 s, 8.7 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.0419 s, 12.8 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.15032 s, 8.3 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.50084 s, 10.5 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.90951 s, 9.0 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.2553 s, 11.6 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.33191 s, 11.2 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.48824 s, 7.5 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.73655 s, 7.0 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.85979 s, 9.2 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.98514 s, 8.8 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.553 s, 7.4 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.89113 s, 6.7 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 5.28215 s, 5.0 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 5.63498 s, 4.7 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 6.07608 s, 4.3 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 6.07654 s, 4.3 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK PASS 16j (46s) == sanityn test 16k: Parallel FSX and drop caches should not panic ========================================================== 01:49:09 (1788932949) + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.1 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.2 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.3 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.4 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.5 -l 67108864 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported truncating to largest ever: 0x1a7062c fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported truncating to largest ever: 0x232e1d3 skipping zero size read truncating to largest ever: 0x1a7062c truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x3c569db truncating to largest ever: 0x32cce75 truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! + fsx 1 done: rc=0 All operations completed A-OK! Wed Sep 9 01:50:01 EDT 2026 + fsx 2 done: rc=0 Wed Sep 9 01:50:01 EDT 2026 + fsx 3 done: rc=0 All operations completed A-OK! Wed Sep 9 01:50:01 EDT 2026 + fsx 4 done: rc=0 Wed Sep 9 01:50:01 EDT 2026 + fsx 5 done: rc=0 Wed Sep 9 01:50:01 EDT 2026 PASS 16k (63s) == sanityn test 17: resource creation/LVB creation race ========================================================================= 01:50:12 (1788933012) fail_loc=0x8000030a PASS 17 (13s) == sanityn test 18: mmap sanity check =========================================================================================== 01:50:25 (1788933025) mmap test1: basic mmap operation (PASS, 0.21655s) mmap test2: MAP_PRIVATE not write back (PASS, 0.10042s) mmap test3: concurrent mmap ops on two nodes (PASS, 8.6993s) mmap test4: c1 write to f1 from mmapped f2, c2 write to f1 from mmapped f1 (PASS, 4.6395s) mmap test5: read/write file to/from the buffer which mmapped to just this file (PASS, 0.18591s) mmap test6: check mmap write/read content on two nodes (PASS, 0.25206s) mmap test7: file i/o with an unmapped buffer (SKIPPED, 0s) mmap test8: SIGBUS for beyond file size (SKIPPED, 0s) mmap test9: SIGBUS for negative file offset (SKIPPED, 0s) mmap test10: multi-thread mmap access (PASS, 1.9836s) mmap test11: multi-thread shared mmap (PASS, 1.537s) mmap test10: mtime not change for readonly mmap access (PASS, 1.1532s) PASS 18 (29s) == sanityn test 19: test concurrent uncached read races ========================================================================= 01:50:54 (1788933054) oleg645-server: error: get_param: param_path 'osd-*/lustre-OST*/read_cache_enable': No such file or directory pdsh@oleg645-client: oleg645-server: ssh exited with exit code 2 SKIP: sanityn test_19 not cache-capable obdfilter SKIP 19 (5s) == sanityn test 20: test extra readahead page left in cache ============================================================== 01:50:59 (1788933059) PASS 20 (9s) == sanityn test 21: Try to remove mountpoint on another dir ============================================================== 01:51:08 (1788933068) rmdir: removing directory, '/mnt/lustre/d21.sanityn' rmdir: failed to remove '/mnt/lustre/d21.sanityn': Device or resource busy rmdir: removing directory, '/mnt/lustre2/d21.sanityn' Removed mounted directory from another mountpoint, needs to be fixed PASS 21 (8s) == sanityn test 23: others should see updated atime while another read============================================================== 01:51:16 (1788933076) now is 1788933079 starting reads multiop /mnt/lustre/f23.sanityn vor20_c TMPPIPE=/tmp/multiop_open_wait_pipe.7610 new atime is 1788933140 PASS 23 (72s) == sanityn test 24a: lfs df [-ih] [path] test =================================================================================== 01:52:28 (1788933148) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 5120 3764224 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 5120 3764224 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre2 UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 115.3K 403 114.9K 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 114.5K 399 114.1K 1% /mnt/lustre[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.3K 403 114.9K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 399 114.1K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre2 UUID bytes Used Available Use% Mounted on lustre-MDT0000_UUID 2.1G 2.1M 2.1G 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3.6G 5.0M 3.6G 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3.6G 32.0M 3.6G 1% /mnt/lustre[OST:1] filesystem_summary: 7.2G 37.0M 7.2G 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118099 403 117696 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117231 399 116832 1% /mnt/lustre2[OST:1] filesystem_summary: 234914 386 234528 1% /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 5120 3764224 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.3K 403 114.9K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 399 114.1K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 118099 403 117696 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 117231 399 116832 1% /mnt/lustre[OST:1] filesystem_summary: 234914 386 234528 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118099 403 117696 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117231 399 116832 1% /mnt/lustre2[OST:1] filesystem_summary: 234914 386 234528 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 5120 3764224 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 5120 3764224 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre2 PASS 24a (7s) == sanityn test 24b: lfs df should show both filesystems ========================================================================= 01:52:35 (1788933155) PASS 24b (8s) == sanityn test 25a: change ACL on one mountpoint be seen on another ============================================================= 01:52:43 (1788933163) running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] PASS 25a (9s) == sanityn test 25b: change ACL under remote dir on one mountpoint be seen on another ========================================================== 01:52:52 (1788933172) SKIP: sanityn test_25b needs >= 2 MDTs SKIP 25b (3s) == sanityn test 26a: allow mtime to get older ============ 01:52:56 (1788933176) utime: good mknod times 1788933177 <= 1788933178 <= 1788933178 for /mnt/lustre/f26a utime: good mknod times 1788933177 <= 1788933178 <= 1788933178 for /mnt/lustre2/f26a utime: good utime mtimes 100000, atime 200000 utime: updated times for /mnt/lustre2/f26a PASS 26a (9s) == sanityn test 26b: sync mtime between ost and mds ====== 01:53:04 (1788933184) PASS 26b (9s) == sanityn test 26c: set-in-past on open file is not lost on close ========================================================== 01:53:13 (1788933193) multiop /mnt/lustre/f26c.sanityn vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7610 File: /mnt/lustre/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502398 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-09-09 01:53:14.000000000 -0400 Birth: 2026-09-09 01:53:13.000000000 -0400 File: /mnt/lustre2/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502398 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-09-09 01:53:14.000000000 -0400 Birth: 2026-09-09 01:53:13.000000000 -0400 PASS 26c (7s) SKIP: sanityn test_27 skipping excluded test 27 == sanityn test 30: recreate file race =================== 01:53:22 (1788933202) 0 PASS 30 (9s) == sanityn test 31a: voluntary cancel / blocking ast race======================================================================== 01:53:31 (1788933211) fail_loc=0x314 PASS 31a (8s) == sanityn test 31b: voluntary OST cancel / blocking ast race======================================================================== 01:53:39 (1788933219) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg645-server mds-ost sync done. sleep 5 for ZFS MDS Waiting for MDT destroys to complete fail_loc=0x314 fail_loc=0x316 fail_loc=0 fail_loc=0 Filesystem 1K-blocks Used Available Use% Mounted on 192.168.206.145@tcp:/lustre 7542784 38912 7497728 1% /mnt/lustre2 PASS 31b (24s) == sanityn test 31r: open-rename(replace) race =========== 01:54:03 (1788933243) -rw-r--r-- 1 root root 0 Sep 9 01:54 /mnt/lustre/f31r.sanityn.target fail_loc=0x1419 fail_val=3 PASS 31r (10s) == sanityn test 31s: open should not revalidate invalid dentry ========================================================== 01:54:13 (1788933253) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 1205: 43152 Terminated tail -f $tfile PASS 31s (9s) == sanityn test 31t: getattr should not revalidate invalid dentry ========================================================== 01:54:22 (1788933262) PASS 31t (10s) == sanityn test 32b: lockless i/o ======================== 01:54:32 (1788933272) SKIP: sanityn test_32b max_nolock_bytes is removed >= 2.17.53 SKIP 32b (5s) == sanityn test 32c: contention detection ================ 01:54:37 (1788933277) ldlm.namespaces.filter-lustre-OST0000_UUID.contended_locks=2 ldlm.namespaces.filter-lustre-OST0001_UUID.contended_locks=2 ldlm.namespaces.filter-lustre-OST0000_UUID.contention_events=0 ldlm.namespaces.filter-lustre-OST0001_UUID.contention_events=0 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.76112 s, 19.0 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.90444 s, 27.5 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.72071 s, 30.5 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.93547 s, 27.1 MB/s contention_seconds 4, contention_events 16 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.08238 s, 25.2 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.40179 s, 21.8 MB/s ldlm.namespaces.filter-lustre-OST0000_UUID.contention_events=0 ldlm.namespaces.filter-lustre-OST0001_UUID.contention_events=0 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.55621 s, 33.7 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.56375 s, 33.5 MB/s PASS 32c (34s) SKIP: sanityn test_33a skipping SLOW test 33a == sanityn test 33b: COS: cross create/delete, 2 clients, benchmark under remote dir ========================================================== 01:55:13 (1788933313) SKIP: sanityn test_33b Need two or more clients, have 1 SKIP 33b (4s) == sanityn test 33c: Cancel cross-MDT lock should trigger Sync-on-Lock-Cancel ========================================================== 01:55:18 (1788933318) SKIP: sanityn test_33c needs >= 2 MDTs SKIP 33c (5s) == sanityn test 33d: dependent transactions should trigger COS ========================================================== 01:55:22 (1788933322) SKIP: sanityn test_33d needs >= 2 MDTs SKIP 33d (4s) == sanityn test 33e: independent transactions shouldn't trigger COS ========================================================== 01:55:26 (1788933326) SKIP: sanityn test_33e needs >= 2 MDTs SKIP 33e (4s) == sanityn test 34: no lock timeout under IO ============= 01:55:30 (1788933330) fail_loc=0x512 fail_loc=0x512 lock should not expire writing on client1 reading on client2 success fail_loc=0x511 fail_loc=0x511 lock should expire writing on client1 reading on client2 success oleg645-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9fbb42a51000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9fbb42a51000.ost_server_uuid in IDLE state after 0 sec oleg645-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9fbb42a51000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9fbb42a51000.ost_server_uuid in IDLE state after 0 sec oleg645-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9fbb42a51000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9fbb42a51000.ost_server_uuid in IDLE state after 0 sec oleg645-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9fbb42a51000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9fbb42a51000.ost_server_uuid in IDLE state after 0 sec oleg645-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9fbb42a51000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9fbb42a51000.ost_server_uuid in IDLE state after 0 sec oleg645-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9fbb42a51000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9fbb42a51000.ost_server_uuid in IDLE state after 0 sec PASS 34 (148s) == sanityn test 35: -EINTR cp_ast vs. bl_ast race does not evict client ========================================================== 01:57:58 (1788933478) fail_loc=0x80000317 Race attempt 0 Wait for 50457 50480 for 60 sec... /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 1855: 50457 Killed createmany -o $MOUNT2/$tdir/a 4000 fail_loc=0x0 Filesystem Size Used Avail Use% Mounted on 192.168.206.145@tcp:/lustre 7.2G 39M 7.2G 1% /mnt/lustre 192.168.206.145@tcp:/lustre 7.2G 39M 7.2G 1% /mnt/lustre2 - open/close 635 (time 1788933496.36 total 10.01 last 63.44) - open/close 1329 (time 1788933506.36 total 20.01 last 69.39) - open/close 1944 (time 1788933516.38 total 30.03 last 61.40) - open/close 2616 (time 1788933526.39 total 40.03 last 67.15) - open/close 3212 (time 1788933536.39 total 50.04 last 59.57) - open/close 3714 (time 1788933546.42 total 60.07 last 50.05) total: 4000 open/close in 68.12 seconds: 58.72 ops/second PASS 35 (76s) == sanityn test 36: handle ESTALE/open-unlink correctly == 01:59:14 (1788933554) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg645-server mds-ost sync done. Waiting for MDT destroys to complete 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.837324 s, 62.6 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7610 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(0) *** before(7498752) after_dd(7440384) after(7491584) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.597303 s, 87.8 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7610 sleep 5 for ZFS MDS Waiting for MDT destroys to complete *** cycle(1) *** before(7491584) after_dd(7428096) after(7479296) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.801666 s, 65.4 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7610 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(2) *** before(7479296) after_dd(7435264) after(7486464) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.18335 s, 44.3 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7610 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(3) *** before(7486464) after_dd(7440384) after(7498752) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.863513 s, 60.7 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7610 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(4) *** before(7498752) after_dd(7440384) after(7491584) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.591744 s, 88.6 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7610 sleep 5 for ZFS MDS Waiting for MDT destroys to complete *** cycle(5) *** before(7491584) after_dd(7441408) after(7492608) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.625057 s, 83.9 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7610 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(6) *** before(7492608) after_dd(7431168) after(7482368) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.61829 s, 84.8 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7610 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(7) *** before(7482368) after_dd(7440384) after(7491584) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.590111 s, 88.8 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7610 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(8) *** before(7491584) after_dd(7440384) after(7491584) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.633916 s, 82.7 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7610 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(9) *** before(7491584) after_dd(7443456) after(7494656) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.686834 s, 76.3 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7610 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(10) *** before(7494656) after_dd(7431168) after(7498752) PASS 36 (299s) == sanityn test 37: check i_size is not updated for directory on close (bug 18695) ======================================================================== 02:04:13 (1788933853) multiop /mnt/lustre/d37.sanityn vD_c TMPPIPE=/tmp/multiop_open_wait_pipe.7610 - create 1290 (time 1788933867.81 total 10.01 last 128.88) - create 2508 (time 1788933877.82 total 20.02 last 121.71) - create 3963 (time 1788933887.83 total 30.02 last 145.43) - create 5195 (time 1788933897.83 total 40.03 last 123.13) - create 6524 (time 1788933907.84 total 50.04 last 132.72) - create 7794 (time 1788933917.85 total 60.04 last 126.98) - create 9102 (time 1788933927.85 total 70.05 last 130.73) total: 10000 create in 77.25 seconds: 129.46 ops/second PASS 37 (89s) == sanityn test 39a: file mtime does not change after rename ========================================================== 02:05:42 (1788933942) repeat after cancel_lru_locks PASS 39a (8s) == sanityn test 39b: file mtime the same on clients with/out lock ========================================================== 02:05:51 (1788933951) repeat after cancel_lru_locks PASS 39b (7s) == sanityn test 39c: check truncate mtime update ================================================================================ 02:05:58 (1788933958) repeat after cancel_lru_locks PASS 39c (8s) == sanityn test 39d: sync write should update mtime ====== 02:06:07 (1788933967) fail_loc=0x411 fail_loc=0 PASS 39d (8s) SKIP: sanityn test_40a skipping ALWAYS excluded test 40a == sanityn test 40b: pdirops: open|create and others ======================================================================== 02:06:16 (1788933976) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40b (19s) == sanityn test 40c: pdirops: link and others ======================================================================== 02:06:35 (1788933995) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40c (19s) == sanityn test 40d: pdirops: unlink and others ======================================================================== 02:06:54 (1788934014) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40d (18s) == sanityn test 40e: pdirops: rename and others ======================================================================== 02:07:12 (1788934032) No conflict No conflict No conflict No conflict No conflict No conflict PASS 40e (17s) == sanityn test 41a: pdirops: create vs mkdir ======================================================================== 02:07:30 (1788934050) mkdir: cannot create directory '/mnt/lustre2/f41a.sanityn': File exists Conflict PASS 41a (13s) == sanityn test 41b: pdirops: create vs create ======================================================================== 02:07:43 (1788934063) open: File exists Conflict PASS 41b (13s) == sanityn test 41c: pdirops: create vs link ======================================================================== 02:07:56 (1788934076) link: cannot create link '/mnt/lustre2/f41c.sanityn' to '/mnt/lustre2/f41c.sanityn-2': File exists Conflict PASS 41c (14s) == sanityn test 41d: pdirops: create vs unlink ======================================================================== 02:08:11 (1788934091) Conflict PASS 41d (14s) == sanityn test 41e: pdirops: create and rename (tgt) ======================================================================== 02:08:25 (1788934105) Conflict PASS 41e (14s) == sanityn test 41f: pdirops: create and rename (src) ======================================================================== 02:08:39 (1788934119) Conflict PASS 41f (14s) == sanityn test 41g: pdirops: create vs getattr ======================================================================== 02:08:53 (1788934133) Conflict PASS 41g (14s) == sanityn test 41h: pdirops: create vs readdir ======================================================================== 02:09:07 (1788934147) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 2427: lctl set_param -n fail_loc=0 2>/dev/null || true: No such file or directory Conflict PASS 41h (14s) == sanityn test 41i: reint_open: create vs create ======== 02:09:21 (1788934161) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 41i (1087s) == sanityn test 42a: pdirops: mkdir vs mkdir ======================================================================== 02:27:28 (1788935248) mkdir: cannot create directory '/mnt/lustre2/f42a.sanityn': File exists Conflict PASS 42a (13s) == sanityn test 42b: pdirops: mkdir vs create ======================================================================== 02:27:41 (1788935261) open: File exists Conflict PASS 42b (13s) == sanityn test 42c: pdirops: mkdir vs link ======================================================================== 02:27:55 (1788935275) link: cannot create link '/mnt/lustre2/f42c.sanityn' to '/mnt/lustre2/f42c.sanityn-2': File exists Conflict PASS 42c (15s) == sanityn test 42d: pdirops: mkdir vs unlink ======================================================================== 02:28:09 (1788935289) Conflict PASS 42d (12s) == sanityn test 42e: pdirops: mkdir and rename (tgt) ======================================================================== 02:28:21 (1788935301) mv: cannot overwrite directory '/mnt/lustre2/f42e.sanityn' with non-directory Conflict PASS 42e (11s) == sanityn test 42f: pdirops: mkdir and rename (src) ======================================================================== 02:28:32 (1788935312) Conflict PASS 42f (13s) == sanityn test 42g: pdirops: mkdir vs getattr ======================================================================== 02:28:45 (1788935325) Conflict PASS 42g (13s) == sanityn test 42h: pdirops: mkdir vs readdir ======================================================================== 02:28:58 (1788935338) Conflict PASS 42h (13s) == sanityn test 43a: rmdir,mkdir doesn't return -EEXIST ======================================================================== 02:29:11 (1788935351) PASS 43a (61s) == sanityn test 43b: pdirops: unlink vs create ======================================================================== 02:30:12 (1788935412) Conflict PASS 43b (12s) == sanityn test 43c: pdirops: unlink vs link ======================================================================== 02:30:24 (1788935424) Conflict PASS 43c (12s) == sanityn test 43d: pdirops: unlink vs unlink ======================================================================== 02:30:36 (1788935436) rm: cannot remove '/mnt/lustre2/f43d.sanityn': No such file or directory Conflict PASS 43d (12s) == sanityn test 43e: pdirops: unlink and rename (tgt) ======================================================================== 02:30:48 (1788935448) Conflict PASS 43e (12s) == sanityn test 43f: pdirops: unlink and rename (src) ======================================================================== 02:31:00 (1788935460) mv: cannot stat '/mnt/lustre2/f43f.sanityn': No such file or directory Conflict PASS 43f (14s) == sanityn test 43g: pdirops: unlink vs getattr ======================================================================== 02:31:15 (1788935475) stat: cannot statx '/mnt/lustre2/f43g.sanityn': No such file or directory Conflict PASS 43g (14s) == sanityn test 43h: pdirops: unlink vs readdir ======================================================================== 02:31:28 (1788935488) Conflict PASS 43h (14s) == sanityn test 43i: pdirops: unlink vs remote mkdir ===== 02:31:42 (1788935502) SKIP: sanityn test_43i needs >= 2 MDTs SKIP 43i (3s) == sanityn test 43j: racy mkdir return EEXIST ======================================================================== 02:31:45 (1788935505) mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists PASS 43j (124s) == sanityn test 43k: unlink vs create ==================== 02:33:49 (1788935629) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 43k (1189s) == sanityn test 44a: pdirops: rename tgt vs mkdir ======================================================================== 02:53:38 (1788936818) mkdir: cannot create directory '/mnt/lustre2/f44a.sanityn': File exists Conflict Wed Sep 9 02:53:47 EDT 2026 PASS 44a (13s) == sanityn test 44b: pdirops: rename tgt vs create ======================================================================== 02:53:52 (1788936832) open: File exists Conflict PASS 44b (13s) == sanityn test 44c: pdirops: rename tgt vs link ======================================================================== 02:54:05 (1788936845) link: cannot create link '/mnt/lustre2/f44c.sanityn' to '/mnt/lustre2/f44c.sanityn-3': File exists Conflict PASS 44c (12s) == sanityn test 44d: pdirops: rename tgt vs unlink ======================================================================== 02:54:17 (1788936857) Conflict PASS 44d (15s) == sanityn test 44e: pdirops: rename tgt and rename (tgt) ======================================================================== 02:54:32 (1788936872) Conflict PASS 44e (14s) == sanityn test 44f: pdirops: rename tgt and rename (src) ======================================================================== 02:54:46 (1788936886) Conflict PASS 44f (14s) == sanityn test 44g: pdirops: rename tgt vs getattr ======================================================================== 02:55:00 (1788936900) Conflict PASS 44g (14s) == sanityn test 44h: pdirops: rename tgt vs readdir ======================================================================== 02:55:14 (1788936914) Conflict PASS 44h (16s) == sanityn test 44i: pdirops: rename tgt vs remote mkdir ========================================================== 02:55:30 (1788936930) SKIP: sanityn test_44i needs >= 2 MDTs SKIP 44i (4s) == sanityn test 45a: rename,mkdir doesn't return -EEXIST ======================================================================== 02:55:34 (1788936934) PASS 45a (118s) == sanityn test 45b: pdirops: rename src vs create ======================================================================== 02:57:33 (1788937053) Conflict PASS 45b (13s) == sanityn test 45c: pdirops: rename src vs link ======================================================================== 02:57:46 (1788937066) Conflict PASS 45c (14s) == sanityn test 45d: pdirops: rename src vs unlink ======================================================================== 02:58:00 (1788937080) rm: cannot remove '/mnt/lustre2/f45d.sanityn': No such file or directory Conflict PASS 45d (13s) == sanityn test 45e: pdirops: rename src and rename (tgt) ======================================================================== 02:58:14 (1788937094) Conflict PASS 45e (15s) == sanityn test 45f: pdirops: rename src and rename (src) ======================================================================== 02:58:29 (1788937109) mv: cannot stat '/mnt/lustre2/f45f.sanityn': No such file or directory Conflict PASS 45f (14s) == sanityn test 45g: pdirops: rename src vs getattr ======================================================================== 02:58:43 (1788937123) stat: cannot statx '/mnt/lustre2/f45g.sanityn': No such file or directory Conflict PASS 45g (12s) == sanityn test 45h: pdirops: unlink vs readdir ======================================================================== 02:58:55 (1788937135) Conflict PASS 45h (13s) == sanityn test 45i: pdirops: rename src vs remote mkdir ========================================================== 02:59:08 (1788937148) SKIP: sanityn test_45i needs >= 2 MDTs SKIP 45i (4s) == sanityn test 45j: read vs rename ====================== 02:59:12 (1788937152) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 45j (1244s) == sanityn test 46a: pdirops: link vs mkdir ======================================================================== 03:19:56 (1788938396) mkdir: cannot create directory '/mnt/lustre2/f46a.sanityn': File exists Conflict PASS 46a (15s) == sanityn test 46b: pdirops: link vs create ======================================================================== 03:20:11 (1788938411) open: File exists Conflict PASS 46b (16s) == sanityn test 46c: pdirops: link vs link ======================================================================== 03:20:27 (1788938427) link: cannot create link '/mnt/lustre2/f46c.sanityn' to '/mnt/lustre2/f46c.sanityn': File exists Conflict PASS 46c (15s) == sanityn test 46d: pdirops: link vs unlink ======================================================================== 03:20:42 (1788938442) Conflict PASS 46d (13s) == sanityn test 46e: pdirops: link and rename (tgt) ======================================================================== 03:20:56 (1788938456) Conflict PASS 46e (14s) == sanityn test 46f: pdirops: link and rename (src) ======================================================================== 03:21:10 (1788938470) Conflict PASS 46f (15s) == sanityn test 46g: pdirops: link vs getattr ======================================================================== 03:21:25 (1788938485) Conflict PASS 46g (15s) == sanityn test 46h: pdirops: link vs readdir ======================================================================== 03:21:40 (1788938500) Conflict PASS 46h (16s) == sanityn test 46i: pdirops: link vs remote mkdir ======= 03:21:57 (1788938517) SKIP: sanityn test_46i needs >= 2 MDTs SKIP 46i (5s) == sanityn test 47a: pdirops: remote mkdir vs mkdir ====== 03:22:01 (1788938521) SKIP: sanityn test_47a needs >= 2 MDTs SKIP 47a (3s) == sanityn test 47b: pdirops: remote mkdir vs create ===== 03:22:05 (1788938525) SKIP: sanityn test_47b needs >= 2 MDTs SKIP 47b (3s) == sanityn test 47c: pdirops: remote mkdir vs link ======= 03:22:08 (1788938528) SKIP: sanityn test_47c needs >= 2 MDTs SKIP 47c (4s) == sanityn test 47d: pdirops: remote mkdir vs unlink ===== 03:22:12 (1788938532) SKIP: sanityn test_47d needs >= 2 MDTs SKIP 47d (3s) == sanityn test 47e: pdirops: remote mkdir and rename (tgt) ========================================================== 03:22:15 (1788938535) SKIP: sanityn test_47e needs >= 2 MDTs SKIP 47e (3s) == sanityn test 47f: pdirops: remote mkdir and rename (src) ========================================================== 03:22:19 (1788938539) SKIP: sanityn test_47f needs >= 2 MDTs SKIP 47f (3s) == sanityn test 47g: pdirops: remote mkdir vs getattr ==== 03:22:22 (1788938542) SKIP: sanityn test_47g needs >= 2 MDTs SKIP 47g (4s) == sanityn test 50: osc lvb attrs: enqueue vs. CP AST ======================================================================== 03:22:26 (1788938546) 10+0 records in 10+0 records out 10240 bytes (10 kB, 10 KiB) copied, 0.0785769 s, 130 kB/s fail_loc=0x410 fail_loc=0x0 PASS 50 (13s) == sanityn test 51a: layout lock: refresh layout should work ========================================================== 03:22:39 (1788938559) 0+1 records in 0+1 records out 158 bytes copied, 0.0531292 s, 3.0 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 3690: kill: (218464) - No such process PASS 51a (11s) == sanityn test 51b: layout lock: glimpse should be able to restart if layout changed ========================================================== 03:22:50 (1788938570) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0138195 s, 74.1 kB/s fail_loc=0x1404 fail_val=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140738 s, 7.5 MB/s 3145728 PASS 51b (21s) == sanityn test 51c: layout lock: IT_LAYOUT blocked and correct layout can be returned ========================================================== 03:23:11 (1788938591) fail_loc=0x172 Setting layout to have 2 stripes ... 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0250147 s, 40.9 kB/s PASS 51c (15s) == sanityn test 51d: layout lock: losing layout lock should clean up memory map region ========================================================== 03:23:26 (1788938606) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0360519 s, 29.1 MB/s Before revoking layout lock: 1024 KB mapped PASS 51d (9s) == sanityn test 51e: lfs getstripe does not break leases, part 2 ========================================================== 03:23:35 (1788938615) /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 2454 0x996 0x240000400 write lease(2) has applied. /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 2454 0x996 0x240000400 read lease(1) has applied. PASS 51e (9s) == sanityn test 54: rename locking ======================= 03:23:45 (1788938625) llite.lustre-ffff9fbb42a51000.xattr_cache=1 llite.lustre-ffff9fbb50d9c000.xattr_cache=1 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505015 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-09 03:23:47.000000000 -0400 Modify: 2026-09-09 03:23:47.000000000 -0400 Change: 2026-09-09 03:23:47.000000000 -0400 Birth: 2026-09-09 03:23:46.000000000 -0400 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000154 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505018 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-09 03:23:53.000000000 -0400 Modify: 2026-09-09 03:23:54.000000000 -0400 Change: 2026-09-09 03:23:54.000000000 -0400 Birth: 2026-09-09 03:23:52.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505021 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-09 03:24:00.000000000 -0400 Modify: 2026-09-09 03:24:00.000000000 -0400 Change: 2026-09-09 03:24:00.000000000 -0400 Birth: 2026-09-09 03:23:59.000000000 -0400 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505020 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-09 03:24:00.000000000 -0400 Modify: 2026-09-09 03:24:01.000000000 -0400 Change: 2026-09-09 03:24:01.000000000 -0400 Birth: 2026-09-09 03:23:59.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000154 File: /mnt/lustre/d1 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505023 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-09 03:24:06.000000000 -0400 Modify: 2026-09-09 03:24:07.000000000 -0400 Change: 2026-09-09 03:24:07.000000000 -0400 Birth: 2026-09-09 03:24:05.000000000 -0400 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505024 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-09 03:24:06.000000000 -0400 Modify: 2026-09-09 03:24:07.000000000 -0400 Change: 2026-09-09 03:24:07.000000000 -0400 Birth: 2026-09-09 03:24:05.000000000 -0400 PASS 54 (34s) == sanityn test 55a: rename vs unlink target dir ========= 03:24:18 (1788938658) fail_loc=0x80000156 mv: cannot move '/mnt/lustre/d55a.sanityn/d1/d2' to '/mnt/lustre/d55a.sanityn/d3/d2': No such file or directory PASS 55a (13s) == sanityn test 55b: rename vs unlink source dir ========= 03:24:32 (1788938672) fail_loc=0x80000156 mv: cannot stat '/mnt/lustre/d55b.sanityn/d1/d2': No such file or directory PASS 55b (14s) == sanityn test 55c: rename vs unlink orphan target dir == 03:24:46 (1788938686) fail_loc=0x156 mv: cannot move '/mnt/lustre/d55c.sanityn/d1/d2' to '/mnt/lustre/d55c.sanityn/d3/d2': No such file or directory PASS 55c (20s) == sanityn test 55d: rename file vs link ================= 03:25:06 (1788938706) fail_loc=0x155 mv: '/mnt/lustre/d55d.sanityn/f1' and '/mnt/lustre/d55d.sanityn/d55d.sanityn/f1' are the same file PASS 55d (16s) == sanityn test 55e: rename race AB/BA under the same parent dir ========================================================== 03:25:22 (1788938722) SKIP: sanityn test_55e needs >= 2 MDTs SKIP 55e (4s) == sanityn test 55f: rename: (P1/A -> P2/B) race with (P2/B -> P1/A) ========================================================== 03:25:27 (1788938727) fail_loc=0x155 We survived after P1A_P2B/P2B_P1A race test fail_loc=0x0 PASS 55f (20s) == sanityn test 55g: rename: race with trylock =========== 03:25:47 (1788938747) mdt.lustre-MDT0000.enable_rename_trylock=1 fail_loc=0x155 mv: cannot move '/mnt/lustre2/d55g.sanityn/b1/b2' to a subdirectory of itself, '/mnt/lustre2/d55g.sanityn/a1/a2/a3/a4/b2' We survived race with trylock test fail_loc=0x0 PASS 55g (24s) == sanityn test 56a: test llverdev with single large stripe ========================================================== 03:26:11 (1788938771) run_llverdev /mnt/lustre/f56a.sanityn -p -s 16000000 -c 4k llverdev: /mnt/lustre/f56a.sanityn is 16777216000000 bytes (15625 GB) in size timestamp: 1788938773 chunksize: 4096 size: 16777216000000 write offset: 16384000000kB 0.53543 MB/s read offset: 16384000000kB inf MB/s llverdev: data verified successfully PASS 56a (124s) == sanityn test 56b: test llverdev and partial verify of wide stripe file ========================================================== 03:28:15 (1788938895) run_llverdev /mnt/lustre/f56b.sanityn -p -s 32000000 -c 4k llverdev: /mnt/lustre/f56b.sanityn is 33554432000000 bytes (31250 GB) in size timestamp: 1788938901 chunksize: 4096 size: 33554432000000 write offset: 32768000000kB 0.953705 MB/s read offset: 32768000000kB inf MB/s llverdev: data verified successfully PASS 56b (143s) == sanityn test 60: Verify data_version behaviour ======== 03:30:39 (1788939039) PASS 60 (10s) == sanityn test 70a: cd directory && rm directory ======== 03:30:48 (1788939048) PASS 70a (10s) == sanityn test 70b: remove files after calling rm_entry ========================================================== 03:30:58 (1788939058) PASS 70b (10s) == sanityn test 71a: correct file map just after write operation is finished ========================================================== 03:31:08 (1788939068) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0116641 s, 5.6 MB/s SKIP: sanityn test_71a ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71a (4s) == sanityn test 71b: check fiemap support for stripecount > 1 ========================================================== 03:31:12 (1788939072) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0084389 s, 7.8 MB/s SKIP: sanityn test_71b ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71b (4s) == sanityn test 71c: check FIEMAP_EXTENT_LAST flag with different extents number ========================================================== 03:31:16 (1788939076) SKIP: sanityn test_71c support only ldiskfs ost SKIP 71c (3s) == sanityn test 71d: fiemap corruption test with fm_extent_count=0 ========================================================== 03:31:19 (1788939079) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0100313 s, 6.5 MB/s SKIP: sanityn test_71d ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71d (3s) == sanityn test 72: getxattr/setxattr cache should be consistent between nodes ========================================================== 03:31:23 (1788939083) llite.lustre-ffff9fbb42a51000.xattr_cache=1 llite.lustre-ffff9fbb50d9c000.xattr_cache=1 getfattr: Removing leading '/' from absolute path names user.attr1="value1" getfattr: Removing leading '/' from absolute path names user.attr1="value2" getfattr: Removing leading '/' from absolute path names getfattr: Removing leading '/' from absolute path names b362740ad371eae580e7f85452e8830b - 870e072da1ebfc20ea8c0cd3d6bdf3b6 - PASS 72 (6s) == sanityn test 73: getxattr should not cause xattr lock cancellation ========================================================== 03:31:30 (1788939090) llite.lustre-ffff9fbb42a51000.xattr_cache=1 llite.lustre-ffff9fbb50d9c000.xattr_cache=1 getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" PASS 73 (7s) == sanityn test 74: flock deadlock: different mounts ======================================================================== 03:31:37 (1788939097) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0356816 s, 28.7 kB/s 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0248017 s, 41.3 kB/s 231285: get lock1 231284: get lock2 231285: done 231285: get lock2 231284: done 231284: get lock1 fcntl cmd 7 failed: Resource deadlock avoided 231284: cannot get lock1: Resource deadlock avoided 231285: done 231285: exit rc=0 231284: done 231284: exit rc=0 PASS 74 (13s) == sanityn test 75: osc: upcall after unuse lock============================================================================= 03:31:50 (1788939110) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.052817 s, 39.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0256117 s, 40.9 MB/s fail_loc=0x31d /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 4373: 231867 Killed stat -c %s $DIR1/$tfile fail_loc=0 PASS 75 (14s) == sanityn test 76: Verify MDT open_files listing ======== 03:32:04 (1788939124) opening files: ........ closing files: ........ PASS 76 (72s) == sanityn test 77a: check FIFO NRS policy =============== 03:33:17 (1788939197) ost.OSS.ost.nrs_policies=fifo ost.OSS.ost_create.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_out.nrs_policies=fifo ost.OSS.ost_seq.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.215004 s, 78.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0413259 s, 25.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0506354 s, 20.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0900254 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0730155 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11617 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150354 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0188381 s, 55.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103099 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0418058 s, 25.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198913 s, 5.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10512 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0467631 s, 22.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194087 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115402 s, 9.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9fbb42a51000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9fbb42a51000.lock_unused_count=7 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16043 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145481 s, 7.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13661 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121245 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128885 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124793 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132126 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143528 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0661876 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151816 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15649 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159126 s, 6.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149105 s, 7.0 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173738 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156981 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179415 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102831 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123886 s, 8.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9fbb42a51000.lock_unused_count=9 removed '/mnt/lustre/d77a.sanityn/nrs_r_oleg645-client.virtnet' removed '/mnt/lustre/d77a.sanityn/nrs_w_oleg645-client.virtnet' removed directory '/mnt/lustre/d77a.sanityn' PASS 77a (10s) == sanityn test 77b: check CRR-N NRS policy ============== 03:33:27 (1788939207) ost.OSS.ost.nrs_policies=crrn ost.OSS.ost_create.nrs_policies=crrn ost.OSS.ost_io.nrs_policies=crrn ost.OSS.ost_out.nrs_policies=crrn ost.OSS.ost_seq.nrs_policies=crrn ost.OSS.ost.nrs_crrn_quantum=1 ost.OSS.ost_create.nrs_crrn_quantum=1 ost.OSS.ost_io.nrs_crrn_quantum=1 ost.OSS.ost_out.nrs_crrn_quantum=1 ost.OSS.ost_seq.nrs_crrn_quantum=1 policy: crr-n, crrn_quantum 1 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.216128 s, 77.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.063445 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0767245 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101997 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0935919 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171349 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159334 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20511 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23247 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135589 s, 7.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.241032 s, 4.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189733 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.262324 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148711 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140731 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0391989 s, 26.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9fbb42a51000.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff9fbb42a51000.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.226549 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100874 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0822964 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0448047 s, 23.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114238 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0924006 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0521308 s, 20.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0597432 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0707178 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0506392 s, 20.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0932963 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0743734 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0494209 s, 21.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0917361 s, 11.4 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0862394 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0787771 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0771706 s, 13.6 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg645-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg645-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost.nrs_crrn_quantum=64 ost.OSS.ost_create.nrs_crrn_quantum=64 ost.OSS.ost_io.nrs_crrn_quantum=64 ost.OSS.ost_out.nrs_crrn_quantum=64 ost.OSS.ost_seq.nrs_crrn_quantum=64 policy: crr-n, crrn_quantum 64 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.263168 s, 63.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.06014 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0809549 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156607 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1479 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114503 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112436 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101198 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11193 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150114 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.253464 s, 4.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.262724 s, 4.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.234498 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.26101 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.252842 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.282801 s, 3.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9fbb42a51000.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff9fbb42a51000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.331931 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00916811 s, 114 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.039567 s, 26.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0408519 s, 25.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00608551 s, 172 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121922 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00816261 s, 128 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00912341 s, 115 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00632882 s, 166 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00794225 s, 132 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00727542 s, 144 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0697628 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.084543 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00792394 s, 132 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0040471 s, 259 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118713 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0104546 s, 100 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg645-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg645-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77b (14s) == sanityn test 77c: check ORR NRS policy ================ 03:33:41 (1788939221) ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_orr_quantum=1 ost.OSS.ost_io.nrs_orr_offset_type=physical ost.OSS.ost_io.nrs_orr_supported=reads policy: orr, orr_quantum 1, orr_offset_type physical, orr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.271983 s, 61.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.030755 s, 34.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0464982 s, 22.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0648244 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0945525 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18604 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0950247 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103029 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198814 s, 5.3 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9fbb42a51000.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23513 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21568 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.226502 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.24899 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.239237 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.309206 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.231835 s, 4.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9fbb42a51000.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff9fbb42a51000.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.319813 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0536963 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105686 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0262238 s, 40.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.004294 s, 244 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.095905 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00823751 s, 127 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0116008 s, 90.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113522 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0954764 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0699322 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0829177 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0663685 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0535211 s, 19.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0485755 s, 21.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0621719 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0447046 s, 23.5 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg645-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg645-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=writes ost.OSS.ost_io.nrs_orr_quantum=64 policy: orr, orr_quantum 64, orr_offset_type physical, orr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.272643 s, 61.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100553 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124039 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112754 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158291 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197683 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186687 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140204 s, 7.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.274329 s, 3.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.238868 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221009 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.3057 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184674 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.278803 s, 3.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21029 s, 5.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.354733 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.486476 s, 2.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9fbb42a51000.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00243179 s, 431 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126227 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00237333 s, 442 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171455 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00341519 s, 307 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135563 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168726 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237913 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.219736 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144286 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131133 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135364 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0057655 s, 182 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109461 s, 9.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100948 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0708291 s, 14.8 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg645-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg645-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=reads_and_writes ost.OSS.ost_io.nrs_orr_offset_type=logical policy: orr, orr_quantum 64, orr_offset_type logical, orr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.401966 s, 41.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0460703 s, 22.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.048193 s, 21.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0758872 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0836565 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105816 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203001 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134831 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102229 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189236 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188405 s, 5.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168146 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136682 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22093 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185947 s, 5.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165534 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216918 s, 4.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9fbb42a51000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0263913 s, 39.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0689994 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0440869 s, 23.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0339896 s, 30.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0563941 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0467243 s, 22.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0500088 s, 21.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0649832 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0554392 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.060025 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0659965 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0700328 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0736563 s, 14.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0966381 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.090022 s, 11.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.082052 s, 12.8 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg645-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg645-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77c (17s) == sanityn test 77d: check TRR nrs policy ================ 03:33:58 (1788939238) ost.OSS.ost_io.nrs_policies=trr ost.OSS.ost_io.nrs_trr_quantum=1 ost.OSS.ost_io.nrs_trr_offset_type=physical ost.OSS.ost_io.nrs_trr_supported=reads policy: trr, trr_quantum 1, trr_offset_type physical, trr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.30113 s, 55.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0262127 s, 40.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0403081 s, 26.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0378201 s, 27.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0813033 s, 12.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10131 s, 10.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0622058 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0991265 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00809907 s, 129 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0741045 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115306 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103478 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0936413 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.231596 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220649 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.303763 s, 3.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9fbb42a51000.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17263 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0577111 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0862952 s, 12.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105581 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101316 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0727542 s, 14.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0987575 s, 10.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0798643 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0772116 s, 13.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114891 s, 9.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0939069 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0935866 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0950835 s, 11.0 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0888058 s, 11.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0748808 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0794345 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13764 s, 7.6 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg645-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg645-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=writes ost.OSS.ost_io.nrs_trr_quantum=64 policy: trr, trr_quantum 64, trr_offset_type physical, trr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.353817 s, 47.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0494489 s, 21.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0465298 s, 22.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0828608 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0825893 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0379381 s, 27.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0412043 s, 25.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0418063 s, 25.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0860258 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215357 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0236243 s, 44.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151121 s, 6.9 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9fbb42a51000.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0213411 s, 49.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105606 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.054943 s, 19.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157619 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0600933 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0854487 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129378 s, 8.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152802 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0534423 s, 19.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119819 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10358 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0921107 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00918146 s, 114 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0983806 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0980271 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0923124 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0435847 s, 24.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0360312 s, 29.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0299793 s, 35.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0254833 s, 41.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0617893 s, 17.0 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg645-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg645-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=reads_and_writes ost.OSS.ost_io.nrs_trr_offset_type=logical policy: trr, trr_quantum 64, trr_offset_type logical, trr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.309275 s, 54.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0311229 s, 33.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0746037 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0831601 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0938578 s, 11.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150207 s, 7.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0778695 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0942759 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1379 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0545358 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175894 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153539 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129452 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165847 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193936 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176817 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16776 s, 6.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0985842 s, 10.6 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.076835 s, 13.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0647129 s, 16.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109163 s, 9.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.076146 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118796 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0785103 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0939703 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0945226 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0710258 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0833825 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0974192 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0785082 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125619 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0879143 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144975 s, 7.2 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg645-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg645-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77d (16s) == sanityn test 77e: check TBF NID nrs policy ============ 03:34:14 (1788939254) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_tbf_rule=start localhost nid={0@lo} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start clients nid={192.168.206.45@tcp} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start others nid={*.*.*.*@tcp} rate=50 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.308397 s, 54.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0530036 s, 19.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0187199 s, 56.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122207 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0256409 s, 40.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0298518 s, 35.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.02812 s, 37.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184467 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0189126 s, 55.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00650589 s, 161 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134585 s, 7.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.204789 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0982014 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156149 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0845042 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0478232 s, 21.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220733 s, 4.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9fbb42a51000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9fbb42a51000.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0839682 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0797158 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0587125 s, 17.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114565 s, 9.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0816375 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0661329 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0965467 s, 10.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129889 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112588 s, 9.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111967 s, 9.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0882612 s, 11.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0870939 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144557 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0985592 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125832 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112331 s, 9.3 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg645-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg645-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change localhost rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change clients rate=101 ost.OSS.ost_io.nrs_tbf_rule=change others rate=51 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.335758 s, 50.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0299056 s, 35.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0286484 s, 36.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0548825 s, 19.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0819135 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0430193 s, 24.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0817361 s, 12.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0361048 s, 29.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0482306 s, 21.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0650046 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172859 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105613 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123184 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184103 s, 5.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0860283 s, 12.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185294 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223924 s, 4.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9fbb42a51000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9fbb42a51000.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0453642 s, 23.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0325505 s, 32.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0596794 s, 17.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0376675 s, 27.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0521971 s, 20.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0392418 s, 26.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0360602 s, 29.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0429559 s, 24.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0619976 s, 16.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0647614 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0495698 s, 21.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0656033 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.04976 s, 21.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0551629 s, 19.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0555994 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0637327 s, 16.5 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg645-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg645-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop localhost ost.OSS.ost_io.nrs_tbf_rule=stop clients ost.OSS.ost_io.nrs_tbf_rule=stop others 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.226061 s, 74.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0254448 s, 41.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0579095 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120384 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126594 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139719 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171062 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0507519 s, 20.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0889942 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0553768 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0514167 s, 20.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9fbb42a51000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154857 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199235 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150268 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190182 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143948 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.238179 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0735711 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0179699 s, 58.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150644 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0591775 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0520192 s, 20.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0544752 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.038292 s, 27.4 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157044 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0495249 s, 21.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0637984 s, 16.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0627261 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19104 s, 5.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104743 s, 10.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104497 s, 10.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107233 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0810899 s, 12.9 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg645-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg645-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.268115 s, 62.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.044542 s, 23.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0355803 s, 29.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0597468 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132031 s, 7.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154081 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12542 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168408 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190999 s, 5.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197724 s, 5.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202103 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187682 s, 5.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185385 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179005 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18661 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.229364 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.285852 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0843079 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101395 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0279201 s, 37.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0371717 s, 28.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.037993 s, 27.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0486698 s, 21.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0485415 s, 21.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0504788 s, 20.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0524607 s, 20.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0568233 s, 18.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0547766 s, 19.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.084447 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0675958 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0713137 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130498 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0832279 s, 12.6 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg645-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg645-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' PASS 77e (24s) == sanityn test 77f: check TBF JobID nrs policy ========== 03:34:38 (1788939278) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start runas jobid={iozone.500 dd.500 tiotest.500} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start iozone_runas jobid={iozone.500} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg645-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.24487 s, 68.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0298348 s, 35.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0440573 s, 23.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105085 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0696079 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10431 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0874671 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147467 s, 7.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183938 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157613 s, 6.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166291 s, 6.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193743 s, 5.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173314 s, 6.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9fbb42a51000.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff9fbb42a51000.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169916 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209846 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211742 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.240529 s, 4.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied ldlm.namespaces.lustre-OST0000-osc-ffff9fbb42a51000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00504394 s, 208 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00507888 s, 206 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00215286 s, 487 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0731803 s, 14.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0626704 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0848953 s, 12.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116171 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0555956 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.047319 s, 22.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0406426 s, 25.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0513828 s, 20.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0461287 s, 22.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0428218 s, 24.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0265964 s, 39.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0357516 s, 29.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0443564 s, 23.6 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg645-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg645-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change runas rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change iozone_runas rate=101 ost.OSS.ost_io.nrs_tbf_rule=change dd_runas rate=51 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg645-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.319008 s, 52.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0393436 s, 26.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0488663 s, 21.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0363687 s, 28.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0913187 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127432 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.229549 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173785 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199114 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19284 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.226937 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20487 s, 5.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.275683 s, 3.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.249022 s, 4.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.266452 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.27888 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.366263 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0450166 s, 23.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.065553 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0623169 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0641621 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0714835 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.073251 s, 14.3 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207329 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0731571 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.259124 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0877306 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0868239 s, 12.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0896136 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0915735 s, 11.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0912094 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.098475 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101148 s, 10.4 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg645-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg645-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop runas ost.OSS.ost_io.nrs_tbf_rule=stop iozone_runas ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg645-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.300124 s, 55.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0222981 s, 47.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0541254 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0694851 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110071 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106411 s, 9.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9fbb42a51000.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0945476 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184137 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140108 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0998575 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167543 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171178 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.266492 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147194 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213259 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.305502 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213737 s, 4.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9fbb42a51000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9fbb42a51000.lock_unused_count=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0618315 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0659963 s, 15.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0812857 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0626194 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111403 s, 9.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0798069 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0931185 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.076447 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0684164 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105998 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0860371 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0762939 s, 13.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0910164 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0867021 s, 12.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0740417 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10154 s, 10.3 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg645-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg645-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_policies=fifo running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg645-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.313488 s, 53.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0366139 s, 28.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0429633 s, 24.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.025988 s, 40.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147299 s, 71.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0759967 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173686 s, 60.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135062 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129489 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0983205 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0897539 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161667 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10417 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0499264 s, 21.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0177815 s, 59.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9fbb42a51000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9fbb42a51000.lock_unused_count=4 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149559 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.246433 s, 4.3 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9fbb42a51000.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0229209 s, 45.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0804366 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0263934 s, 39.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0336709 s, 31.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0570958 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0310432 s, 33.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0599118 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0357635 s, 29.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0514624 s, 20.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.050842 s, 20.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0591117 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0538696 s, 19.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0603822 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0617405 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0620671 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0848743 s, 12.4 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg645-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg645-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' PASS 77f (24s) == sanityn test 77g: Change TBF type directly ============ 03:35:02 (1788939302) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 ost.OSS.ost_io.nrs_policies=fifo PASS 77g (14s) == sanityn test 77h: Wrong policy name should report error, not LBUG ========================================================== 03:35:17 (1788939317) oleg645-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=abc: No such file or directory oleg645-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='abc': No such file or directory pdsh@oleg645-client: oleg645-server: ssh exited with exit code 2 oleg645-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc: Invalid argument oleg645-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc': Invalid argument pdsh@oleg645-client: oleg645-server: ssh exited with exit code 22 oleg645-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf reg abc: Invalid argument oleg645-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf reg abc': Invalid argument pdsh@oleg645-client: oleg645-server: ssh exited with exit code 22 oleg645-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc efg: Invalid argument oleg645-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc efg': Invalid argument pdsh@oleg645-client: oleg645-server: ssh exited with exit code 22 PASS 77h (22s) == sanityn test 77i: Change rank of TBF rule ============= 03:35:38 (1788939338) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start before jobid={jobid} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start after jobid={jobid} rate=1000 rank=default ost.OSS.ost_io.nrs_tbf_rule=start target jobid={jobid} rate=1000 rank=after Move before itself ost.OSS.ost_io.nrs_tbf_rule=change target rank=target Move to higher rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=before Move to lower rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=after Move before default ost.OSS.ost_io.nrs_tbf_rule=change target rank=default ost.OSS.ost_io.nrs_policies=fifo PASS 77i (28s) == sanityn test 77j: check TBF-OPCode NRS policy ========= 03:36:07 (1788939367) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf opcode ost.OSS.ost_io.nrs_tbf_rule=start ost_r opcode={ost_read} rate=5 ost.OSS.ost_io.nrs_tbf_rule=start ost_w opcode={ost_write} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.377319 s, 44.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0484849 s, 21.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.047198 s, 22.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0775247 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0933619 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0980616 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155231 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123229 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124453 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112752 s, 9.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221131 s, 4.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164445 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159224 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182304 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169651 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145958 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207211 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0237927 s, 44.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116611 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113397 s, 9.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197367 s, 5.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118176 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103176 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.219303 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141312 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144435 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129448 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142454 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.230446 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142121 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124366 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110438 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.250423 s, 4.2 MB/s removed '/mnt/lustre/d77j.sanityn/nrs_w_oleg645-client.virtnet' removed '/mnt/lustre/d77j.sanityn/nrs_r_oleg645-client.virtnet' removed directory '/mnt/lustre/d77j.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 5 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.1709 s, 10.3 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.6584 s, 5.3 MB/s Read runtime is 20 s, speed is 5.000000 IOPS cleanup_tbf /mnt/lustre/d77j.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_r ost.OSS.ost_io.nrs_tbf_rule=stop ost_w ost.OSS.ost_io.nrs_policies=fifo PASS 77j (71s) == sanityn test 77ja: check TBF-UID/GID NRS policy ======= 03:37:17 (1788939437) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg645-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg645-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg645-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg645-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg645-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.29124 s, 13.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0334744 s, 31.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109189 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0765093 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106786 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145089 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127815 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138816 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133683 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183173 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162075 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147646 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180773 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152168 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13853 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188076 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.228469 s, 4.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0876388 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0930987 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104967 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109856 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106878 s, 9.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13317 s, 7.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114445 s, 9.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111421 s, 9.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142107 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148004 s, 7.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122667 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118706 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155074 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15745 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13247 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163473 s, 6.4 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg645-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg645-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.2621 s, 5.2 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9651 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg645-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg645-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg645-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg645-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg645-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.27346 s, 13.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0901449 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0983296 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115702 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125759 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1689 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220414 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162495 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220557 s, 4.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.231534 s, 4.5 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.219131 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.266991 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216305 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23481 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.272919 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.249484 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.297296 s, 3.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0833779 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0950883 s, 11.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0839151 s, 12.5 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0971473 s, 10.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119842 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123559 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0962913 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111405 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1362 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141493 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117011 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135976 s, 7.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111077 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100378 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117241 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128711 s, 8.1 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg645-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg645-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9726 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9592 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77ja (155s) == sanityn test 77jb: check TBF-UID/GID NRS policy on files that don't belong to us ========================================================== 03:39:52 (1788939592) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg645-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg645-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg645-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg645-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg645-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.2783 s, 13.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0456725 s, 23.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0732623 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105832 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158193 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160584 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126777 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0802408 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153994 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134463 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0962976 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125868 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102723 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159305 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178703 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177759 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192189 s, 5.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0398558 s, 26.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00598599 s, 175 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0482337 s, 21.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0781035 s, 13.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0685377 s, 15.3 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0498486 s, 21.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0537599 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0693805 s, 15.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0760751 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168809 s, 62.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0641538 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0717895 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162098 s, 64.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0107147 s, 97.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0109503 s, 95.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00756954 s, 139 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg645-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg645-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9613 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9567 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg645-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg645-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg645-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg645-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg645-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.29119 s, 13.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0952601 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0995924 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152893 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0690531 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157246 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0589288 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144495 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1971 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147868 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.25019 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100675 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124599 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.284948 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185508 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180907 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.441167 s, 2.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0449009 s, 23.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0489431 s, 21.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0640542 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0404088 s, 25.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.053584 s, 19.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0705278 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0410489 s, 25.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0374959 s, 28.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0596636 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00387374 s, 271 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0657672 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.053382 s, 19.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0977153 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0603592 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0622343 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0730157 s, 14.4 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg645-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg645-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.91 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9272 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77jb (157s) == sanityn test 77k: check TBF policy with UID/GID/JobID/OPCode expression ========================================================== 03:42:30 (1788939750) ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg645-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.416852 s, 40.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0451559 s, 23.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0697492 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107974 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131519 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144494 s, 7.3 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156244 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182812 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187714 s, 5.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176584 s, 5.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220742 s, 4.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221722 s, 4.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.233624 s, 4.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.24813 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185974 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223693 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.277816 s, 3.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.040214 s, 26.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117064 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0984034 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.083008 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0116308 s, 90.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0907176 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130667 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139252 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0749436 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136424 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126002 s, 8.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.03701 s, 28.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117666 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0648519 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152147 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0777134 s, 13.5 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg645-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg645-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.5703 s, 9.9 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.94999 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.206.45@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.206.45@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.441292 s, 38.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0400105 s, 26.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0761788 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0747391 s, 14.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0841952 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0732171 s, 14.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0482191 s, 21.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0790923 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0715206 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134452 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0441161 s, 23.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0226274 s, 46.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.070834 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.020281 s, 51.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182954 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141117 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.234041 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0761341 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100601 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0742629 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0612597 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0996841 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0878009 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0768319 s, 13.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0950971 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106606 s, 9.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0945749 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.098928 s, 10.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108453 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0999294 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111267 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10413 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148831 s, 7.0 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg645-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg645-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.48065 s, 14.0 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0072 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.206.45@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg645-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.449114 s, 37.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0545141 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0594221 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0311119 s, 33.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0765747 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187044 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0833977 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189418 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197004 s, 5.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151669 s, 6.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121706 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120532 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191986 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127569 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.238945 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113263 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15754 s, 6.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00239594 s, 438 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0709995 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00713644 s, 147 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0188777 s, 55.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.024152 s, 43.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00430832 s, 243 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0176551 s, 59.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0114412 s, 91.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00883131 s, 119 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0115661 s, 90.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0665127 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0873398 s, 12.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160204 s, 65.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140779 s, 74.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111454 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0232719 s, 45.1 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg645-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg645-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.8423 s, 9.7 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.16 s, 20.3 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg645-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.613205 s, 27.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107469 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126146 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0566514 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0588531 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0807962 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127651 s, 8.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190332 s, 5.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0725607 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0674918 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0286956 s, 36.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.403276 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.217461 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.254518 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.275922 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.378315 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242552 s, 4.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15124 s, 6.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199672 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13422 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143781 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125724 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136028 s, 7.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163409 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127712 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112965 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205884 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127975 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127577 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155573 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175364 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165435 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18166 s, 5.8 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg645-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg645-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.6295 s, 9.0 MB/s Write runtime is 13 s, speed is 7.692307 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0377 s, 10.4 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.11298 s, 11.5 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0059 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg645-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.22576 s, 13.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0515255 s, 20.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0348079 s, 30.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0620903 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0729684 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0721518 s, 14.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161729 s, 6.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157719 s, 6.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180566 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133726 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0939884 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103354 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210383 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210213 s, 5.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.236757 s, 4.4 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.224654 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.243474 s, 4.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0337976 s, 31.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148907 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0341252 s, 30.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0439968 s, 23.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106677 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0486289 s, 21.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11623 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00288936 s, 363 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179356 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0114478 s, 91.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163654 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0264842 s, 39.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130885 s, 80.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126513 s, 8.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157467 s, 66.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116276 s, 9.0 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg645-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg645-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.3115 s, 5.2 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.6127 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_ug ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg645-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.390448 s, 43.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0674153 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0484868 s, 21.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0783217 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0974675 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107371 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122125 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.239567 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237755 s, 4.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223133 s, 4.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.245426 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.27616 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.288999 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.295229 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.264007 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.249221 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.345692 s, 3.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0891761 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0464715 s, 22.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0590391 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0652911 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0448416 s, 23.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0434189 s, 24.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.092403 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0356042 s, 29.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113038 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.064956 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106406 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0576425 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103601 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0795777 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0608011 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0847993 s, 12.4 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg645-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg645-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.45063 s, 11.1 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0131 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg645-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.57756 s, 29.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0235698 s, 44.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0670001 s, 15.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0644481 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112217 s, 9.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117208 s, 8.9 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148062 s, 7.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106371 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0782702 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120726 s, 8.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112957 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136403 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172395 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119387 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119252 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170892 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106084 s, 9.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0586657 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0613037 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0897728 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0723546 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102431 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105652 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0618796 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0939913 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0993481 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0992427 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108325 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0856397 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111384 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115845 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0934719 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131274 s, 8.0 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg645-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg645-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.7289 s, 8.9 MB/s Write runtime is 13 s, speed is 7.692307 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0213 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.6585 s, 9.0 MB/s Write runtime is 13 s, speed is 7.692307 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.96584 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_policies=fifo PASS 77k (449s) == sanityn test 77kb: Check different granular TBF type combination: jobid+opcode ========================================================== 03:50:00 (1788940200) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/nrs_r_oleg645-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.585166 s, 28.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0330385 s, 31.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0419231 s, 25.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0626246 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148481 s, 70.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0635173 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0609658 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157471 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150431 s, 7.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188549 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202102 s, 5.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209613 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.262466 s, 4.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.276333 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.337032 s, 3.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.277854 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.303252 s, 3.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0975533 s, 10.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0369574 s, 28.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0331092 s, 31.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111925 s, 9.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0479474 s, 21.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0467785 s, 22.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0489661 s, 21.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0989861 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0600315 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0626249 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106903 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0641371 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0723549 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0748538 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0812934 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.087177 s, 12.0 MB/s removed '/mnt/lustre/d77kb.sanityn/nrs_w_oleg645-client.virtnet' removed '/mnt/lustre/d77kb.sanityn/nrs_r_oleg645-client.virtnet' removed directory '/mnt/lustre/d77kb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.43656 s, 11.1 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.8804 s, 10.6 MB/s Read runtime is 10 s, speed is 10.000000 IOPS cleanup_tbf /mnt/lustre/d77kb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kb (59s) == sanityn test 77kc: Check different granular TBF type combination: nid+opcode ========================================================== 03:50:59 (1788940259) ost.OSS.ost_io.nrs_policies=tbf nid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.206.45@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.206.45@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.720718 s, 23.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0344292 s, 30.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0490965 s, 21.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0998242 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.204689 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151509 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.238313 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222549 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20253 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212384 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.256317 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.248986 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174349 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206759 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.256705 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.2119 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.261378 s, 4.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0485163 s, 21.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0730774 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0763652 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0729498 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0685864 s, 15.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0769564 s, 13.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0798761 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157175 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136574 s, 7.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0925801 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0949715 s, 11.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17481 s, 6.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0872806 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113316 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121014 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113043 s, 9.3 MB/s removed '/mnt/lustre/d77kc.sanityn/nrs_r_oleg645-client.virtnet' removed '/mnt/lustre/d77kc.sanityn/nrs_w_oleg645-client.virtnet' removed directory '/mnt/lustre/d77kc.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 13.191 s, 7.9 MB/s Write runtime is 14 s, speed is 7.142857 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.2698 s, 10.2 MB/s Read runtime is 12 s, speed is 8.333333 IOPS cleanup_tbf /mnt/lustre/d77kc.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kc (66s) == sanityn test 77kd: Check different granular TBF type combination: nid+jobid ========================================================== 03:52:05 (1788940325) ost.OSS.ost_io.nrs_policies=tbf nid+jobid ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.206.45@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/nrs_r_oleg645-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.454764 s, 36.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0584237 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0682895 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143167 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174575 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158974 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191826 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201001 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207505 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215169 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.218851 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.233215 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.224991 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.25142 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.245686 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.295358 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.383895 s, 2.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0606599 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0428602 s, 24.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101145 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136191 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0539614 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0543719 s, 19.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0793466 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0958673 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0728341 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.097162 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0811275 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0979721 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0957222 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0924333 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195647 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112772 s, 9.3 MB/s removed '/mnt/lustre/d77kd.sanityn/nrs_r_oleg645-client.virtnet' removed '/mnt/lustre/d77kd.sanityn/nrs_w_oleg645-client.virtnet' removed directory '/mnt/lustre/d77kd.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.1762 s, 9.4 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kd.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.56389 s, 18.8 MB/s Read runtime is 7 s, speed is 14.285714 IOPS cleanup_tbf /mnt/lustre/d77kd.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kd (56s) == sanityn test 77ke: Check different granular TBF type combination: jobid+opcode ========================================================== 03:53:01 (1788940381) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/nrs_r_oleg645-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.783401 s, 21.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0635427 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0757658 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0774759 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127051 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198975 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195238 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223552 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171395 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183373 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172549 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.230975 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216105 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.266565 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.322159 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.285048 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.238873 s, 4.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157497 s, 66.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0696605 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0930069 s, 11.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0955423 s, 11.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0487181 s, 21.5 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0198547 s, 52.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0844811 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170885 s, 61.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0297719 s, 35.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120706 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0764287 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0592126 s, 17.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0209326 s, 50.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0283551 s, 37.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149412 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129352 s, 8.1 MB/s removed '/mnt/lustre/d77ke.sanityn/nrs_r_oleg645-client.virtnet' removed '/mnt/lustre/d77ke.sanityn/nrs_w_oleg645-client.virtnet' removed directory '/mnt/lustre/d77ke.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.4068 s, 9.2 MB/s Write runtime is 13 s, speed is 7.692307 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ke.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.91568 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.6415 s, 9.9 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.99062 s, 10.5 MB/s Read runtime is 12 s, speed is 8.333333 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77ke (105s) == sanityn test 77kf: Check different granular TBF type combination: uid+gid ========================================================== 03:54:46 (1788940486) ost.OSS.ost_io.nrs_policies=tbf uid+gid ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/nrs_r_oleg645-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.09292 s, 15.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0574318 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0486791 s, 21.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0585224 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209521 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19946 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242534 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.285856 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.304914 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.281417 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.342296 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.310881 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.337494 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.340032 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.384516 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.327829 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.440436 s, 2.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0802873 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162328 s, 64.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0270178 s, 38.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.096628 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12768 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12822 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129349 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0685547 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211168 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.073675 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0818996 s, 12.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170503 s, 6.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174988 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10396 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193674 s, 5.4 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128604 s, 8.2 MB/s removed '/mnt/lustre/d77kf.sanityn/nrs_w_oleg645-client.virtnet' removed '/mnt/lustre/d77kf.sanityn/nrs_r_oleg645-client.virtnet' removed directory '/mnt/lustre/d77kf.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.1075 s, 5.2 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kf.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8806 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77kf.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kf (82s) == sanityn test 77kg: check different granular TBF type combination: uid+opcode ========================================================== 03:56:09 (1788940569) ost.OSS.ost_io.nrs_policies=tbf uid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg645-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.81934 s, 20.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0856894 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0911827 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0901822 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13222 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116631 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153503 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201676 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209376 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152106 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.234844 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195091 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227045 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.25901 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187147 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.287356 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.293239 s, 3.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9fbb42a51000.lock_unused_count=5 ldlm.namespaces.lustre-OST0001-osc-ffff9fbb42a51000.lock_unused_count=2 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101883 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123837 s, 8.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0977406 s, 10.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105213 s, 10.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0822348 s, 12.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108373 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0869985 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116457 s, 9.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12565 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132962 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110842 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121203 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123017 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119001 s, 8.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151369 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137229 s, 7.6 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg645-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg645-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 12.3534 s, 8.5 MB/s Write runtime is 14 s, speed is 7.142857 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0821 s, 10.4 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg645-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.688392 s, 24.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0480016 s, 21.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.040184 s, 26.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0931913 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1313 s, 8.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140245 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180784 s, 5.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163354 s, 6.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.261051 s, 4.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.302374 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209477 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.283415 s, 3.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208916 s, 5.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.292695 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.258588 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.287311 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.332851 s, 3.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0209186 s, 50.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.080581 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0954669 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0321117 s, 32.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0532168 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148305 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109874 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0108727 s, 96.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115643 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107074 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116393 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0574332 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0188808 s, 55.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0808161 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153177 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0412508 s, 25.4 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg645-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg645-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 12.2542 s, 8.6 MB/s Write runtime is 14 s, speed is 7.142857 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.98064 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 13.0289 s, 8.0 MB/s Write runtime is 14 s, speed is 7.142857 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.90168 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kg (154s) == sanityn test 77kh: Verify that Project ID support for NRS TBF rule ========================================================== 03:58:43 (1788940723) using SAVE_PROJECT_SUPPORTED=0 Stopping clients: oleg645-client.virtnet /mnt/lustre (opts:) Stopping client oleg645-client.virtnet /mnt/lustre opts: Stopping clients: oleg645-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg645-client.virtnet /mnt/lustre2 opts: Stopping /mnt/lustre-mds1 (opts:-f) on oleg645-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg645-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg645-server sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime) proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) devtmpfs on /dev type devtmpfs (rw,nosuid,mode=755) securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime) tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev) devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000) tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755) tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755) cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd) pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime) bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700) cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices) cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct) cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio) cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma) cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb) cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids) cgroup on /sys/fs/cgroup/misc type cgroup (rw,nosuid,nodev,noexec,relatime,misc) cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event) cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory) cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio) cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer) cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset) none on /sys/kernel/tracing type tracefs (rw,relatime) configfs on /sys/kernel/config type configfs (rw,relatime) /dev/nbd0 on / type ext4 (ro,relatime,stripe=32) rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) debugfs on /sys/kernel/debug type debugfs (rw,relatime) mqueue on /dev/mqueue type mqueue (rw,relatime) hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M) none on /mnt type ramfs (rw,relatime) none on /var/lib/stateless/writable type tmpfs (rw,relatime) /dev/vda on /home/green/git/lustre-release type squashfs (ro,relatime) none on /var/cache/man type tmpfs (rw,relatime) none on /var/log type tmpfs (rw,relatime) none on /var/lib/dbus type tmpfs (rw,relatime) none on /tmp type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) none on /var/tmp type tmpfs (rw,relatime) none on /var/lib/NetworkManager type tmpfs (rw,relatime) none on /var/lib/systemd/random-seed type tmpfs (rw,relatime) none on /var/spool type tmpfs (rw,relatime) none on /var/lib/nfs type tmpfs (rw,relatime) none on /var/lib/gssproxy type tmpfs (rw,relatime) none on /var/lib/logrotate type tmpfs (rw,relatime) none on /etc type tmpfs (rw,relatime) none on /var/lib/rsyslog type tmpfs (rw,relatime) none on /root/.ssh type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) 192.168.200.253:/exports/state/oleg645-client.virtnet on /var/lib/stateless/state type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.206.45,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg645-client.virtnet/boot on /boot type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.206.45,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg645-client.virtnet/etc/kdump.conf on /etc/kdump.conf type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.206.45,local_lock=none,addr=192.168.200.253) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=378180k,mode=700) 192.168.200.253:/exports/testreports/69916/testresults/sanityn-zfs-rocky8.10_x86_64-rocky8.10_x86_64-retry1 on /tmp/testlogs type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.206.45,local_lock=none,addr=192.168.200.253) /dev/vda on /usr/sbin/mount.lustre type squashfs (ro,relatime) Checking servers environments Checking clients oleg645-client.virtnet environments Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg645-server' oleg645-server: oleg645-server.virtnet: executing load_modules_local oleg645-server: kmemleak disabled oleg645-server: Loading modules from /home/green/git/lustre-release/lustre oleg645-server: detected 4 online CPUs by sysfs oleg645-server: MODOPTS_LIBCFS= oleg645-server: Force libcfs to create 2 CPU partitions oleg645-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Setup mgs, mdt, osts Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg645-server: oleg645-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg645-client: oleg645-server: ssh exited with exit code 1 Started lustre-MDT0000 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg645-server: oleg645-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg645-client: oleg645-server: ssh exited with exit code 1 Started lustre-OST0000 Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg645-server: oleg645-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg645-client: oleg645-server: ssh exited with exit code 1 Started lustre-OST0001 Starting client: oleg645-client.virtnet: -o user_xattr,flock 192.168.206.145@tcp:/lustre /mnt/lustre Starting client oleg645-client.virtnet: -o user_xattr,flock 192.168.206.145@tcp:/lustre /mnt/lustre Started clients oleg645-client.virtnet: 192.168.206.145@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Starting client: oleg645-client.virtnet: -o user_xattr,flock 192.168.206.145@tcp:/lustre /mnt/lustre2 Starting client oleg645-client.virtnet: -o user_xattr,flock 192.168.206.145@tcp:/lustre /mnt/lustre2 Started clients oleg645-client.virtnet: 192.168.206.145@tcp:/lustre on /mnt/lustre2 type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9fbb4559d000.idle_timeout=debug osc.lustre-OST0000-osc-ffff9fbb50d9a800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9fbb4559d000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9fbb50d9a800.idle_timeout=debug disable quota as required ost.OSS.ost_io.nrs_policies=tbf projid ost.OSS.ost_io.nrs_tbf_rule=start ext_proj_rw projid={100} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.520999 s, 32.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10135 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132051 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.300071 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.331344 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.253004 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.273333 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.317164 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.399911 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.297489 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.414054 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.349291 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.407661 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.378389 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.455811 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.437692 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.404968 s, 2.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.030185 s, 34.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016409 s, 63.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0698855 s, 15.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0744446 s, 14.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00998895 s, 105 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0848215 s, 12.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0779594 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0952321 s, 11.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0710105 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0880511 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0843195 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0918657 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00922776 s, 114 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0475921 s, 22.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00532438 s, 197 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124452 s, 8.4 MB/s removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg645-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg645-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 12.0121 s, 8.7 MB/s Write runtime is 13 s, speed is 7.692307 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.51333 s, 19.0 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_proj_rw ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf projid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w projid={100}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r projid={100}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.787326 s, 21.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0909745 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100094 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0753001 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.250429 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.245514 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.29558 s, 3.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.341188 s, 3.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.341451 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.33481 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.348305 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.378933 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.391934 s, 2.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.417671 s, 2.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.401995 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.394289 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.435949 s, 2.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9fbb4559d000.lock_unused_count=5 ldlm.namespaces.lustre-OST0001-osc-ffff9fbb4559d000.lock_unused_count=2 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14692 s, 7.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209189 s, 5.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145446 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222756 s, 4.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195889 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175822 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.252476 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182898 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201536 s, 5.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.228726 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193645 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.218957 s, 4.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.228771 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213181 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210727 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23629 s, 4.4 MB/s removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg645-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg645-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.0357 s, 11.6 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0387 s, 10.4 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77kh (220s) == sanityn test 77ki: Add rule with unsupported TBF type should fail for generic TBF ========================================================== 04:02:23 (1788940943) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 oleg645-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_fail projid={100}&opcode={ost_write} rate=20: Invalid argument oleg645-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_fail projid={100}&opcode={ost_write} rate=20': Invalid argument pdsh@oleg645-client: oleg645-server: ssh exited with exit code 22 ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77ki (19s) == sanityn test 77kj: Verify nodemap support for NRS TBF rule ========================================================== 04:02:42 (1788940962) On MGS 192.168.206.145, active = nodemap.active=1 On MGS 192.168.206.145, TBF = ost.OSS.ost_io.nrs_policies=tbf nodemap ost.OSS.ost_io.nrs_tbf_rule=start ext_nm_rw nodemap={TBF} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.851495 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0535074 s, 19.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0857931 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0577979 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0851774 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16191 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17651 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199273 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190149 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221283 s, 4.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179996 s, 5.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9fbb4559d000.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.276029 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.333572 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.37307 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.366936 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.411557 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.460395 s, 2.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9fbb4559d000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9fbb4559d000.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.311349 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.349014 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.329014 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.32981 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220265 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.30153 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22633 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.346852 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.235292 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.365696 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.252952 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.33237 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.264383 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.365462 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.413228 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.359136 s, 2.9 MB/s removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg645-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg645-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.84238 s, 10.7 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.22061 s, 20.1 MB/s Read runtime is 7 s, speed is 14.285714 IOPS cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete On MGS 192.168.206.145, TBF.id = On MGS 192.168.206.145, active = nodemap.active=0 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.621064 s, 27.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0367981 s, 28.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0728699 s, 14.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.061398 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0607632 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171703 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.239703 s, 4.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.251245 s, 4.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.284754 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.30895 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.307149 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.303327 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.298169 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.28061 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.312835 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.28286 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.397411 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146825 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158824 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205897 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160408 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189045 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205914 s, 5.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160384 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.219648 s, 4.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164536 s, 6.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189752 s, 5.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153366 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210386 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.261407 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23386 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170206 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200115 s, 5.2 MB/s removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg645-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg645-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 1 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 13.268 s, 7.9 MB/s Write runtime is 14 s, speed is 7.142857 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 3.48465 s, 30.1 MB/s Read runtime is 5 s, speed is 20.000000 IOPS cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_nm_rw ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf nodemap+opcode oleg645-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_w nodemap={TBF}&opcode={ost_write} rate=20: Invalid argument oleg645-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_w nodemap={TBF}&opcode={ost_write} rate=20': Invalid argument oleg645-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_r nodemap={TBF}&opcode={ost_read} rate=10: Invalid argument oleg645-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_r nodemap={TBF}&opcode={ost_read} rate=10': Invalid argument pdsh@oleg645-client: oleg645-server: ssh exited with exit code 22 On MGS 192.168.206.145, active = nodemap.active=1 On MGS 192.168.206.145, TBF = ost.OSS.ost_io.nrs_tbf_rule=start ext_w nodemap={TBF}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nodemap={TBF}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.770208 s, 21.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105933 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193418 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107507 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.270053 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185832 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176079 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.272325 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.288173 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.279404 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.346659 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.457582 s, 2.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9fbb4559d000.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.426786 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.484821 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.564292 s, 1.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.496282 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.675205 s, 1.6 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9fbb4559d000.lock_unused_count=2 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106877 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158957 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106359 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136131 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128188 s, 8.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0378266 s, 27.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0249549 s, 42.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154594 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0660949 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.088959 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0628008 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.088583 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.089981 s, 11.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121376 s, 8.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128861 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0598524 s, 17.5 MB/s removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg645-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg645-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.97749 s, 11.7 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.029 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo On MGS 192.168.206.145, TBF.id = On MGS 192.168.206.145, active = nodemap.active=0 PASS 77kj (202s) == sanityn test 77l: check the output of NRS policies for generic TBF ========================================================== 04:06:04 (1788941164) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_policies=fifo PASS 77l (11s) == sanityn test 77m: check NRS Delay slows write RPC processing ========================================================== 04:06:15 (1788941175) ost.OSS.ost_io.nrs_policies=delay ost.OSS.ost_io.nrs_delay_min=4 ost.OSS.ost_io.nrs_delay_max=4 ost.OSS.ost_io.nrs_delay_pct=100 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg645-client.virtnet] [bs=1M] [count=1] [oflag=direct] [conv=fdatasync] 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 3.71426 s, 282 kB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg645-client.virtnet] [bs=1M] [count=10] [oflag=direct] [conv=fdatasync] 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 40.029 s, 262 kB/s ost.OSS.ost_io.nrs_policies=fifo PASS 77m (59s) == sanityn test 77n: check wildcard support for TBF JobID NRS policy ========================================================== 04:07:14 (1788941234) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={*.500} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.560686 s, 29.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0786907 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137934 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106631 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201193 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.261069 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.299989 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.310906 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.258323 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.309378 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.320041 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.302954 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.326533 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.278737 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.307636 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.379147 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.363923 s, 2.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9fbb4559d000.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff9fbb4559d000.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0244908 s, 42.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0301883 s, 34.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0310346 s, 33.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0603953 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0296114 s, 35.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0729483 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0466683 s, 22.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0920585 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0475988 s, 22.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0777299 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0933 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0603774 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0617785 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0623746 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0632233 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0712264 s, 14.7 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg645-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg645-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77n.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.31734 s, 12.6 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77n.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.21161 s, 20.1 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.*} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.453867 s, 37.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0350173 s, 29.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0800829 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0790367 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.229029 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.252296 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.272496 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.264537 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.274794 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.282536 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.290544 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.304492 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.263989 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.299873 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.337031 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.334184 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.359184 s, 2.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9fbb4559d000.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff9fbb4559d000.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0619381 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102729 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11324 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0766672 s, 13.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1283 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121857 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123772 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0986036 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110631 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11081 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180302 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16645 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17228 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126307 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128491 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161395 s, 6.5 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg645-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg645-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.58378 s, 10.9 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.23413 s, 20.0 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_policies=fifo PASS 77n (90s) == sanityn test 77o: Changing rank should not panic ====== 04:08:44 (1788941324) mds.MDS.mdt.nrs_policies=tbf nid mds.MDS.mdt.nrs_tbf_rule=start name nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=start name1 nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=change name1 rank=name mds.MDS.mdt.nrs_tbf_rule=stop name mds.MDS.mdt.nrs_policies=fifo PASS 77o (11s) == sanityn test 77q: Parallel TBF rule definitions should not panic ========================================================== 04:08:56 (1788941336) mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_policies=fifo PASS 77q (127s) == sanityn test 77p: Check validity of rule names for TBF policies ========================================================== 04:11:02 (1788941462) mds.MDS.mdt.nrs_policies=tbf oleg645-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_overflo uid={500} rate=500: File name too long oleg645-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_overflo uid={500} rate=500': File name too long pdsh@oleg645-client: oleg645-server: ssh exited with exit code 36 oleg645-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start uid={500} rate=500: Invalid argument oleg645-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start uid={500} rate=500': Invalid argument pdsh@oleg645-client: oleg645-server: ssh exited with exit code 22 oleg645-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_empty: Invalid argument oleg645-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_empty': Invalid argument pdsh@oleg645-client: oleg645-server: ssh exited with exit code 22 oleg645-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p@spec uid={500} rate=500: Invalid argument oleg645-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p@spec uid={500} rate=500': Invalid argument pdsh@oleg645-client: oleg645-server: ssh exited with exit code 22 oleg645-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p.spec uid={500} rate=500: Invalid argument oleg645-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p.spec uid={500} rate=500': Invalid argument pdsh@oleg645-client: oleg645-server: ssh exited with exit code 22 oleg645-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p~spec uid={500} rate=500: Invalid argument oleg645-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p~spec uid={500} rate=500': Invalid argument pdsh@oleg645-client: oleg645-server: ssh exited with exit code 22 oleg645-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p#spec uid={500} rate=500: Invalid argument oleg645-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p#spec uid={500} rate=500': Invalid argument pdsh@oleg645-client: oleg645-server: ssh exited with exit code 22 oleg645-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p/spec uid={500} rate=500: Invalid argument oleg645-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p/spec uid={500} rate=500': Invalid argument pdsh@oleg645-client: oleg645-server: ssh exited with exit code 22 oleg645-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p^spec uid={500} rate=500: Invalid argument oleg645-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p^spec uid={500} rate=500': Invalid argument pdsh@oleg645-client: oleg645-server: ssh exited with exit code 22 oleg645-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p%spec uid={500} rate=500: Invalid argument oleg645-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p%spec uid={500} rate=500': Invalid argument pdsh@oleg645-client: oleg645-server: ssh exited with exit code 22 oleg645-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p*spec uid={500} rate=500: Invalid argument oleg645-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p*spec uid={500} rate=500': Invalid argument pdsh@oleg645-client: oleg645-server: ssh exited with exit code 22 oleg645-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p;spec uid={500} rate=500: Invalid argument oleg645-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p;spec uid={500} rate=500': Invalid argument pdsh@oleg645-client: oleg645-server: ssh exited with exit code 22 oleg645-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p,spec uid={500} rate=500: Invalid argument oleg645-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p,spec uid={500} rate=500': Invalid argument pdsh@oleg645-client: oleg645-server: ssh exited with exit code 22 oleg645-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p?spec uid={500} rate=500: Invalid argument oleg645-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p?spec uid={500} rate=500': Invalid argument pdsh@oleg645-client: oleg645-server: ssh exited with exit code 22 oleg645-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77pspec uid={500} rate=500: Invalid argument oleg645-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p>spec uid={500} rate=500': Invalid argument pdsh@oleg645-client: oleg645-server: ssh exited with exit code 22 oleg645-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p:spec uid={500} rate=500: Invalid argument oleg645-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p:spec uid={500} rate=500': Invalid argument pdsh@oleg645-client: oleg645-server: ssh exited with exit code 22 oleg645-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p+spec uid={500} rate=500: Invalid argument oleg645-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p+spec uid={500} rate=500': Invalid argument pdsh@oleg645-client: oleg645-server: ssh exited with exit code 22 oleg645-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p=spec uid={500} rate=500: Invalid argument oleg645-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p=spec uid={500} rate=500': Invalid argument pdsh@oleg645-client: oleg645-server: ssh exited with exit code 22 oleg645-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p)spec uid={500} rate=500: Invalid argument oleg645-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p)spec uid={500} rate=500': Invalid argument pdsh@oleg645-client: oleg645-server: ssh exited with exit code 22 oleg645-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p(spec uid={500} rate=500: Invalid argument oleg645-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p(spec uid={500} rate=500': Invalid argument pdsh@oleg645-client: oleg645-server: ssh exited with exit code 22 oleg645-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p{spec uid={500} rate=500: Invalid argument oleg645-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p{spec uid={500} rate=500': Invalid argument pdsh@oleg645-client: oleg645-server: ssh exited with exit code 22 oleg645-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p}spec uid={500} rate=500: Invalid argument oleg645-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p}spec uid={500} rate=500': Invalid argument pdsh@oleg645-client: oleg645-server: ssh exited with exit code 22 oleg645-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p|spec uid={500} rate=500: Invalid argument oleg645-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p|spec uid={500} rate=500': Invalid argument pdsh@oleg645-client: oleg645-server: ssh exited with exit code 22 oleg645-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p[spec uid={500} rate=500: Invalid argument oleg645-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p[spec uid={500} rate=500': Invalid argument pdsh@oleg645-client: oleg645-server: ssh exited with exit code 22 oleg645-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p]spec uid={500} rate=500: Invalid argument oleg645-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p]spec uid={500} rate=500': Invalid argument pdsh@oleg645-client: oleg645-server: ssh exited with exit code 22 oleg645-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p!spec uid={500} rate=500: Invalid argument oleg645-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p!spec uid={500} rate=500': Invalid argument pdsh@oleg645-client: oleg645-server: ssh exited with exit code 22 oleg645-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p&spec uid={500} rate=500: Invalid argument oleg645-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p&spec uid={500} rate=500': Invalid argument pdsh@oleg645-client: oleg645-server: ssh exited with exit code 22 oleg645-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p$spec uid={500} rate=500: Invalid argument oleg645-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p$spec uid={500} rate=500': Invalid argument pdsh@oleg645-client: oleg645-server: ssh exited with exit code 22 oleg645-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p`spec uid={500} rate=500: Invalid argument oleg645-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p`spec uid={500} rate=500': Invalid argument pdsh@oleg645-client: oleg645-server: ssh exited with exit code 22 oleg645-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p\spec uid={500} rate=500: Invalid argument oleg645-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p\spec uid={500} rate=500': Invalid argument pdsh@oleg645-client: oleg645-server: ssh exited with exit code 22 mds.MDS.mdt.nrs_policies=fifo PASS 77p (31s) == sanityn test 77r: Change type of tbf policy at run time ========================================================== 04:11:33 (1788941493) mds.MDS.mdt.nrs_policies=tbf uid mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf gid mds.MDS.mdt.nrs_tbf_rule=start md_rule gid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start md_rule jobid={*.500} rate=1 mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=fifo check the number of requests in queue: fifo: 1 crrn: 0 tbf: 0 delay: 0 fifo: 0 crrn: 0 tbf: 0 delay: 0 cleanup 77r 272920 kill md_thread xargs PASS 77r (58s) == sanityn test 77s: Check TBF LRU shrinker ============== 04:12:32 (1788941552) jobid_name=%e.%p jobid_var=TEST_77s_JOBVAR mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start touch jobid={touch.*} rate=1000 mds.MDS.mdt.nrs_tbf_rule=start unlink jobid={unlink.*} rate=1000 touch jobid={touch.*} 1000, ref 9 touch jobid={touch.*} 1000, ref 0 unlink jobid={unlink.*} 1000, ref 9 unlink jobid={unlink.*} 1000, ref 0 mds.MDS.mdt.nrs_policies=fifo jobid_var=procname_uid jobid_name=%e.%u PASS 77s (74s) == sanityn test 78: Enable policy and specify tunings right away ========================================================== 04:13:46 (1788941626) oleg645-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_orr_quantum=1: No such device oleg645-server: error: set_param: setting 'ost/OSS/*/nrs_orr_quantum'='1': No such device pdsh@oleg645-client: oleg645-server: ssh exited with exit code 19 ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_policies=fifo PASS 78 (9s) == sanityn test 79: xattr: intent error ================== 04:13:55 (1788941635) File: /mnt/lustre/d79.sanityn Size: 16384 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115238826935667 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-09 04:13:57.000000000 -0400 Modify: 2026-09-09 04:13:57.000000000 -0400 Change: 2026-09-09 04:13:57.000000000 -0400 Birth: 2026-09-09 04:13:56.000000000 -0400 fail_val=0 fail_loc=0x80000160 fail_val=0 fail_loc=0x80000131 # file: mnt/lustre/d79.sanityn trusted.name1="value1" PASS 79 (17s) == sanityn test 80a: migrate directory when some children is being opened ========================================================== 04:14:12 (1788941652) SKIP: sanityn test_80a needs >= 2 MDTs SKIP 80a (3s) == sanityn test 80b: Accessing directory during migration ========================================================== 04:14:15 (1788941655) SKIP: sanityn test_80b needs >= 2 MDTs SKIP 80b (3s) == sanityn test 81a: rename and stat under striped directory ========================================================== 04:14:18 (1788941658) SKIP: sanityn test_81a needs >= 2 MDTs SKIP 81a (4s) == sanityn test 81b: rename under striped directory doesn't deadlock ========================================================== 04:14:22 (1788941662) SKIP: sanityn test_81b We need at least 2 MDTs for this test SKIP 81b (3s) == sanityn test 81c: rename revoke LOOKUP lock for remote object ========================================================== 04:14:25 (1788941665) SKIP: sanityn test_81c needs >= 4 MDTs SKIP 81c (3s) == sanityn test 81d: parallel rename file cross-dir on same MDT ========================================================== 04:14:28 (1788941668) - open/close 732 (time 1788941683.62 total 10.00 last 73.20) - open/close 1479 (time 1788941693.65 total 20.03 last 74.51) - open/close 2167 (time 1788941703.65 total 30.03 last 68.77) total: 2500 open/close in 34.76 seconds: 71.92 ops/second snapshot_time 1788941793.145154507 secs.nsecs start_time 1788941670.475459007 secs.nsecs elapsed_time 122.669695500 secs.nsecs open 2505 samples [usecs] 126 13713 3879556 9776288970 close 2505 samples [usecs] 195 5268 899282 601521206 mknod 2501 samples [usecs] 569 11333 3349266 7361515434 unlink 2500 samples [usecs] 709 101979 6650622 87031927680 mkdir 2 samples [usecs] 986 1005 1991 1982221 rmdir 2 samples [usecs] 7084 7494 14578 106343092 rename 2500 samples [usecs] 508 52356 17362765 230971576615 getattr 3762 samples [usecs] 10 8250 240079 331804719 parallel_rename_file 2500 samples [usecs] 508 52356 17362765 230971576615 crossdir_rename 2500 samples [usecs] 508 52356 17362765 230971576615 readpage 32 samples [usecs] 3755 73674 810278 28081279346 PASS 81d (132s) == sanityn test 82: fsetxattr and fgetxattr on orphan files ========================================================== 04:16:40 (1788941800) multiop /mnt/lustre/f82.sanityn vO_ac TMPPIPE=/tmp/multiop_open_wait_pipe.7610 multiop /mnt/lustre2/f82.sanityn vo_Ac TMPPIPE=/tmp/multiop_open_wait_pipe.7610 PASS 82 (7s) == sanityn test 83: access striped directory while it is being created/unlinked ========================================================== 04:16:47 (1788941807) SKIP: sanityn test_83 needs >= 2 MDTs SKIP 83 (3s) == sanityn test 84: 0-nlink race in lu_object_find() ===== 04:16:50 (1788941810) fail_loc=0x8000060b PASS 84 (14s) == sanityn test 85: Lustre API root cache race =========== 04:17:04 (1788941824) Starting: llapi_root_test /mnt/lustre/f85.sanityn /mnt/lustre2/f85.sanityn Starting test test1 at 1788941825 Finishing test test1 at 1788941830 PASS 85 (12s) == sanityn test 90: open/create and unlink striped directory ========================================================== 04:17:16 (1788941836) SKIP: sanityn test_90 needs >= 2 MDTs SKIP 90 (4s) == sanityn test 91: chmod and unlink striped directory === 04:17:20 (1788941840) SKIP: sanityn test_91 needs >= 2 MDTs SKIP 91 (4s) == sanityn test 92: create remote directory under orphan directory ========================================================== 04:17:24 (1788941844) SKIP: sanityn test_92 needs >= 2 MDTs SKIP 92 (4s) == sanityn test 93: alloc_rr should not allocate on same ost ========================================================== 04:17:28 (1788941848) fail_loc=0x00000163 fail_loc=0x0 /mnt/lustre/f93.sanityn-1/file1 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5798 0x16a6 0x240000400 1 5760 0x1680 0x280000400 /mnt/lustre2/f93.sanityn-2/file2 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5799 0x16a7 0x240000400 1 5761 0x1681 0x280000400 rc1=0 and rc2=0 PASS 93 (20s) == sanityn test 94: signal vs CP callback race =========== 04:17:48 (1788941868) osc.lustre-OST0000-osc-ffff9fbb4559d000.idle_timeout=0 osc.lustre-OST0000-osc-ffff9fbb50d9a800.idle_timeout=0 osc.lustre-OST0001-osc-ffff9fbb4559d000.idle_timeout=0 osc.lustre-OST0001-osc-ffff9fbb50d9a800.idle_timeout=0 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.246825 s, 33.2 kB/s fail_val=5 fail_loc=0x80000312 fail_val=6 fail_loc=0x80000329 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 2.92875 s, 1.4 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 6389: 289801 Alarm clock dd if=/dev/zero of=$DIR/$tfile conv=notrunc oflag=append bs=4k count=1 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0815965 s, 50.2 kB/s osc.lustre-OST0000-osc-ffff9fbb4559d000.idle_timeout=debug osc.lustre-OST0000-osc-ffff9fbb50d9a800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9fbb4559d000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9fbb50d9a800.idle_timeout=debug PASS 94 (16s) == sanityn test 95a: Check readpage() on a page that was removed from page cache ========================================================== 04:18:04 (1788941884) llite.lustre-ffff9fbb4559d000.fast_read=0 llite.lustre-ffff9fbb50d9a800.fast_read=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0655022 s, 32.0 MB/s fail_loc=0x80001421 fail_val=7 fail_loc=0x80001422 fail_val=10 llite.lustre-ffff9fbb4559d000.fast_read=1 llite.lustre-ffff9fbb50d9a800.fast_read=1 PASS 95a (20s) == sanityn test 95b: Check readpage() on a page that is no longer uptodate ========================================================== 04:18:25 (1788941905) llite.lustre-ffff9fbb4559d000.fast_read=0 llite.lustre-ffff9fbb50d9a800.fast_read=0 1+0 records in 1+0 records out 12288 bytes (12 kB, 12 KiB) copied, 0.00326943 s, 3.8 MB/s fail_loc=0x80001424 fail_val=5 fail_loc=0 llite.lustre-ffff9fbb4559d000.fast_read=1 llite.lustre-ffff9fbb50d9a800.fast_read=1 PASS 95b (12s) == sanityn test 100a: DoM: glimpse RPCs for stat without IO lock (DoM only file) ========================================================== 04:18:38 (1788941918) SKIP: sanityn test_100a Reserved for glimpse-ahead SKIP 100a (3s) == sanityn test 100b: DoM: no glimpse RPC for stat with IO lock (DoM only file) ========================================================== 04:18:41 (1788941921) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0124529 s, 329 kB/s /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK PASS 100b (8s) == sanityn test 100c: DoM: write vs stat without IO lock (combined file) ========================================================== 04:18:49 (1788941929) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.196551 s, 10.7 MB/s /mnt/lustre/d100c.sanityn/dom has type file OK /mnt/lustre/d100c.sanityn/dom has size 2097152 OK PASS 100c (8s) == sanityn test 100d: DoM: write+truncate vs stat without IO lock (combined file) ========================================================== 04:18:57 (1788941937) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.181681 s, 11.5 MB/s /mnt/lustre/d100d.sanityn/dom has type file OK /mnt/lustre/d100d.sanityn/dom has size 4096 OK PASS 100d (8s) == sanityn test 100e: DoM: read on open and file size ==== 04:19:05 (1788941945) 1+0 records in 1+0 records out 12000 bytes (12 kB, 12 KiB) copied, 0.00861666 s, 1.4 MB/s PASS 100e (8s) == sanityn test 101a: Discard DoM data on unlink ========= 04:19:13 (1788941953) /mnt/lustre/f101a.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0250029 s, 164 kB/s PASS 101a (8s) == sanityn test 101b: Discard DoM data on rename ========= 04:19:22 (1788941962) /mnt/lustre/f101b.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0171762 s, 238 kB/s PASS 101b (7s) == sanityn test 101c: Discard DoM data on close-unlink === 04:19:28 (1788941968) /mnt/lustre/f101c.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0194612 s, 210 kB/s PASS 101c (9s) == sanityn test 102: Test open by handle of unlinked file ========================================================== 04:19:37 (1788941977) file: /mnt/lustre/f102.sanityn fh_bytes: 32 fh_type: 151 fh_data: d1 0b 00 00 02 00 00 00 54 0f 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 file's parent FID is [0:0x0:0x0] file FID is [0x200000bd1:0xf54:0x0] access via mount point '/mnt/lustre2' - OK access after unlink - OK check_fhandle_syscalls test Passed! PASS 102 (8s) == sanityn test 103: Test size correctness with lockahead ========================================================== 04:19:45 (1788941985) fail_loc=0x415 fail_loc=0x415 fail_val=2 Incorrect size expected (no glimpse fix): Starting test test23 at 1788941988 lockahead_test: lockahead_test.c:1171: test23: assertion 'sb.st_size == sb2.st_size' failed: size on /mnt/lustre/d103.sanityn/f103.sanityn and /mnt/lustre2/d103.sanityn/f103.sanityn differs: 0 vs 1048576 fail_loc=0 fail_loc=0x214 fail_val=2 Starting test test23 at 1788941991 Finishing test test23 at 1788941991 PASS 103 (13s) == sanityn test 104: Verify that MDS stores atime/mtime/ctime during close ========================================================== 04:19:58 (1788941998) SKIP: sanityn test_104 ldiskfs only test SKIP 104 (3s) == sanityn test 105: Glimpse and lock cancel race ======== 04:20:02 (1788942002) fail_loc=0x416 PASS 105 (24s) == sanityn test 106a: Verify the btime via statx() ======= 04:20:25 (1788942025) SKIP: sanityn test_106a Test only for ldiskfs and statx() supported SKIP 106a (4s) == sanityn test 106b: Glimpse RPCs test for statx ======== 04:20:29 (1788942029) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.013038 s, 78.5 kB/s File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238826938203 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-09 04:20:30.000000000 -0400 Modify: 2026-09-09 04:20:31.000000000 -0400 Change: 2026-09-09 04:20:31.000000000 -0400 Birth: 2026-09-09 04:20:30.000000000 -0400 /mnt/lustre/f106b.sanityn 144115238826938203 -rw-r--r-- 1 0 0 1788942030 1788942030 1788942031 File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238826938203 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-09 04:20:30.000000000 -0400 Modify: 2026-09-09 04:20:31.000000000 -0400 Change: 2026-09-09 04:20:31.000000000 -0400 Birth: 2026-09-09 04:20:30.000000000 -0400 1788942031 1024 1 PASS 106b (7s) == sanityn test 106c: Verify statx attributes mask ======= 04:20:37 (1788942037) PASS 106c (7s) == sanityn test 106d: Verify POSIX immutable interface for non-root users ========================================================== 04:20:44 (1788942044) mdd.lustre-MDT0000.posix_immutable=1 mdd.lustre-MDT0000.posix_immutable_min_atime_delta=30 mdd.lustre-MDT0000.posix_immutable=0 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d106d.sanityn/f106d.sanityn] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [-ac] [--date=1 hour] [/mnt/lustre/d106d.sanityn/f106d.sanityn] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [a-w] [/mnt/lustre/d106d.sanityn/f106d.sanityn] mdd.lustre-MDT0000.posix_immutable=1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d106d.sanityn/f106d.sanityn] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [-ac] [--date=1 hour] [/mnt/lustre/d106d.sanityn/f106d.sanityn] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [a-w] [/mnt/lustre/d106d.sanityn/f106d.sanityn] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d106d.sanityn/f106d.sanityn] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [a-w] [/mnt/lustre/d106d.sanityn/f106d.sanityn] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [-ac] [--date=1 hour] [/mnt/lustre/d106d.sanityn/f106d.sanityn] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d106d.sanityn/f106d.sanityn] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [-ac] [--date=65 seconds] [/mnt/lustre/d106d.sanityn/f106d.sanityn] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [a-w] [/mnt/lustre/d106d.sanityn/f106d.sanityn] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d106d.sanityn/f106d.sanityn] mdd.lustre-MDT0000.posix_immutable_min_atime_delta=86400 mdd.lustre-MDT0000.posix_immutable=0 PASS 106d (166s) == sanityn test 106e: Verify lfs migrate/swap honors POSIX immutable expiry/cap rules ========================================================== 04:23:30 (1788942210) mdd.lustre-MDT0000.posix_immutable=1 mdd.lustre-MDT0000.posix_immutable_min_atime_delta=30 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d106e.sanityn/f106e.sanityn] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [-ac] [--date=1 hour] [/mnt/lustre/d106e.sanityn/f106e.sanityn] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [a-w] [/mnt/lustre/d106e.sanityn/f106e.sanityn] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d106e.sanityn/f106e.sanityn] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d106e.sanityn/f106e.sanityn] [bs=4k] [count=1] 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0126743 s, 323 kB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [-ac] [--date=65 seconds] [/mnt/lustre/d106e.sanityn/f106e.sanityn] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [a-w] [/mnt/lustre/d106e.sanityn/f106e.sanityn] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [u+w] [/mnt/lustre/d106e.sanityn/f106e.sanityn] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-c] [2] [/mnt/lustre/d106e.sanityn/f106e.sanityn] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d106e.sanityn/f106e.sanityn] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d106e.sanityn/f106e.sanityn] [bs=4k] [count=1] 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00798467 s, 513 kB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [-ac] [--date=65 seconds] [/mnt/lustre/d106e.sanityn/f106e.sanityn] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [a-w] [/mnt/lustre/d106e.sanityn/f106e.sanityn] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d106e.sanityn/f106e.sanityn] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d106e.sanityn/f106e.sanityn] [bs=4k] [count=1] 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00408033 s, 1.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [-ac] [--date=60 seconds] [/mnt/lustre/d106e.sanityn/f106e.sanityn] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [a-w] [/mnt/lustre/d106e.sanityn/f106e.sanityn] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [u+w] [/mnt/lustre/d106e.sanityn/f106e.sanityn] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [swap_layouts] [/mnt/lustre/d106e.sanityn/f106e.sanityn] [/mnt/lustre/d106e.sanityn/f106e.sanityn_swap] mdd.lustre-MDT0000.posix_immutable_min_atime_delta=86400 mdd.lustre-MDT0000.posix_immutable=0 PASS 106e (229s) == sanityn test 106f: Verify non-owner denial after POSIX immutable expiry ========================================================== 04:27:19 (1788942439) mdd.lustre-MDT0000.posix_immutable=1 mdd.lustre-MDT0000.posix_immutable_min_atime_delta=30 mdd.lustre-MDT0000.posix_immutable_min_atime_delta=86400 mdd.lustre-MDT0000.posix_immutable=0 PASS 106f (157s) == sanityn test 107a: Basic grouplock conflict =========== 04:29:57 (1788942597) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.309153 s, 33.9 MB/s /mnt/lustre/f107a.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5816 0x16b8 0x240000400 multiop /mnt/lustre/f107a.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7610 multiop /mnt/lustre2/f107a.sanityn vO_G16022000r10g16022000c TMPPIPE=/tmp/multiop_open_wait_pipe.7610 First grouplock blocks second one PASS 107a (9s) == sanityn test 107b: Grouplock is added to the head of waiting list ========================================================== 04:30:06 (1788942606) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.364176 s, 28.8 MB/s /mnt/lustre/f107b.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 5777 0x1691 0x280000400 multiop /mnt/lustre/f107b.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7610 Grouplock blocks IO First grouplock blocks second one Second grouplock blocks IO PASS 107b (14s) == sanityn test 108a: lseek: parallel updates ============ 04:30:20 (1788942620) 1+0 records in 1+0 records out 10000 bytes (10 kB, 9.8 KiB) copied, 0.0127692 s, 783 kB/s llite.lustre-ffff9fbb4559d000.hybrid_io=0 llite.lustre-ffff9fbb50d9a800.hybrid_io=0 fail_val=4 fail_loc=0x80000414 1+0 records in 1+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 4.93646 s, 1.7 MB/s PASS 108a (13s) == sanityn test 109: Race with several mount instances on 1 node ========================================================== 04:30:34 (1788942634) 192.168.206.145@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg645-client.virtnet /mnt/lustre (opts:) 192.168.206.145@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg645-client.virtnet /mnt/lustre2 (opts:) Starting race between client mounts (duration 30) Iteration 0 Starting client: oleg645-client.virtnet: -o user_xattr,flock 192.168.206.145@tcp:/lustre /mnt/lustre Starting client: oleg645-client.virtnet: -o user_xattr,flock 192.168.206.145@tcp:/lustre /mnt/lustre2 192.168.206.145@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 192.168.206.145@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg645-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg645-client.virtnet /mnt/lustre (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg645-server' oleg645-server: oleg645-server.virtnet: executing load_modules_local oleg645-server: kmemleak disabled oleg645-server: Loading modules from /home/green/git/lustre-release/lustre oleg645-server: detected 4 online CPUs by sysfs oleg645-server: MODOPTS_LIBCFS= oleg645-server: Force libcfs to create 2 CPU partitions oleg645-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Iteration 1 Starting client: oleg645-client.virtnet: -o user_xattr,flock 192.168.206.145@tcp:/lustre /mnt/lustre2 Starting client: oleg645-client.virtnet: -o user_xattr,flock 192.168.206.145@tcp:/lustre /mnt/lustre 192.168.206.145@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 192.168.206.145@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg645-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg645-client.virtnet /mnt/lustre (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg645-server' oleg645-server: oleg645-server.virtnet: executing load_modules_local oleg645-server: kmemleak disabled oleg645-server: Loading modules from /home/green/git/lustre-release/lustre oleg645-server: detected 4 online CPUs by sysfs oleg645-server: MODOPTS_LIBCFS= oleg645-server: Force libcfs to create 2 CPU partitions oleg645-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Starting client: oleg645-client.virtnet: -o user_xattr,flock 192.168.206.145@tcp:/lustre /mnt/lustre Starting client: oleg645-client.virtnet: -o user_xattr,flock 192.168.206.145@tcp:/lustre /mnt/lustre2 PASS 109 (71s) == sanityn test 110: do not grant another lock on resend ========================================================== 04:31:45 (1788942705) fail_loc=0x525 fail_val=3 fail_loc=0x534 File: /mnt/lustre/f110.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238860488707 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-09 04:31:47.000000000 -0400 Modify: 2026-09-09 04:31:47.000000000 -0400 Change: 2026-09-09 04:31:47.000000000 -0400 Birth: 2026-09-09 04:31:46.000000000 -0400 fail_loc=0 fail_val=0 Connected clients: oleg645-client.virtnet oleg645-client.virtnet PASS 110 (83s) == sanityn test 111: A racy rename/link an open file should not cause fs corruption ========================================================== 04:33:08 (1788942788) SKIP: sanityn test_111 needs >= 2 MDTs SKIP 111 (4s) == sanityn test 112: update max-inherit in default LMV === 04:33:12 (1788942792) SKIP: sanityn test_112 We need at least 2 MDTs for this test SKIP 112 (3s) == sanityn test 113: check servers of specified fs ======= 04:33:15 (1788942795) lustre-OST0000-osc-ffff9fbb530a8800 active. lustre-OST0001-osc-ffff9fbb530a8800 active. lustre-OST0000-osc-ffff9fbb530a8800 active. lustre-OST0001-osc-ffff9fbb530a8800 active. MGC192.168.206.145@tcp active. PASS 113 (6s) == sanityn test 114: implicit default LMV inherit ======== 04:33:21 (1788942801) SKIP: sanityn test_114 We need at least 2 MDTs for this test SKIP 114 (3s) == sanityn test 115: ldiskfs doesn't check direntry for uniqueness ========================================================== 04:33:24 (1788942804) SKIP: sanityn test_115 ldiskfs only test SKIP 115 (3s) == sanityn test 116: DNE: Set default LMV layout from a remote client ========================================================== 04:33:27 (1788942807) SKIP: sanityn test_116 needs >= 2 MDTs SKIP 116 (3s) == sanityn test 121: trunc append race =================== 04:33:30 (1788942810) 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00965172 s, 2.1 MB/s fail_loc=0x80001436 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0690544 s, 297 kB/s 0000000 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0005000 0000 0000 0000 0000 0000 0000 0000 0000 * 000c800 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0011800 PASS 121 (8s) == sanityn test 122: directory size is consistent across mounts ========================================================== 04:33:38 (1788942818) PASS 122 (8s) == sanityn test 200: service remains healthy while able to process request ========================================================== 04:33:46 (1788942826) at_max=10 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=9 fail_loc=0x1e1 fail_val=20 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1116 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0972 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.2163 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1919 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.2777 s, 0.2 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.263 s, 0.2 kB/s 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.2329 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1559 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1969 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.2271 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.2985 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.297 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.2713 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.258 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.3667 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.3999 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.3046 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.3462 s, 0.1 kB/s at_max=600 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=116 oleg645-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9fbb4883b000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9fbb4883b000.ost_server_uuid in IDLE state after 0 sec PASS 200 (79s) cleanup: ====================================================== == sanityn test complete, duration 10653 sec ============= 04:35:06 (1788942906) === sanityn: start cleanup 04:35:08 (1788942908) === Stopping clients: oleg645-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg645-client.virtnet /mnt/lustre2 opts: === sanityn: finish cleanup 04:39:19 (1788943159) ===