-----============= acceptance-small: sanityn ============----- Wed Sep 30 08:54:19 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky use SANITYN_ONLY='' SANITYN_EXCEPT='' oleg255-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanityn.*ex': No such file or directory excepting tests: 27 40a 56b skipping tests SLOW=no: 33a === sanityn: start setup 08:54:53 (1790772893) === Starting client oleg255-client.virtnet: -o user_xattr,flock 192.168.202.155@tcp:/lustre /mnt/lustre2 Mount client oleg255-client.virtnet: mount -t lustre -o user_xattr,flock 192.168.202.155@tcp:/lustre /mnt/lustre2 Started clients oleg255-client.virtnet: 192.168.202.155@tcp:/lustre on /mnt/lustre2 type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) oleg255-client.virtnet: executing check_config_client /mnt/lustre oleg255-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg255-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8c4950f45000.idle_timeout=debug osc.lustre-OST0000-osc-ffff8c4953479800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8c4950f45000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8c4953479800.idle_timeout=debug disable quota as required oleg255-server: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanityn: finish setup 08:55:34 (1790772934) === 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0382594 s, 27.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7627] == sanityn test 0: do_node_vp() and do_facet_vp() do the right thing ========================================================== 08:55:37 (1790772937) PASS 0 (10s) == sanityn test 1: Check attribute updates on 2 mount points ========================================================== 08:55:46 (1790772946) /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0777 OK /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0666 OK /mnt/lustre/f1.sanityn: absent OK PASS 1 (8s) == sanityn test 2a: check cached attribute updates on 2 mtpt's ================================================================== 08:55:54 (1790772954) -rw-r--r-- 1 root root 0 Sep 30 08:55 /mnt/lustre2/f2a /mnt/lustre/f2a has type file OK /mnt/lustre/f2a has perms 0777 OK PASS 2a (7s) == sanityn test 2b: check cached attribute updates on 2 mtpt's ================================================================== 08:56:01 (1790772961) -rw-r--r-- 1 root root 0 Sep 30 08:56 /mnt/lustre2/f2b /mnt/lustre2/f2b has type file OK /mnt/lustre2/f2b has perms 0777 OK PASS 2b (8s) == sanityn test 2c: check cached attribute updates on 2 mtpt's root ============================================================= 08:56:09 (1790772969) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0777 OK PASS 2c (8s) == sanityn test 2d: check cached attribute updates on 2 mtpt's root ============================================================= 08:56:17 (1790772977) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0755 OK PASS 2d (7s) == sanityn test 2e: check chmod on root is propagated to others ========================================================== 08:56:24 (1790772984) total 1 -rwxrwxrwx 1 root root 0 Sep 30 08:55 f2a -rwxrwxrwx 1 root root 0 Sep 30 08:56 f2b total 1 -rwxrwxrwx 1 root root 0 Sep 30 08:55 f2a -rwxrwxrwx 1 root root 0 Sep 30 08:56 f2b running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre2/f2e.sanityn] [count=1] 1+0 records in 1+0 records out 512 bytes copied, 0.0135508 s, 37.8 kB/s PASS 2e (6s) == sanityn test 2f: check attr/owner updates on DNE with 2 mtpt's ========================================================== 08:56:31 (1790772991) SKIP: sanityn test_2f needs >= 2 MDTs SKIP 2f (4s) == sanityn test 2g: check blocks update on sync write ==== 08:56:34 (1790772994) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.575781 s, 3.6 MB/s /mnt/lustre/f2g.sanityn has 4103 blocks /mnt/lustre2/f2g.sanityn has 4103 blocks PASS 2g (7s) == sanityn test 3: symlink on one mtpt, readlink on another ===================================================================== 08:56:41 (1790773001) PASS 3 (6s) == sanityn test 4: fstat validation on multiple mount points ==================================================================== 08:56:47 (1790773007) PASS 4 (9s) == sanityn test 5: create a file on one mount, truncate it on the other ========================================================== 08:56:56 (1790773016) /mnt/lustre/f5 has type file OK /mnt/lustre/f5 has size 100 OK PASS 5 (9s) == sanityn test 6: remove of open file on other node ============================================================================ 08:57:06 (1790773026) opening writing unlinking /mnt/lustre2/f6.sanityn accessing (1) seeking (1) accessing (2) fstat... reading comparing data truncating seeking (2) writing again seeking (3) reading again comparing data again closing SUCCESS - goto beer PASS 6 (8s) == sanityn test 7: remove of open directory on other node ======================================================================= 08:57:13 (1790773033) creating directory /mnt/lustre/d7 opening directory unlinking /mnt/lustre/d7 Ok, everything goes well. PASS 7 (7s) == sanityn test 8: remove of open special file on other node ==================================================================== 08:57:20 (1790773040) creating special file /mnt/lustre/f8.sanityn opening file unlinking /mnt/lustre/f8.sanityn Ok, everything goes well. PASS 8 (6s) == sanityn test 9a: append of file with sub-page size on multiple mounts ========================================================== 08:57:26 (1790773046) PASS 9a (7s) == sanityn test 9b: append to striped sparse file ======== 08:57:33 (1790773053) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0457904 s, 22.9 MB/s 3+0 records in 3+0 records out 3 bytes copied, 0.0675042 s, 0.0 kB/s Data read (expecting 'foo'): foo PASS 9b (7s) == sanityn test 10a: write of file with sub-page size on multiple mounts ========================================================== 08:57:40 (1790773060) 1+0 records in 1+0 records out 1 byte copied, 0.00606851 s, 0.2 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0115716 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0162109 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0123538 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0193688 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.016428 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0146822 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0227132 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0120768 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0115985 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00824825 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00790279 s, 0.1 kB/s PASS 10a (8s) == sanityn test 10b: write of file with sub-page size on multiple mounts ========================================================== 08:57:48 (1790773068) 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.0160106 s, 192 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0492479 s, 83.2 kB/s 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.00023685 s, 13.0 MB/s PASS 10b (7s) == sanityn test 11: execution of file opened for write should return error ============================================================== 08:57:55 (1790773075) multiop /mnt/lustre/d11/f vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7627 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 304: /mnt/lustre2/d11/f: Text file busy PASS 11 (8s) == sanityn test 12: test lock ordering (link, stat, unlink) ========================================================== 08:58:03 (1790773083) warning: '-runtime' deprecated, use '-t runtime' instead - link 335 (time 1790773095.48 total 10.02 last 33.44) - link 626 (time 1790773105.49 total 20.03 last 29.06) - link 966 (time 1790773115.50 total 30.05 last 33.96) - link 1286 (time 1790773125.50 total 40.05 last 31.99) - link 1616 (time 1790773135.50 total 50.05 last 33.00) - link 1919 (time 1790773145.52 total 60.07 last 30.24) - link 2238 (time 1790773155.53 total 70.07 last 31.88) - link 2610 (time 1790773165.55 total 80.09 last 37.13) - link 2953 (time 1790773175.58 total 90.13 last 34.19) total: 3309 link in 100.01 seconds: 33.09 ops/second using seed 2646679165 running for 100 seconds - stat 4534 (time 1790773097 ; total 11 ; last 11) - stat 10000 (time 1790773107 ; total 21 ; last 10) - stat 16316 (time 1790773118 ; total 32 ; last 11) - stat 20000 (time 1790773124 ; total 38 ; last 6) - stat 25690 (time 1790773135 ; total 49 ; last 11) - stat 28817 (time 1790773146 ; total 60 ; last 11) - stat 30000 (time 1790773150 ; total 64 ; last 4) - stat 32728 (time 1790773161 ; total 75 ; last 11) - stat 35426 (time 1790773172 ; total 86 ; last 11) - stat 38648 (time 1790773183 ; total 97 ; last 11) total: 39723 stats in 100 seconds: 397.230011 stats/second - unlinked 0 (time 1790773136 ; total 0 ; last 0) unlink(/mnt/lustre2/lockdir/lockfile3309) error: No such file or directory total: 3308 unlinks in 89 seconds: 37.168541 unlinks/second /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 77: kill: (21451) - No such process /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 78: kill: (21453) - No such process PASS 12 (149s) == sanityn test 13: test directory page revocation ======= 09:00:32 (1790773232) f13 f13 PASS 13 (10s) == sanityn test 14aa: execution of file open for write returns -ETXTBSY ========================================================== 09:00:42 (1790773242) multiop /mnt/lustre/d14aa.sanityn/f14aa.sanityn vOw_c TMPPIPE=/tmp/multiop_open_wait_pipe.7627 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 339: /mnt/lustre2/d14aa.sanityn/f14aa.sanityn: Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5095: 22638 Terminated $MULTIOP_PROG $FILE v$ARGS > $TMPPIPE (wd: ~) PASS 14aa (8s) == sanityn test 14ab: open(RDWR) of executing file returns -ETXTBSY ========================================================== 09:00:50 (1790773250) open(O_RDWR|O_CREAT): Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5095: 23217 Terminated $DIR1/$tdir/sleep 60 (wd: ~) PASS 14ab (7s) == sanityn test 14b: truncate of executing file returns -ETXTBSY ================================================================ 09:00:57 (1790773257) truncate: cannot truncate '/mnt/lustre2/d14b.sanityn/sleep' to length 60: Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 354: 23796 Terminated $DIR1/$tdir/sleep 60 PASS 14b (8s) == sanityn test 14c: open(O_TRUNC) of executing file return -ETXTBSY ============================================================ 09:01:05 (1790773265) cp: cannot create regular file '/mnt/lustre2/d14c.sanityn/sleep': Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 366: 24392 Terminated $DIR1/$tdir/sleep 60 PASS 14c (7s) == sanityn test 14d: chmod of executing file is still possible ================================================================== 09:01:12 (1790773272) chmod /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 377: 24975 Terminated $DIR1/$tdir/sleep 60 PASS 14d (10s) == sanityn test 15: test out-of-space with multiple writers ===================================================================== 09:01:22 (1790773282) PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin: MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky sleep 5 for ZFS MDS Waiting for MDT destroys to complete fallocate on zfs doesn't consume space STRIPECOUNT=2 ORIGFREE=7531520 MAXFREE=800000 SKIP: oos2 test_15 oos2.sh: 7531520KB free > 800000KB, increase MAXFREE (or reduce fs size) sleep 5 for ZFS MDS Waiting for MDT destroys to complete SKIP 15 (52s) == sanityn test 16a: 500 iterations of dual-mount fsx ==== 09:02:15 (1790773335) keep default fallocate mode: 2 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.489236 s, 17.1 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.331509 s, 25.3 MB/s Chance of close/open is 1 in 50 Seed set to 3340 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read skipping zero length punch hole truncating to largest ever: 0x34ad1d 000006[1] 1790773340.917296 trunc from 00000000 to 0x34ad1c (0x34ad1d bytes) 000009[1] 1790773341.026822 write 0x988100 thru 0x9932f5 (0xb1f6 bytes) 000010[0] 1790773341.055742 mapread 0x1c2702 thru 0x1d09b7 (0xe2b6 bytes) 000012[0] 1790773341.160861 trunc from 0x9932f6 to 0x0e45fe (0x8aecf7 bytes) 000013[1] 1790773341.229857 mapwrite 0x958448 thru 0x95ef13 (0x6acc bytes) 000017[1] 1790773341.432183 trunc from 0x95ef14 to 0x09ad52 (0x8c41c1 bytes) 000019[0] 1790773341.453244 punch from 0x07c19f to 0x07daa6 (0x1908 bytes) 000020[1] 1790773341.485356 trunc from 0x09ad53 to 0x09eb48 (0x3df6 bytes) 000021[0] 1790773341.508490 mapread 0x029d1f thru 0x033455 (0x9737 bytes) 000023[0] 1790773341.565767 trunc from 0x09eb49 to 0x156e7f (0xb8337 bytes) 000026[0] 1790773341.652187 read 0x048b99 thru 0x0573a7 (0xe80f bytes) 000029[1] 1790773341.694670 punch from 0x095ea2 to 0x098fe3 (0x3142 bytes) 000032[0] 1790773341.764687 mapwrite 0x138c2f thru 0x1447ab (0xbb7d bytes) 000033[1] 1790773341.872083 write 0x3be683 thru 0x3cbe35 (0xd7b3 bytes) 000035[0] 1790773341.925812 mapwrite 0x8487a4 thru 0x84bb19 (0x3376 bytes) 000036[0] 1790773342.065135 read 0x45ef23 thru 0x4651b7 (0x6295 bytes) 000037[0] 1790773342.079749 trunc from 0x84bb1a to 0x10f9a9 (0x73c170 bytes) 000038[1] 1790773342.098720 read 0x03cf64 thru 0x04ad37 (0xddd4 bytes) 000043[1] 1790773342.137047 mapread 0x03c94d thru 0x03d376 (0xa2a bytes) 000044[1] 1790773342.141876 write 0x542d9b thru 0x546a70 (0x3cd6 bytes) 000046[0] 1790773342.168822 read 0x2d15cb thru 0x2dc808 (0xb23e bytes) 000047[0] 1790773342.247798 mapread 0x131cfd thru 0x132b23 (0xe27 bytes) 000049[0] 1790773342.393481 read 0x31f537 thru 0x32c230 (0xccfa bytes) 000050[0] 1790773342.482299 punch from 0x3d1328 to 0x3d5d5f (0x4a38 bytes) 000051[0] 1790773342.520681 write 0x735e39 thru 0x737b7c (0x1d44 bytes) 000054[1] 1790773342.605833 write 0x8cf8d5 thru 0x8da768 (0xae94 bytes) 000056[0] 1790773342.666636 read 0x41f3b6 thru 0x425130 (0x5d7b bytes) 000058[1] 1790773342.733916 write 0x54d981 thru 0x554b9b (0x721b bytes) 000060[1] 1790773342.795758 write 0x4a82e4 thru 0x4b7a50 (0xf76d bytes) 000061[1] 1790773342.855762 mapread 0x284d20 thru 0x2901fa (0xb4db bytes) 000062[1] 1790773342.928992 write 0x58f982 thru 0x59d9bc (0xe03b bytes) 000066[0] 1790773343.023043 mapread 0x500c49 thru 0x507b4c (0x6f04 bytes) 000068[1] 1790773343.152463 read 0x2011dd thru 0x20bb52 (0xa976 bytes) truncating to largest ever: 0x4bac85 000069[1] 1790773343.185065 trunc from 0x8da769 to 0x4bac84 (0x41fae4 bytes) 000071[1] 1790773343.273175 trunc from 0x4bac85 to 0x3267d7 (0x1944ad bytes) 000072[1] 1790773343.320589 mapwrite 0x986c41 thru 0x98a2d6 (0x3696 bytes) truncating to largest ever: 0x774a79 000074[0] 1790773343.463049 trunc from 0x98a2d7 to 0x774a78 (0x21585e bytes) 000076[1] 1790773343.577767 mapread 0x519045 thru 0x524744 (0xb700 bytes) 000077[1] 1790773343.670089 read 0x5f5370 thru 0x5f6caa (0x193b bytes) 000080[0] 1790773343.737033 mapread 0x236239 thru 0x2363f3 (0x1bb bytes) 000082[1] 1790773343.761191 trunc from 0x774a79 to 0x39a3cd (0x3da6ab bytes) 000083[1] 1790773343.845380 mapread 0x28719f thru 0x2891c2 (0x2024 bytes) 000085[1] 1790773343.895264 mapwrite 0x60ea91 thru 0x610212 (0x1782 bytes) 000087[1] 1790773344.005469 mapwrite 0x82aade thru 0x8330a0 (0x85c3 bytes) 000090[1] 1790773344.144729 trunc from 0x8330a1 to 0x0e26d0 (0x7509d0 bytes) 000092[0] 1790773344.191495 trunc from 0x0e26d1 to 0x6b210a (0x5cfa3a bytes) 000094[0] 1790773344.270606 mapread 0x69b70a thru 0x6a4ca0 (0x9597 bytes) 000095[1] 1790773344.319297 write 0x12ffed thru 0x132e3d (0x2e51 bytes) truncating to largest ever: 0x7d6670 000096[1] 1790773344.359851 trunc from 0x6b210b to 0x7d666f (0x124565 bytes) 000097[1] 1790773344.423878 trunc from 0x7d6670 to 0x2894f4 (0x54d17b bytes) 000098[1] 1790773344.466202 punch from 0x2793ba to 0x27d518 (0x415f bytes) 000099[0] 1790773344.498238 read 0x1d694d thru 0x1d956c (0x2c20 bytes) 000100[1] 1790773344.712866 trunc from 0x2894f5 to 0x1c5b9d (0xc3957 bytes) 000103[1] 1790773344.817263 mapread 0x0b49fd thru 0x0b5902 (0xf06 bytes) 000105[0] 1790773344.876243 read 0x0788e9 thru 0x080523 (0x7c3b bytes) 000106[0] 1790773344.959750 trunc from 0x1c5b9e to 0x583b06 (0x3bdf69 bytes) 000108[1] 1790773345.048617 write 0x710706 thru 0x716ea8 (0x67a3 bytes) 000109[1] 1790773345.084353 mapread 0x14559d thru 0x14a1f0 (0x4c54 bytes) 000110[0] 1790773345.125561 mapwrite 0x1987a0 thru 0x1a5213 (0xca74 bytes) 000112[1] 1790773345.364914 mapwrite 0x1fb6d9 thru 0x1ff8ee (0x4216 bytes) 000114[0] 1790773345.499696 trunc from 0x716ea9 to 0x78566c (0x6e7c4 bytes) 000116[1] 1790773345.615122 mapwrite 0x1205e3 thru 0x129313 (0x8d31 bytes) 000119[0] 1790773345.789210 trunc from 0x78566d to 0x6df9be (0xa5cae bytes) 000120[1] 1790773345.860699 write 0x8f9def thru 0x8fc940 (0x2b52 bytes) 000121[1] 1790773345.901872 write 0x3994f2 thru 0x39dc7f (0x478e bytes) truncating to largest ever: 0x9aafdd 000122[0] 1790773345.928123 trunc from 0x8fc941 to 0x9aafdc (0xae69c bytes) 000123[0] 1790773345.982737 write 0x90a1c0 thru 0x9120a2 (0x7ee3 bytes) 000126[1] 1790773346.153578 read 0x33ba9f thru 0x3474e3 (0xba45 bytes) 000127[0] 1790773346.193792 punch from 0x550fcb to 0x558808 (0x783e bytes) 000129[1] 1790773346.199718 read 0x2b1ed4 thru 0x2b3268 (0x1395 bytes) 000130[1] 1790773346.205461 punch from 0x466bff to 0x4689fd (0x1dff bytes) 000132[1] 1790773346.242797 write 0x173c3a thru 0x1801a3 (0xc56a bytes) 000133[1] 1790773346.272199 mapread 0x1f890a thru 0x1fff9a (0x7691 bytes) 000134[1] 1790773346.335048 punch from 0x6e12f5 to 0x6e3200 (0x1f0c bytes) 000137[0] 1790773346.386974 trunc from 0x9aafdd to 0x0da067 (0x8d0f75 bytes) 000138[1] 1790773346.442165 mapwrite 0x5b7c4d thru 0x5badd1 (0x3185 bytes) 000139[0] 1790773346.598193 read 0x02695e thru 0x03549d (0xeb40 bytes) 000140[0] 1790773346.697607 read 0x5405d8 thru 0x540905 (0x32e bytes) 000141[1] 1790773346.721647 write 0x643926 thru 0x65232a (0xea05 bytes) 000143[0] 1790773346.754135 write 0x4103d3 thru 0x41eec6 (0xeaf4 bytes) 000145[0] 1790773346.816897 punch from 0x2d3904 to 0x2e1d21 (0xe41e bytes) 000146[0] 1790773346.869663 mapread 0x250c29 thru 0x25b1f9 (0xa5d1 bytes) 000148[0] 1790773346.929680 mapwrite 0x5a165c thru 0x5a6896 (0x523b bytes) 000149[1] 1790773347.028640 punch from 0x07fe60 to 0x086d98 (0x6f39 bytes) 000152[0] 1790773347.102205 write 0x601095 thru 0x60b17f (0xa0eb bytes) 000155[0] 1790773347.156767 mapread 0x05a225 thru 0x05bcaf (0x1a8b bytes) 000158[0] 1790773347.230059 trunc from 0x65232b to 0x3c30c2 (0x28f268 bytes) 000159[1] 1790773347.265099 mapread 0x32ee45 thru 0x330f44 (0x2100 bytes) 000160[1] 1790773347.341834 trunc from 0x3c30c3 to 0x6bfbb7 (0x2fcaf5 bytes) 000162[1] 1790773347.388886 read 0x66d27a thru 0x674758 (0x74df bytes) 000163[1] 1790773347.397464 trunc from 0x6bfbb8 to 0x1973ae (0x528809 bytes) 000164[0] 1790773347.417689 mapread 0x033f82 thru 0x03654a (0x25c9 bytes) 000165[1] 1790773347.457462 read 0x0d7521 thru 0x0e5255 (0xdd35 bytes) 000166[1] 1790773347.507077 write 0x553fae thru 0x555a4d (0x1aa0 bytes) 000167[1] 1790773347.510047 mapread 0x09343f thru 0x0a19a9 (0xe56b bytes) 000168[0] 1790773347.529049 write 0x371491 thru 0x375202 (0x3d72 bytes) 000169[0] 1790773347.562117 mapread 0x1e80fb thru 0x1eb264 (0x316a bytes) 000170[1] 1790773347.602275 mapread 0x2f613c thru 0x2f6f3f (0xe04 bytes) 000172[0] 1790773347.700492 mapwrite 0x2f2e50 thru 0x30292a (0xfadb bytes) 000173[1] 1790773347.837678 read 0x08b0da thru 0x0926ba (0x75e1 bytes) 000175[1] 1790773347.876981 write 0x3c0b08 thru 0x3c4583 (0x3a7c bytes) 000177[1] 1790773347.905067 mapread 0x0349dc thru 0x03edb5 (0xa3da bytes) 000178[1] 1790773347.930117 trunc from 0x555a4e to 0x3cc639 (0x189414 bytes) 000179[0] 1790773347.962780 mapread 0x282edb thru 0x28f041 (0xc167 bytes) 000180[0] 1790773348.037102 punch from 0x1e79d0 to 0x1e9af2 (0x2123 bytes) 000181[1] 1790773348.069285 punch from 0x0e7fd0 to 0x0f3d03 (0xbd34 bytes) 000182[1] 1790773348.104770 read 0x231f9b thru 0x23ad90 (0x8df6 bytes) 000187[0] 1790773348.252895 mapread 0x21a132 thru 0x21e1c0 (0x408f bytes) 000188[0] 1790773348.312826 mapread 0x0b2e39 thru 0x0b3fa1 (0x1169 bytes) 000189[0] 1790773348.351692 punch from 0x1e1dd5 to 0x1ea17c (0x83a8 bytes) 000190[1] 1790773348.395011 write 0x631dff thru 0x63a176 (0x8378 bytes) 000191[0] 1790773348.418570 mapread 0x437fbd thru 0x441e8e (0x9ed2 bytes) 000193[1] 1790773348.507139 punch from 0x19243e to 0x19a278 (0x7e3b bytes) 000194[0] 1790773348.535142 trunc from 0x63a177 to 0x5e3279 (0x56efd bytes) 000196[1] 1790773348.670377 write 0x7a5148 thru 0x7aa1e2 (0x509b bytes) 000197[0] 1790773348.705397 trunc from 0x7aa1e3 to 0x8c610d (0x11bf2b bytes) 000198[1] 1790773348.755413 write 0x25b16e thru 0x26308d (0x7f20 bytes) 000199[1] 1790773348.809578 punch from 0x56ab5a to 0x578e24 (0xe2cb bytes) 000201[1] 1790773348.830969 mapread 0x0dd9fa thru 0x0e0f60 (0x3567 bytes) 000203[1] 1790773348.854059 read 0x53f592 thru 0x548371 (0x8de0 bytes) 000204[0] 1790773348.905452 mapwrite 0x9ee47e thru 0x9f4a6e (0x65f1 bytes) 000208[1] 1790773349.316923 mapwrite 0x3e258b thru 0x3efe5d (0xd8d3 bytes) 000209[1] 1790773349.597259 trunc from 0x9f4a6f to 0x14ae2b (0x8a9c43 bytes) 000210[0] 1790773349.662035 mapwrite 0x1e3b45 thru 0x1ed44d (0x9909 bytes) 000212[0] 1790773349.810846 mapread 0x01a271 thru 0x0213b0 (0x7140 bytes) 000213[1] 1790773349.833948 mapwrite 0x652b00 thru 0x65b3f7 (0x88f8 bytes) 000217[0] 1790773349.978027 write 0x0ab32d thru 0x0b019b (0x4e6f bytes) 000220[0] 1790773350.043790 write 0x5211bb thru 0x530ede (0xfd24 bytes) 000221[0] 1790773350.087672 mapwrite 0x495238 thru 0x4a10f9 (0xbec2 bytes) 000222[0] 1790773350.223792 write 0x119eb8 thru 0x12813f (0xe288 bytes) 000223[1] 1790773350.271619 punch from 0x1081bc to 0x109462 (0x12a7 bytes) 000226[0] 1790773350.378713 write 0x6d2274 thru 0x6d7a5b (0x57e8 bytes) 000228[1] 1790773350.418771 mapwrite 0x3e06fc thru 0x3ec0ea (0xb9ef bytes) 000229[0] 1790773350.546386 read 0x5fb157 thru 0x607a71 (0xc91b bytes) 000231[1] 1790773350.597257 trunc from 0x6d7a5c to 0x0fb1fd (0x5dc85e bytes) 000232[1] 1790773350.667097 punch from 0x0e3d69 to 0x0ec548 (0x87e0 bytes) 000235[1] 1790773350.718751 mapread 0x0f745c thru 0x0fb1fd (0x3da2 bytes) 000238[0] 1790773350.787106 mapread 0x0ad516 thru 0x0b5556 (0x8041 bytes) 000240[1] 1790773350.874269 read 0x0154da thru 0x024dbc (0xf8e3 bytes) 000242[1] 1790773350.929646 write 0x41e43f thru 0x422b01 (0x46c3 bytes) 000243[1] 1790773350.931546 punch from 0x21f329 to 0x22671b (0x73f3 bytes) 000244[0] 1790773350.981685 read 0x3413a6 thru 0x349b8c (0x87e7 bytes) 000245[0] 1790773351.065583 punch from 0x24efbe to 0x25288d (0x38d0 bytes) 000246[0] 1790773351.103881 read 0x315727 thru 0x31a882 (0x515c bytes) 000247[1] 1790773351.155983 punch from 0x32f2b7 to 0x33061a (0x1364 bytes) 000249[0] 1790773351.208531 trunc from 0x422b02 to 0x0d8403 (0x34a6fe bytes) 000250[0] 1790773351.329873 trunc from 0x0d8404 to 0x6bc229 (0x5e3e26 bytes) 000251[0] 1790773351.369439 mapwrite 0x1b83ed thru 0x1c04aa (0x80be bytes) 000252[0] 1790773351.552751 punch from 0x0adf7a to 0x0b8193 (0xa21a bytes) 000254[0] 1790773351.583929 mapwrite 0x35cf94 thru 0x360c63 (0x3cd0 bytes) 000255[0] 1790773351.729940 punch from 0x022411 to 0x031de0 (0xf9d0 bytes) 000256[0] 1790773351.753920 mapwrite 0x4dcdc0 thru 0x4e63fa (0x963b bytes) 000258[1] 1790773351.877888 punch from 0x36c3d6 to 0x3791af (0xcdda bytes) 000259[1] 1790773351.904259 trunc from 0x6bc22a to 0x2fb686 (0x3c0ba3 bytes) 000263[1] 1790773352.008892 mapwrite 0x9a1379 thru 0x9a6308 (0x4f90 bytes) 000264[0] 1790773352.247816 write 0x923290 thru 0x93271f (0xf490 bytes) 000265[1] 1790773352.301146 mapread 0x5446d7 thru 0x5459b3 (0x12dd bytes) 000272[1] 1790773352.445701 mapread 0x8bdf8b thru 0x8bfe52 (0x1ec8 bytes) 000274[0] 1790773352.548601 mapwrite 0x3b57fe thru 0x3c2d03 (0xd506 bytes) 000275[1] 1790773352.682088 trunc from 0x9a6309 to 0x94e716 (0x57bf2 bytes) 000276[1] 1790773352.749130 write 0x574981 thru 0x581dfb (0xd47b bytes) 000278[0] 1790773352.783361 punch from 0x149468 to 0x157a89 (0xe622 bytes) 000281[0] 1790773352.866031 mapread 0x54a526 thru 0x557365 (0xce40 bytes) 000282[0] 1790773352.979712 mapread 0x602f9b thru 0x606391 (0x33f7 bytes) 000283[1] 1790773353.027964 mapwrite 0x170625 thru 0x17e4d2 (0xdeae bytes) 000287[0] 1790773353.288247 mapwrite 0x01d492 thru 0x02135a (0x3ec9 bytes) 000288[0] 1790773353.458245 mapwrite 0x2dac39 thru 0x2e434a (0x9712 bytes) 000289[1] 1790773353.544541 write 0x8e978c thru 0x8f726c (0xdae1 bytes) 000290[0] 1790773353.594080 read 0x59f629 thru 0x5ad907 (0xe2df bytes) 000294[1] 1790773353.699753 punch from 0x3369e4 to 0x33a01e (0x363b bytes) 000296[1] 1790773353.719790 mapread 0x7fa785 thru 0x805467 (0xace3 bytes) 000297[1] 1790773353.777669 write 0x9d89f6 thru 0x9e651e (0xdb29 bytes) 000300[0] 1790773353.821762 punch from 0x3d23b1 to 0x3db2c1 (0x8f11 bytes) 000302[0] 1790773353.892019 punch from 0x5b9f45 to 0x5c1ab7 (0x7b73 bytes) 000304[1] 1790773353.921770 mapread 0x57909f thru 0x57d540 (0x44a2 bytes) 000306[1] 1790773353.998530 mapread 0x5245f6 thru 0x526b88 (0x2593 bytes) 000307[1] 1790773354.022011 write 0x8eb04c thru 0x8faaf2 (0xfaa7 bytes) 000310[1] 1790773354.061594 trunc from 0x9e651f to 0x25956c (0x78cfb2 bytes) 000311[0] 1790773354.117077 read 0x22767a thru 0x2302d7 (0x8c5e bytes) 000313[0] 1790773354.188687 mapwrite 0x959ee7 thru 0x95b57c (0x1696 bytes) 000314[1] 1790773354.522529 read 0x1c6388 thru 0x1ce926 (0x859f bytes) 000317[0] 1790773354.586197 mapwrite 0x19f0d5 thru 0x1aed6d (0xfc99 bytes) 000318[0] 1790773354.800072 read 0x2581ae thru 0x25d1d7 (0x502a bytes) 000321[1] 1790773354.868817 write 0x875a14 thru 0x8853f6 (0xf9e3 bytes) 000323[1] 1790773354.951663 mapwrite 0x2787f9 thru 0x279435 (0xc3d bytes) 000326[1] 1790773355.103736 mapread 0x4a18a4 thru 0x4aedfa (0xd557 bytes) 000327[0] 1790773355.169229 trunc from 0x95b57d to 0x356725 (0x604e57 bytes) 000328[1] 1790773355.226598 mapwrite 0x6b9f03 thru 0x6c4b1b (0xac19 bytes) 000329[1] 1790773355.362556 punch from 0x229eab to 0x230e26 (0x6f7c bytes) 000330[0] 1790773355.372353 mapread 0x4b029e thru 0x4b85cd (0x8330 bytes) 000332[1] 1790773355.421101 write 0x97d952 thru 0x97f86a (0x1f19 bytes) 000333[0] 1790773355.434835 trunc from 0x97f86b to 0x4755bb (0x50a2af bytes) 000337[1] 1790773355.532046 write 0x0deec9 thru 0x0e50e0 (0x6218 bytes) 000338[0] 1790773355.578029 mapwrite 0x7991c4 thru 0x7a7a9b (0xe8d8 bytes) 000339[0] 1790773355.790998 trunc from 0x7a7a9c to 0x2bb6b9 (0x4ec3e2 bytes) 000340[1] 1790773355.812908 mapread 0x10370f thru 0x105c52 (0x2544 bytes) 000342[1] 1790773355.867890 write 0x4fa023 thru 0x509e1c (0xfdfa bytes) 000343[0] 1790773355.902514 mapwrite 0x4d915f thru 0x4de77a (0x561c bytes) 000344[0] 1790773356.003131 punch from 0x419fd8 to 0x428838 (0xe861 bytes) 000345[1] 1790773356.036152 mapwrite 0x61bb15 thru 0x6264c1 (0xa9ad bytes) 000349[1] 1790773356.212676 read 0x433fb9 thru 0x43c0ef (0x8137 bytes) 000351[0] 1790773356.274860 mapwrite 0x5989f6 thru 0x5a3633 (0xac3e bytes) 000352[0] 1790773356.410842 read 0x434f76 thru 0x444900 (0xf98b bytes) 000354[1] 1790773356.458117 mapread 0x265f17 thru 0x26b61f (0x5709 bytes) 000355[0] 1790773356.506856 write 0x91c2a3 thru 0x923472 (0x71d0 bytes) 000356[1] 1790773356.526773 punch from 0x0d8da7 to 0x0dc246 (0x34a0 bytes) 000357[0] 1790773356.574797 trunc from 0x923473 to 0x1863f2 (0x79d080 bytes) truncating to largest ever: 0x9f4fc5 000358[1] 1790773356.643206 trunc from 0x1863f3 to 0x9f4fc4 (0x86ebd2 bytes) 000359[0] 1790773356.713542 mapwrite 0x719726 thru 0x728add (0xf3b8 bytes) 000360[0] 1790773356.843162 write 0x55f9f8 thru 0x5699ba (0x9fc3 bytes) 000362[0] 1790773356.887190 mapread 0x8e6e9d thru 0x8f6ad4 (0xfc38 bytes) 000364[0] 1790773356.968039 mapread 0x2de981 thru 0x2e0c6e (0x22ee bytes) 000367[0] 1790773357.005950 punch from 0x025d1e to 0x0292a4 (0x3587 bytes) 000368[1] 1790773357.022108 mapread 0x465efc thru 0x46924e (0x3353 bytes) 000370[1] 1790773357.097873 read 0x9136b4 thru 0x916247 (0x2b94 bytes) 000372[1] 1790773357.127508 mapread 0x29a557 thru 0x2a1596 (0x7040 bytes) 000373[0] 1790773357.155156 read 0x9480c5 thru 0x94c192 (0x40ce bytes) 000375[1] 1790773357.165485 trunc from 0x9f4fc5 to 0x5a96d6 (0x44b8ee bytes) 000376[0] 1790773357.211836 write 0x215cc0 thru 0x21ca2d (0x6d6e bytes) 000379[0] 1790773357.277856 write 0x88e8bf thru 0x898f4d (0xa68f bytes) 000380[1] 1790773357.299612 punch from 0x34a284 to 0x34add0 (0xb4d bytes) 000385[1] 1790773357.385936 mapread 0x7aae4a thru 0x7ad4ac (0x2663 bytes) 000386[1] 1790773357.409614 write 0x945edc thru 0x94ea96 (0x8bbb bytes) 000387[1] 1790773357.436750 punch from 0x8b3108 to 0x8bbfd0 (0x8ec9 bytes) 000388[1] 1790773357.442701 punch from 0x5690af to 0x5727a5 (0x96f7 bytes) 000392[1] 1790773357.493103 write 0x0d6fad thru 0x0dc568 (0x55bc bytes) 000394[1] 1790773357.511905 mapwrite 0x77eb69 thru 0x782abb (0x3f53 bytes) 000395[1] 1790773357.648820 mapread 0x5ccf76 thru 0x5d28f1 (0x597c bytes) 000396[1] 1790773357.746436 trunc from 0x94ea97 to 0x896006 (0xb8a90 bytes) 000398[1] 1790773357.854013 write 0x7aaf36 thru 0x7ba0d1 (0xf19c bytes) 000400[1] 1790773357.962116 mapread 0x2b73c9 thru 0x2bae46 (0x3a7e bytes) 000401[1] 1790773358.088046 read 0x53203f thru 0x536cfb (0x4cbd bytes) 000403[1] 1790773358.150666 punch from 0x58c0be to 0x58e1f5 (0x2138 bytes) 000404[0] 1790773358.190473 mapwrite 0x45d2f3 thru 0x4695c1 (0xc2cf bytes) 000405[1] 1790773358.426566 trunc from 0x896007 to 0x29380d (0x6027f9 bytes) 000406[1] 1790773358.495938 mapwrite 0x1120bb thru 0x11e912 (0xc858 bytes) 000407[1] 1790773358.602173 mapread 0x1de8d6 thru 0x1e430f (0x5a3a bytes) 000410[0] 1790773358.649619 mapwrite 0x0f89a4 thru 0x104c68 (0xc2c5 bytes) 000411[0] 1790773358.799716 mapwrite 0x247ccd thru 0x248470 (0x7a4 bytes) 000412[0] 1790773358.873836 write 0x60f668 thru 0x612f27 (0x38c0 bytes) 000414[1] 1790773358.936058 write 0x403854 thru 0x40698f (0x313c bytes) 000415[0] 1790773358.984192 read 0x00a16b thru 0x016e0f (0xcca5 bytes) 000416[0] 1790773359.010609 mapwrite 0x17b3a1 thru 0x185022 (0x9c82 bytes) 000417[1] 1790773359.114034 punch from 0x35e5f5 to 0x369584 (0xaf90 bytes) 000418[1] 1790773359.132215 mapread 0x1845f0 thru 0x188e47 (0x4858 bytes) 000420[0] 1790773359.165386 trunc from 0x612f28 to 0x537f25 (0xdb002 bytes) 000421[1] 1790773359.232921 mapwrite 0x95b33c thru 0x96abb5 (0xf87a bytes) 000422[1] 1790773359.424551 write 0x37f94c thru 0x3860ac (0x6761 bytes) 000424[1] 1790773359.452621 punch from 0x957423 to 0x95dab6 (0x6694 bytes) 000427[1] 1790773359.564851 punch from 0x912a27 to 0x916c55 (0x422f bytes) 000428[0] 1790773359.595540 read 0x8edfb6 thru 0x8f4624 (0x666f bytes) 000430[1] 1790773359.680909 punch from 0x75350a to 0x75ef51 (0xba48 bytes) 000431[1] 1790773359.721931 punch from 0x956f01 to 0x9620e9 (0xb1e9 bytes) 000432[1] 1790773359.793643 write 0x572cc6 thru 0x579b7c (0x6eb7 bytes) 000434[0] 1790773359.848157 mapwrite 0x67d785 thru 0x67d8f3 (0x16f bytes) 000435[0] 1790773360.050394 mapwrite 0x67b524 thru 0x68646e (0xaf4b bytes) 000436[0] 1790773360.159751 read 0x292062 thru 0x298d8a (0x6d29 bytes) 000437[1] 1790773360.182210 read 0x41b48d thru 0x428b89 (0xd6fd bytes) 000440[0] 1790773360.211962 read 0x5d04cf thru 0x5d36d3 (0x3205 bytes) 000445[1] 1790773360.221221 read 0x2b7618 thru 0x2bc8ee (0x52d7 bytes) 000447[0] 1790773360.230770 mapwrite 0x9b448e thru 0x9b9ff4 (0x5b67 bytes) 000448[1] 1790773360.360466 write 0x063c7e thru 0x068944 (0x4cc7 bytes) 000449[1] 1790773360.400457 mapwrite 0x99389f thru 0x994a4f (0x11b1 bytes) 000451[0] 1790773360.490834 mapread 0x114e14 thru 0x11e773 (0x9960 bytes) 000452[1] 1790773360.556505 mapread 0x8f8df5 thru 0x905050 (0xc25c bytes) 000453[1] 1790773360.596578 read 0x867a6e thru 0x872c2e (0xb1c1 bytes) 000456[1] 1790773360.637101 read 0x659fc9 thru 0x6679f3 (0xda2b bytes) 000457[0] 1790773360.663005 write 0x5de8fc thru 0x5df32e (0xa33 bytes) 000458[1] 1790773360.682581 mapread 0x31704b thru 0x319de6 (0x2d9c bytes) 000459[1] 1790773360.746789 read 0x7572d3 thru 0x75c500 (0x522e bytes) 000464[1] 1790773360.777833 punch from 0x832e14 to 0x83e4bc (0xb6a9 bytes) 000467[0] 1790773360.799269 punch from 0x2b8b74 to 0x2c25ca (0x9a57 bytes) 000468[0] 1790773360.829264 punch from 0x6a85f1 to 0x6ad84d (0x525d bytes) 000469[0] 1790773360.848651 mapread 0x1330f5 thru 0x134934 (0x1840 bytes) 000470[1] 1790773360.867869 punch from 0x95fcd3 to 0x96780d (0x7b3b bytes) 000472[0] 1790773360.916791 punch from 0x3f82ed to 0x404b12 (0xc826 bytes) 000473[0] 1790773360.964659 mapwrite 0x53d4fc thru 0x543532 (0x6037 bytes) 000474[0] 1790773361.031758 mapread 0x2250be thru 0x22ccc2 (0x7c05 bytes) 000479[1] 1790773361.151009 mapwrite 0x8719f6 thru 0x87665f (0x4c6a bytes) 000480[1] 1790773361.267555 read 0x34c3cc thru 0x353e18 (0x7a4d bytes) 000482[0] 1790773361.313015 punch from 0x1d7784 to 0x1e647c (0xecf9 bytes) 000483[0] 1790773361.333736 mapwrite 0x4c90e5 thru 0x4d27ce (0x96ea bytes) 000486[0] 1790773361.420204 mapread 0x84f38d thru 0x850ce2 (0x1956 bytes) 000487[0] 1790773361.455587 punch from 0x5b2443 to 0x5b9b78 (0x7736 bytes) 000488[0] 1790773361.462621 read 0x2c7373 thru 0x2ced44 (0x79d2 bytes) 000491[0] 1790773361.526640 mapwrite 0x02eb15 thru 0x03644a (0x7936 bytes) 000492[1] 1790773361.631140 read 0x281a54 thru 0x2862e5 (0x4892 bytes) 000496[0] 1790773361.688064 mapwrite 0x272310 thru 0x273f51 (0x1c42 bytes) 000497[0] 1790773361.771148 mapwrite 0x5ee254 thru 0x5f5813 (0x75c0 bytes) 000499[1] 1790773361.889290 mapread 0x86ffb4 thru 0x87a9ec (0xaa39 bytes) 000500[0] 1790773361.978260 write 0x6a48bd thru 0x6acb95 (0x82d9 bytes) All operations completed A-OK! Chance of close/open is 1 in 50 Seed set to 3362 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000002[1] 1790773362.846784 write 0x1cc000 thru 0x1d6b96 (0xab97 bytes) 000003[1] 1790773362.921826 read 0x032000 thru 0x0400f2 (0xe0f3 bytes) 000005[1] 1790773362.987018 mapread 0x0d0000 thru 0x0da85c (0xa85d bytes) 000007[0] 1790773363.070857 write 0x2be000 thru 0x2c725f (0x9260 bytes) 000009[0] 1790773363.175111 mapread 0x016000 thru 0x01b2a9 (0x52aa bytes) 000011[1] 1790773363.206368 read 0x105000 thru 0x106885 (0x1886 bytes) 000012[0] 1790773363.303773 punch from 0x25784c to 0x25de75 (0x662a bytes) 000013[1] 1790773363.382001 write 0x1c1000 thru 0x1c8bd2 (0x7bd3 bytes) 000014[0] 1790773363.420908 mapread 0x052000 thru 0x060cb3 (0xecb4 bytes) 000015[0] 1790773363.483615 read 0x0be000 thru 0x0c3157 (0x5158 bytes) 000017[0] 1790773363.501437 mapwrite 0x16e000 thru 0x17d8e8 (0xf8e9 bytes) truncating to largest ever: 0x2d45eb 000018[0] 1790773363.632476 trunc from 0x2c7260 to 0x2d45ea (0xd38b bytes) 000019[0] 1790773363.658073 read 0x145000 thru 0x14c3d6 (0x73d7 bytes) 000020[1] 1790773363.731597 write 0x1d7000 thru 0x1d9519 (0x251a bytes) 000021[0] 1790773363.770731 write 0x663000 thru 0x672efd (0xfefe bytes) 000023[0] 1790773363.837446 mapread 0x617000 thru 0x621e25 (0xae26 bytes) 000025[1] 1790773363.887028 mapwrite 0x3a3000 thru 0x3ac695 (0x9696 bytes) 000026[0] 1790773364.128481 read 0x3c3000 thru 0x3d248d (0xf48e bytes) truncating to largest ever: 0x75ac5f 000029[1] 1790773364.198732 trunc from 0x672efe to 0x75ac5e (0xe7d61 bytes) truncating to largest ever: 0x8e1cfe 000031[1] 1790773364.298275 trunc from 0x75ac5f to 0x8e1cfd (0x18709f bytes) 000032[0] 1790773364.369980 trunc from 0x8e1cfe to 0x0f921d (0x7e8ae0 bytes) 000033[1] 1790773364.440828 trunc from 0x0f921e to 0x2f32ef (0x1fa0d2 bytes) 000034[0] 1790773364.530432 trunc from 0x2f32f0 to 0x382527 (0x8f238 bytes) 000035[1] 1790773364.616147 mapread 00000000 thru 0x00b9b4 (0xb9b5 bytes) 000036[1] 1790773364.797560 write 0x260000 thru 0x263390 (0x3391 bytes) 000041[0] 1790773364.863573 punch from 0x1eb9f7 to 0x1fa95a (0xef64 bytes) 000042[0] 1790773364.918152 read 0x264000 thru 0x2672a6 (0x32a7 bytes) 000043[0] 1790773365.049078 mapwrite 0x232000 thru 0x23ea1e (0xca1f bytes) 000049[1] 1790773365.299091 write 0x8c6000 thru 0x8cd6a3 (0x76a4 bytes) 000050[1] 1790773365.321469 punch from 0x3b47f7 to 0x3c413b (0xf945 bytes) 000051[0] 1790773365.336204 write 0x8a2000 thru 0x8b1fb3 (0xffb4 bytes) 000052[1] 1790773365.403220 read 0x710000 thru 0x71bee7 (0xbee8 bytes) 000054[0] 1790773365.472123 write 0x76d000 thru 0x773919 (0x691a bytes) 000055[0] 1790773365.499222 mapwrite 0x42c000 thru 0x43a15b (0xe15c bytes) 000056[0] 1790773365.619233 mapwrite 0x975000 thru 0x97acd4 (0x5cd5 bytes) 000057[1] 1790773365.733230 mapwrite 0x664000 thru 0x66b5cb (0x75cc bytes) 000059[0] 1790773365.826836 mapwrite 0x8fb000 thru 0x9086cc (0xd6cd bytes) 000060[1] 1790773365.997666 punch from 0x2dd7ee to 0x2e6dee (0x9601 bytes) 000061[0] 1790773366.034813 mapread 0x030000 thru 0x03573b (0x573c bytes) 000062[0] 1790773366.087396 read 0x234000 thru 0x23a158 (0x6159 bytes) 000063[1] 1790773366.094805 write 0x880000 thru 0x881119 (0x111a bytes) 000064[1] 1790773366.114413 mapwrite 0x055000 thru 0x058a13 (0x3a14 bytes) 000065[0] 1790773366.177582 mapwrite 0x182000 thru 0x184704 (0x2705 bytes) 000066[0] 1790773366.251536 mapread 0x7bd000 thru 0x7bf93b (0x293c bytes) 000067[1] 1790773366.282445 mapwrite 0x4b4000 thru 0x4b84fb (0x44fc bytes) 000068[1] 1790773366.400688 punch from 0x3a81e9 to 0x3aac50 (0x2a68 bytes) 000069[1] 1790773366.441224 read 0x4d6000 thru 0x4e3e42 (0xde43 bytes) truncating to largest ever: 0x97f976 000074[0] 1790773366.532198 trunc from 0x97acd5 to 0x97f975 (0x4ca1 bytes) 000078[1] 1790773366.621633 mapwrite 0x936000 thru 0x94061c (0xa61d bytes) 000079[1] 1790773366.736290 write 0x2b9000 thru 0x2c21f7 (0x91f8 bytes) 000080[0] 1790773366.761862 read 0x76b000 thru 0x7777e9 (0xc7ea bytes) 000081[0] 1790773366.798126 write 0x5a6000 thru 0x5a75df (0x15e0 bytes) 000082[0] 1790773366.822559 punch from 0x48e583 to 0x4975bf (0x903d bytes) 000083[0] 1790773366.830003 trunc from 0x97f976 to 0x35a259 (0x62571c bytes) 000085[1] 1790773366.851658 read 0x2e2000 thru 0x2e48d2 (0x28d3 bytes) 000086[0] 1790773366.877892 mapread 0x302000 thru 0x30b6a4 (0x96a5 bytes) 000087[0] 1790773366.903504 mapread 0x349000 thru 0x3505ec (0x75ed bytes) 000091[1] 1790773366.921132 write 0x3fe000 thru 0x4034b5 (0x54b6 bytes) 000092[1] 1790773366.967958 read 0x051000 thru 0x057b7e (0x6b7f bytes) 000093[1] 1790773367.022729 trunc from 0x4034b6 to 0x7a39e6 (0x3a0531 bytes) 000094[1] 1790773367.051812 trunc from 0x7a39e7 to 0x534354 (0x26f692 bytes) 000095[0] 1790773367.087592 write 0x0bf000 thru 0x0bf08e (0x8f bytes) 000097[0] 1790773367.143181 read 0x162000 thru 0x1702a2 (0xe2a3 bytes) 000098[0] 1790773367.162145 mapread 0x089000 thru 0x097fc2 (0xefc3 bytes) 000099[0] 1790773367.204576 mapwrite 0x251000 thru 0x25ba02 (0xaa03 bytes) 000100[0] 1790773367.307311 trunc from 0x534355 to 0x8a9624 (0x3752d0 bytes) 000101[0] 1790773367.343012 mapread 0x3e9000 thru 0x3f88b2 (0xf8b3 bytes) 000102[0] 1790773367.391247 mapwrite 0x363000 thru 0x36568d (0x268e bytes) 000103[0] 1790773367.452183 punch from 0x2e8afd to 0x2f7ba5 (0xf0a9 bytes) 000104[1] 1790773367.502368 mapwrite 0x962000 thru 0x9656ac (0x36ad bytes) 000105[0] 1790773367.730249 trunc from 0x9656ad to 0x167a32 (0x7fdc7a bytes) 000108[0] 1790773367.823358 read 0x148000 thru 0x1482d8 (0x2d9 bytes) 000109[1] 1790773367.907215 mapread 0x0f6000 thru 0x1014a9 (0xb4aa bytes) 000110[1] 1790773367.978694 read 0x04c000 thru 0x04d4fa (0x14fb bytes) 000111[1] 1790773368.015436 mapread 0x12e000 thru 0x13058f (0x2590 bytes) 000112[1] 1790773368.029274 trunc from 0x167a33 to 0x311395 (0x1a9963 bytes) 000113[1] 1790773368.098094 read 0x1c9000 thru 0x1d0be1 (0x7be2 bytes) 000114[1] 1790773368.181091 punch from 0x00677e to 0x00cf16 (0x6799 bytes) 000116[0] 1790773368.211537 punch from 0x0f6c5b to 0x0f7376 (0x71c bytes) 000117[1] 1790773368.262006 trunc from 0x311396 to 0x5b436b (0x2a2fd6 bytes) 000118[1] 1790773368.364214 mapread 0x30a000 thru 0x30a9b9 (0x9ba bytes) 000119[0] 1790773368.426908 write 0x7e5000 thru 0x7f48aa (0xf8ab bytes) 000123[1] 1790773368.508157 mapread 0x2e2000 thru 0x2e91f4 (0x71f5 bytes) 000124[1] 1790773368.636087 trunc from 0x7f48ab to 0x5196ca (0x2db1e0 bytes) 000126[0] 1790773368.716637 trunc from 0x5196cb to 0x534cae (0x1b5e4 bytes) 000128[0] 1790773368.791644 mapread 0x226000 thru 0x2294dc (0x34dd bytes) 000129[0] 1790773368.897396 mapread 0x495000 thru 0x4a3745 (0xe746 bytes) 000133[1] 1790773368.951593 read 0x24a000 thru 0x251d07 (0x7d08 bytes) 000135[1] 1790773368.992457 write 0x026000 thru 0x032106 (0xc107 bytes) 000136[1] 1790773369.031960 write 0x806000 thru 0x80da60 (0x7a61 bytes) 000138[0] 1790773369.038097 mapwrite 0x65a000 thru 0x660a70 (0x6a71 bytes) 000139[1] 1790773369.187936 mapread 0x64c000 thru 0x653777 (0x7778 bytes) 000140[0] 1790773369.267163 trunc from 0x80da61 to 0x329322 (0x4e473e bytes) 000141[0] 1790773369.312415 mapread 0x1c5000 thru 0x1c7ec5 (0x2ec6 bytes) 000142[1] 1790773369.335274 read 0x224000 thru 0x22889a (0x489b bytes) 000146[1] 1790773369.394402 mapwrite 0x09c000 thru 0x0abf74 (0xff75 bytes) 000147[1] 1790773369.557687 mapwrite 0x098000 thru 0x0a21dd (0xa1de bytes) 000148[0] 1790773369.655227 read 0x239000 thru 0x24152f (0x8530 bytes) 000149[1] 1790773369.748756 punch from 0x2d5e68 to 0x2d8944 (0x2add bytes) 000151[0] 1790773369.791117 punch from 0x037066 to 0x03de79 (0x6e14 bytes) 000152[1] 1790773369.810423 mapread 0x29a000 thru 0x29a6bd (0x6be bytes) 000154[0] 1790773369.838569 mapread 0x117000 thru 0x11f8e3 (0x88e4 bytes) 000155[1] 1790773369.882464 punch from 0x1cf736 to 0x1d3757 (0x4022 bytes) 000156[0] 1790773369.913788 write 0x98b000 thru 0x99788e (0xc88f bytes) 000157[0] 1790773369.956833 mapwrite 0x6a2000 thru 0x6abfb3 (0x9fb4 bytes) 000159[0] 1790773370.144434 write 0x839000 thru 0x846683 (0xd684 bytes) 000160[1] 1790773370.182827 write 0x5e0000 thru 0x5e8c7c (0x8c7d bytes) 000161[0] 1790773370.251959 trunc from 0x99788f to 0x4c4c4b (0x4d2c43 bytes) 000162[1] 1790773370.318834 write 0x10f000 thru 0x1106d5 (0x16d6 bytes) 000164[1] 1790773370.366185 mapread 0x0d1000 thru 0x0e0647 (0xf648 bytes) 000165[1] 1790773370.398617 read 0x465000 thru 0x47188f (0xc890 bytes) 000170[0] 1790773370.441580 read 0x3df000 thru 0x3e1e63 (0x2e64 bytes) 000171[0] 1790773370.467528 punch from 0x3bf3c2 to 0x3c7859 (0x8498 bytes) 000173[1] 1790773370.502473 mapwrite 0x49a000 thru 0x4a4dd4 (0xadd5 bytes) 000174[1] 1790773370.601905 mapread 0x44c000 thru 0x44e178 (0x2179 bytes) 000175[1] 1790773370.613648 write 0x58f000 thru 0x58f0d0 (0xd1 bytes) 000176[1] 1790773370.629014 mapwrite 0x11e000 thru 0x120a8e (0x2a8f bytes) 000177[1] 1790773370.696488 mapwrite 0x8d5000 thru 0x8d8e41 (0x3e42 bytes) 000182[1] 1790773370.818795 write 0x4ef000 thru 0x4fe9e6 (0xf9e7 bytes) 000183[1] 1790773370.825931 mapwrite 0x756000 thru 0x764a44 (0xea45 bytes) 000184[0] 1790773370.913399 trunc from 0x8d8e42 to 0x87c1e7 (0x5cc5a bytes) 000185[0] 1790773370.968129 mapwrite 0x6d7000 thru 0x6db5bf (0x45c0 bytes) 000187[1] 1790773371.110642 read 0x454000 thru 0x46241f (0xe420 bytes) 000188[0] 1790773371.130947 trunc from 0x87c1e8 to 0x25992c (0x6228bb bytes) 000189[1] 1790773371.187140 punch from 0x07d5c2 to 0x08d0ec (0xfb2b bytes) 000191[0] 1790773371.222161 read 0x0cb000 thru 0x0cfb65 (0x4b66 bytes) 000192[0] 1790773371.284905 trunc from 0x25992d to 0x4c3513 (0x269be7 bytes) 000193[0] 1790773371.333958 mapwrite 0x7bf000 thru 0x7cbac1 (0xcac2 bytes) 000195[1] 1790773371.528494 mapwrite 0x142000 thru 0x148792 (0x6793 bytes) 000196[1] 1790773371.684687 mapread 0x289000 thru 0x29243e (0x943f bytes) 000199[1] 1790773371.748797 mapread 0x319000 thru 0x31c439 (0x343a bytes) 000201[1] 1790773371.790750 write 0x94d000 thru 0x957c38 (0xac39 bytes) 000202[1] 1790773371.816127 trunc from 0x957c39 to 0x44817c (0x50fabc bytes) 000203[0] 1790773371.841208 mapwrite 0x9d1000 thru 0x9de028 (0xd029 bytes) 000204[0] 1790773371.968474 punch from 0x13749d to 0x145b7b (0xe6df bytes) 000205[0] 1790773371.986252 mapwrite 0x7b4000 thru 0x7c1b77 (0xdb78 bytes) 000207[0] 1790773372.121408 read 0x4fc000 thru 0x50758a (0xb58b bytes) 000208[0] 1790773372.149654 mapwrite 0x4e1000 thru 0x4e3b22 (0x2b23 bytes) 000209[0] 1790773372.247440 write 0x3da000 thru 0x3dc84f (0x2850 bytes) 000211[0] 1790773372.293663 write 0x79f000 thru 0x7aef34 (0xff35 bytes) 000214[0] 1790773372.333998 write 0x0a7000 thru 0x0b14b6 (0xa4b7 bytes) 000220[0] 1790773372.417849 trunc from 0x9de029 to 0x33807c (0x6a5fac bytes) 000221[1] 1790773372.462576 punch from 0x307c43 to 0x31683d (0xebfb bytes) 000222[0] 1790773372.507107 mapwrite 0x6b5000 thru 0x6b7d38 (0x2d39 bytes) 000225[1] 1790773372.650262 read_OD 0x53a000 thru 0x544fff (0xb000 bytes) 000227[0] 1790773372.701463 punch from 0x076783 to 0x079b20 (0x339e bytes) 000234[0] 1790773372.770223 trunc from 0x6b7d39 to 0x207a88 (0x4b02b0 bytes) 000235[1] 1790773372.806233 mapread 0x004000 thru 0x00cade (0x8adf bytes) 000236[0] 1790773372.856304 mapwrite 0x6ed000 thru 0x6fc3f4 (0xf3f5 bytes) 000237[1] 1790773372.996669 punch from 0x36fe16 to 0x37fc4b (0xfe36 bytes) 000238[1] 1790773373.044086 write_OD 0x7fd000 thru 0x802fff (0x6000 bytes) 000240[1] 1790773373.139196 mapwrite 0x32a000 thru 0x339baf (0xfbb0 bytes) 000241[1] 1790773373.314288 punch from 0x46c5d3 to 0x479f7f (0xd9ad bytes) 000243[0] 1790773373.386024 read 0x45c000 thru 0x465db5 (0x9db6 bytes) 000245[1] 1790773373.433807 read_OD 0x533000 thru 0x53ffff (0xd000 bytes) 000250[1] 1790773373.460852 mapread 0x3c2000 thru 0x3c8942 (0x6943 bytes) 000254[1] 1790773373.490501 write_OD 0x18d000 thru 0x195fff (0x9000 bytes) 000255[1] 1790773373.546450 punch from 0x238445 to 0x238e20 (0x9dc bytes) 000256[0] 1790773373.575177 mapread 0x07a000 thru 0x082545 (0x8546 bytes) 000259[1] 1790773373.632545 read_OD 0x44a000 thru 0x44bfff (0x2000 bytes) 000262[1] 1790773373.644286 trunc from 0x803000 to 0x3d5643 (0x42d9bc bytes) 000264[0] 1790773373.706094 mapread 0x072000 thru 0x07dd5e (0xbd5f bytes) 000266[1] 1790773373.781546 mapwrite 0x440000 thru 0x449b78 (0x9b79 bytes) 000269[0] 1790773373.880947 write 0x03f000 thru 0x042704 (0x3705 bytes) 000270[1] 1790773373.912267 mapread 0x10e000 thru 0x10ee6c (0xe6d bytes) 000271[1] 1790773373.956765 trunc from 0x449b79 to 0x41494b (0x3522d bytes) 000272[1] 1790773374.013110 trunc from 0x41494c to 0x83944b (0x424b00 bytes) 000274[1] 1790773374.059782 punch from 0x42ec21 to 0x4328e8 (0x3cc8 bytes) 000277[0] 1790773374.083136 trunc from 0x83944c to 0x8762cc (0x3ce81 bytes) 000278[1] 1790773374.118055 punch from 0x67d6cc to 0x68b2e7 (0xdc1c bytes) 000279[1] 1790773374.139240 punch from 0x82fb38 to 0x836692 (0x6b5b bytes) 000281[0] 1790773374.161277 trunc from 0x8762cd to 0x8ecdbf (0x76af3 bytes) 000284[1] 1790773374.230300 write_OD 0x8d8000 thru 0x8e0fff (0x9000 bytes) 000285[1] 1790773374.277544 write_OD 0x59c000 thru 0x5a2fff (0x7000 bytes) 000287[0] 1790773374.329946 write 0x5ce000 thru 0x5dd8fc (0xf8fd bytes) 000289[1] 1790773374.399452 mapread 0x747000 thru 0x74be29 (0x4e2a bytes) 000291[1] 1790773374.489462 mapread 0x091000 thru 0x0a02a4 (0xf2a5 bytes) 000293[0] 1790773374.528082 trunc from 0x8ecdc0 to 0x41d91e (0x4cf4a1 bytes) 000295[1] 1790773374.604947 write_OD 0x94b000 thru 0x953fff (0x9000 bytes) 000297[1] 1790773374.704305 write_OD 0x37b000 thru 0x388fff (0xe000 bytes) 000299[1] 1790773374.843297 read_OD 0x3c2000 thru 0x3c2fff (0x1000 bytes) 000304[0] 1790773374.853286 mapread 0x34b000 thru 0x350f62 (0x5f63 bytes) 000305[1] 1790773374.894451 read_OD 0x0a0000 thru 0x0a0fff (0x1000 bytes) 000308[0] 1790773374.899987 write 0x914000 thru 0x91fcd0 (0xbcd1 bytes) 000309[1] 1790773374.937057 mapread 0x481000 thru 0x48dc66 (0xcc67 bytes) 000311[1] 1790773375.042422 write_OD 0x443000 thru 0x44cfff (0xa000 bytes) 000313[1] 1790773375.146058 mapread 0x168000 thru 0x169b0b (0x1b0c bytes) 000316[1] 1790773375.169439 write_OD 0x564000 thru 0x564fff (0x1000 bytes) 000317[0] 1790773375.244378 mapwrite 0x88e000 thru 0x896837 (0x8838 bytes) 000318[1] 1790773375.426048 trunc from 0x954000 to 0x5a34ca (0x3b0b35 bytes) 000319[1] 1790773375.509174 read_OD 0x03e000 thru 0x045fff (0x8000 bytes) 000320[1] 1790773375.559436 trunc from 0x5a34cb to 0x570834 (0x32c96 bytes) 000325[0] 1790773375.711227 mapread 0x367000 thru 0x36ec6b (0x7c6c bytes) 000328[0] 1790773375.774737 write 0x752000 thru 0x753ab8 (0x1ab9 bytes) 000329[0] 1790773375.808894 punch from 0x16f967 to 0x174105 (0x479f bytes) 000331[1] 1790773375.830390 write_OD 0x16e000 thru 0x17bfff (0xe000 bytes) 000333[1] 1790773375.921921 mapread 0x296000 thru 0x2a4b6e (0xeb6f bytes) 000335[0] 1790773375.976510 mapread 0x733000 thru 0x73a743 (0x7744 bytes) 000337[1] 1790773376.029864 punch from 0x0b2e1e to 0x0c297e (0xfb61 bytes) 000338[1] 1790773376.066290 read_OD 0x6a8000 thru 0x6b1fff (0xa000 bytes) truncating to largest ever: 0x99228b 000339[0] 1790773376.100830 trunc from 0x753ab9 to 0x99228a (0x23e7d2 bytes) 000340[0] 1790773376.136892 punch from 0x1cdae0 to 0x1da9fb (0xcf1c bytes) 000341[1] 1790773376.142274 write_OD 0x26c000 thru 0x273fff (0x8000 bytes) 000344[0] 1790773376.228563 mapwrite 0x5a6000 thru 0x5a8292 (0x2293 bytes) 000346[1] 1790773376.347463 mapread 0x635000 thru 0x63f67f (0xa680 bytes) 000349[1] 1790773376.433483 trunc from 0x99228b to 0x390444 (0x601e46 bytes) 000351[1] 1790773376.515055 trunc from 0x390445 to 0x80fbeb (0x47f7a7 bytes) 000352[1] 1790773376.566836 trunc from 0x80fbec to 0x04c8fc (0x7c32ef bytes) 000353[0] 1790773376.630835 trunc from 0x04c8fd to 0x399092 (0x34c796 bytes) 000356[1] 1790773376.721238 trunc from 0x399093 to 0x6256b1 (0x28c61f bytes) 000357[1] 1790773376.758902 mapread 0x3c1000 thru 0x3cd1fa (0xc1fb bytes) 000359[1] 1790773376.970269 mapwrite 0x466000 thru 0x46fe80 (0x9e81 bytes) 000362[1] 1790773377.254899 read_OD 0x3af000 thru 0x3b5fff (0x7000 bytes) 000363[1] 1790773377.287112 write_OD 0x852000 thru 0x858fff (0x7000 bytes) 000365[1] 1790773377.334269 write_OD 0x2a7000 thru 0x2b1fff (0xb000 bytes) 000367[1] 1790773377.432534 mapread 0x6c2000 thru 0x6c4ee8 (0x2ee9 bytes) 000368[0] 1790773377.488084 mapread 0x68e000 thru 0x69d46c (0xf46d bytes) 000369[1] 1790773377.529581 read_OD 0x6fd000 thru 0x708fff (0xc000 bytes) 000370[1] 1790773377.547527 write_OD 0x2e1000 thru 0x2eafff (0xa000 bytes) 000372[0] 1790773377.669340 punch from 0x0420db to 0x043714 (0x163a bytes) 000373[1] 1790773377.704367 read_OD 0x02d000 thru 0x03afff (0xe000 bytes) 000377[0] 1790773377.732161 read 0x2f0000 thru 0x2feab0 (0xeab1 bytes) 000378[0] 1790773377.759792 trunc from 0x859000 to 0x390e8d (0x4c8172 bytes) 000379[1] 1790773377.805450 punch from 0x073fce to 0x0781c7 (0x41fa bytes) 000380[1] 1790773377.819476 read_OD 0x24f000 thru 0x256fff (0x8000 bytes) 000381[0] 1790773377.838873 mapread 0x2be000 thru 0x2c4669 (0x666a bytes) 000382[0] 1790773377.864899 write 0x446000 thru 0x44ea8a (0x8a8b bytes) 000386[1] 1790773377.880026 mapwrite 0x282000 thru 0x29082d (0xe82e bytes) 000389[0] 1790773378.058884 read 0x3c8000 thru 0x3c8f46 (0xf47 bytes) 000390[0] 1790773378.086009 punch from 0x0460f8 to 0x04705a (0xf63 bytes) 000392[1] 1790773378.127007 write_OD 0x5b2000 thru 0x5bafff (0x9000 bytes) 000393[1] 1790773378.179122 mapread 0x1a9000 thru 0x1b4e09 (0xbe0a bytes) 000394[0] 1790773378.231433 punch from 0x30d461 to 0x318e33 (0xb9d3 bytes) 000395[0] 1790773378.252895 punch from 0x474354 to 0x483ac5 (0xf772 bytes) 000397[1] 1790773378.271315 trunc from 0x5bb000 to 0x6001d1 (0x451d2 bytes) 000400[0] 1790773378.337262 mapwrite 0x8cf000 thru 0x8d0291 (0x1292 bytes) 000401[1] 1790773378.434431 punch from 0x1ee0fa to 0x1fa737 (0xc63e bytes) 000402[0] 1790773378.451465 mapwrite 0x168000 thru 0x172b53 (0xab54 bytes) 000403[1] 1790773378.530130 mapwrite 0x304000 thru 0x30602a (0x202b bytes) 000404[0] 1790773378.629130 mapread 0x557000 thru 0x558e90 (0x1e91 bytes) 000405[1] 1790773378.666368 write_OD 0x572000 thru 0x578fff (0x7000 bytes) 000406[0] 1790773378.724720 trunc from 0x8d0292 to 0x99074e (0xc04bd bytes) 000409[0] 1790773378.804585 mapwrite 0x6ac000 thru 0x6b5e2e (0x9e2f bytes) 000411[0] 1790773378.905477 punch from 0x86494d to 0x86b87a (0x6f2e bytes) 000413[0] 1790773378.933804 mapread 0x183000 thru 0x18de44 (0xae45 bytes) 000416[1] 1790773378.960501 trunc from 0x99074f to 0x424ebe (0x56b890 bytes) 000417[0] 1790773379.040166 punch from 0x195320 to 0x1a295c (0xd63d bytes) 000418[0] 1790773379.060724 punch from 0x15af01 to 0x166527 (0xb627 bytes) 000421[0] 1790773379.107261 trunc from 0x424ebf to 0x8ddd06 (0x4b8e48 bytes) 000422[0] 1790773379.148897 mapwrite 0x26b000 thru 0x271b77 (0x6b78 bytes) 000427[1] 1790773379.273862 trunc from 0x8ddd07 to 0x90e62c (0x30926 bytes) 000428[0] 1790773379.325279 write 0x3aa000 thru 0x3ae25a (0x425b bytes) 000429[0] 1790773379.349744 read 0x83e000 thru 0x84330e (0x530f bytes) 000430[1] 1790773379.376206 mapread 0x71f000 thru 0x7225e5 (0x35e6 bytes) 000431[0] 1790773379.478641 mapwrite 0x682000 thru 0x68df5f (0xbf60 bytes) 000432[0] 1790773379.579231 punch from 0x801c3a to 0x81124d (0xf614 bytes) 000436[1] 1790773379.603548 mapwrite 0x88e000 thru 0x8938d8 (0x58d9 bytes) 000437[0] 1790773379.681760 mapread 0x7ad000 thru 0x7b7bdc (0xabdd bytes) 000438[0] 1790773379.728158 punch from 0x6efbde to 0x6fd0a9 (0xd4cc bytes) 000441[0] 1790773379.747820 mapread 0x3c3000 thru 0x3c50bf (0x20c0 bytes) 000443[0] 1790773379.791196 trunc from 0x90e62d to 0x64d6dd (0x2c0f4f bytes) 000444[1] 1790773379.828082 read_OD 0x3bf000 thru 0x3c5fff (0x7000 bytes) 000446[0] 1790773379.859444 punch from 0x109b03 to 0x115c8c (0xc18a bytes) 000448[1] 1790773379.900975 write_OD 0x498000 thru 0x49afff (0x3000 bytes) 000449[0] 1790773379.953051 punch from 0x4d2814 to 0x4d525e (0x2a4b bytes) 000450[1] 1790773379.981074 punch from 0x3ac0f5 to 0x3affe1 (0x3eed bytes) 000452[1] 1790773380.053711 write_OD 0x1ee000 thru 0x1effff (0x2000 bytes) 000454[0] 1790773380.127751 punch from 0x4c7c5f to 0x4ccb98 (0x4f3a bytes) 000455[1] 1790773380.156694 read_OD 0x314000 thru 0x320fff (0xd000 bytes) 000457[1] 1790773380.192006 mapwrite 0x363000 thru 0x36cdfc (0x9dfd bytes) 000460[0] 1790773380.363147 punch from 0x103cf2 to 0x1089dc (0x4ceb bytes) 000462[0] 1790773380.391548 punch from 0x421e60 to 0x4268aa (0x4a4b bytes) 000464[1] 1790773380.396709 mapread 0x279000 thru 0x27dae6 (0x4ae7 bytes) 000466[0] 1790773380.430757 read 0x543000 thru 0x54c756 (0x9757 bytes) 000468[1] 1790773380.441022 write_OD 0x3d4000 thru 0x3d9fff (0x6000 bytes) 000469[1] 1790773380.487448 punch from 0x49e416 to 0x4ad9af (0xf59a bytes) 000473[0] 1790773380.520153 write 0x2ff000 thru 0x30a544 (0xb545 bytes) 000474[1] 1790773380.566862 mapread 0x30b000 thru 0x3187cd (0xd7ce bytes) 000475[1] 1790773380.638950 read_OD 0x0d0000 thru 0x0dbfff (0xc000 bytes) 000476[1] 1790773380.654741 write_OD 0x1eb000 thru 0x1ebfff (0x1000 bytes) 000478[0] 1790773380.711188 read 0x1f8000 thru 0x205c53 (0xdc54 bytes) 000480[1] 1790773380.725077 read_OD 0x060000 thru 0x068fff (0x9000 bytes) 000481[1] 1790773380.734176 read_OD 0x02b000 thru 0x02ffff (0x5000 bytes) 000483[1] 1790773380.742933 read_OD 0x088000 thru 0x090fff (0x9000 bytes) 000484[1] 1790773380.752111 mapwrite 0x207000 thru 0x213749 (0xc74a bytes) 000485[0] 1790773380.898736 trunc from 0x64d6de to 0x4ad439 (0x1a02a4 bytes) 000488[0] 1790773381.005942 read 0x349000 thru 0x35541b (0xc41c bytes) 000491[1] 1790773381.052641 punch from 0x355b51 to 0x357ffe (0x24ae bytes) 000493[0] 1790773381.089884 trunc from 0x4ad43a to 0x24a9c4 (0x262a75 bytes) 000494[1] 1790773381.167812 punch from 0x1c03d0 to 0x1cee7f (0xeab0 bytes) 000498[1] 1790773381.232658 punch from 0x0a4d26 to 0x0ab9de (0x6cb9 bytes) All operations completed A-OK! PASS 16a (52s) == sanityn test 16b: 500 iterations of dual-mount fsx at small size ========================================================== 09:03:07 (1790773387) keep default fallocate mode: 2 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.363206 s, 23.1 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.283712 s, 29.6 MB/s Chance of close/open is 1 in 50 Seed set to 3390 fd 0: /mnt/lustre/f16b.sanityn fd 1: /mnt/lustre2/f16b.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000001[1] 1790773391.750131 write 0x1f9be5 thru 0x1fb3fa (0x1816 bytes) 000003[0] 1790773391.762855 mapread 0x17cf9a thru 0x17d853 (0x8ba bytes) 000004[0] 1790773391.815232 mapread 0x1c1513 thru 0x1c2ab3 (0x15a1 bytes) 000005[0] 1790773391.832071 punch from 0x1d9c0e to 0x1daabf (0xeb2 bytes) 000006[1] 1790773391.867882 mapread 0x16c478 thru 0x16de07 (0x1990 bytes) truncating to largest ever: 0x211f34 000010[0] 1790773391.957312 trunc from 0x1fb3fb to 0x211f33 (0x16b39 bytes) 000013[0] 1790773392.085693 write 0x6ac641 thru 0x6ac8a8 (0x268 bytes) 000014[1] 1790773392.105039 mapread 0x652b3c thru 0x65338d (0x852 bytes) 000017[1] 1790773392.150918 punch from 0x0e5ef5 to 0x0e6e79 (0xf85 bytes) 000018[0] 1790773392.178954 mapwrite 0x382fb5 thru 0x383a02 (0xa4e bytes) 000019[0] 1790773392.286373 punch from 0x5abdd7 to 0x5ad71e (0x1948 bytes) truncating to largest ever: 0x4db597 000021[0] 1790773392.331720 trunc from 0x6ac8a9 to 0x4db596 (0x1d1312 bytes) 000022[1] 1790773392.352119 punch from 0x2adde1 to 0x2afbfb (0x1e1b bytes) 000023[1] 1790773392.389776 read 0x027937 thru 0x028341 (0xa0b bytes) 000024[0] 1790773392.412186 mapwrite 0x8847cf thru 0x885909 (0x113b bytes) truncating to largest ever: 0x909ad9 000027[1] 1790773392.574031 trunc from 0x88590a to 0x909ad8 (0x841cf bytes) 000028[0] 1790773392.660680 trunc from 0x909ad9 to 0x46e064 (0x49ba74 bytes) 000029[1] 1790773392.718057 punch from 0x1a0746 to 0x1a22a9 (0x1b64 bytes) 000030[0] 1790773392.750296 write 0x229d1d thru 0x22ad33 (0x1017 bytes) 000031[0] 1790773392.795764 mapwrite 0x5acfa9 thru 0x5ad4a2 (0x4fa bytes) 000033[0] 1790773392.942130 read 0x3d0b7a thru 0x3d1d71 (0x11f8 bytes) 000036[1] 1790773392.968394 mapwrite 0x2f7709 thru 0x2f8580 (0xe78 bytes) 000037[0] 1790773393.074273 read 0x3c8feb thru 0x3ca9e2 (0x19f8 bytes) 000038[0] 1790773393.126876 read 0x037c40 thru 0x037c63 (0x24 bytes) 000039[0] 1790773393.133206 punch from 0x3a8781 to 0x3a97a9 (0x1029 bytes) 000040[1] 1790773393.156457 write 0x52f8ca thru 0x530962 (0x1099 bytes) 000042[0] 1790773393.226224 trunc from 0x5ad4a3 to 0x6d4e27 (0x127985 bytes) 000044[0] 1790773393.352717 trunc from 0x6d4e28 to 0x08971d (0x64b70a bytes) 000049[0] 1790773393.456160 punch from 0x047205 to 0x048355 (0x1151 bytes) 000051[1] 1790773393.505074 mapwrite 0x3f5264 thru 0x3f5503 (0x2a0 bytes) 000052[0] 1790773393.645418 read 0x22740a thru 0x227fe9 (0xbe0 bytes) 000054[1] 1790773393.752903 mapwrite 0x91e405 thru 0x91e7a6 (0x3a2 bytes) 000056[0] 1790773393.897058 trunc from 0x91e7a7 to 0x890e61 (0x8d945 bytes) 000058[0] 1790773393.984864 mapread 0x77439b thru 0x77567b (0x12e1 bytes) 000061[1] 1790773394.117661 punch from 0x61f5ce to 0x6202b4 (0xce7 bytes) 000063[1] 1790773394.190596 trunc from 0x890e62 to 0x4d2b81 (0x3be2e0 bytes) 000064[1] 1790773394.221226 read 0x4490ef thru 0x44a267 (0x1179 bytes) 000065[0] 1790773394.263643 mapwrite 0x31b3f6 thru 0x31b568 (0x173 bytes) 000067[0] 1790773394.371995 punch from 0x236961 to 0x23718e (0x82e bytes) 000069[1] 1790773394.413065 trunc from 0x4d2b82 to 0x39b059 (0x137b28 bytes) 000071[1] 1790773394.451853 read 0x00d1e2 thru 0x00ddc7 (0xbe6 bytes) 000074[0] 1790773394.561082 mapwrite 0x25bfd6 thru 0x25cd00 (0xd2b bytes) 000076[0] 1790773394.677247 mapwrite 0x879da2 thru 0x87a6de (0x93d bytes) 000079[1] 1790773394.826902 mapwrite 0x077d1b thru 0x078065 (0x34b bytes) 000080[1] 1790773394.919106 write 0x59d915 thru 0x59dcb6 (0x3a2 bytes) 000081[1] 1790773395.019972 mapread 0x583d28 thru 0x5840a4 (0x37d bytes) 000082[1] 1790773395.023419 punch from 0x64b51c to 0x64ca8f (0x1574 bytes) 000084[0] 1790773395.051881 trunc from 0x87a6df to 0x7baece (0xbf810 bytes) 000087[0] 1790773395.115042 mapread 0x5cc3fe thru 0x5cd49f (0x10a2 bytes) 000089[0] 1790773395.126037 mapwrite 0x8bc335 thru 0x8bc7d6 (0x4a2 bytes) 000091[1] 1790773395.199802 write 0x1ea45b thru 0x1ea511 (0xb7 bytes) 000092[1] 1790773395.231100 mapread 0x6560af thru 0x656a4a (0x99c bytes) 000093[1] 1790773395.259126 punch from 0x523593 to 0x523bc0 (0x62e bytes) 000095[0] 1790773395.329907 mapwrite 0x200031 thru 0x200f61 (0xf31 bytes) 000097[1] 1790773395.429187 punch from 0x8551c7 to 0x856d7f (0x1bb9 bytes) 000098[1] 1790773395.465039 trunc from 0x8bc7d7 to 0x1056d5 (0x7b7101 bytes) 000099[1] 1790773395.507197 write 0x737b92 thru 0x7391fc (0x166b bytes) 000101[1] 1790773395.605037 trunc from 0x7391fd to 0x749342 (0x10146 bytes) 000102[1] 1790773395.630554 trunc from 0x749343 to 0x23970f (0x50fc33 bytes) 000103[1] 1790773395.669147 mapread 0x170258 thru 0x170ec0 (0xc69 bytes) 000104[1] 1790773395.716700 trunc from 0x239710 to 0x382699 (0x148f8a bytes) 000105[0] 1790773395.740651 mapread 0x2845f3 thru 0x285583 (0xf91 bytes) 000106[1] 1790773395.811927 mapwrite 0x5c33ff thru 0x5c4249 (0xe4b bytes) 000109[0] 1790773395.943958 write 0x7940db thru 0x794af8 (0xa1e bytes) 000110[0] 1790773395.975656 mapwrite 0x6a9549 thru 0x6aa2fb (0xdb3 bytes) 000112[1] 1790773396.083100 write 0x3ec12b thru 0x3edb0d (0x19e3 bytes) 000114[0] 1790773396.135777 mapwrite 0x2b23c1 thru 0x2b3ad9 (0x1719 bytes) 000118[0] 1790773396.287003 read 0x300613 thru 0x30246a (0x1e58 bytes) 000121[1] 1790773396.351613 mapwrite 0x2c9504 thru 0x2ca205 (0xd02 bytes) 000123[0] 1790773396.459929 mapread 0x011bb7 thru 0x012ada (0xf24 bytes) 000125[0] 1790773396.512093 trunc from 0x794af9 to 0x122a61 (0x672097 bytes) 000126[0] 1790773396.555887 mapwrite 0x54efab thru 0x550492 (0x14e8 bytes) 000129[1] 1790773396.650568 punch from 0x177192 to 0x1775b6 (0x425 bytes) 000132[1] 1790773396.723109 read 0x205704 thru 0x20738e (0x1c8b bytes) truncating to largest ever: 0x99cdbf 000133[1] 1790773396.747765 trunc from 0x550493 to 0x99cdbe (0x44c92c bytes) 000137[1] 1790773396.826271 mapwrite 0x685752 thru 0x6865aa (0xe59 bytes) 000138[1] 1790773396.923090 mapwrite 0x525a59 thru 0x5262bf (0x867 bytes) 000139[0] 1790773397.029472 write 0x6af83a thru 0x6b094f (0x1116 bytes) 000140[1] 1790773397.127135 mapwrite 0x4baf0a thru 0x4bc8aa (0x19a1 bytes) 000141[0] 1790773397.276612 write 0x80c095 thru 0x80d762 (0x16ce bytes) 000142[1] 1790773397.336077 mapwrite 0x53e7b3 thru 0x53e876 (0xc4 bytes) 000143[1] 1790773397.425699 read 0x0b5348 thru 0x0b6b5b (0x1814 bytes) 000144[0] 1790773397.460244 mapwrite 0x343b36 thru 0x345080 (0x154b bytes) 000145[1] 1790773397.613185 mapwrite 0x662d3c thru 0x6630c0 (0x385 bytes) 000147[0] 1790773397.740910 mapwrite 0x61d19e thru 0x61d857 (0x6ba bytes) 000148[0] 1790773397.939187 mapread 0x71b224 thru 0x71d1a2 (0x1f7f bytes) 000150[0] 1790773397.986164 mapread 0x7a70fb thru 0x7a7cb3 (0xbb9 bytes) 000152[1] 1790773398.000037 write 0x2ea577 thru 0x2ebe6f (0x18f9 bytes) 000153[0] 1790773398.054167 mapread 0x98317b thru 0x984493 (0x1319 bytes) 000155[1] 1790773398.121219 punch from 0x9773cb to 0x97881b (0x1451 bytes) 000156[0] 1790773398.164874 mapread 0x7f205f thru 0x7f3122 (0x10c4 bytes) 000157[1] 1790773398.206755 mapwrite 0x69b002 thru 0x69cfa7 (0x1fa6 bytes) 000158[0] 1790773398.306445 mapwrite 0x8dc2a6 thru 0x8dc5dc (0x337 bytes) 000159[0] 1790773398.392918 punch from 0x75aec5 to 0x75ca7b (0x1bb7 bytes) 000160[0] 1790773398.405850 trunc from 0x99cdbf to 0x1ecab8 (0x7b0306 bytes) 000164[0] 1790773398.484235 punch from 0x1b6a39 to 0x1b8778 (0x1d40 bytes) 000166[0] 1790773398.489453 mapwrite 0x8dc37b thru 0x8dc927 (0x5ad bytes) 000167[0] 1790773398.585046 trunc from 0x8dc928 to 0x583725 (0x359202 bytes) 000168[0] 1790773398.630691 mapwrite 0x3de288 thru 0x3df879 (0x15f2 bytes) 000169[1] 1790773398.745105 mapwrite 0x6af470 thru 0x6b123e (0x1dcf bytes) 000171[1] 1790773398.963923 write 0x83bb10 thru 0x83bf4c (0x43d bytes) 000172[0] 1790773398.967920 punch from 0x2a5991 to 0x2a70f6 (0x1766 bytes) 000173[1] 1790773399.015714 mapwrite 0x0bbc7d thru 0x0bcb37 (0xebb bytes) 000174[1] 1790773399.148521 trunc from 0x83bf4d to 0x6f65fd (0x14594f bytes) 000175[0] 1790773399.206202 mapread 0x43f9dd thru 0x440d8d (0x13b1 bytes) 000176[1] 1790773399.251703 write 0x6daeed thru 0x6dcb39 (0x1c4d bytes) 000178[1] 1790773399.324199 punch from 0x03a979 to 0x03bd18 (0x13a0 bytes) 000179[1] 1790773399.392705 mapread 0x4bbf7e thru 0x4bce34 (0xeb7 bytes) 000182[1] 1790773399.503760 write 0x47350a thru 0x4748fd (0x13f4 bytes) 000183[0] 1790773399.515396 trunc from 0x6f65fe to 0x319ec9 (0x3dc734 bytes) 000187[0] 1790773399.615105 write 0x8f9f36 thru 0x8fa4cb (0x596 bytes) 000189[0] 1790773399.646307 read 0x1ffee4 thru 0x20049f (0x5bc bytes) 000195[0] 1790773399.718232 punch from 0x169d10 to 0x16b1a7 (0x1498 bytes) 000196[1] 1790773399.724200 punch from 0x6b2505 to 0x6b39f1 (0x14ed bytes) 000198[1] 1790773399.773801 mapread 0x57500d thru 0x57663e (0x1632 bytes) 000199[0] 1790773399.853510 write 0x97cc37 thru 0x97d2fa (0x6c4 bytes) 000200[1] 1790773399.868022 read 0x37aea9 thru 0x37b49d (0x5f5 bytes) 000201[1] 1790773399.893059 write 0x851d7d thru 0x853b62 (0x1de6 bytes) 000205[1] 1790773400.007162 mapwrite 0x74f041 thru 0x750337 (0x12f7 bytes) 000206[0] 1790773400.085250 punch from 0x82aa0b to 0x82b0ca (0x6c0 bytes) 000207[1] 1790773400.192053 mapwrite 0x8db4f9 thru 0x8dcd06 (0x180e bytes) 000208[0] 1790773400.430702 trunc from 0x97d2fb to 0x7882a8 (0x1f5052 bytes) 000209[0] 1790773400.537788 mapread 0x0d429b thru 0x0d48dc (0x642 bytes) 000210[1] 1790773400.611194 trunc from 0x7882a9 to 0x720de0 (0x674c8 bytes) 000212[0] 1790773400.802171 read 0x475330 thru 0x4765a3 (0x1274 bytes) 000214[1] 1790773400.922234 mapread 0x2d1215 thru 0x2d1576 (0x362 bytes) 000215[0] 1790773400.943457 mapwrite 0x68321c thru 0x683386 (0x16b bytes) 000216[1] 1790773401.103268 mapwrite 0x0a2ef2 thru 0x0a3d5f (0xe6e bytes) 000218[0] 1790773401.209816 write 0x4dd626 thru 0x4de8f4 (0x12cf bytes) 000219[1] 1790773401.242654 trunc from 0x720de1 to 0x65e0b5 (0xc2d2b bytes) 000220[1] 1790773401.319215 punch from 0x26c31f to 0x26ccc7 (0x9a9 bytes) 000221[1] 1790773401.355268 mapwrite 0x4571c0 thru 0x4587d1 (0x1612 bytes) 000222[1] 1790773401.414492 mapread 0x65bf51 thru 0x65c56d (0x61d bytes) 000225[1] 1790773401.426678 trunc from 0x65e0b6 to 0x01434c (0x649d69 bytes) 000227[0] 1790773401.453417 write 0x7cbf7e thru 0x7cc46e (0x4f1 bytes) 000228[1] 1790773401.492047 punch from 0x1bce5c to 0x1bd1b4 (0x359 bytes) 000229[0] 1790773401.510480 mapread 0x3d31d4 thru 0x3d3ae0 (0x90d bytes) 000230[0] 1790773401.555922 punch from 0x283010 to 0x283fe2 (0xfd3 bytes) 000233[1] 1790773401.649224 mapwrite 0x584bc4 thru 0x585d99 (0x11d6 bytes) 000235[1] 1790773401.808623 write 0x98a9ce thru 0x98ab65 (0x198 bytes) 000239[1] 1790773401.847851 punch from 0x135d6b to 0x1374c8 (0x175e bytes) 000240[0] 1790773401.882144 read 0x74d5bd thru 0x74ee51 (0x1895 bytes) 000241[1] 1790773401.896951 punch from 0x42a610 to 0x42a7f6 (0x1e7 bytes) 000243[1] 1790773401.917981 write 0x6a907c thru 0x6a92b5 (0x23a bytes) 000244[0] 1790773401.948036 mapwrite 0x70a2da thru 0x70ae09 (0xb30 bytes) 000246[1] 1790773402.085704 mapwrite 0x407da6 thru 0x409cc0 (0x1f1b bytes) 000247[0] 1790773402.204584 write 0x65ea53 thru 0x66071e (0x1ccc bytes) 000248[1] 1790773402.246275 punch from 0x8582a2 to 0x859e4e (0x1bad bytes) 000251[0] 1790773402.293560 punch from 0x7e4da6 to 0x7e5e52 (0x10ad bytes) 000254[0] 1790773402.340874 mapread 0x1618eb thru 0x16320d (0x1923 bytes) 000255[0] 1790773402.390051 mapwrite 0x64e65b thru 0x64e7d5 (0x17b bytes) 000256[0] 1790773402.438850 mapwrite 0x345b05 thru 0x346111 (0x60d bytes) 000258[1] 1790773402.518935 read 0x9685b2 thru 0x969970 (0x13bf bytes) 000260[1] 1790773402.559949 mapwrite 0x5d1784 thru 0x5d3037 (0x18b4 bytes) 000262[1] 1790773402.661900 trunc from 0x98ab66 to 0x235db3 (0x754db2 bytes) 000265[1] 1790773402.751852 write 0x30f833 thru 0x30f971 (0x13f bytes) 000267[1] 1790773402.778533 mapread 0x0c74e9 thru 0x0c8d01 (0x1819 bytes) 000268[1] 1790773402.821546 write 0x2244eb thru 0x224650 (0x166 bytes) 000269[1] 1790773402.848463 mapwrite 0x4fc666 thru 0x4fe02a (0x19c5 bytes) 000271[1] 1790773403.022582 read 0x1d6fff thru 0x1d88e8 (0x18ea bytes) 000273[1] 1790773403.060087 mapread 0x35c24f thru 0x35d5c5 (0x1377 bytes) 000274[0] 1790773403.088993 punch from 0x32c2b2 to 0x32c34c (0x9b bytes) 000275[0] 1790773403.131434 read 0x32a69c thru 0x32ace9 (0x64e bytes) 000276[1] 1790773403.155036 mapread 0x0d20b1 thru 0x0d20b6 (0x6 bytes) 000277[1] 1790773403.205486 write 0x91b5d9 thru 0x91c8e8 (0x1310 bytes) 000281[0] 1790773403.299251 read 0x29fff4 thru 0x2a0d2c (0xd39 bytes) 000283[0] 1790773403.325634 mapread 0x55ed94 thru 0x55f66e (0x8db bytes) 000285[0] 1790773403.370167 read 0x48ec36 thru 0x48f656 (0xa21 bytes) 000286[0] 1790773403.376595 punch from 0x8fc8d1 to 0x8fd106 (0x836 bytes) 000287[0] 1790773403.406967 punch from 0x900db2 to 0x9014fa (0x749 bytes) 000290[0] 1790773403.477260 punch from 0x154c0b to 0x155a86 (0xe7c bytes) 000291[1] 1790773403.486131 mapwrite 0x75d394 thru 0x75d697 (0x304 bytes) 000293[0] 1790773403.645828 trunc from 0x91c8e9 to 0x42da02 (0x4eeee6 bytes) 000294[0] 1790773403.685241 read 0x2552e5 thru 0x25542e (0x14a bytes) 000295[1] 1790773403.728121 write 0x12119b thru 0x12214a (0xfb0 bytes) 000296[1] 1790773403.810377 read 0x3bb97e thru 0x3bc2a6 (0x929 bytes) 000297[0] 1790773403.837050 trunc from 0x42da03 to 0x8583ef (0x42a9ed bytes) 000298[1] 1790773403.883572 mapread 0x660449 thru 0x6623d9 (0x1f91 bytes) 000299[1] 1790773403.958074 mapread 0x42d312 thru 0x42de41 (0xb30 bytes) 000301[0] 1790773404.040997 read 0x614c16 thru 0x61565b (0xa46 bytes) 000303[1] 1790773404.134665 write 0x7a3fe1 thru 0x7a435c (0x37c bytes) 000304[0] 1790773404.220751 mapread 0x31f946 thru 0x320d44 (0x13ff bytes) 000306[1] 1790773404.316048 mapwrite 0x8c9923 thru 0x8c9ee5 (0x5c3 bytes) 000307[0] 1790773404.470623 write 0x8c1922 thru 0x8c1b22 (0x201 bytes) 000309[0] 1790773404.549227 trunc from 0x8c9ee6 to 0x118088 (0x7b1e5d bytes) 000310[1] 1790773404.612912 mapwrite 0x283615 thru 0x2839b8 (0x3a4 bytes) 000312[0] 1790773404.729200 trunc from 0x2839b9 to 0x688b6c (0x4051b4 bytes) 000315[0] 1790773404.895527 read 0x42fd20 thru 0x4316db (0x19bc bytes) 000317[1] 1790773404.930994 write 0x80bc98 thru 0x80d749 (0x1ab2 bytes) 000318[1] 1790773404.975190 mapread 0x697939 thru 0x698e1c (0x14e4 bytes) 000319[1] 1790773405.090063 punch from 0x094379 to 0x09535d (0xfe5 bytes) 000320[1] 1790773405.117755 write 0x845e3a thru 0x847c2c (0x1df3 bytes) 000322[0] 1790773405.135702 mapread 0x093f26 thru 0x095539 (0x1614 bytes) 000324[1] 1790773405.197737 mapwrite 0x804e80 thru 0x805452 (0x5d3 bytes) 000325[1] 1790773405.324572 mapwrite 0x2dc9e4 thru 0x2de331 (0x194e bytes) 000326[0] 1790773405.404873 punch from 0x1522c0 to 0x152ee9 (0xc2a bytes) 000328[1] 1790773405.426279 read 0x0ab131 thru 0x0ac5c9 (0x1499 bytes) 000330[1] 1790773405.445805 punch from 0x0ba913 to 0x0bc2d5 (0x19c3 bytes) 000332[1] 1790773405.481876 mapwrite 0x2bb024 thru 0x2bcf85 (0x1f62 bytes) 000335[1] 1790773405.550087 punch from 0x74f69e to 0x75157b (0x1ede bytes) 000336[1] 1790773405.587847 trunc from 0x847c2d to 0x16f706 (0x6d8526 bytes) 000337[0] 1790773405.612948 write 0x4ebfe1 thru 0x4ec47a (0x49a bytes) 000338[0] 1790773405.645752 read 0x4b82d5 thru 0x4b8b88 (0x8b4 bytes) 000339[0] 1790773405.675139 read 0x0b0ef4 thru 0x0b29e7 (0x1af4 bytes) 000340[0] 1790773405.699176 punch from 0x42099d to 0x422580 (0x1be4 bytes) 000341[0] 1790773405.709993 mapread 0x4b3c95 thru 0x4b4231 (0x59d bytes) 000343[0] 1790773405.749558 read 0x1cdb08 thru 0x1ce69c (0xb95 bytes) 000346[1] 1790773405.785864 trunc from 0x4ec47b to 0x4ebad4 (0x9a6 bytes) 000349[1] 1790773405.884135 mapwrite 0x792027 thru 0x79363a (0x1614 bytes) 000351[1] 1790773405.987774 mapwrite 0x8b193c thru 0x8b2ff6 (0x16bb bytes) 000353[0] 1790773406.111447 read 0x37149d thru 0x3726b2 (0x1216 bytes) 000354[0] 1790773406.149562 punch from 0x581a4a to 0x583983 (0x1f3a bytes) 000355[1] 1790773406.162495 trunc from 0x8b2ff7 to 0x0f08f7 (0x7c26ff bytes) 000356[1] 1790773406.201512 write 0x0e9108 thru 0x0eab7a (0x1a73 bytes) 000357[1] 1790773406.237021 trunc from 0x0f08f8 to 0x15ba16 (0x6b11f bytes) 000359[1] 1790773406.285735 mapread 0x127ff3 thru 0x12870b (0x719 bytes) 000360[1] 1790773406.326569 read 0x0720e8 thru 0x072671 (0x58a bytes) 000362[0] 1790773406.438916 mapwrite 0x7e0356 thru 0x7e05e6 (0x291 bytes) 000366[1] 1790773406.626846 mapwrite 0x95a45f thru 0x95b534 (0x10d6 bytes) 000367[1] 1790773406.918066 read 0x6d6370 thru 0x6d7140 (0xdd1 bytes) 000368[0] 1790773406.935447 write 0x4ed1cd thru 0x4ed991 (0x7c5 bytes) 000372[1] 1790773407.016599 read 0x56057a thru 0x56214e (0x1bd5 bytes) 000373[1] 1790773407.043625 mapread 0x8cc6ca thru 0x8cd1ef (0xb26 bytes) 000376[0] 1790773407.046180 write 0x62d62a thru 0x62d7b4 (0x18b bytes) 000377[1] 1790773407.073800 write 0x0cf918 thru 0x0d11b8 (0x18a1 bytes) 000378[0] 1790773407.092702 read 0x427ec4 thru 0x428d7f (0xebc bytes) 000379[0] 1790773407.116986 trunc from 0x95b535 to 0x073d3c (0x8e77f8 bytes) 000382[0] 1790773407.199191 punch from 0x05dcd4 to 0x05e7bb (0xae8 bytes) 000384[0] 1790773407.242764 mapread 0x023718 thru 0x024082 (0x96b bytes) 000385[0] 1790773407.261275 trunc from 0x073d3d to 0x3020f4 (0x28e3b8 bytes) 000388[0] 1790773407.320856 mapread 0x117fdc thru 0x11919d (0x11c2 bytes) 000390[0] 1790773407.331547 punch from 0x0450a5 to 0x0452b6 (0x212 bytes) 000391[1] 1790773407.336251 mapwrite 0x6a3cde thru 0x6a5849 (0x1b6c bytes) 000393[0] 1790773407.487213 read 0x34f866 thru 0x34fe4b (0x5e6 bytes) 000394[1] 1790773407.513634 mapread 0x6152c5 thru 0x615b54 (0x890 bytes) 000395[1] 1790773407.540166 mapwrite 0x80da92 thru 0x80e6eb (0xc5a bytes) 000397[1] 1790773407.682959 punch from 0x45258b to 0x45430e (0x1d84 bytes) 000398[1] 1790773407.687466 write 0x930a3e thru 0x931093 (0x656 bytes) 000399[0] 1790773407.721108 write 0x1d3f25 thru 0x1d4d0d (0xde9 bytes) 000400[1] 1790773407.767508 write 0x6e66db thru 0x6e71ec (0xb12 bytes) 000401[0] 1790773407.796175 read 0x7876fc thru 0x7881fd (0xb02 bytes) 000402[1] 1790773407.816527 mapread 0x19dab7 thru 0x19e54b (0xa95 bytes) 000403[0] 1790773407.853239 read 0x317dc1 thru 0x318d2d (0xf6d bytes) 000404[0] 1790773407.862850 write 0x0c95c2 thru 0x0ca777 (0x11b6 bytes) 000407[0] 1790773407.908804 write 0x33bfab thru 0x33c0b3 (0x109 bytes) 000409[1] 1790773407.937861 read 0x09438f thru 0x0945f9 (0x26b bytes) 000410[0] 1790773407.962130 mapwrite 0x9562e5 thru 0x957a62 (0x177e bytes) 000411[1] 1790773408.075233 trunc from 0x957a63 to 0x14d05d (0x80aa05 bytes) 000413[0] 1790773408.136613 mapread 0x1003c0 thru 0x10131f (0xf60 bytes) 000414[1] 1790773408.201586 trunc from 0x14d05e to 0x69fb9e (0x552b41 bytes) 000416[0] 1790773408.281160 punch from 0x61484a to 0x616609 (0x1dc0 bytes) 000417[0] 1790773408.330652 read 0x0ece4d thru 0x0edfe9 (0x119d bytes) 000420[0] 1790773408.366463 read 0x2e9a58 thru 0x2eb1e9 (0x1792 bytes) 000423[0] 1790773408.417457 mapread 0x6664cf thru 0x667578 (0x10aa bytes) 000424[0] 1790773408.430833 mapread 0x2c7824 thru 0x2c9391 (0x1b6e bytes) 000425[0] 1790773408.451222 read 0x27298d thru 0x273ebf (0x1533 bytes) 000426[1] 1790773408.458072 trunc from 0x69fb9f to 0x0df873 (0x5c032b bytes) 000427[0] 1790773408.511835 mapread 0x04a3b9 thru 0x04ba1f (0x1667 bytes) 000429[0] 1790773408.578576 trunc from 0x0df874 to 0x87c4c6 (0x79cc53 bytes) 000433[1] 1790773408.726599 punch from 0x49aa26 to 0x49bae7 (0x10c2 bytes) 000434[0] 1790773408.771251 write 0x715c64 thru 0x715f89 (0x326 bytes) 000436[0] 1790773408.853168 trunc from 0x87c4c7 to 0x758d5d (0x123769 bytes) 000439[0] 1790773408.913778 mapread 0x3dda44 thru 0x3df479 (0x1a36 bytes) 000440[1] 1790773408.944364 read 0x42ff59 thru 0x430a49 (0xaf1 bytes) 000441[1] 1790773408.993774 trunc from 0x758d5e to 0x34bac5 (0x40d298 bytes) 000444[0] 1790773409.058424 write 0x8029c4 thru 0x8043d7 (0x1a14 bytes) 000446[0] 1790773409.115534 mapread 0x01f7da thru 0x01fb83 (0x3aa bytes) 000449[1] 1790773409.156877 mapread 0x0aae93 thru 0x0ab7db (0x949 bytes) 000450[0] 1790773409.185681 mapread 0x4c7050 thru 0x4c855a (0x150b bytes) 000452[1] 1790773409.201453 punch from 0x2f30c3 to 0x2f380f (0x74d bytes) 000453[0] 1790773409.214686 punch from 0x12d8cc to 0x12e23c (0x971 bytes) 000455[0] 1790773409.241533 mapwrite 0x5b1c1d thru 0x5b2a8c (0xe70 bytes) 000456[0] 1790773409.301863 punch from 0x596dad to 0x596ec9 (0x11d bytes) 000457[1] 1790773409.306828 mapread 0x1e0071 thru 0x1e04ea (0x47a bytes) 000458[0] 1790773409.340161 read 0x383df6 thru 0x3848e4 (0xaef bytes) 000459[0] 1790773409.345906 read 0x04132c thru 0x041d10 (0x9e5 bytes) 000460[1] 1790773409.351759 mapread 0x6e38ec thru 0x6e540f (0x1b24 bytes) 000465[1] 1790773409.389082 write 0x7f7549 thru 0x7f811f (0xbd7 bytes) 000466[0] 1790773409.445724 write 0x903a84 thru 0x905663 (0x1be0 bytes) 000467[0] 1790773409.487305 punch from 0x4d11fa to 0x4d1f62 (0xd69 bytes) 000468[1] 1790773409.519292 trunc from 0x905664 to 0x253d28 (0x6b193b bytes) 000469[1] 1790773409.575087 write 0x732eac thru 0x73445e (0x15b3 bytes) 000470[0] 1790773409.580295 punch from 0x311dc6 to 0x31226b (0x4a6 bytes) 000472[1] 1790773409.631683 read 0x1f53e2 thru 0x1f6ce2 (0x1901 bytes) 000473[1] 1790773409.655219 punch from 0x6fdb2c to 0x6fe36d (0x842 bytes) 000474[1] 1790773409.660557 mapread 0x23a382 thru 0x23b79b (0x141a bytes) 000475[1] 1790773409.675932 write 0x374ed2 thru 0x375e98 (0xfc7 bytes) 000480[1] 1790773409.702562 write 0x7a91f9 thru 0x7aa423 (0x122b bytes) 000481[0] 1790773409.704109 mapwrite 0x3af1ad thru 0x3af27c (0xd0 bytes) 000482[0] 1790773409.770895 mapread 0x64b003 thru 0x64ce3a (0x1e38 bytes) 000483[0] 1790773409.788407 mapwrite 0x897e26 thru 0x897fa8 (0x183 bytes) 000484[1] 1790773409.870729 punch from 0x7896a7 to 0x78b03b (0x1995 bytes) 000485[1] 1790773409.903112 punch from 0x2e69dd to 0x2e7506 (0xb2a bytes) 000486[1] 1790773409.916799 trunc from 0x897fa9 to 0x6c97d3 (0x1ce7d5 bytes) 000487[1] 1790773409.937437 trunc from 0x6c97d4 to 0x6af169 (0x1a66a bytes) 000488[0] 1790773409.948398 mapread 0x274db0 thru 0x274f72 (0x1c3 bytes) 000489[1] 1790773409.969867 mapwrite 0x15cdbb thru 0x15d011 (0x257 bytes) 000490[1] 1790773410.047593 punch from 0x0bd905 to 0x0be40a (0xb06 bytes) 000492[0] 1790773410.077835 trunc from 0x6af16a to 0x4f6663 (0x1b8b06 bytes) 000493[0] 1790773410.155069 punch from 0x0cb1b7 to 0x0cbfd2 (0xe1c bytes) 000494[0] 1790773410.164424 write 0x8421ad thru 0x8425db (0x42f bytes) 000495[0] 1790773410.167796 read 0x1a1f9a thru 0x1a3d82 (0x1de9 bytes) 000498[1] 1790773410.172848 trunc from 0x8425dc to 0x2b314c (0x58f48f bytes) 000499[0] 1790773410.245718 read 0x21467b thru 0x216286 (0x1c0c bytes) All operations completed A-OK! PASS 16b (28s) == sanityn test 16c: verify data consistency on ldiskfs with cache disabled (b=17397) ========================================================== 09:03:35 (1790773415) SKIP: sanityn test_16c dio on ldiskfs only SKIP 16c (5s) == sanityn test 16d: Verify DIO and buffer IO with two clients ========================================================== 09:03:40 (1790773420) 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.407648 s, 20.6 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.320935 s, 26.1 MB/s ldlm.namespaces.MGC192.168.202.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8c4950f45000.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8c4953479800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8c4950f45000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8c4953479800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8c4950f45000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8c4953479800.lru_size=clear 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 14.0128 s, 7.5 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 3.95485 s, 26.5 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 3.0592 s, 34.3 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.78093 s, 13.5 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.94848 s, 13.2 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.11157 s, 25.5 MB/s PASS 16d (55s) == sanityn test 16e: Verify size consistency for O_DIRECT write ========================================================== 09:04:35 (1790773475) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.299928 s, 35.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107754 s, 9.7 MB/s PASS 16e (8s) == sanityn test 16f: rw sequential consistency vs drop_caches ========================================================== 09:04:43 (1790773483) TIMEOUT PASS 16f (28s) == sanityn test 16g: mmap rw sequential consistency vs drop_caches ========================================================== 09:05:11 (1790773511) TIMEOUT PASS 16g (28s) == sanityn test 16h: mmap read after truncate file ======= 09:05:39 (1790773539) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16h (8s) == sanityn test 16i: read after truncate file ============ 09:05:47 (1790773547) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16i (7s) == sanityn test 16j: race dio with buffered i/o ========== 09:05:54 (1790773554) 1+0 records in 1+0 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.23234 s, 21.3 MB/s bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.73137 s, 15.1 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.90924 s, 9.0 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.98123 s, 13.2 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.52673 s, 10.4 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.29208 s, 11.4 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.11634 s, 8.4 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.56745 s, 16.7 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.83208 s, 14.3 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.03293 s, 8.6 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.07357 s, 8.5 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.90808 s, 13.7 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.08661 s, 12.6 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.35442 s, 11.1 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.45006 s, 10.7 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.35382 s, 7.8 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.70568 s, 7.1 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.94011 s, 6.7 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.94185 s, 6.7 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK PASS 16j (37s) == sanityn test 16k: Parallel FSX and drop caches should not panic ========================================================== 09:06:31 (1790773591) + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.1 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.2 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.3 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.4 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.5 -l 67108864 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported truncating to largest ever: 0x1a7062c fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x32cce75 truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! + fsx 1 done: rc=0 All operations completed A-OK! Wed Sep 30 09:07:19 EDT 2026 + fsx 2 done: rc=0 Wed Sep 30 09:07:19 EDT 2026 + fsx 3 done: rc=0 Wed Sep 30 09:07:19 EDT 2026 + fsx 4 done: rc=0 Wed Sep 30 09:07:19 EDT 2026 + fsx 5 done: rc=0 Wed Sep 30 09:07:19 EDT 2026 PASS 16k (57s) == sanityn test 17: resource creation/LVB creation race ========================================================================= 09:07:28 (1790773648) fail_loc=0x8000030a PASS 17 (11s) == sanityn test 18: mmap sanity check =========================================================================================== 09:07:39 (1790773659) mmap test1: basic mmap operation (PASS, 0.10521s) mmap test2: MAP_PRIVATE not write back (PASS, 0.049346s) mmap test3: concurrent mmap ops on two nodes (PASS, 5.2067s) mmap test4: c1 write to f1 from mmapped f2, c2 write to f1 from mmapped f1 (PASS, 4.5241s) mmap test5: read/write file to/from the buffer which mmapped to just this file (PASS, 0.16571s) mmap test6: check mmap write/read content on two nodes (PASS, 0.19032s) mmap test7: file i/o with an unmapped buffer (SKIPPED, 0s) mmap test8: SIGBUS for beyond file size (SKIPPED, 0s) mmap test9: SIGBUS for negative file offset (SKIPPED, 0s) mmap test10: multi-thread mmap access (PASS, 0.91893s) mmap test11: multi-thread shared mmap (PASS, 1.5186s) mmap test10: mtime not change for readonly mmap access (PASS, 1.1767s) PASS 18 (23s) == sanityn test 19: test concurrent uncached read races ========================================================================= 09:08:02 (1790773682) oleg255-server: error: get_param: param_path 'osd-*/lustre-OST*/read_cache_enable': No such file or directory pdsh@oleg255-client: oleg255-server: ssh exited with exit code 2 SKIP: sanityn test_19 not cache-capable obdfilter SKIP 19 (5s) == sanityn test 20: test extra readahead page left in cache ============================================================== 09:08:07 (1790773687) PASS 20 (8s) == sanityn test 21: Try to remove mountpoint on another dir ============================================================== 09:08:15 (1790773695) rmdir: removing directory, '/mnt/lustre/d21.sanityn' rmdir: failed to remove '/mnt/lustre/d21.sanityn': Device or resource busy rmdir: removing directory, '/mnt/lustre2/d21.sanityn' Removed mounted directory from another mountpoint, needs to be fixed PASS 21 (7s) == sanityn test 23: others should see updated atime while another read============================================================== 09:08:22 (1790773702) now is 1790773705 starting reads multiop /mnt/lustre/f23.sanityn vor20_c TMPPIPE=/tmp/multiop_open_wait_pipe.7627 new atime is 1790773766 PASS 23 (70s) == sanityn test 24a: lfs df [-ih] [path] test =================================================================================== 09:09:32 (1790773772) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2048 2206592 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 30720 3738624 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2048 2206592 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 30720 3738624 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre2 UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.9K 386 524.5K 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 115.2K 400 114.8K 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 114.5K 402 114.2K 1% /mnt/lustre[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.9K 386 524.5K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.2K 400 114.8K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 402 114.2K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre2 UUID bytes Used Available Use% Mounted on lustre-MDT0000_UUID 2.1G 2.0M 2.1G 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3.6G 8.0M 3.6G 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3.6G 30.0M 3.6G 1% /mnt/lustre[OST:1] filesystem_summary: 7.2G 38.0M 7.2G 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537447 386 537061 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118000 400 117600 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117298 402 116896 1% /mnt/lustre2[OST:1] filesystem_summary: 234882 386 234496 1% /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2048 2206592 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 30720 3738624 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.9K 386 524.5K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.2K 400 114.8K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 402 114.2K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537447 386 537061 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 118000 400 117600 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 117298 402 116896 1% /mnt/lustre[OST:1] filesystem_summary: 234882 386 234496 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537447 386 537061 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118000 400 117600 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117298 402 116896 1% /mnt/lustre2[OST:1] filesystem_summary: 234882 386 234496 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2048 2206592 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 30720 3738624 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2048 2206592 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 30720 3738624 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre2 PASS 24a (7s) == sanityn test 24b: lfs df should show both filesystems ========================================================================= 09:09:39 (1790773779) PASS 24b (6s) == sanityn test 25a: change ACL on one mountpoint be seen on another ============================================================= 09:09:45 (1790773785) running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] PASS 25a (8s) == sanityn test 25b: change ACL under remote dir on one mountpoint be seen on another ========================================================== 09:09:53 (1790773793) SKIP: sanityn test_25b needs >= 2 MDTs SKIP 25b (3s) == sanityn test 26a: allow mtime to get older ============ 09:09:57 (1790773797) utime: good mknod times 1790773797 <= 1790773798 <= 1790773798 for /mnt/lustre/f26a utime: good mknod times 1790773797 <= 1790773798 <= 1790773798 for /mnt/lustre2/f26a utime: good utime mtimes 100000, atime 200000 utime: updated times for /mnt/lustre2/f26a PASS 26a (9s) == sanityn test 26b: sync mtime between ost and mds ====== 09:10:05 (1790773805) PASS 26b (9s) == sanityn test 26c: set-in-past on open file is not lost on close ========================================================== 09:10:14 (1790773814) multiop /mnt/lustre/f26c.sanityn vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7627 File: /mnt/lustre/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502398 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-09-30 09:10:16.000000000 -0400 Birth: 2026-09-30 09:10:16.000000000 -0400 File: /mnt/lustre2/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502398 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-09-30 09:10:16.000000000 -0400 Birth: 2026-09-30 09:10:16.000000000 -0400 PASS 26c (8s) SKIP: sanityn test_27 skipping excluded test 27 == sanityn test 30: recreate file race =================== 09:10:24 (1790773824) 0 PASS 30 (9s) == sanityn test 31a: voluntary cancel / blocking ast race======================================================================== 09:10:33 (1790773833) fail_loc=0x314 PASS 31a (8s) == sanityn test 31b: voluntary OST cancel / blocking ast race======================================================================== 09:10:41 (1790773841) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg255-server mds-ost sync done. sleep 5 for ZFS MDS Waiting for MDT destroys to complete fail_loc=0x314 fail_loc=0x316 fail_loc=0 fail_loc=0 Filesystem 1K-blocks Used Available Use% Mounted on 192.168.202.155@tcp:/lustre 7542784 39936 7496704 1% /mnt/lustre2 PASS 31b (25s) == sanityn test 31r: open-rename(replace) race =========== 09:11:06 (1790773866) -rw-r--r-- 1 root root 0 Sep 30 09:11 /mnt/lustre/f31r.sanityn.target fail_loc=0x1419 fail_val=3 PASS 31r (9s) == sanityn test 31s: open should not revalidate invalid dentry ========================================================== 09:11:15 (1790773875) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 1211: 43199 Terminated tail -f $tfile PASS 31s (7s) == sanityn test 31t: getattr should not revalidate invalid dentry ========================================================== 09:11:22 (1790773882) PASS 31t (8s) == sanityn test 32b: lockless i/o ======================== 09:11:30 (1790773890) SKIP: sanityn test_32b max_nolock_bytes is removed >= 2.17.53 SKIP 32b (3s) == sanityn test 32c: contention detection ================ 09:11:33 (1790773893) ldlm.namespaces.filter-lustre-OST0000_UUID.contended_locks=2 ldlm.namespaces.filter-lustre-OST0001_UUID.contended_locks=2 ldlm.namespaces.filter-lustre-OST0000_UUID.contention_events=0 ldlm.namespaces.filter-lustre-OST0001_UUID.contention_events=0 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.3387 s, 39.2 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.4273 s, 36.7 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.95777 s, 26.8 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.2843 s, 23.0 MB/s contention_seconds 4, contention_events 17 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.83408 s, 28.6 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.18649 s, 24.0 MB/s ldlm.namespaces.filter-lustre-OST0000_UUID.contention_events=0 ldlm.namespaces.filter-lustre-OST0001_UUID.contention_events=0 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.26221 s, 41.5 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.26787 s, 41.4 MB/s PASS 32c (29s) SKIP: sanityn test_33a skipping SLOW test 33a == sanityn test 33b: COS: cross create/delete, 2 clients, benchmark under remote dir ========================================================== 09:12:04 (1790773924) SKIP: sanityn test_33b Need two or more clients, have 1 SKIP 33b (3s) == sanityn test 33c: Cancel cross-MDT lock should trigger Sync-on-Lock-Cancel ========================================================== 09:12:07 (1790773927) SKIP: sanityn test_33c needs >= 2 MDTs SKIP 33c (3s) == sanityn test 33d: dependent transactions should trigger COS ========================================================== 09:12:10 (1790773930) SKIP: sanityn test_33d needs >= 2 MDTs SKIP 33d (2s) == sanityn test 33e: independent transactions shouldn't trigger COS ========================================================== 09:12:13 (1790773933) SKIP: sanityn test_33e needs >= 2 MDTs SKIP 33e (2s) == sanityn test 34: no lock timeout under IO ============= 09:12:15 (1790773935) fail_loc=0x512 fail_loc=0x512 lock should not expire writing on client1 reading on client2 success fail_loc=0x511 fail_loc=0x511 lock should expire writing on client1 reading on client2 success oleg255-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8c4950f45000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8c4950f45000.ost_server_uuid in IDLE state after 0 sec oleg255-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8c4950f45000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8c4950f45000.ost_server_uuid in IDLE state after 0 sec oleg255-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8c4950f45000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8c4950f45000.ost_server_uuid in IDLE state after 0 sec oleg255-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8c4950f45000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8c4950f45000.ost_server_uuid in IDLE state after 0 sec oleg255-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8c4950f45000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8c4950f45000.ost_server_uuid in IDLE state after 0 sec oleg255-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8c4950f45000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8c4950f45000.ost_server_uuid in IDLE state after 0 sec PASS 34 (143s) == sanityn test 35: -EINTR cp_ast vs. bl_ast race does not evict client ========================================================== 09:14:38 (1790774078) fail_loc=0x80000317 Race attempt 0 Wait for 50501 50534 for 60 sec... - open/close 858 (time 1790774095.46 total 10.00 last 85.76) - open/close 1693 (time 1790774105.46 total 20.01 last 83.48) - open/close 2628 (time 1790774115.47 total 30.01 last 93.43) - open/close 3575 (time 1790774125.47 total 40.02 last 94.67) total: 4000 open/close in 44.35 seconds: 90.20 ops/second fail_loc=0x0 Filesystem Size Used Avail Use% Mounted on 192.168.202.155@tcp:/lustre 7.2G 40M 7.2G 1% /mnt/lustre 192.168.202.155@tcp:/lustre 7.2G 40M 7.2G 1% /mnt/lustre2 PASS 35 (73s) == sanityn test 36: handle ESTALE/open-unlink correctly == 09:15:52 (1790774152) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg255-server mds-ost sync done. Waiting for MDT destroys to complete 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.88729 s, 27.8 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7627 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(0) *** before(7497728) after_dd(7444480) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.30139 s, 40.3 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7627 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(1) *** before(7497728) after_dd(7442432) after(7493632) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.25364 s, 41.8 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7627 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(2) *** before(7493632) after_dd(7442432) after(7493632) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.33387 s, 39.3 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7627 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(3) *** before(7493632) after_dd(7442432) after(7493632) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.21192 s, 43.3 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7627 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(4) *** before(7493632) after_dd(7440384) after(7491584) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.30707 s, 40.1 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7627 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(5) *** before(7491584) after_dd(7446528) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.21825 s, 43.0 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7627 sleep 5 for ZFS MDS Waiting for MDT destroys to complete *** cycle(6) *** before(7497728) after_dd(7444480) after(7495680) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.37428 s, 38.2 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7627 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(7) *** before(7495680) after_dd(7444480) after(7495680) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.61792 s, 32.4 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7627 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(8) *** before(7495680) after_dd(7444480) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.37258 s, 38.2 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7627 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(9) *** before(7497728) after_dd(7446528) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.35936 s, 38.6 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7627 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(10) *** before(7497728) after_dd(7444480) after(7495680) PASS 36 (322s) == sanityn test 37: check i_size is not updated for directory on close (bug 18695) ======================================================================== 09:21:13 (1790774473) multiop /mnt/lustre/d37.sanityn vD_c TMPPIPE=/tmp/multiop_open_wait_pipe.7627 - create 1979 (time 1790774486.83 total 10.00 last 197.89) - create 3725 (time 1790774496.83 total 20.00 last 174.59) - create 5350 (time 1790774506.83 total 30.00 last 162.47) - create 6887 (time 1790774516.84 total 40.01 last 153.64) - create 8520 (time 1790774526.84 total 50.01 last 163.23) total: 10000 create in 59.39 seconds: 168.36 ops/second PASS 37 (71s) == sanityn test 39a: file mtime does not change after rename ========================================================== 09:22:24 (1790774544) repeat after cancel_lru_locks PASS 39a (8s) == sanityn test 39b: file mtime the same on clients with/out lock ========================================================== 09:22:32 (1790774552) repeat after cancel_lru_locks PASS 39b (9s) == sanityn test 39c: check truncate mtime update ================================================================================ 09:22:41 (1790774561) repeat after cancel_lru_locks PASS 39c (7s) == sanityn test 39d: sync write should update mtime ====== 09:22:48 (1790774568) fail_loc=0x411 fail_loc=0 PASS 39d (7s) SKIP: sanityn test_40a skipping ALWAYS excluded test 40a == sanityn test 40b: pdirops: open|create and others ======================================================================== 09:22:57 (1790774577) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40b (18s) == sanityn test 40c: pdirops: link and others ======================================================================== 09:23:16 (1790774596) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40c (18s) == sanityn test 40d: pdirops: unlink and others ======================================================================== 09:23:34 (1790774614) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40d (19s) == sanityn test 40e: pdirops: rename and others ======================================================================== 09:23:53 (1790774633) No conflict No conflict No conflict No conflict No conflict No conflict PASS 40e (18s) == sanityn test 41a: pdirops: create vs mkdir ======================================================================== 09:24:11 (1790774651) mkdir: cannot create directory '/mnt/lustre2/f41a.sanityn': File exists Conflict PASS 41a (14s) == sanityn test 41b: pdirops: create vs create ======================================================================== 09:24:25 (1790774665) open: File exists Conflict PASS 41b (12s) == sanityn test 41c: pdirops: create vs link ======================================================================== 09:24:38 (1790774678) link: cannot create link '/mnt/lustre2/f41c.sanityn' to '/mnt/lustre2/f41c.sanityn-2': File exists Conflict PASS 41c (14s) == sanityn test 41d: pdirops: create vs unlink ======================================================================== 09:24:52 (1790774692) Conflict PASS 41d (15s) == sanityn test 41e: pdirops: create and rename (tgt) ======================================================================== 09:25:07 (1790774707) Conflict PASS 41e (12s) == sanityn test 41f: pdirops: create and rename (src) ======================================================================== 09:25:19 (1790774719) Conflict PASS 41f (14s) == sanityn test 41g: pdirops: create vs getattr ======================================================================== 09:25:33 (1790774733) Conflict PASS 41g (13s) == sanityn test 41h: pdirops: create vs readdir ======================================================================== 09:25:46 (1790774746) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 2433: lctl set_param -n fail_loc=0 2>/dev/null || true: No such file or directory Conflict PASS 41h (14s) == sanityn test 41i: reint_open: create vs create ======== 09:26:00 (1790774760) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 41i (1088s) == sanityn test 42a: pdirops: mkdir vs mkdir ======================================================================== 09:44:08 (1790775848) mkdir: cannot create directory '/mnt/lustre2/f42a.sanityn': File exists Conflict PASS 42a (12s) == sanityn test 42b: pdirops: mkdir vs create ======================================================================== 09:44:20 (1790775860) open: File exists Conflict PASS 42b (11s) == sanityn test 42c: pdirops: mkdir vs link ======================================================================== 09:44:32 (1790775872) link: cannot create link '/mnt/lustre2/f42c.sanityn' to '/mnt/lustre2/f42c.sanityn-2': File exists Conflict PASS 42c (14s) == sanityn test 42d: pdirops: mkdir vs unlink ======================================================================== 09:44:45 (1790775885) Conflict PASS 42d (13s) == sanityn test 42e: pdirops: mkdir and rename (tgt) ======================================================================== 09:44:58 (1790775898) mv: cannot overwrite directory '/mnt/lustre2/f42e.sanityn' with non-directory Conflict PASS 42e (13s) == sanityn test 42f: pdirops: mkdir and rename (src) ======================================================================== 09:45:11 (1790775911) Conflict PASS 42f (14s) == sanityn test 42g: pdirops: mkdir vs getattr ======================================================================== 09:45:25 (1790775925) Conflict PASS 42g (13s) == sanityn test 42h: pdirops: mkdir vs readdir ======================================================================== 09:45:38 (1790775938) Conflict PASS 42h (13s) == sanityn test 43a: rmdir,mkdir doesn't return -EEXIST ======================================================================== 09:45:51 (1790775951) PASS 43a (63s) == sanityn test 43b: pdirops: unlink vs create ======================================================================== 09:46:54 (1790776014) Conflict PASS 43b (14s) == sanityn test 43c: pdirops: unlink vs link ======================================================================== 09:47:09 (1790776029) Conflict PASS 43c (14s) == sanityn test 43d: pdirops: unlink vs unlink ======================================================================== 09:47:23 (1790776043) rm: cannot remove '/mnt/lustre2/f43d.sanityn': No such file or directory Conflict PASS 43d (13s) == sanityn test 43e: pdirops: unlink and rename (tgt) ======================================================================== 09:47:36 (1790776056) Conflict PASS 43e (14s) == sanityn test 43f: pdirops: unlink and rename (src) ======================================================================== 09:47:50 (1790776070) mv: cannot stat '/mnt/lustre2/f43f.sanityn': No such file or directory Conflict PASS 43f (14s) == sanityn test 43g: pdirops: unlink vs getattr ======================================================================== 09:48:04 (1790776084) stat: cannot statx '/mnt/lustre2/f43g.sanityn': No such file or directory Conflict PASS 43g (13s) == sanityn test 43h: pdirops: unlink vs readdir ======================================================================== 09:48:17 (1790776097) Conflict PASS 43h (14s) == sanityn test 43i: pdirops: unlink vs remote mkdir ===== 09:48:31 (1790776111) SKIP: sanityn test_43i needs >= 2 MDTs SKIP 43i (3s) == sanityn test 43j: racy mkdir return EEXIST ======================================================================== 09:48:34 (1790776114) mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists PASS 43j (127s) == sanityn test 43k: unlink vs create ==================== 09:50:41 (1790776241) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 43k (1226s) == sanityn test 44a: pdirops: rename tgt vs mkdir ======================================================================== 10:11:07 (1790777467) mkdir: cannot create directory '/mnt/lustre2/f44a.sanityn': File exists Conflict Wed Sep 30 10:11:16 EDT 2026 PASS 44a (13s) == sanityn test 44b: pdirops: rename tgt vs create ======================================================================== 10:11:20 (1790777480) open: File exists Conflict PASS 44b (13s) == sanityn test 44c: pdirops: rename tgt vs link ======================================================================== 10:11:33 (1790777493) link: cannot create link '/mnt/lustre2/f44c.sanityn' to '/mnt/lustre2/f44c.sanityn-3': File exists Conflict PASS 44c (14s) == sanityn test 44d: pdirops: rename tgt vs unlink ======================================================================== 10:11:47 (1790777507) Conflict PASS 44d (15s) == sanityn test 44e: pdirops: rename tgt and rename (tgt) ======================================================================== 10:12:02 (1790777522) Conflict PASS 44e (15s) == sanityn test 44f: pdirops: rename tgt and rename (src) ======================================================================== 10:12:17 (1790777537) Conflict PASS 44f (13s) == sanityn test 44g: pdirops: rename tgt vs getattr ======================================================================== 10:12:30 (1790777550) Conflict PASS 44g (14s) == sanityn test 44h: pdirops: rename tgt vs readdir ======================================================================== 10:12:44 (1790777564) Conflict PASS 44h (13s) == sanityn test 44i: pdirops: rename tgt vs remote mkdir ========================================================== 10:12:57 (1790777577) SKIP: sanityn test_44i needs >= 2 MDTs SKIP 44i (2s) == sanityn test 45a: rename,mkdir doesn't return -EEXIST ======================================================================== 10:12:59 (1790777579) PASS 45a (103s) == sanityn test 45b: pdirops: rename src vs create ======================================================================== 10:14:42 (1790777682) Conflict PASS 45b (13s) == sanityn test 45c: pdirops: rename src vs link ======================================================================== 10:14:55 (1790777695) Conflict PASS 45c (12s) == sanityn test 45d: pdirops: rename src vs unlink ======================================================================== 10:15:08 (1790777708) rm: cannot remove '/mnt/lustre2/f45d.sanityn': No such file or directory Conflict PASS 45d (14s) == sanityn test 45e: pdirops: rename src and rename (tgt) ======================================================================== 10:15:22 (1790777722) Conflict PASS 45e (14s) == sanityn test 45f: pdirops: rename src and rename (src) ======================================================================== 10:15:36 (1790777736) mv: cannot stat '/mnt/lustre2/f45f.sanityn': No such file or directory Conflict PASS 45f (13s) == sanityn test 45g: pdirops: rename src vs getattr ======================================================================== 10:15:49 (1790777749) stat: cannot statx '/mnt/lustre2/f45g.sanityn': No such file or directory Conflict PASS 45g (13s) == sanityn test 45h: pdirops: unlink vs readdir ======================================================================== 10:16:02 (1790777762) Conflict PASS 45h (11s) == sanityn test 45i: pdirops: rename src vs remote mkdir ========================================================== 10:16:13 (1790777773) SKIP: sanityn test_45i needs >= 2 MDTs SKIP 45i (3s) == sanityn test 45j: read vs rename ====================== 10:16:16 (1790777776) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 45j (1178s) == sanityn test 46a: pdirops: link vs mkdir ======================================================================== 10:35:55 (1790778955) mkdir: cannot create directory '/mnt/lustre2/f46a.sanityn': File exists Conflict PASS 46a (14s) == sanityn test 46b: pdirops: link vs create ======================================================================== 10:36:08 (1790778968) open: File exists Conflict PASS 46b (14s) == sanityn test 46c: pdirops: link vs link ======================================================================== 10:36:22 (1790778982) link: cannot create link '/mnt/lustre2/f46c.sanityn' to '/mnt/lustre2/f46c.sanityn': File exists Conflict PASS 46c (13s) == sanityn test 46d: pdirops: link vs unlink ======================================================================== 10:36:35 (1790778995) Conflict PASS 46d (13s) == sanityn test 46e: pdirops: link and rename (tgt) ======================================================================== 10:36:48 (1790779008) Conflict PASS 46e (15s) == sanityn test 46f: pdirops: link and rename (src) ======================================================================== 10:37:03 (1790779023) Conflict PASS 46f (13s) == sanityn test 46g: pdirops: link vs getattr ======================================================================== 10:37:17 (1790779037) Conflict PASS 46g (14s) == sanityn test 46h: pdirops: link vs readdir ======================================================================== 10:37:31 (1790779051) Conflict PASS 46h (14s) == sanityn test 46i: pdirops: link vs remote mkdir ======= 10:37:45 (1790779065) SKIP: sanityn test_46i needs >= 2 MDTs SKIP 46i (3s) == sanityn test 47a: pdirops: remote mkdir vs mkdir ====== 10:37:48 (1790779068) SKIP: sanityn test_47a needs >= 2 MDTs SKIP 47a (4s) == sanityn test 47b: pdirops: remote mkdir vs create ===== 10:37:53 (1790779073) SKIP: sanityn test_47b needs >= 2 MDTs SKIP 47b (3s) == sanityn test 47c: pdirops: remote mkdir vs link ======= 10:37:56 (1790779076) SKIP: sanityn test_47c needs >= 2 MDTs SKIP 47c (4s) == sanityn test 47d: pdirops: remote mkdir vs unlink ===== 10:38:00 (1790779080) SKIP: sanityn test_47d needs >= 2 MDTs SKIP 47d (4s) == sanityn test 47e: pdirops: remote mkdir and rename (tgt) ========================================================== 10:38:04 (1790779084) SKIP: sanityn test_47e needs >= 2 MDTs SKIP 47e (3s) == sanityn test 47f: pdirops: remote mkdir and rename (src) ========================================================== 10:38:07 (1790779087) SKIP: sanityn test_47f needs >= 2 MDTs SKIP 47f (4s) == sanityn test 47g: pdirops: remote mkdir vs getattr ==== 10:38:11 (1790779091) SKIP: sanityn test_47g needs >= 2 MDTs SKIP 47g (3s) == sanityn test 50: osc lvb attrs: enqueue vs. CP AST ======================================================================== 10:38:15 (1790779095) 10+0 records in 10+0 records out 10240 bytes (10 kB, 10 KiB) copied, 0.13919 s, 73.6 kB/s fail_loc=0x410 fail_loc=0x0 PASS 50 (15s) == sanityn test 51a: layout lock: refresh layout should work ========================================================== 10:38:29 (1790779109) 0+1 records in 0+1 records out 158 bytes copied, 0.0247408 s, 6.4 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 3696: kill: (218710) - No such process PASS 51a (12s) == sanityn test 51b: layout lock: glimpse should be able to restart if layout changed ========================================================== 10:38:41 (1790779121) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0362729 s, 28.2 kB/s fail_loc=0x1404 fail_val=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.292496 s, 3.6 MB/s 3145728 PASS 51b (22s) == sanityn test 51c: layout lock: IT_LAYOUT blocked and correct layout can be returned ========================================================== 10:39:03 (1790779143) fail_loc=0x172 Setting layout to have 2 stripes ... 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0281314 s, 36.4 kB/s PASS 51c (13s) == sanityn test 51d: layout lock: losing layout lock should clean up memory map region ========================================================== 10:39:16 (1790779156) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0652007 s, 16.1 MB/s Before revoking layout lock: 1024 KB mapped PASS 51d (9s) == sanityn test 51e: lfs getstripe does not break leases, part 2 ========================================================== 10:39:25 (1790779165) /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 2451 0x993 0x240000400 write lease(2) has applied. /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 2451 0x993 0x240000400 read lease(1) has applied. PASS 51e (10s) == sanityn test 54: rename locking ======================= 10:39:35 (1790779175) llite.lustre-ffff8c4950f45000.xattr_cache=1 llite.lustre-ffff8c4953479800.xattr_cache=1 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 14 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505015 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-30 10:39:37.000000000 -0400 Modify: 2026-09-30 10:39:37.000000000 -0400 Change: 2026-09-30 10:39:37.000000000 -0400 Birth: 2026-09-30 10:39:37.000000000 -0400 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000154 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 14 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505018 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-30 10:39:44.000000000 -0400 Modify: 2026-09-30 10:39:45.000000000 -0400 Change: 2026-09-30 10:39:45.000000000 -0400 Birth: 2026-09-30 10:39:44.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 2 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505021 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-30 10:39:51.000000000 -0400 Modify: 2026-09-30 10:39:51.000000000 -0400 Change: 2026-09-30 10:39:51.000000000 -0400 Birth: 2026-09-30 10:39:50.000000000 -0400 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1 Size: 16384 Blocks: 14 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505020 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-30 10:39:51.000000000 -0400 Modify: 2026-09-30 10:39:52.000000000 -0400 Change: 2026-09-30 10:39:52.000000000 -0400 Birth: 2026-09-30 10:39:50.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000154 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 14 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505024 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-30 10:39:57.000000000 -0400 Modify: 2026-09-30 10:39:58.000000000 -0400 Change: 2026-09-30 10:39:58.000000000 -0400 Birth: 2026-09-30 10:39:57.000000000 -0400 File: /mnt/lustre/d1 Size: 16384 Blocks: 14 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505023 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-30 10:39:57.000000000 -0400 Modify: 2026-09-30 10:39:58.000000000 -0400 Change: 2026-09-30 10:39:58.000000000 -0400 Birth: 2026-09-30 10:39:57.000000000 -0400 PASS 54 (33s) == sanityn test 55a: rename vs unlink target dir ========= 10:40:08 (1790779208) fail_loc=0x80000156 mv: cannot move '/mnt/lustre/d55a.sanityn/d1/d2' to '/mnt/lustre/d55a.sanityn/d3/d2': No such file or directory PASS 55a (12s) == sanityn test 55b: rename vs unlink source dir ========= 10:40:20 (1790779220) fail_loc=0x80000156 mv: cannot stat '/mnt/lustre/d55b.sanityn/d1/d2': No such file or directory PASS 55b (12s) == sanityn test 55c: rename vs unlink orphan target dir == 10:40:32 (1790779232) fail_loc=0x156 mv: cannot move '/mnt/lustre/d55c.sanityn/d1/d2' to '/mnt/lustre/d55c.sanityn/d3/d2': No such file or directory PASS 55c (18s) == sanityn test 55d: rename file vs link ================= 10:40:50 (1790779250) fail_loc=0x155 mv: '/mnt/lustre/d55d.sanityn/f1' and '/mnt/lustre/d55d.sanityn/d55d.sanityn/f1' are the same file PASS 55d (15s) == sanityn test 55e: rename race AB/BA under the same parent dir ========================================================== 10:41:05 (1790779265) SKIP: sanityn test_55e needs >= 2 MDTs SKIP 55e (4s) == sanityn test 55f: rename: (P1/A -> P2/B) race with (P2/B -> P1/A) ========================================================== 10:41:09 (1790779269) fail_loc=0x155 We survived after P1A_P2B/P2B_P1A race test fail_loc=0x0 PASS 55f (18s) == sanityn test 55g: rename: race with trylock =========== 10:41:28 (1790779288) mdt.lustre-MDT0000.enable_rename_trylock=1 fail_loc=0x155 mv: cannot move '/mnt/lustre2/d55g.sanityn/b1/b2' to a subdirectory of itself, '/mnt/lustre2/d55g.sanityn/a1/a2/a3/a4/b2' We survived race with trylock test fail_loc=0x0 PASS 55g (22s) == sanityn test 56a: test llverdev with single large stripe ========================================================== 10:41:50 (1790779310) run_llverdev /mnt/lustre/f56a.sanityn -p -s 16000000 -c 4k llverdev: /mnt/lustre/f56a.sanityn is 16777216000000 bytes (15625 GB) in size timestamp: 1790779311 chunksize: 4096 size: 16777216000000 write offset: 16384000000kB 0.670759 MB/s read offset: 16384000000kB inf MB/s llverdev: data verified successfully PASS 56a (99s) SKIP: sanityn test_56b skipping ALWAYS excluded test 56b == sanityn test 60: Verify data_version behaviour ======== 10:43:32 (1790779412) PASS 60 (10s) == sanityn test 70a: cd directory && rm directory ======== 10:43:42 (1790779422) PASS 70a (7s) == sanityn test 70b: remove files after calling rm_entry ========================================================== 10:43:49 (1790779429) PASS 70b (7s) == sanityn test 71a: correct file map just after write operation is finished ========================================================== 10:43:56 (1790779436) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0177517 s, 3.7 MB/s SKIP: sanityn test_71a ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71a (3s) == sanityn test 71b: check fiemap support for stripecount > 1 ========================================================== 10:44:00 (1790779440) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0147553 s, 4.4 MB/s SKIP: sanityn test_71b ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71b (3s) == sanityn test 71c: check FIEMAP_EXTENT_LAST flag with different extents number ========================================================== 10:44:03 (1790779443) SKIP: sanityn test_71c support only ldiskfs ost SKIP 71c (3s) == sanityn test 71d: fiemap corruption test with fm_extent_count=0 ========================================================== 10:44:06 (1790779446) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0421475 s, 1.6 MB/s SKIP: sanityn test_71d ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71d (3s) == sanityn test 72: getxattr/setxattr cache should be consistent between nodes ========================================================== 10:44:09 (1790779449) llite.lustre-ffff8c4950f45000.xattr_cache=1 llite.lustre-ffff8c4953479800.xattr_cache=1 getfattr: Removing leading '/' from absolute path names user.attr1="value1" getfattr: Removing leading '/' from absolute path names user.attr1="value2" getfattr: Removing leading '/' from absolute path names getfattr: Removing leading '/' from absolute path names b362740ad371eae580e7f85452e8830b - 870e072da1ebfc20ea8c0cd3d6bdf3b6 - PASS 72 (7s) == sanityn test 73: getxattr should not cause xattr lock cancellation ========================================================== 10:44:16 (1790779456) llite.lustre-ffff8c4950f45000.xattr_cache=1 llite.lustre-ffff8c4953479800.xattr_cache=1 getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" PASS 73 (7s) == sanityn test 74: flock deadlock: different mounts ======================================================================== 10:44:23 (1790779463) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0170732 s, 60.0 kB/s 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0100719 s, 102 kB/s 231060: get lock2 231061: get lock1 231060: done 231060: get lock1 231061: done 231061: get lock2 fcntl cmd 7 failed: Resource deadlock avoided 231061: cannot get lock2: Resource deadlock avoided 231061: exit rc=0 231060: done 231060: exit rc=0 PASS 74 (10s) == sanityn test 75: osc: upcall after unuse lock============================================================================= 10:44:33 (1790779473) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.07191 s, 29.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0337806 s, 31.0 MB/s fail_loc=0x31d /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 4379: 231642 Killed stat -c %s $DIR1/$tfile fail_loc=0 PASS 75 (12s) == sanityn test 76: Verify MDT open_files listing ======== 10:44:45 (1790779485) opening files: ........ closing files: ........ PASS 76 (59s) == sanityn test 77a: check FIFO NRS policy =============== 10:45:44 (1790779544) ost.OSS.ost.nrs_policies=fifo ost.OSS.ost_create.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_out.nrs_policies=fifo ost.OSS.ost_seq.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.429083 s, 39.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0644354 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0563256 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0867635 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116987 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127391 s, 8.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120535 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.225411 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153181 s, 6.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209897 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.246218 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.265816 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.229688 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207514 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.263302 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.253986 s, 4.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8c4950f45000.lock_unused_count=3 ldlm.namespaces.lustre-OST0001-osc-ffff8c4950f45000.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.332547 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0646707 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0888924 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00883925 s, 119 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00867495 s, 121 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0980462 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0959089 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0591468 s, 17.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0565665 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104178 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115757 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0655095 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0933851 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134262 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113666 s, 9.2 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8c4950f45000.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132075 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00602338 s, 174 MB/s removed '/mnt/lustre/d77a.sanityn/nrs_w_oleg255-client.virtnet' removed '/mnt/lustre/d77a.sanityn/nrs_r_oleg255-client.virtnet' removed directory '/mnt/lustre/d77a.sanityn' PASS 77a (7s) == sanityn test 77b: check CRR-N NRS policy ============== 10:45:52 (1790779552) ost.OSS.ost.nrs_policies=crrn ost.OSS.ost_create.nrs_policies=crrn ost.OSS.ost_io.nrs_policies=crrn ost.OSS.ost_out.nrs_policies=crrn ost.OSS.ost_seq.nrs_policies=crrn ost.OSS.ost.nrs_crrn_quantum=1 ost.OSS.ost_create.nrs_crrn_quantum=1 ost.OSS.ost_io.nrs_crrn_quantum=1 ost.OSS.ost_out.nrs_crrn_quantum=1 ost.OSS.ost_seq.nrs_crrn_quantum=1 policy: crr-n, crrn_quantum 1 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.383215 s, 43.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0918325 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0929607 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.281093 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.224896 s, 4.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.234843 s, 4.5 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8c4950f45000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.26792 s, 3.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216407 s, 4.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223832 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23269 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.245277 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.233227 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.238805 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.230095 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.317979 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.291311 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.290021 s, 3.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8c4950f45000.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0327548 s, 32.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.038304 s, 27.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0470133 s, 22.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.069354 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0696036 s, 15.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.038759 s, 27.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0570717 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111011 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0638266 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.055275 s, 19.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0577543 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0756653 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.06499 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0812001 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0705146 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0693732 s, 15.1 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg255-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg255-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost.nrs_crrn_quantum=64 ost.OSS.ost_create.nrs_crrn_quantum=64 ost.OSS.ost_io.nrs_crrn_quantum=64 ost.OSS.ost_out.nrs_crrn_quantum=64 ost.OSS.ost_seq.nrs_crrn_quantum=64 policy: crr-n, crrn_quantum 64 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.493688 s, 34.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119801 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105089 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.274535 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.353366 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.335521 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.352674 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.372051 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.39695 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.371965 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.455016 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.430243 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.43981 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.476814 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.030989 s, 33.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.431198 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105304 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0999194 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00394104 s, 266 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0970578 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0504405 s, 20.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.011247 s, 93.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114732 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.055265 s, 19.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00655116 s, 160 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128201 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00946163 s, 111 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00719957 s, 146 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.548392 s, 1.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15297 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00455999 s, 230 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.618379 s, 1.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00313283 s, 335 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8c4950f45000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff8c4950f45000.lock_unused_count=9 removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg255-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg255-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77b (12s) == sanityn test 77c: check ORR NRS policy ================ 10:46:04 (1790779564) ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_orr_quantum=1 ost.OSS.ost_io.nrs_orr_offset_type=physical ost.OSS.ost_io.nrs_orr_supported=reads policy: orr, orr_quantum 1, orr_offset_type physical, orr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.392738 s, 42.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.072746 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101204 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0664171 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122846 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0888704 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208164 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.234464 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203863 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.241765 s, 4.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.313731 s, 3.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222549 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.234836 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.251839 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.330231 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21395 s, 4.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8c4950f45000.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffff8c4950f45000.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.268538 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00258471 s, 406 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00261363 s, 401 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0025173 s, 417 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117044 s, 9.0 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0881935 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0103644 s, 101 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0878461 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0261166 s, 40.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0589404 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146647 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0837791 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157904 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15635 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112626 s, 9.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168754 s, 6.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0537427 s, 19.5 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg255-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg255-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=writes ost.OSS.ost_io.nrs_orr_quantum=64 policy: orr, orr_quantum 64, orr_offset_type physical, orr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.376529 s, 44.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.047847 s, 21.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0837911 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.066415 s, 15.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144012 s, 7.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129226 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115105 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0841986 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112521 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104983 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148106 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157611 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.226904 s, 4.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8c4950f45000.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffff8c4950f45000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164137 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110669 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.246888 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.261117 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.089859 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0557157 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0448951 s, 23.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0900564 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0693559 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0725289 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0681099 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0388413 s, 27.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0425861 s, 24.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0723736 s, 14.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0502516 s, 20.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0543716 s, 19.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0464099 s, 22.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13258 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0644149 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0868191 s, 12.1 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg255-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg255-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=reads_and_writes ost.OSS.ost_io.nrs_orr_offset_type=logical policy: orr, orr_quantum 64, orr_offset_type logical, orr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.333505 s, 50.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0792305 s, 13.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0924532 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0806825 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0768563 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116918 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.229754 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23354 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212382 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156016 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.219792 s, 4.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8c4950f45000.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199138 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.253089 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.304392 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.30162 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.230869 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.372629 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00845259 s, 124 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.050891 s, 20.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0265023 s, 39.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158328 s, 66.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0864938 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0730384 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0626661 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0464079 s, 22.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.064255 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016579 s, 63.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0665347 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0655141 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0437141 s, 24.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0779206 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0758136 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00797181 s, 132 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg255-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg255-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77c (12s) == sanityn test 77d: check TRR nrs policy ================ 10:46:16 (1790779576) ost.OSS.ost_io.nrs_policies=trr ost.OSS.ost_io.nrs_trr_quantum=1 ost.OSS.ost_io.nrs_trr_offset_type=physical ost.OSS.ost_io.nrs_trr_supported=reads policy: trr, trr_quantum 1, trr_offset_type physical, trr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.377964 s, 44.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102167 s, 10.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141509 s, 7.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146788 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139068 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133474 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163807 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146418 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125695 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134195 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.225967 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0446787 s, 23.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0332175 s, 31.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0274756 s, 38.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0108855 s, 96.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0293741 s, 35.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00404684 s, 259 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00825932 s, 127 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00327181 s, 320 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00484427 s, 216 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0717165 s, 14.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00493154 s, 213 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.288443 s, 3.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0891131 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0476852 s, 22.0 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0323772 s, 32.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.236059 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.287437 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0884801 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.322714 s, 3.2 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8c4950f45000.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.268619 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.370961 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135398 s, 77.4 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg255-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg255-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=writes ost.OSS.ost_io.nrs_trr_quantum=64 policy: trr, trr_quantum 64, trr_offset_type physical, trr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.40023 s, 41.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114548 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129434 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172355 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153533 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167158 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181191 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177183 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153012 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150384 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206298 s, 5.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.243133 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203789 s, 5.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202249 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205038 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0999696 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213296 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0022036 s, 476 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137109 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00385288 s, 272 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0782607 s, 13.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0915576 s, 11.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0113938 s, 92.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127121 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0707159 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0671149 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0568391 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0532076 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0467736 s, 22.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0538433 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0362247 s, 28.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0411581 s, 25.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0347235 s, 30.2 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg255-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg255-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=reads_and_writes ost.OSS.ost_io.nrs_trr_offset_type=logical policy: trr, trr_quantum 64, trr_offset_type logical, trr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.381372 s, 44.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124039 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164667 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0938956 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19698 s, 5.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0909259 s, 11.5 MB/s 1+0 records in 1+0 records out ldlm.namespaces.lustre-OST0000-osc-ffff8c4950f45000.lock_unused_count=5 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.229016 s, 4.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.231905 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.235939 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128323 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182891 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16346 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.308181 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.298052 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194539 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.217543 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.389771 s, 2.7 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8c4950f45000.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0256533 s, 40.9 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.038214 s, 27.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0909331 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00522594 s, 201 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0593951 s, 17.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109938 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00560522 s, 187 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0840386 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0469664 s, 22.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0394882 s, 26.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0332882 s, 31.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0376726 s, 27.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0313887 s, 33.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0304486 s, 34.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0207551 s, 50.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0308794 s, 34.0 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg255-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg255-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77d (12s) == sanityn test 77e: check TBF NID nrs policy ============ 10:46:28 (1790779588) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_tbf_rule=start localhost nid={0@lo} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start clients nid={192.168.202.55@tcp} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start others nid={*.*.*.*@tcp} rate=50 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.44343 s, 37.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0994503 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102913 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0935924 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202405 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222394 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173813 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.231932 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211478 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.2031 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.309154 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.287053 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.367209 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.255446 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.353801 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.26035 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.257432 s, 4.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8c4950f45000.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff8c4950f45000.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0808976 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111914 s, 9.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0511065 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0606039 s, 17.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0544203 s, 19.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.058592 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0783799 s, 13.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0504705 s, 20.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0854253 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0756128 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0905833 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0840923 s, 12.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100284 s, 10.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0852385 s, 12.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127844 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104396 s, 10.0 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8c4950f45000.lock_unused_count=1 removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg255-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg255-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change localhost rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change clients rate=101 ost.OSS.ost_io.nrs_tbf_rule=change others rate=51 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.412725 s, 40.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0588157 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0809181 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108244 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0710236 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139951 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0700083 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0899505 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0911581 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0808848 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108906 s, 9.6 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8c4950f45000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174647 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.259141 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.232174 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.317347 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.322324 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.300535 s, 3.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8c4950f45000.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0432574 s, 24.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0610741 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108624 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0594975 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132854 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128829 s, 8.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0858618 s, 12.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0779771 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127848 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0939477 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0908367 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0918909 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106604 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0971721 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114531 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0964763 s, 10.9 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg255-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg255-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop localhost ost.OSS.ost_io.nrs_tbf_rule=stop clients ost.OSS.ost_io.nrs_tbf_rule=stop others 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.570491 s, 29.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0692378 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0764508 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101672 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0841692 s, 12.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175689 s, 6.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155252 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167689 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169148 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211321 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203564 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14919 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165806 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.254035 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.24766 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.236518 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.298161 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0591754 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0374114 s, 28.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0608894 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.068739 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0447455 s, 23.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0469979 s, 22.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0574709 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.063627 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0563335 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0692875 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0610143 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0378418 s, 27.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00475652 s, 220 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0754505 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0732953 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.078291 s, 13.4 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg255-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg255-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.389657 s, 43.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0758959 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11701 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0996158 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.217467 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188046 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.236105 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.251052 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220919 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.228067 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.246401 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.277613 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.26671 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.323143 s, 3.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8c4950f45000.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffff8c4950f45000.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.316159 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.34529 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0965748 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127319 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0446831 s, 23.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106236 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.414538 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0479514 s, 21.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0344825 s, 30.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.060411 s, 17.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0778428 s, 13.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117687 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0964163 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0537714 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133731 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0732784 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0862483 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0521359 s, 20.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0880348 s, 11.9 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg255-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg255-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' PASS 77e (21s) == sanityn test 77f: check TBF JobID nrs policy ========== 10:46:50 (1790779610) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start runas jobid={iozone.500 dd.500 tiotest.500} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start iozone_runas jobid={iozone.500} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg255-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.342887 s, 48.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0439826 s, 23.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0650204 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.075004 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100573 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123969 s, 8.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158285 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203202 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.204244 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147381 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.233759 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186956 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20193 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.259866 s, 4.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.272269 s, 3.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.233604 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.241088 s, 4.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115246 s, 9.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0918965 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.07063 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10815 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0576521 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0758068 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0668371 s, 15.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0824392 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0655523 s, 16.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0820252 s, 12.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0866432 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0890466 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.087554 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0944097 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0965047 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112984 s, 9.3 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8c4950f45000.lock_unused_count=2 removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg255-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg255-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change runas rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change iozone_runas rate=101 ost.OSS.ost_io.nrs_tbf_rule=change dd_runas rate=51 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg255-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.337877 s, 49.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.050977 s, 20.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0568855 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0630188 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0665028 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10395 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0880589 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157554 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150231 s, 7.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0502383 s, 20.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101671 s, 10.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.275874 s, 3.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.281538 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221345 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.24317 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.35876 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.388558 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120236 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0649312 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0731619 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113907 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0745858 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0953972 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0825957 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0811473 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0450118 s, 23.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00340692 s, 308 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00293091 s, 358 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0047682 s, 220 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10582 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111823 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109211 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183971 s, 57.0 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg255-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg255-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop runas ost.OSS.ost_io.nrs_tbf_rule=stop iozone_runas ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg255-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.412778 s, 40.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0603973 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0828046 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146482 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196963 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163759 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211791 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174919 s, 6.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied ldlm.namespaces.lustre-OST0000-osc-ffff8c4950f45000.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172034 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.313221 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.268985 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.287585 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.300636 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.36165 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.330708 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.370131 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.485636 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0328699 s, 31.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132175 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0394243 s, 26.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.265571 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119469 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0808517 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.264606 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0626854 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0945042 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0689441 s, 15.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.063056 s, 16.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0756339 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.30299 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0678287 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0675712 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.284766 s, 3.7 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg255-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg255-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_policies=fifo running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg255-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.626476 s, 26.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0318479 s, 32.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103792 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116082 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166459 s, 6.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.204728 s, 5.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211082 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.228168 s, 4.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8c4950f45000.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120161 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201022 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166799 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.243139 s, 4.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.330449 s, 3.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.357118 s, 2.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.397877 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.318586 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.320768 s, 3.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8c4950f45000.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff8c4950f45000.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0995337 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0492046 s, 21.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00542667 s, 193 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0551513 s, 19.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0922661 s, 11.4 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0544704 s, 19.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0931013 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0601692 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114942 s, 9.1 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0665728 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114675 s, 9.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115157 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0704804 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0718549 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0696908 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0755029 s, 13.9 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg255-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg255-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' PASS 77f (21s) == sanityn test 77g: Change TBF type directly ============ 10:47:11 (1790779631) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 ost.OSS.ost_io.nrs_policies=fifo PASS 77g (15s) == sanityn test 77h: Wrong policy name should report error, not LBUG ========================================================== 10:47:26 (1790779646) oleg255-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=abc: No such file or directory oleg255-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='abc': No such file or directory pdsh@oleg255-client: oleg255-server: ssh exited with exit code 2 oleg255-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc: Invalid argument oleg255-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc': Invalid argument pdsh@oleg255-client: oleg255-server: ssh exited with exit code 22 oleg255-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf reg abc: Invalid argument oleg255-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf reg abc': Invalid argument pdsh@oleg255-client: oleg255-server: ssh exited with exit code 22 oleg255-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc efg: Invalid argument oleg255-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc efg': Invalid argument pdsh@oleg255-client: oleg255-server: ssh exited with exit code 22 PASS 77h (15s) == sanityn test 77i: Change rank of TBF rule ============= 10:47:41 (1790779661) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start before jobid={jobid} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start after jobid={jobid} rate=1000 rank=default ost.OSS.ost_io.nrs_tbf_rule=start target jobid={jobid} rate=1000 rank=after Move before itself ost.OSS.ost_io.nrs_tbf_rule=change target rank=target Move to higher rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=before Move to lower rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=after Move before default ost.OSS.ost_io.nrs_tbf_rule=change target rank=default ost.OSS.ost_io.nrs_policies=fifo PASS 77i (23s) == sanityn test 77j: check TBF-OPCode NRS policy ========= 10:48:04 (1790779684) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf opcode ost.OSS.ost_io.nrs_tbf_rule=start ost_r opcode={ost_read} rate=5 ost.OSS.ost_io.nrs_tbf_rule=start ost_w opcode={ost_write} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.520667 s, 32.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0802245 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142309 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.233543 s, 4.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.240485 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.270707 s, 3.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222285 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.268279 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207927 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.253032 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.214384 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177673 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.298529 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.26883 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.234248 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.283395 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120637 s, 8.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0334107 s, 31.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0511411 s, 20.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0388964 s, 27.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0689141 s, 15.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0823344 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0525468 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0619818 s, 16.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0869208 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0723607 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0569666 s, 18.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0743283 s, 14.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0569376 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0623941 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0955888 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0766185 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0954045 s, 11.0 MB/s removed '/mnt/lustre/d77j.sanityn/nrs_r_oleg255-client.virtnet' removed '/mnt/lustre/d77j.sanityn/nrs_w_oleg255-client.virtnet' removed directory '/mnt/lustre/d77j.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 5 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.65716 s, 15.8 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.5701 s, 5.4 MB/s Read runtime is 20 s, speed is 5.000000 IOPS cleanup_tbf /mnt/lustre/d77j.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_r ost.OSS.ost_io.nrs_tbf_rule=stop ost_w ost.OSS.ost_io.nrs_policies=fifo PASS 77j (65s) == sanityn test 77ja: check TBF-UID/GID NRS policy ======= 10:49:09 (1790779749) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg255-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg255-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg255-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg255-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg255-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.31046 s, 12.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0377232 s, 27.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0792574 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114648 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112233 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14384 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209019 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121418 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.236766 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177731 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202723 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169747 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.254757 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199775 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195195 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.232281 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181125 s, 5.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0646847 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0391898 s, 26.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0715482 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0537644 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0529346 s, 19.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.076425 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0649761 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0660618 s, 15.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0787425 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0712314 s, 14.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106281 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0768908 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103583 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0993059 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103403 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178724 s, 5.9 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg255-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg255-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9359 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.999 s, 5.2 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg255-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg255-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg255-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg255-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg255-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.32698 s, 12.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0933399 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109061 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11534 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166401 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196814 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0561823 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206308 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210081 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167634 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.324819 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.243951 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209859 s, 5.0 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196555 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.234634 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185769 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.252764 s, 4.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0950317 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0967784 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0972814 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0876323 s, 12.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0394284 s, 26.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0489961 s, 21.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0654227 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0513627 s, 20.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121112 s, 8.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0596327 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138879 s, 7.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0624087 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.077382 s, 13.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0851521 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0735894 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0787949 s, 13.3 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg255-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg255-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8645 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.0074 s, 5.2 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77ja (153s) == sanityn test 77jb: check TBF-UID/GID NRS policy on files that don't belong to us ========================================================== 10:51:42 (1790779902) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg255-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg255-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg255-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg255-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg255-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.28947 s, 13.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0952087 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126996 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136709 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209883 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.233535 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.231628 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186285 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.265884 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.26191 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.231648 s, 4.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.297963 s, 3.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.234928 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.373535 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.367456 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.434561 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.487734 s, 2.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0737328 s, 14.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0854302 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.019034 s, 55.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104911 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0551586 s, 19.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0568656 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123247 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0893119 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0894989 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158407 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0941523 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169077 s, 6.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116487 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123449 s, 8.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124938 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151892 s, 6.9 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg255-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg255-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8937 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9649 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg255-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg255-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg255-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg255-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg255-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.33325 s, 12.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135382 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1697 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183599 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.274037 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195913 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.262071 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.309562 s, 3.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.218785 s, 4.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.310796 s, 3.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.234416 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.230738 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196065 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.383551 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.315292 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.334421 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.277833 s, 3.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0851847 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.092407 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0948226 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0909618 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101585 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0999233 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113479 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114409 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106108 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122152 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115276 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0948417 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140298 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0981065 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121025 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135083 s, 7.8 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg255-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg255-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.1175 s, 5.2 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9663 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77jb (158s) == sanityn test 77k: check TBF policy with UID/GID/JobID/OPCode expression ========================================================== 10:54:20 (1790780060) ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg255-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.614454 s, 27.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0820033 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.075092 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144271 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167568 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20292 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198382 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237748 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141916 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206641 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.366535 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.381503 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.364164 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.440969 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.259387 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.362501 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.466832 s, 2.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8c4950f45000.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff8c4950f45000.lock_unused_count=1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0471388 s, 22.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0331509 s, 31.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0243503 s, 43.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0648801 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0425173 s, 24.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0433199 s, 24.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0343822 s, 30.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0376923 s, 27.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10824 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0459246 s, 22.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0485364 s, 21.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0562191 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0711906 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0103176 s, 102 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8c4950f45000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0706335 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0868635 s, 12.1 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg255-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg255-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.80991 s, 11.9 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0027 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.202.55@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.202.55@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.508395 s, 33.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0781931 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0664087 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0706863 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150954 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201277 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106038 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195847 s, 5.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0994492 s, 10.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.239265 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.235512 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177762 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13933 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.251766 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13689 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0606643 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151144 s, 6.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8c4950f45000.lock_unused_count=3 ldlm.namespaces.lustre-OST0001-osc-ffff8c4950f45000.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140518 s, 7.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143115 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161633 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124475 s, 8.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12478 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130103 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154164 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123814 s, 8.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170052 s, 6.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136224 s, 7.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133434 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166929 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138634 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144396 s, 7.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148624 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155031 s, 6.8 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8c4950f45000.lock_unused_count=5 removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg255-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg255-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.99291 s, 15.0 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.93033 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.202.55@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg255-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.555644 s, 30.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0982443 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108269 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126724 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.04984 s, 21.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103045 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102199 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106421 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101049 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105058 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.30634 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.37063 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121705 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.379647 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146807 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.278797 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158833 s, 6.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0562219 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0945948 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0496812 s, 21.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00720681 s, 145 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0119505 s, 87.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00560242 s, 187 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00930997 s, 113 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0044966 s, 233 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00338447 s, 310 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00878534 s, 119 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0107122 s, 97.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132065 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00662145 s, 158 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139346 s, 75.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0663679 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.071855 s, 14.6 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg255-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg255-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.39014 s, 14.2 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.52758 s, 19.0 MB/s Read runtime is 7 s, speed is 14.285714 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg255-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.638448 s, 26.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.083435 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0696724 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0861027 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180383 s, 5.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21289 s, 4.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237464 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210521 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.292357 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.258082 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.246638 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.303758 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.336642 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.335445 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.311485 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.359383 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.355889 s, 2.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.040354 s, 26.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0694683 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.07127 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0333152 s, 31.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0344829 s, 30.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0409419 s, 25.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0458171 s, 22.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0486677 s, 21.5 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0483779 s, 21.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0768827 s, 13.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.058221 s, 18.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0646816 s, 16.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0694734 s, 15.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0767613 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0847268 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0855507 s, 12.3 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg255-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg255-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.9218 s, 9.6 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.9757 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.65633 s, 13.7 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.94489 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg255-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.503819 s, 33.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.041468 s, 25.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.056184 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0844376 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0502609 s, 20.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0751136 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.226408 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.243546 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.241742 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.243652 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221669 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.229589 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213066 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.277913 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.299858 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.293741 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.319393 s, 3.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0350033 s, 30.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0423803 s, 24.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0469304 s, 22.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0630291 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0404898 s, 25.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0422986 s, 24.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0365445 s, 28.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0417301 s, 25.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.04404 s, 23.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0492866 s, 21.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0617288 s, 17.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0673229 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0586201 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0692591 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0519645 s, 20.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0631975 s, 16.6 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg255-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg255-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9529 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.7314 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_ug ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg255-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.468633 s, 35.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0852012 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0756231 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0962365 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137898 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18364 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131976 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223967 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17397 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223526 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.248779 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242593 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.217107 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203648 s, 5.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209604 s, 5.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188544 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213043 s, 4.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0678072 s, 15.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0517919 s, 20.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0734317 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0433252 s, 24.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.081517 s, 12.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0541154 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0464659 s, 22.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0555878 s, 18.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0540474 s, 19.4 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0605735 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0554908 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0606039 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0766617 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0616829 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0666087 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0651336 s, 16.1 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg255-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg255-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.79867 s, 15.4 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.91216 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg255-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.879749 s, 19.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0970364 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15131 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126473 s, 8.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136364 s, 7.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186416 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180979 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.342293 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.330761 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.375576 s, 2.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.324648 s, 3.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.351717 s, 3.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.269975 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.422429 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.373981 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.396807 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.40959 s, 2.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0566639 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0495054 s, 21.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0234858 s, 44.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0465305 s, 22.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0577857 s, 18.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00486322 s, 216 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0579555 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.059881 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0658749 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0464194 s, 22.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0499195 s, 21.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0825557 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0762799 s, 13.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0631114 s, 16.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109028 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0836306 s, 12.5 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg255-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg255-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.0632 s, 9.5 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.94245 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.35 s, 9.2 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.99689 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_policies=fifo PASS 77k (434s) == sanityn test 77kb: Check different granular TBF type combination: jobid+opcode ========================================================== 11:01:34 (1790780494) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/nrs_r_oleg255-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.476852 s, 35.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0715749 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0866453 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141463 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.231921 s, 4.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.258815 s, 4.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.2544 s, 4.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.279887 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.266186 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.245836 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220137 s, 4.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.274887 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.27409 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.30478 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.281802 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.271439 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.316519 s, 3.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00240206 s, 437 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0713701 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00237765 s, 441 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0346277 s, 30.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0885394 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0371586 s, 28.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0381691 s, 27.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0114128 s, 91.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0602148 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0107398 s, 97.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00687179 s, 153 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.024177 s, 43.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.055353 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0706796 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0787902 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0601851 s, 17.4 MB/s removed '/mnt/lustre/d77kb.sanityn/nrs_w_oleg255-client.virtnet' removed '/mnt/lustre/d77kb.sanityn/nrs_r_oleg255-client.virtnet' removed directory '/mnt/lustre/d77kb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.53233 s, 16.1 MB/s Write runtime is 7 s, speed is 14.285714 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.85679 s, 10.6 MB/s Read runtime is 10 s, speed is 10.000000 IOPS cleanup_tbf /mnt/lustre/d77kb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kb (48s) == sanityn test 77kc: Check different granular TBF type combination: nid+opcode ========================================================== 11:02:22 (1790780542) ost.OSS.ost_io.nrs_policies=tbf nid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.202.55@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.202.55@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.689534 s, 24.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0898496 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0787091 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13238 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0889242 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0935588 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0710792 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166819 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0943307 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185938 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223303 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.266886 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.287435 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.224892 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.345432 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.353659 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.298252 s, 3.5 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8c4950f45000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0736396 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0558319 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0965126 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110433 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0879794 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0666591 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0687045 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0691222 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0878331 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0811998 s, 12.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0763803 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0887844 s, 11.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122044 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0928467 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111831 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104615 s, 10.0 MB/s removed '/mnt/lustre/d77kc.sanityn/nrs_w_oleg255-client.virtnet' removed '/mnt/lustre/d77kc.sanityn/nrs_r_oleg255-client.virtnet' removed directory '/mnt/lustre/d77kc.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.27457 s, 14.4 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.97097 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kc.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kc (51s) == sanityn test 77kd: Check different granular TBF type combination: nid+jobid ========================================================== 11:03:13 (1790780593) ost.OSS.ost_io.nrs_policies=tbf nid+jobid ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.202.55@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/nrs_r_oleg255-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.637079 s, 26.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0815533 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0493044 s, 21.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0969169 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.069252 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192543 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237064 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192083 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.307097 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.277762 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23789 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.298502 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.318922 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.298521 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.296543 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.337917 s, 3.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8c4950f45000.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff8c4950f45000.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.391831 s, 2.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0341007 s, 30.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0395772 s, 26.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0607207 s, 17.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.032371 s, 32.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0423666 s, 24.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0551405 s, 19.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0850208 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0427576 s, 24.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101416 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0642135 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0545289 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.068382 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0664021 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0638841 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0649989 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0690993 s, 15.2 MB/s removed '/mnt/lustre/d77kd.sanityn/nrs_w_oleg255-client.virtnet' removed '/mnt/lustre/d77kd.sanityn/nrs_r_oleg255-client.virtnet' removed directory '/mnt/lustre/d77kd.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.37388 s, 16.5 MB/s Write runtime is 7 s, speed is 14.285714 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kd.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.09597 s, 20.6 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kd.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kd (42s) == sanityn test 77ke: Check different granular TBF type combination: jobid+opcode ========================================================== 11:03:55 (1790780635) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/nrs_r_oleg255-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.696544 s, 24.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0597697 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0646646 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116458 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151043 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227337 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.289855 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.294229 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.300687 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.319867 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.31157 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.282036 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.336219 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.326439 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.328249 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.354303 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.374968 s, 2.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0193601 s, 54.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0570344 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0216477 s, 48.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0917803 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0397192 s, 26.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0433824 s, 24.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0443513 s, 23.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.057314 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.058298 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0517076 s, 20.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.07251 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0676528 s, 15.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0881103 s, 11.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0880575 s, 11.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165128 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178017 s, 5.9 MB/s removed '/mnt/lustre/d77ke.sanityn/nrs_r_oleg255-client.virtnet' removed '/mnt/lustre/d77ke.sanityn/nrs_w_oleg255-client.virtnet' removed directory '/mnt/lustre/d77ke.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.8055 s, 8.9 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ke.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.95988 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.07595 s, 11.6 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.93205 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77ke (96s) == sanityn test 77kf: Check different granular TBF type combination: uid+gid ========================================================== 11:05:31 (1790780731) ost.OSS.ost_io.nrs_policies=tbf uid+gid ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/nrs_r_oleg255-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.28043 s, 13.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0352625 s, 29.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0742248 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0777238 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0944553 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189033 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.255897 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180385 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222677 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.318705 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.293083 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.314091 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.277841 s, 3.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.339817 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.371249 s, 2.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.336931 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.341596 s, 3.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.055188 s, 19.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0394586 s, 26.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.050397 s, 20.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0776586 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0553889 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0557008 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0575039 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109806 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0675982 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0714787 s, 14.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.08342 s, 12.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0839254 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0895895 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0891881 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0994184 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102605 s, 10.2 MB/s removed '/mnt/lustre/d77kf.sanityn/nrs_r_oleg255-client.virtnet' removed '/mnt/lustre/d77kf.sanityn/nrs_w_oleg255-client.virtnet' removed directory '/mnt/lustre/d77kf.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.0156 s, 5.2 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kf.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.7422 s, 5.3 MB/s Read runtime is 20 s, speed is 5.000000 IOPS cleanup_tbf /mnt/lustre/d77kf.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kf (79s) == sanityn test 77kg: check different granular TBF type combination: uid+opcode ========================================================== 11:06:50 (1790780810) ost.OSS.ost_io.nrs_policies=tbf uid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg255-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.619306 s, 27.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0534499 s, 19.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0549641 s, 19.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0698947 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172722 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173275 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20186 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213417 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209638 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.26266 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.289229 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.275931 s, 3.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.271785 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.31267 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.309307 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.320562 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.328269 s, 3.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8c4950f45000.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff8c4950f45000.lock_unused_count=1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.069562 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0463334 s, 22.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0708605 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0462544 s, 22.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0253606 s, 41.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.053405 s, 19.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0329602 s, 31.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0580402 s, 18.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0318842 s, 32.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0684571 s, 15.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0747302 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0542516 s, 19.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0824306 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0817721 s, 12.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0875601 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0903738 s, 11.6 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg255-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg255-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.04616 s, 14.9 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.85882 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg255-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.942033 s, 17.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0892595 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105643 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.256362 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.292736 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.295039 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.325045 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.30841 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.342064 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.319732 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.357953 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.275919 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.298654 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.353467 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.26213 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.30877 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.338364 s, 3.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0576819 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0720093 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0256555 s, 40.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0484083 s, 21.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0567831 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0105528 s, 99.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0621573 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0884104 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0797526 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0717642 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0559518 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0803566 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0629038 s, 16.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0873997 s, 12.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0661241 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0932108 s, 11.2 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg255-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg255-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.873 s, 9.6 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0038 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.024 s, 9.5 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.93685 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kg (143s) == sanityn test 77kh: Verify that Project ID support for NRS TBF rule ========================================================== 11:09:13 (1790780953) using SAVE_PROJECT_SUPPORTED=0 Stopping clients: oleg255-client.virtnet /mnt/lustre (opts:) Stopping client oleg255-client.virtnet /mnt/lustre opts: Stopping clients: oleg255-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg255-client.virtnet /mnt/lustre2 opts: Stopping /mnt/lustre-mds1 (opts:-f) on oleg255-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg255-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg255-server sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime) proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) devtmpfs on /dev type devtmpfs (rw,nosuid,mode=755) securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime) tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev) devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000) tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755) tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755) cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd) pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime) bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700) cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio) cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma) cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids) cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct) cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer) cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory) cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices) cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event) cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio) cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset) cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb) cgroup on /sys/fs/cgroup/misc type cgroup (rw,nosuid,nodev,noexec,relatime,misc) none on /sys/kernel/tracing type tracefs (rw,relatime) configfs on /sys/kernel/config type configfs (rw,relatime) /dev/nbd0 on / type ext4 (ro,relatime,stripe=32) rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) mqueue on /dev/mqueue type mqueue (rw,relatime) debugfs on /sys/kernel/debug type debugfs (rw,relatime) hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M) none on /mnt type ramfs (rw,relatime) none on /var/lib/stateless/writable type tmpfs (rw,relatime) /dev/vda on /home/green/git/lustre-release type squashfs (ro,relatime) none on /var/cache/man type tmpfs (rw,relatime) none on /var/log type tmpfs (rw,relatime) none on /var/lib/dbus type tmpfs (rw,relatime) none on /tmp type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) none on /var/tmp type tmpfs (rw,relatime) none on /var/lib/NetworkManager type tmpfs (rw,relatime) none on /var/lib/systemd/random-seed type tmpfs (rw,relatime) none on /var/spool type tmpfs (rw,relatime) none on /var/lib/nfs type tmpfs (rw,relatime) none on /var/lib/gssproxy type tmpfs (rw,relatime) none on /var/lib/logrotate type tmpfs (rw,relatime) none on /etc type tmpfs (rw,relatime) none on /var/lib/rsyslog type tmpfs (rw,relatime) none on /root/.ssh type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) 192.168.200.253:/exports/state/oleg255-client.virtnet on /var/lib/stateless/state type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.202.55,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg255-client.virtnet/boot on /boot type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.202.55,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg255-client.virtnet/etc/kdump.conf on /etc/kdump.conf type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.202.55,local_lock=none,addr=192.168.200.253) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=378180k,mode=700) 192.168.200.253:/exports/testreports/71132/testresults/sanityn-zfs-rocky8.10_x86_64-rocky8.10_x86_64 on /tmp/testlogs type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.202.55,local_lock=none,addr=192.168.200.253) /dev/vda on /usr/sbin/mount.lustre type squashfs (ro,relatime) Checking servers environments Checking clients oleg255-client.virtnet environments Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg255-server' oleg255-server: oleg255-server.virtnet: executing load_modules_local oleg255-server: kmemleak disabled oleg255-server: Loading modules from /home/green/git/lustre-release/lustre oleg255-server: detected 4 online CPUs by sysfs oleg255-server: MODOPTS_LIBCFS= oleg255-server: Force libcfs to create 2 CPU partitions oleg255-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Setup mgs, mdt, osts Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg255-server: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0000 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg255-server: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-OST0000 Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg255-server: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-OST0001 Starting client: oleg255-client.virtnet: -o user_xattr,flock 192.168.202.155@tcp:/lustre /mnt/lustre Starting client oleg255-client.virtnet: -o user_xattr,flock 192.168.202.155@tcp:/lustre /mnt/lustre Started clients oleg255-client.virtnet: 192.168.202.155@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Starting client: oleg255-client.virtnet: -o user_xattr,flock 192.168.202.155@tcp:/lustre /mnt/lustre2 Starting client oleg255-client.virtnet: -o user_xattr,flock 192.168.202.155@tcp:/lustre /mnt/lustre2 Started clients oleg255-client.virtnet: 192.168.202.155@tcp:/lustre on /mnt/lustre2 type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8c4949be7800.idle_timeout=debug osc.lustre-OST0000-osc-ffff8c495929c800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8c4949be7800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8c495929c800.idle_timeout=debug disable quota as required ost.OSS.ost_io.nrs_policies=tbf projid ost.OSS.ost_io.nrs_tbf_rule=start ext_proj_rw projid={100} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.436964 s, 38.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0599336 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0800819 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0674685 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0821816 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153733 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187634 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206327 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186983 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.226984 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164976 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177119 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.318461 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.316119 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.275034 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.298596 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.328684 s, 3.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8c4949be7800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0587811 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0708378 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0559303 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0497254 s, 21.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0674083 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0337628 s, 31.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0506534 s, 20.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0732794 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.044856 s, 23.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.063568 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0677701 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0728406 s, 14.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0723969 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.080305 s, 13.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0808032 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0763035 s, 13.7 MB/s removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg255-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg255-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.64015 s, 10.9 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.59968 s, 18.7 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_proj_rw ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf projid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w projid={100}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r projid={100}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.24876 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1095 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132416 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.229074 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.290054 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.316275 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.292324 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.310631 s, 3.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.370654 s, 2.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.383189 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.335623 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.359208 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.389826 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.387004 s, 2.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8c4949be7800.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff8c4949be7800.lock_unused_count=3 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.465163 s, 2.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.407112 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.473751 s, 2.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118542 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140229 s, 7.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101364 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109269 s, 9.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117788 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140532 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120511 s, 8.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126641 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143118 s, 7.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134259 s, 7.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134586 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154918 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141272 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152682 s, 6.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158082 s, 6.6 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159193 s, 6.6 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8c4949be7800.lock_unused_count=4 removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg255-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg255-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.5225 s, 11.0 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0555 s, 10.4 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77kh (211s) == sanityn test 77ki: Add rule with unsupported TBF type should fail for generic TBF ========================================================== 11:12:45 (1790781165) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 oleg255-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_fail projid={100}&opcode={ost_write} rate=20: Invalid argument oleg255-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_fail projid={100}&opcode={ost_write} rate=20': Invalid argument pdsh@oleg255-client: oleg255-server: ssh exited with exit code 22 ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77ki (18s) == sanityn test 77kj: Verify nodemap support for NRS TBF rule ========================================================== 11:13:03 (1790781183) On MGS 192.168.202.155, active = nodemap.active=1 On MGS 192.168.202.155, TBF = ost.OSS.ost_io.nrs_policies=tbf nodemap ost.OSS.ost_io.nrs_tbf_rule=start ext_nm_rw nodemap={TBF} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.678158 s, 24.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0716363 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0401278 s, 26.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0866523 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14798 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15327 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.263349 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.301196 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.377597 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.396826 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.280154 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.482203 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.385255 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.48525 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.459917 s, 2.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8c4949be7800.lock_unused_count=5 ldlm.namespaces.lustre-OST0001-osc-ffff8c4949be7800.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.475386 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.362168 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209577 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.240002 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.253198 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.247046 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.292991 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.247821 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.254264 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.233124 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.219144 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223958 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.226081 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.240957 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.287736 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.245023 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.32952 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.30093 s, 3.5 MB/s removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg255-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg255-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.61958 s, 12.2 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.49233 s, 19.1 MB/s Read runtime is 7 s, speed is 14.285714 IOPS cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete On MGS 192.168.202.155, TBF.id = On MGS 192.168.202.155, active = nodemap.active=0 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.851084 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0585185 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.093045 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223575 s, 4.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22141 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196647 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1771 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175877 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211021 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.231249 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.246547 s, 4.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.268585 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.24683 s, 4.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8c4949be7800.lock_unused_count=5 ldlm.namespaces.lustre-OST0001-osc-ffff8c4949be7800.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208545 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.241311 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.322193 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.299655 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118484 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0613048 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0575387 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.064629 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0660404 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0850252 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102103 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0714042 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0811929 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117695 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0855671 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0911172 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124571 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0927129 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0952791 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100126 s, 10.5 MB/s removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg255-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg255-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 1 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.01262 s, 13.1 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.43603 s, 23.6 MB/s Read runtime is 5 s, speed is 20.000000 IOPS cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_nm_rw ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf nodemap+opcode oleg255-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_w nodemap={TBF}&opcode={ost_write} rate=20: Invalid argument oleg255-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_w nodemap={TBF}&opcode={ost_write} rate=20': Invalid argument oleg255-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_r nodemap={TBF}&opcode={ost_read} rate=10: Invalid argument oleg255-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_r nodemap={TBF}&opcode={ost_read} rate=10': Invalid argument pdsh@oleg255-client: oleg255-server: ssh exited with exit code 22 On MGS 192.168.202.155, active = nodemap.active=1 On MGS 192.168.202.155, TBF = ost.OSS.ost_io.nrs_tbf_rule=start ext_w nodemap={TBF}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nodemap={TBF}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.556939 s, 30.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0758832 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0884427 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167753 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208882 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199832 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195336 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210114 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.241075 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.29135 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.253589 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.298307 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.312038 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.320257 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.344157 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.362282 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.348846 s, 3.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8c4949be7800.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff8c4949be7800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0888388 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116117 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0485867 s, 21.6 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8c4949be7800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0064246 s, 163 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0100313 s, 105 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00243477 s, 431 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00232214 s, 452 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00281035 s, 373 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111225 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0805941 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0770467 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0998514 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0468822 s, 22.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0723464 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0392678 s, 26.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0695801 s, 15.1 MB/s removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg255-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg255-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.70914 s, 15.6 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.95618 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo On MGS 192.168.202.155, TBF.id = On MGS 192.168.202.155, active = nodemap.active=0 PASS 77kj (183s) == sanityn test 77l: check the output of NRS policies for generic TBF ========================================================== 11:16:06 (1790781366) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_policies=fifo PASS 77l (10s) == sanityn test 77m: check NRS Delay slows write RPC processing ========================================================== 11:16:17 (1790781377) ost.OSS.ost_io.nrs_policies=delay ost.OSS.ost_io.nrs_delay_min=4 ost.OSS.ost_io.nrs_delay_max=4 ost.OSS.ost_io.nrs_delay_pct=100 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg255-client.virtnet] [bs=1M] [count=1] [oflag=direct] [conv=fdatasync] 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 3.73917 s, 280 kB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg255-client.virtnet] [bs=1M] [count=10] [oflag=direct] [conv=fdatasync] 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 40.083 s, 262 kB/s ost.OSS.ost_io.nrs_policies=fifo PASS 77m (58s) == sanityn test 77n: check wildcard support for TBF JobID NRS policy ========================================================== 11:17:15 (1790781435) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={*.500} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.5967 s, 28.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0645125 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115222 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0511949 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0802381 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169974 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142562 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123962 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159568 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139258 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.249845 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.304426 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.288797 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.228866 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0877934 s, 11.9 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8c4949be7800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.303251 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161119 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0925674 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00414627 s, 253 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8c4949be7800.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0573864 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115669 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00453773 s, 231 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110196 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0117099 s, 89.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0643868 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125085 s, 83.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0453418 s, 23.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0920082 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132703 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135614 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137774 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.087068 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103426 s, 10.1 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg255-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg255-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77n.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.81937 s, 11.9 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77n.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.40931 s, 19.4 MB/s Read runtime is 7 s, speed is 14.285714 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.*} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.564302 s, 29.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0563677 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112884 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.097948 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206165 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130865 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203485 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.214542 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210128 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220774 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.28241 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.250971 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.296817 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.2613 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.268309 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.304515 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.351765 s, 3.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8c4949be7800.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff8c4949be7800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115243 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00846399 s, 124 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133839 s, 78.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133651 s, 78.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0542798 s, 19.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0557598 s, 18.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8c4949be7800.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0113487 s, 92.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164302 s, 63.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.021252 s, 49.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0246497 s, 42.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131241 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18962 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202154 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201195 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00812882 s, 129 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0293703 s, 35.7 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg255-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg255-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.46194 s, 14.1 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.3228 s, 19.7 MB/s Read runtime is 7 s, speed is 14.285714 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_policies=fifo PASS 77n (96s) == sanityn test 77o: Changing rank should not panic ====== 11:18:52 (1790781532) mds.MDS.mdt.nrs_policies=tbf nid mds.MDS.mdt.nrs_tbf_rule=start name nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=start name1 nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=change name1 rank=name mds.MDS.mdt.nrs_tbf_rule=stop name mds.MDS.mdt.nrs_policies=fifo PASS 77o (12s) == sanityn test 77q: Parallel TBF rule definitions should not panic ========================================================== 11:19:05 (1790781545) mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_policies=fifo PASS 77q (128s) == sanityn test 77p: Check validity of rule names for TBF policies ========================================================== 11:21:13 (1790781673) mds.MDS.mdt.nrs_policies=tbf oleg255-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_overflo uid={500} rate=500: File name too long oleg255-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_overflo uid={500} rate=500': File name too long pdsh@oleg255-client: oleg255-server: ssh exited with exit code 36 oleg255-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start uid={500} rate=500: Invalid argument oleg255-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start uid={500} rate=500': Invalid argument pdsh@oleg255-client: oleg255-server: ssh exited with exit code 22 oleg255-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_empty: Invalid argument oleg255-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_empty': Invalid argument pdsh@oleg255-client: oleg255-server: ssh exited with exit code 22 oleg255-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p@spec uid={500} rate=500: Invalid argument oleg255-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p@spec uid={500} rate=500': Invalid argument pdsh@oleg255-client: oleg255-server: ssh exited with exit code 22 oleg255-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p.spec uid={500} rate=500: Invalid argument oleg255-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p.spec uid={500} rate=500': Invalid argument pdsh@oleg255-client: oleg255-server: ssh exited with exit code 22 oleg255-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p~spec uid={500} rate=500: Invalid argument oleg255-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p~spec uid={500} rate=500': Invalid argument pdsh@oleg255-client: oleg255-server: ssh exited with exit code 22 oleg255-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p#spec uid={500} rate=500: Invalid argument oleg255-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p#spec uid={500} rate=500': Invalid argument pdsh@oleg255-client: oleg255-server: ssh exited with exit code 22 oleg255-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p/spec uid={500} rate=500: Invalid argument oleg255-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p/spec uid={500} rate=500': Invalid argument pdsh@oleg255-client: oleg255-server: ssh exited with exit code 22 oleg255-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p^spec uid={500} rate=500: Invalid argument oleg255-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p^spec uid={500} rate=500': Invalid argument pdsh@oleg255-client: oleg255-server: ssh exited with exit code 22 oleg255-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p%spec uid={500} rate=500: Invalid argument oleg255-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p%spec uid={500} rate=500': Invalid argument pdsh@oleg255-client: oleg255-server: ssh exited with exit code 22 oleg255-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p*spec uid={500} rate=500: Invalid argument oleg255-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p*spec uid={500} rate=500': Invalid argument pdsh@oleg255-client: oleg255-server: ssh exited with exit code 22 oleg255-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p;spec uid={500} rate=500: Invalid argument oleg255-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p;spec uid={500} rate=500': Invalid argument pdsh@oleg255-client: oleg255-server: ssh exited with exit code 22 oleg255-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p,spec uid={500} rate=500: Invalid argument oleg255-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p,spec uid={500} rate=500': Invalid argument pdsh@oleg255-client: oleg255-server: ssh exited with exit code 22 oleg255-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p?spec uid={500} rate=500: Invalid argument oleg255-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p?spec uid={500} rate=500': Invalid argument pdsh@oleg255-client: oleg255-server: ssh exited with exit code 22 oleg255-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77pspec uid={500} rate=500: Invalid argument oleg255-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p>spec uid={500} rate=500': Invalid argument pdsh@oleg255-client: oleg255-server: ssh exited with exit code 22 oleg255-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p:spec uid={500} rate=500: Invalid argument oleg255-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p:spec uid={500} rate=500': Invalid argument pdsh@oleg255-client: oleg255-server: ssh exited with exit code 22 oleg255-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p+spec uid={500} rate=500: Invalid argument oleg255-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p+spec uid={500} rate=500': Invalid argument pdsh@oleg255-client: oleg255-server: ssh exited with exit code 22 oleg255-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p=spec uid={500} rate=500: Invalid argument oleg255-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p=spec uid={500} rate=500': Invalid argument pdsh@oleg255-client: oleg255-server: ssh exited with exit code 22 oleg255-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p)spec uid={500} rate=500: Invalid argument oleg255-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p)spec uid={500} rate=500': Invalid argument pdsh@oleg255-client: oleg255-server: ssh exited with exit code 22 oleg255-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p(spec uid={500} rate=500: Invalid argument oleg255-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p(spec uid={500} rate=500': Invalid argument pdsh@oleg255-client: oleg255-server: ssh exited with exit code 22 oleg255-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p{spec uid={500} rate=500: Invalid argument oleg255-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p{spec uid={500} rate=500': Invalid argument pdsh@oleg255-client: oleg255-server: ssh exited with exit code 22 oleg255-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p}spec uid={500} rate=500: Invalid argument oleg255-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p}spec uid={500} rate=500': Invalid argument pdsh@oleg255-client: oleg255-server: ssh exited with exit code 22 oleg255-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p|spec uid={500} rate=500: Invalid argument oleg255-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p|spec uid={500} rate=500': Invalid argument pdsh@oleg255-client: oleg255-server: ssh exited with exit code 22 oleg255-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p[spec uid={500} rate=500: Invalid argument oleg255-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p[spec uid={500} rate=500': Invalid argument pdsh@oleg255-client: oleg255-server: ssh exited with exit code 22 oleg255-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p]spec uid={500} rate=500: Invalid argument oleg255-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p]spec uid={500} rate=500': Invalid argument pdsh@oleg255-client: oleg255-server: ssh exited with exit code 22 oleg255-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p!spec uid={500} rate=500: Invalid argument oleg255-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p!spec uid={500} rate=500': Invalid argument pdsh@oleg255-client: oleg255-server: ssh exited with exit code 22 oleg255-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p&spec uid={500} rate=500: Invalid argument oleg255-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p&spec uid={500} rate=500': Invalid argument pdsh@oleg255-client: oleg255-server: ssh exited with exit code 22 oleg255-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p$spec uid={500} rate=500: Invalid argument oleg255-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p$spec uid={500} rate=500': Invalid argument pdsh@oleg255-client: oleg255-server: ssh exited with exit code 22 oleg255-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p`spec uid={500} rate=500: Invalid argument oleg255-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p`spec uid={500} rate=500': Invalid argument pdsh@oleg255-client: oleg255-server: ssh exited with exit code 22 oleg255-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p\spec uid={500} rate=500: Invalid argument oleg255-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p\spec uid={500} rate=500': Invalid argument pdsh@oleg255-client: oleg255-server: ssh exited with exit code 22 mds.MDS.mdt.nrs_policies=fifo PASS 77p (41s) == sanityn test 77r: Change type of tbf policy at run time ========================================================== 11:21:54 (1790781714) mds.MDS.mdt.nrs_policies=tbf uid mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf gid mds.MDS.mdt.nrs_tbf_rule=start md_rule gid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start md_rule jobid={*.500} rate=1 mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=fifo check the number of requests in queue: fifo: 0 crrn: 0 tbf: 0 delay: 0 fifo: 0 crrn: 0 tbf: 0 delay: 0 cleanup 77r 272900 kill md_thread xargs PASS 77r (59s) == sanityn test 77s: Check TBF LRU shrinker ============== 11:22:53 (1790781773) jobid_name=%e.%p jobid_var=TEST_77s_JOBVAR mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start touch jobid={touch.*} rate=1000 mds.MDS.mdt.nrs_tbf_rule=start unlink jobid={unlink.*} rate=1000 touch jobid={touch.*} 1000, ref 10 touch jobid={touch.*} 1000, ref 0 unlink jobid={unlink.*} 1000, ref 7 unlink jobid={unlink.*} 1000, ref 0 mds.MDS.mdt.nrs_policies=fifo jobid_var=procname_uid jobid_name=%e.%u PASS 77s (77s) == sanityn test 78: Enable policy and specify tunings right away ========================================================== 11:24:11 (1790781851) oleg255-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_orr_quantum=1: No such device oleg255-server: error: set_param: setting 'ost/OSS/*/nrs_orr_quantum'='1': No such device pdsh@oleg255-client: oleg255-server: ssh exited with exit code 19 ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_policies=fifo PASS 78 (10s) == sanityn test 79: xattr: intent error ================== 11:24:21 (1790781861) File: /mnt/lustre/d79.sanityn Size: 16384 Blocks: 2 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115238826935647 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-30 11:24:24.000000000 -0400 Modify: 2026-09-30 11:24:24.000000000 -0400 Change: 2026-09-30 11:24:24.000000000 -0400 Birth: 2026-09-30 11:24:24.000000000 -0400 fail_val=0 fail_loc=0x80000160 fail_val=0 fail_loc=0x80000131 # file: mnt/lustre/d79.sanityn trusted.name1="value1" PASS 79 (19s) == sanityn test 80a: migrate directory when some children is being opened ========================================================== 11:24:40 (1790781880) SKIP: sanityn test_80a needs >= 2 MDTs SKIP 80a (4s) == sanityn test 80b: Accessing directory during migration ========================================================== 11:24:44 (1790781884) SKIP: sanityn test_80b needs >= 2 MDTs SKIP 80b (5s) == sanityn test 81a: rename and stat under striped directory ========================================================== 11:24:49 (1790781889) SKIP: sanityn test_81a needs >= 2 MDTs SKIP 81a (4s) == sanityn test 81b: rename under striped directory doesn't deadlock ========================================================== 11:24:53 (1790781893) SKIP: sanityn test_81b We need at least 2 MDTs for this test SKIP 81b (5s) == sanityn test 81c: rename revoke LOOKUP lock for remote object ========================================================== 11:24:58 (1790781898) SKIP: sanityn test_81c needs >= 4 MDTs SKIP 81c (4s) == sanityn test 81d: parallel rename file cross-dir on same MDT ========================================================== 11:25:02 (1790781902) - open/close 972 (time 1790781919.59 total 10.01 last 97.12) - open/close 1869 (time 1790781929.59 total 20.02 last 89.63) total: 2500 open/close in 27.05 seconds: 92.42 ops/second snapshot_time 1790782020.862713454 secs.nsecs start_time 1790781906.791519526 secs.nsecs elapsed_time 114.071193928 secs.nsecs open 2505 samples [usecs] 139 14399 3201185 6096044387 close 2505 samples [usecs] 131 15641 845491 667922679 mknod 2501 samples [usecs] 479 14256 2810996 4722465360 unlink 2500 samples [usecs] 717 105922 6064546 81053053216 mkdir 2 samples [usecs] 999 5201 6200 28048402 rmdir 2 samples [usecs] 959 8207 9166 68274530 rename 2500 samples [usecs] 477 51645 15565999 200299678855 getattr 3762 samples [usecs] 7 11804 251573 602888555 parallel_rename_file 2500 samples [usecs] 477 51645 15565999 200299678855 crossdir_rename 2500 samples [usecs] 477 51645 15565999 200299678855 readpage 28 samples [usecs] 6579 89036 742441 26809835071 PASS 81d (125s) == sanityn test 82: fsetxattr and fgetxattr on orphan files ========================================================== 11:27:08 (1790782028) multiop /mnt/lustre/f82.sanityn vO_ac TMPPIPE=/tmp/multiop_open_wait_pipe.7627 multiop /mnt/lustre2/f82.sanityn vo_Ac TMPPIPE=/tmp/multiop_open_wait_pipe.7627 PASS 82 (9s) == sanityn test 83: access striped directory while it is being created/unlinked ========================================================== 11:27:16 (1790782036) SKIP: sanityn test_83 needs >= 2 MDTs SKIP 83 (3s) == sanityn test 84: 0-nlink race in lu_object_find() ===== 11:27:20 (1790782040) fail_loc=0x8000060b PASS 84 (14s) == sanityn test 85: Lustre API root cache race =========== 11:27:33 (1790782053) Starting: llapi_root_test /mnt/lustre/f85.sanityn /mnt/lustre2/f85.sanityn Starting test test1 at 1790782055 Finishing test test1 at 1790782060 PASS 85 (13s) == sanityn test 90: open/create and unlink striped directory ========================================================== 11:27:46 (1790782066) SKIP: sanityn test_90 needs >= 2 MDTs SKIP 90 (4s) == sanityn test 91: chmod and unlink striped directory === 11:27:50 (1790782070) SKIP: sanityn test_91 needs >= 2 MDTs SKIP 91 (4s) == sanityn test 92: create remote directory under orphan directory ========================================================== 11:27:54 (1790782074) SKIP: sanityn test_92 needs >= 2 MDTs SKIP 92 (4s) == sanityn test 93: alloc_rr should not allocate on same ost ========================================================== 11:27:58 (1790782078) fail_loc=0x00000163 fail_loc=0x0 /mnt/lustre/f93.sanityn-1/file1 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 4776 0x12a8 0x240000400 1 4737 0x1281 0x280000400 /mnt/lustre2/f93.sanityn-2/file2 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 4738 0x1282 0x280000400 0 4777 0x12a9 0x240000400 rc1=0 and rc2=0 PASS 93 (18s) == sanityn test 94: signal vs CP callback race =========== 11:28:16 (1790782096) osc.lustre-OST0000-osc-ffff8c4949be7800.idle_timeout=0 osc.lustre-OST0000-osc-ffff8c495929c800.idle_timeout=0 osc.lustre-OST0001-osc-ffff8c4949be7800.idle_timeout=0 osc.lustre-OST0001-osc-ffff8c495929c800.idle_timeout=0 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.315647 s, 26.0 kB/s fail_val=5 fail_loc=0x80000312 fail_val=6 fail_loc=0x80000329 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 3.01223 s, 1.4 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 6395: 289668 Alarm clock dd if=/dev/zero of=$DIR/$tfile conv=notrunc oflag=append bs=4k count=1 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.106068 s, 38.6 kB/s osc.lustre-OST0000-osc-ffff8c4949be7800.idle_timeout=debug osc.lustre-OST0000-osc-ffff8c495929c800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8c4949be7800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8c495929c800.idle_timeout=debug PASS 94 (17s) == sanityn test 95a: Check readpage() on a page that was removed from page cache ========================================================== 11:28:34 (1790782114) llite.lustre-ffff8c4949be7800.fast_read=0 llite.lustre-ffff8c495929c800.fast_read=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0394526 s, 53.2 MB/s fail_loc=0x80001421 fail_val=7 fail_loc=0x80001422 fail_val=10 llite.lustre-ffff8c4949be7800.fast_read=1 llite.lustre-ffff8c495929c800.fast_read=1 PASS 95a (21s) == sanityn test 95b: Check readpage() on a page that is no longer uptodate ========================================================== 11:28:55 (1790782135) llite.lustre-ffff8c4949be7800.fast_read=0 llite.lustre-ffff8c495929c800.fast_read=0 1+0 records in 1+0 records out 12288 bytes (12 kB, 12 KiB) copied, 0.00930119 s, 1.3 MB/s fail_loc=0x80001424 fail_val=5 fail_loc=0 llite.lustre-ffff8c4949be7800.fast_read=1 llite.lustre-ffff8c495929c800.fast_read=1 PASS 95b (15s) == sanityn test 100a: DoM: glimpse RPCs for stat without IO lock (DoM only file) ========================================================== 11:29:10 (1790782150) SKIP: sanityn test_100a Reserved for glimpse-ahead SKIP 100a (4s) == sanityn test 100b: DoM: no glimpse RPC for stat with IO lock (DoM only file) ========================================================== 11:29:14 (1790782154) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00529243 s, 774 kB/s /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK PASS 100b (7s) == sanityn test 100c: DoM: write vs stat without IO lock (combined file) ========================================================== 11:29:21 (1790782161) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.117836 s, 17.8 MB/s /mnt/lustre/d100c.sanityn/dom has type file OK /mnt/lustre/d100c.sanityn/dom has size 2097152 OK PASS 100c (7s) == sanityn test 100d: DoM: write+truncate vs stat without IO lock (combined file) ========================================================== 11:29:28 (1790782168) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.129109 s, 16.2 MB/s /mnt/lustre/d100d.sanityn/dom has type file OK /mnt/lustre/d100d.sanityn/dom has size 4096 OK PASS 100d (8s) == sanityn test 100e: DoM: read on open and file size ==== 11:29:36 (1790782176) 1+0 records in 1+0 records out 12000 bytes (12 kB, 12 KiB) copied, 0.00417146 s, 2.9 MB/s PASS 100e (8s) == sanityn test 101a: Discard DoM data on unlink ========= 11:29:44 (1790782184) /mnt/lustre/f101a.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00828514 s, 494 kB/s PASS 101a (11s) == sanityn test 101b: Discard DoM data on rename ========= 11:29:55 (1790782195) /mnt/lustre/f101b.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0144041 s, 284 kB/s PASS 101b (10s) == sanityn test 101c: Discard DoM data on close-unlink === 11:30:05 (1790782205) /mnt/lustre/f101c.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0109099 s, 375 kB/s PASS 101c (10s) == sanityn test 102a: Test open by handle of unlinked file ========================================================== 11:30:15 (1790782215) file: /mnt/lustre/f102a.sanityn fh_bytes: 32 fh_type: 151 fh_data: d1 0b 00 00 02 00 00 00 40 0f 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 file's parent FID is [0:0x0:0x0] file FID is [0x200000bd1:0xf40:0x0] access via mount point '/mnt/lustre2' - OK access after unlink - OK check_fhandle_syscalls test Passed! PASS 102a (9s) == sanityn test 102b: open by handle of a dir inside an encrypted dir ========================================================== 11:30:25 (1790782225) 192.168.202.155@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg255-client.virtnet /mnt/lustre2 (opts:) 192.168.202.155@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg255-client.virtnet /mnt/lustre (opts:) Starting client: oleg255-client.virtnet: -o user_xattr,flock,test_dummy_encryption 192.168.202.155@tcp:/lustre /mnt/lustre Starting client: oleg255-client.virtnet: -o user_xattr,flock,test_dummy_encryption 192.168.202.155@tcp:/lustre /mnt/lustre2 file: /mnt/lustre/d102b.sanityn/encdir/sub fh_bytes: 32 fh_type: 151 fh_data: d3 0b 00 00 02 00 00 00 03 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 file's parent FID is [0:0x0:0x0] file FID is [0x200000bd3:0x3:0x0] access via mount point '/mnt/lustre2' - OK check_fhandle_syscalls test Passed! 192.168.202.155@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,test_dummy_encryption=%s,test_dummy_encryption,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg255-client.virtnet /mnt/lustre (opts:) Starting client: oleg255-client.virtnet: -o user_xattr,flock 192.168.202.155@tcp:/lustre /mnt/lustre 192.168.202.155@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,test_dummy_encryption=%s,test_dummy_encryption,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg255-client.virtnet /mnt/lustre2 (opts:) Starting client: oleg255-client.virtnet: -o user_xattr,flock 192.168.202.155@tcp:/lustre /mnt/lustre2 PASS 102b (23s) == sanityn test 103: Test size correctness with lockahead ========================================================== 11:30:47 (1790782247) fail_loc=0x415 fail_loc=0x415 fail_val=2 Incorrect size expected (no glimpse fix): Starting test test23 at 1790782250 lockahead_test: lockahead_test.c:1171: test23: assertion 'sb.st_size == sb2.st_size' failed: size on /mnt/lustre/d103.sanityn/f103.sanityn and /mnt/lustre2/d103.sanityn/f103.sanityn differs: 0 vs 1048576 fail_loc=0 fail_loc=0x214 fail_val=2 Starting test test23 at 1790782253 Finishing test test23 at 1790782254 PASS 103 (14s) == sanityn test 104: Verify that MDS stores atime/mtime/ctime during close ========================================================== 11:31:02 (1790782262) SKIP: sanityn test_104 ldiskfs only test SKIP 104 (5s) == sanityn test 105: Glimpse and lock cancel race ======== 11:31:06 (1790782266) fail_loc=0x416 PASS 105 (18s) == sanityn test 106a: Verify the btime via statx() ======= 11:31:24 (1790782284) SKIP: sanityn test_106a Test only for ldiskfs and statx() supported SKIP 106a (3s) == sanityn test 106b: Glimpse RPCs test for statx ======== 11:31:27 (1790782287) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0339206 s, 30.2 kB/s File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238877265927 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-30 11:31:29.000000000 -0400 Modify: 2026-09-30 11:31:29.000000000 -0400 Change: 2026-09-30 11:31:29.000000000 -0400 Birth: 2026-09-30 11:31:29.000000000 -0400 /mnt/lustre/f106b.sanityn 144115238877265927 -rw-r--r-- 1 0 0 1790782289 1790782289 1790782289 File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238877265927 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-30 11:31:29.000000000 -0400 Modify: 2026-09-30 11:31:29.000000000 -0400 Change: 2026-09-30 11:31:29.000000000 -0400 Birth: 2026-09-30 11:31:29.000000000 -0400 1790782289 1024 1 PASS 106b (8s) == sanityn test 106c: Verify statx attributes mask ======= 11:31:35 (1790782295) PASS 106c (6s) == sanityn test 107a: Basic grouplock conflict =========== 11:31:42 (1790782302) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.263563 s, 39.8 MB/s /mnt/lustre/f107a.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 4745 0x1289 0x280000400 multiop /mnt/lustre/f107a.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7627 multiop /mnt/lustre2/f107a.sanityn vO_G16022000r10g16022000c TMPPIPE=/tmp/multiop_open_wait_pipe.7627 First grouplock blocks second one PASS 107a (9s) == sanityn test 107b: Grouplock is added to the head of waiting list ========================================================== 11:31:51 (1790782311) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.35023 s, 29.9 MB/s /mnt/lustre/f107b.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 4786 0x12b2 0x240000400 multiop /mnt/lustre/f107b.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7627 Grouplock blocks IO First grouplock blocks second one Second grouplock blocks IO PASS 107b (13s) == sanityn test 108a: lseek: parallel updates ============ 11:32:04 (1790782324) 1+0 records in 1+0 records out 10000 bytes (10 kB, 9.8 KiB) copied, 0.0120667 s, 829 kB/s llite.lustre-ffff8c49534b6800.hybrid_io=0 llite.lustre-ffff8c495929d000.hybrid_io=0 fail_val=4 fail_loc=0x80000414 1+0 records in 1+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 4.46181 s, 1.9 MB/s PASS 108a (11s) == sanityn test 109: Race with several mount instances on 1 node ========================================================== 11:32:15 (1790782335) 192.168.202.155@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg255-client.virtnet /mnt/lustre (opts:) 192.168.202.155@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg255-client.virtnet /mnt/lustre2 (opts:) Starting race between client mounts (duration 30) Iteration 0 Starting client: oleg255-client.virtnet: -o user_xattr,flock 192.168.202.155@tcp:/lustre /mnt/lustre Starting client: oleg255-client.virtnet: -o user_xattr,flock 192.168.202.155@tcp:/lustre /mnt/lustre2 192.168.202.155@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.155@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg255-client.virtnet /mnt/lustre (opts:) Stopping client oleg255-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg255-server' oleg255-server: oleg255-server.virtnet: executing load_modules_local oleg255-server: kmemleak disabled oleg255-server: Loading modules from /home/green/git/lustre-release/lustre oleg255-server: detected 4 online CPUs by sysfs oleg255-server: MODOPTS_LIBCFS= oleg255-server: Force libcfs to create 2 CPU partitions oleg255-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Iteration 1 Starting client: oleg255-client.virtnet: -o user_xattr,flock 192.168.202.155@tcp:/lustre /mnt/lustre Starting client: oleg255-client.virtnet: -o user_xattr,flock 192.168.202.155@tcp:/lustre /mnt/lustre2 192.168.202.155@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.155@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg255-client.virtnet /mnt/lustre (opts:) Stopping client oleg255-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg255-server' oleg255-server: oleg255-server.virtnet: executing load_modules_local oleg255-server: kmemleak disabled oleg255-server: Loading modules from /home/green/git/lustre-release/lustre oleg255-server: detected 4 online CPUs by sysfs oleg255-server: MODOPTS_LIBCFS= oleg255-server: Force libcfs to create 2 CPU partitions oleg255-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Starting client: oleg255-client.virtnet: -o user_xattr,flock 192.168.202.155@tcp:/lustre /mnt/lustre Starting client: oleg255-client.virtnet: -o user_xattr,flock 192.168.202.155@tcp:/lustre /mnt/lustre2 PASS 109 (72s) == sanityn test 110: do not grant another lock on resend ========================================================== 11:33:27 (1790782407) fail_loc=0x525 fail_val=3 fail_loc=0x534 File: /mnt/lustre/f110.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238910820355 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-30 11:33:29.000000000 -0400 Modify: 2026-09-30 11:33:29.000000000 -0400 Change: 2026-09-30 11:33:29.000000000 -0400 Birth: 2026-09-30 11:33:29.000000000 -0400 fail_loc=0 fail_val=0 Connected clients: oleg255-client.virtnet oleg255-client.virtnet PASS 110 (72s) == sanityn test 111: A racy rename/link an open file should not cause fs corruption ========================================================== 11:34:40 (1790782480) SKIP: sanityn test_111 needs >= 2 MDTs SKIP 111 (3s) == sanityn test 112: update max-inherit in default LMV === 11:34:43 (1790782483) SKIP: sanityn test_112 We need at least 2 MDTs for this test SKIP 112 (3s) == sanityn test 113: check servers of specified fs ======= 11:34:46 (1790782486) lustre-OST0000-osc-ffff8c4951436000 active. lustre-OST0001-osc-ffff8c4951436000 active. lustre-OST0000-osc-ffff8c4951436000 active. lustre-OST0001-osc-ffff8c4951436000 active. MGC192.168.202.155@tcp active. PASS 113 (7s) == sanityn test 114: implicit default LMV inherit ======== 11:34:53 (1790782493) SKIP: sanityn test_114 We need at least 2 MDTs for this test SKIP 114 (3s) == sanityn test 115: ldiskfs doesn't check direntry for uniqueness ========================================================== 11:34:57 (1790782497) SKIP: sanityn test_115 ldiskfs only test SKIP 115 (3s) == sanityn test 116: DNE: Set default LMV layout from a remote client ========================================================== 11:35:01 (1790782501) SKIP: sanityn test_116 needs >= 2 MDTs SKIP 116 (3s) == sanityn test 121: trunc append race =================== 11:35:04 (1790782504) 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0140013 s, 1.5 MB/s fail_loc=0x80001436 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.130732 s, 157 kB/s 0000000 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0005000 0000 0000 0000 0000 0000 0000 0000 0000 * 000c800 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0011800 PASS 121 (10s) == sanityn test 122: directory size is consistent across mounts ========================================================== 11:35:14 (1790782514) PASS 122 (9s) == sanityn test 200: service remains healthy while able to process request ========================================================== 11:35:23 (1790782523) at_max=10 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=9 fail_loc=0x1e1 fail_val=20 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1373 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1661 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1704 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.2038 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.234 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.2208 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.2648 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.074 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.143 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.2118 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.2536 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.2776 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.2805 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.2743 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.3441 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.3869 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.3267 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.3386 s, 0.1 kB/s at_max=600 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=116 oleg255-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8c4951436000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8c4951436000.ost_server_uuid in FULL state after 0 sec PASS 200 (82s) cleanup: ====================================================== == sanityn test complete, duration 9745 sec ============== 11:36:47 (1790782607) === sanityn: start cleanup 11:36:49 (1790782609) === Stopping clients: oleg255-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg255-client.virtnet /mnt/lustre2 opts: === sanityn: finish cleanup 11:40:15 (1790782815) ===