-----============= acceptance-small: sanityn ============----- Wed Apr 22 14:02:36 EDT 2026 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg438-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanityn.*ex': No such file or directory excepting tests: 27 40a 102 skipping tests SLOW=no: 33a === sanityn: start setup 14:02:43 (1776880963) === Starting client oleg438-client.virtnet: -o user_xattr,flock 192.168.204.138@tcp:/lustre /mnt/lustre2 Mount client oleg438-client.virtnet: mount -t lustre -o user_xattr,flock 192.168.204.138@tcp:/lustre /mnt/lustre2 Started clients oleg438-client.virtnet: 192.168.204.138@tcp:/lustre on /mnt/lustre2 type lustre (rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) oleg438-client.virtnet: executing check_config_client /mnt/lustre oleg438-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg438-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9c56c2fa4800.idle_timeout=debug osc.lustre-OST0000-osc-ffff9c56c682c000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9c56c2fa4800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9c56c682c000.idle_timeout=debug disable quota as required oleg438-server: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanityn: finish setup 14:02:53 (1776880973) === 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132272 s, 79.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7529] == sanityn test 0: do_node_vp() and do_facet_vp() do the right thing ========================================================== 14:02:53 (1776880973) PASS 0 (3s) == sanityn test 1: Check attribute updates on 2 mount points ========================================================== 14:02:56 (1776880976) /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0777 OK /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0666 OK /mnt/lustre/f1.sanityn: absent OK PASS 1 (2s) == sanityn test 2a: check cached attribute updates on 2 mtpt's ================================================================== 14:02:58 (1776880978) -rw-r--r-- 1 root root 0 Apr 22 14:02 /mnt/lustre2/f2a /mnt/lustre/f2a has type file OK /mnt/lustre/f2a has perms 0777 OK PASS 2a (2s) == sanityn test 2b: check cached attribute updates on 2 mtpt's ================================================================== 14:03:00 (1776880980) -rw-r--r-- 1 root root 0 Apr 22 14:03 /mnt/lustre2/f2b /mnt/lustre2/f2b has type file OK /mnt/lustre2/f2b has perms 0777 OK PASS 2b (1s) == sanityn test 2c: check cached attribute updates on 2 mtpt's root ============================================================= 14:03:02 (1776880982) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0777 OK PASS 2c (1s) == sanityn test 2d: check cached attribute updates on 2 mtpt's root ============================================================= 14:03:03 (1776880983) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0755 OK PASS 2d (2s) == sanityn test 2e: check chmod on root is propagated to others ========================================================== 14:03:05 (1776880985) total 1 -rwxrwxrwx 1 root root 0 Apr 22 14:02 f2a -rwxrwxrwx 1 root root 0 Apr 22 14:03 f2b total 1 -rwxrwxrwx 1 root root 0 Apr 22 14:02 f2a -rwxrwxrwx 1 root root 0 Apr 22 14:03 f2b running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre2/f2e.sanityn] [count=1] 1+0 records in 1+0 records out 512 bytes copied, 0.0022407 s, 228 kB/s PASS 2e (2s) == sanityn test 2f: check attr/owner updates on DNE with 2 mtpt's ========================================================== 14:03:07 (1776880987) SKIP: sanityn test_2f needs >= 2 MDTs SKIP 2f (1s) == sanityn test 2g: check blocks update on sync write ==== 14:03:08 (1776880988) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0698518 s, 30.0 MB/s /mnt/lustre/f2g.sanityn has 4103 blocks /mnt/lustre2/f2g.sanityn has 4103 blocks PASS 2g (2s) == sanityn test 3: symlink on one mtpt, readlink on another ===================================================================== 14:03:10 (1776880990) PASS 3 (2s) == sanityn test 4: fstat validation on multiple mount points ==================================================================== 14:03:12 (1776880992) PASS 4 (3s) == sanityn test 5: create a file on one mount, truncate it on the other ========================================================== 14:03:15 (1776880995) /mnt/lustre/f5 has type file OK /mnt/lustre/f5 has size 100 OK PASS 5 (2s) == sanityn test 6: remove of open file on other node ============================================================================ 14:03:17 (1776880997) opening writing unlinking /mnt/lustre2/f6.sanityn accessing (1) seeking (1) accessing (2) fstat... reading comparing data truncating seeking (2) writing again seeking (3) reading again comparing data again closing SUCCESS - goto beer PASS 6 (2s) == sanityn test 7: remove of open directory on other node ======================================================================= 14:03:19 (1776880999) creating directory /mnt/lustre/d7 opening directory unlinking /mnt/lustre/d7 Ok, everything goes well. PASS 7 (2s) == sanityn test 8: remove of open special file on other node ==================================================================== 14:03:21 (1776881001) creating special file /mnt/lustre/f8.sanityn opening file unlinking /mnt/lustre/f8.sanityn Ok, everything goes well. PASS 8 (2s) == sanityn test 9a: append of file with sub-page size on multiple mounts ========================================================== 14:03:23 (1776881003) PASS 9a (2s) == sanityn test 9b: append to striped sparse file ======== 14:03:25 (1776881005) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015886 s, 66.0 MB/s 3+0 records in 3+0 records out 3 bytes copied, 0.00796434 s, 0.4 kB/s Data read (expecting 'foo'): foo PASS 9b (1s) == sanityn test 10a: write of file with sub-page size on multiple mounts ========================================================== 14:03:26 (1776881006) 1+0 records in 1+0 records out 1 byte copied, 0.00151337 s, 0.7 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00256637 s, 0.4 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00273166 s, 0.4 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00232428 s, 0.4 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00251527 s, 0.4 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00246777 s, 0.4 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00278289 s, 0.4 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00254643 s, 0.4 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00250529 s, 0.4 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.002547 s, 0.4 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00351137 s, 0.3 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00277259 s, 0.4 kB/s PASS 10a (3s) == sanityn test 10b: write of file with sub-page size on multiple mounts ========================================================== 14:03:29 (1776881009) 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.00384022 s, 800 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0102788 s, 398 kB/s 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.000218423 s, 14.1 MB/s PASS 10b (2s) == sanityn test 11: execution of file opened for write should return error ============================================================== 14:03:31 (1776881011) multiop /mnt/lustre/d11/f vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7529 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 300: /mnt/lustre2/d11/f: Text file busy PASS 11 (2s) == sanityn test 12: test lock ordering (link, stat, unlink) ========================================================== 14:03:33 (1776881013) warning: '-runtime' deprecated, use '-t runtime' instead - link 2321 (time 1776881023.77 total 10.00 last 232.05) - link 4709 (time 1776881033.78 total 20.01 last 238.69) - link 6926 (time 1776881043.78 total 30.01 last 221.69) - link 9181 (time 1776881053.78 total 40.01 last 225.42) - link 10000 (time 1776881057.84 total 44.07 last 201.98) - link 11939 (time 1776881067.84 total 54.07 last 193.84) - link 13983 (time 1776881077.84 total 64.07 last 204.36) - link 15845 (time 1776881087.84 total 74.07 last 186.14) - link 17048 (time 1776881097.85 total 84.08 last 120.22) - link 18176 (time 1776881107.86 total 94.09 last 112.72) total: 18855 link in 100.00 seconds: 188.55 ops/second using seed 3811565140 running for 100 seconds - stat 10000 (time 1776881023 ; total 9 ; last 9) - stat 20000 (time 1776881031 ; total 17 ; last 8) - stat 30000 (time 1776881040 ; total 26 ; last 9) - stat 40000 (time 1776881049 ; total 35 ; last 9) - stat 50000 (time 1776881059 ; total 45 ; last 10) - stat 56115 (time 1776881070 ; total 56 ; last 11) - stat 60000 (time 1776881080 ; total 66 ; last 10) - stat 63099 (time 1776881091 ; total 77 ; last 11) - stat 65791 (time 1776881102 ; total 88 ; last 11) - stat 68604 (time 1776881113 ; total 99 ; last 11) total: 68981 stats in 100 seconds: 689.809998 stats/second - unlinked 0 (time 1776881064 ; total 0 ; last 0) - unlinked 10000 (time 1776881141 ; total 77 ; last 77) unlink(/mnt/lustre2/lockdir/lockfile18855) error: No such file or directory total: 18854 unlinks in 102 seconds: 184.843140 unlinks/second /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 77: kill: (21300) - No such process /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 78: kill: (21302) - No such process PASS 12 (155s) == sanityn test 13: test directory page revocation ======= 14:06:08 (1776881168) f13 f13 PASS 13 (3s) == sanityn test 14aa: execution of file open for write returns -ETXTBSY ========================================================== 14:06:11 (1776881171) multiop /mnt/lustre/d14aa.sanityn/f14aa.sanityn vOw_c TMPPIPE=/tmp/multiop_open_wait_pipe.7529 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 335: /mnt/lustre2/d14aa.sanityn/f14aa.sanityn: Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4991: 22487 Terminated $MULTIOP_PROG $FILE v$ARGS > $TMPPIPE (wd: ~) PASS 14aa (3s) == sanityn test 14ab: open(RDWR) of executing file returns -ETXTBSY ========================================================== 14:06:14 (1776881174) open(O_RDWR|O_CREAT): Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4991: 23066 Terminated $DIR1/$tdir/sleep 60 (wd: ~) PASS 14ab (2s) == sanityn test 14b: truncate of executing file returns -ETXTBSY ================================================================ 14:06:16 (1776881176) truncate: cannot truncate '/mnt/lustre2/d14b.sanityn/sleep' to length 60: Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 350: 23643 Terminated $DIR1/$tdir/sleep 60 PASS 14b (3s) == sanityn test 14c: open(O_TRUNC) of executing file return -ETXTBSY ============================================================ 14:06:19 (1776881179) cp: cannot create regular file '/mnt/lustre2/d14c.sanityn/sleep': Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 362: 24225 Terminated $DIR1/$tdir/sleep 60 PASS 14c (3s) == sanityn test 14d: chmod of executing file is still possible ================================================================== 14:06:22 (1776881182) chmod /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 373: 24808 Terminated $DIR1/$tdir/sleep 60 PASS 14d (4s) == sanityn test 15: test out-of-space with multiple writers ===================================================================== 14:06:26 (1776881186) PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin: mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky sleep 5 for ZFS MDS Waiting for MDT destroys to complete fallocate on zfs doesn't consume space STRIPECOUNT=2 ORIGFREE=7531520 MAXFREE=800000 SKIP: oos2 test_15 oos2.sh: 7531520KB free > 800000KB, increase MAXFREE (or reduce fs size) sleep 5 for ZFS MDS Waiting for MDT destroys to complete SKIP 15 (32s) == sanityn test 16a: 500 iterations of dual-mount fsx ==== 14:06:58 (1776881218) fallocate on zfs doesn't consume space fallocate not supported 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.176379 s, 47.6 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.0996163 s, 84.2 MB/s Chance of close/open is 1 in 50 Seed set to 1220 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000001[1] 1776881220.520569 write 0x3648dc thru 0x36b252 (0x6977 bytes) 000002[0] 1776881220.526778 read 0x022422 thru 0x0263de (0x3fbd bytes) truncating to largest ever: 0x2d2bdb 000004[1] 1776881220.542640 trunc from 0x36b253 to 0x2d2bda (0x98678 bytes) truncating to largest ever: 0x9221b5 000005[0] 1776881220.560028 trunc from 0x2d2bdb to 0x9221b4 (0x64f5da bytes) 000007[0] 1776881220.585827 mapwrite 0x78410d thru 0x793236 (0xf12a bytes) 000008[0] 1776881220.644387 mapread 0x1ce9b2 thru 0x1d4a0b (0x605a bytes) 000010[0] 1776881220.675870 write 0x7cca90 thru 0x7db77a (0xeceb bytes) 000011[1] 1776881220.691200 write 0x963da8 thru 0x969c02 (0x5e5b bytes) 000014[1] 1776881220.720151 write 0x2525bd thru 0x257fda (0x5a1e bytes) 000015[0] 1776881220.732077 mapread 0x61f17e thru 0x61fe8b (0xd0e bytes) 000016[1] 1776881220.747785 mapread 0x10fa4f thru 0x118a51 (0x9003 bytes) 000017[1] 1776881220.772962 mapread 0x509885 thru 0x51632c (0xcaa8 bytes) 000018[0] 1776881220.801721 mapwrite 0x42b99a thru 0x439259 (0xd8c0 bytes) 000020[0] 1776881220.872771 write 0x69da44 thru 0x6aa05b (0xc618 bytes) 000029[1] 1776881220.954639 write 0x2c9083 thru 0x2d3156 (0xa0d4 bytes) 000030[0] 1776881220.968881 mapwrite 0x6d19de thru 0x6dc7d5 (0xadf8 bytes) 000031[1] 1776881221.019281 trunc from 0x969c03 to 0x57d87e (0x3ec384 bytes) 000032[0] 1776881221.035452 mapread 0x050127 thru 0x055ef2 (0x5dcc bytes) 000034[1] 1776881221.076658 mapread 0x2083ab thru 0x21431f (0xbf75 bytes) 000035[0] 1776881221.087685 mapread 0x23e3a8 thru 0x243c3a (0x5893 bytes) 000037[1] 1776881221.120589 read 0x218237 thru 0x2242af (0xc079 bytes) 000039[0] 1776881221.143739 trunc from 0x57d87f to 0x836394 (0x2b8b16 bytes) 000040[0] 1776881221.162561 mapwrite 0x39fc22 thru 0x3a857e (0x895d bytes) 000043[0] 1776881221.207920 write 0x55644d thru 0x55c815 (0x63c9 bytes) truncating to largest ever: 0x99ed82 000045[0] 1776881221.229786 trunc from 0x836395 to 0x99ed81 (0x1689ed bytes) 000047[1] 1776881221.258321 read 0x3e8eef thru 0x3f8add (0xfbef bytes) 000050[1] 1776881221.281404 write 0x74c4de thru 0x74f67b (0x319e bytes) 000052[1] 1776881221.297182 mapwrite 0x3234c7 thru 0x325304 (0x1e3e bytes) 000054[1] 1776881221.336307 read 0x8f08b6 thru 0x8f1a4e (0x1199 bytes) 000059[1] 1776881221.359952 write 0x865355 thru 0x87277e (0xd42a bytes) 000060[0] 1776881221.375988 mapread 0x4e6b39 thru 0x4e9b43 (0x300b bytes) 000061[0] 1776881221.396161 read 0x8a717c thru 0x8b15fe (0xa483 bytes) 000064[0] 1776881221.412398 read 0x227807 thru 0x22f105 (0x78ff bytes) 000066[1] 1776881221.420436 trunc from 0x99ed82 to 0x517605 (0x48777c bytes) 000068[0] 1776881221.453827 mapwrite 0x5a8523 thru 0x5b1ae4 (0x95c2 bytes) 000070[1] 1776881221.525053 read 0x42d79a thru 0x4306af (0x2f16 bytes) 000071[0] 1776881221.542665 mapwrite 0x2a4b6e thru 0x2afe9b (0xb32e bytes) 000078[0] 1776881221.629686 mapwrite 0x02e0ed thru 0x03962c (0xb540 bytes) 000081[1] 1776881221.672899 trunc from 0x5b1ae5 to 0x23203d (0x37faa7 bytes) 000082[1] 1776881221.694636 mapread 0x0d81ad thru 0x0dc779 (0x45cd bytes) 000086[1] 1776881221.724855 write 0x63161d thru 0x6330de (0x1ac2 bytes) 000087[1] 1776881221.727393 read 0x1342b7 thru 0x14381b (0xf565 bytes) 000088[1] 1776881221.740811 trunc from 0x6330df to 0x124d4a (0x50e394 bytes) 000091[0] 1776881221.779710 write 0x0cb5e5 thru 0x0da625 (0xf041 bytes) 000096[0] 1776881221.836626 trunc from 0x124d4b to 0x02ebbe (0xf618c bytes) 000097[1] 1776881221.860038 mapwrite 0x5b2274 thru 0x5b521d (0x2faa bytes) 000099[0] 1776881221.924225 mapread 0x3cab65 thru 0x3d56b3 (0xab4f bytes) 000100[1] 1776881221.952075 read 0x5595d5 thru 0x55f935 (0x6361 bytes) 000102[0] 1776881221.966499 trunc from 0x5b521e to 0x765b15 (0x1b08f8 bytes) 000104[1] 1776881221.989824 write 0x6df4b9 thru 0x6e6dd3 (0x791b bytes) 000105[1] 1776881222.010659 read 0x3b5980 thru 0x3b666b (0xcec bytes) 000108[1] 1776881222.028736 trunc from 0x765b16 to 0x6c639d (0x9f778 bytes) 000110[0] 1776881222.056358 mapwrite 0x5e871a thru 0x5ec902 (0x41e9 bytes) 000111[1] 1776881222.109558 mapread 0x672c82 thru 0x67adb0 (0x812f bytes) 000112[0] 1776881222.136658 mapwrite 0x8220cd thru 0x823ca2 (0x1bd6 bytes) 000114[1] 1776881222.198955 trunc from 0x823ca3 to 0x075df3 (0x7adeaf bytes) 000115[0] 1776881222.218390 write 0x5bf7f1 thru 0x5c4b40 (0x5350 bytes) 000117[0] 1776881222.244604 write 0x65b3e1 thru 0x660107 (0x4d27 bytes) 000118[0] 1776881222.253801 mapwrite 0x933ee7 thru 0x9366e7 (0x2801 bytes) 000119[1] 1776881222.311185 trunc from 0x9366e8 to 0x086601 (0x8b00e6 bytes) 000120[0] 1776881222.340918 trunc from 0x086602 to 0x9850c3 (0x8feac2 bytes) 000122[1] 1776881222.373894 write 0x793d28 thru 0x79bdd4 (0x80ad bytes) 000130[1] 1776881222.451430 write 0x68728f thru 0x6913d8 (0xa14a bytes) 000131[1] 1776881222.465668 mapread 0x81bd28 thru 0x822358 (0x6631 bytes) 000133[0] 1776881222.493392 mapread 0x2268f1 thru 0x227b0e (0x121e bytes) 000134[1] 1776881222.503397 write 0x4a6fbb thru 0x4a735d (0x3a3 bytes) 000136[0] 1776881222.516832 mapwrite 0x31d404 thru 0x3273a3 (0x9fa0 bytes) 000138[1] 1776881222.580656 read 0x05eb32 thru 0x063432 (0x4901 bytes) 000141[1] 1776881222.592057 read 0x63196e thru 0x640c25 (0xf2b8 bytes) 000142[0] 1776881222.605834 write 0x6bf1e8 thru 0x6c9919 (0xa732 bytes) 000144[0] 1776881222.622390 mapread 0x6c07b6 thru 0x6d0642 (0xfe8d bytes) 000147[1] 1776881222.638572 read 0x758759 thru 0x758aee (0x396 bytes) 000150[0] 1776881222.654147 trunc from 0x9850c4 to 0x054105 (0x930fbe bytes) 000151[1] 1776881222.684082 trunc from 0x054106 to 0x7cd2f0 (0x7791eb bytes) 000153[0] 1776881222.714824 mapwrite 0x38d3ff thru 0x399e5e (0xca60 bytes) 000154[1] 1776881222.773479 write 0x3a80e0 thru 0x3b62dd (0xe1fe bytes) 000155[0] 1776881222.792695 mapread 0x76d638 thru 0x77a93e (0xd307 bytes) 000157[1] 1776881222.818785 mapread 0x43f5ef thru 0x44731d (0x7d2f bytes) 000158[0] 1776881222.832693 mapwrite 0x8f2031 thru 0x8f7d43 (0x5d13 bytes) 000159[0] 1776881222.899549 mapread 0x422840 thru 0x427b01 (0x52c2 bytes) 000160[1] 1776881222.912935 write 0x69af3a thru 0x6aa501 (0xf5c8 bytes) 000162[1] 1776881222.941594 mapread 0x7773f5 thru 0x77b3ca (0x3fd6 bytes) 000163[1] 1776881222.963688 mapwrite 0x5a5f23 thru 0x5ad96c (0x7a4a bytes) 000165[0] 1776881223.029847 write 0x5b5f45 thru 0x5bd3da (0x7496 bytes) 000168[0] 1776881223.045221 write 0x55878e thru 0x55f6c7 (0x6f3a bytes) 000169[0] 1776881223.059614 mapwrite 0x755edd thru 0x7635f8 (0xd71c bytes) 000170[1] 1776881223.109612 read 0x55bd15 thru 0x55e17c (0x2468 bytes) 000171[0] 1776881223.120582 write 0x55d6d9 thru 0x56823d (0xab65 bytes) 000172[1] 1776881223.131659 write 0x295ddc thru 0x2a1a93 (0xbcb8 bytes) 000173[0] 1776881223.150946 mapread 0x45c4f7 thru 0x4698da (0xd3e4 bytes) truncating to largest ever: 0x9c702e 000177[1] 1776881223.205418 trunc from 0x8f7d44 to 0x9c702d (0xcf2ea bytes) 000180[1] 1776881223.246739 read 0x79e895 thru 0x7a6d56 (0x84c2 bytes) 000181[1] 1776881223.252093 mapwrite 0x663982 thru 0x66ec8c (0xb30b bytes) 000182[1] 1776881223.294700 read 0x4bf341 thru 0x4c4630 (0x52f0 bytes) 000183[0] 1776881223.300009 read 0x284731 thru 0x28e787 (0xa057 bytes) 000184[0] 1776881223.322443 mapwrite 0x82d96d thru 0x83b6b3 (0xdd47 bytes) 000185[0] 1776881223.383649 mapread 0x85273b thru 0x85f518 (0xcdde bytes) 000187[0] 1776881223.414701 mapwrite 0x4c7d51 thru 0x4cf670 (0x7920 bytes) 000189[0] 1776881223.472043 trunc from 0x9c702e to 0x43a57f (0x58caae bytes) 000190[1] 1776881223.493133 write 0x751abc thru 0x75fe33 (0xe378 bytes) 000195[0] 1776881223.546719 mapwrite 0x5341ae thru 0x535765 (0x15b8 bytes) 000197[0] 1776881223.609298 trunc from 0x75fe34 to 0x554487 (0x20b9ac bytes) 000199[0] 1776881223.637500 read 0x21cc9a thru 0x225ad4 (0x8e3b bytes) 000200[0] 1776881223.650887 mapread 0x04081a thru 0x0462ca (0x5ab1 bytes) 000201[0] 1776881223.666678 trunc from 0x554488 to 0x5dcd20 (0x88899 bytes) 000202[0] 1776881223.683428 mapwrite 0x012ee3 thru 0x021279 (0xe397 bytes) 000205[1] 1776881223.727123 trunc from 0x5dcd21 to 0x0a4c9b (0x538085 bytes) 000206[1] 1776881223.755611 read 0x0870c9 thru 0x091354 (0xa28c bytes) 000207[1] 1776881223.767394 write 0x514cc4 thru 0x5166c8 (0x1a05 bytes) 000208[0] 1776881223.780022 mapwrite 0x03bbe3 thru 0x03f624 (0x3a42 bytes) 000209[0] 1776881223.832615 mapwrite 0x6431c4 thru 0x64ba06 (0x8843 bytes) 000210[0] 1776881223.887420 mapread 0x441d09 thru 0x445973 (0x3c6b bytes) 000214[1] 1776881223.921665 trunc from 0x64ba07 to 0x27719f (0x3d4867 bytes) 000215[1] 1776881223.951009 read 0x0ef49c thru 0x0fdd15 (0xe87a bytes) 000216[1] 1776881223.980190 read 0x020045 thru 0x029154 (0x9110 bytes) 000217[0] 1776881223.990061 write 0x9c62b1 thru 0x9c707b (0xdcb bytes) 000218[0] 1776881224.010700 mapread 0x7a0715 thru 0x7a6e10 (0x66fc bytes) 000221[1] 1776881224.046356 write 0x45decb thru 0x460f17 (0x304d bytes) 000223[1] 1776881224.068648 write 0x36402f thru 0x36ffc7 (0xbf99 bytes) 000225[1] 1776881224.124911 mapwrite 0x02b642 thru 0x031a0f (0x63ce bytes) 000228[1] 1776881224.163669 mapwrite 0x7cc2ef thru 0x7d81fe (0xbf10 bytes) 000232[0] 1776881224.268671 write 0x16597a thru 0x16fe8a (0xa511 bytes) 000233[1] 1776881224.294693 mapread 0x54bca7 thru 0x559d93 (0xe0ed bytes) 000236[0] 1776881224.326729 mapread 0x08835a thru 0x09822a (0xfed1 bytes) 000237[0] 1776881224.349655 read 0x6491c7 thru 0x64bf59 (0x2d93 bytes) truncating to largest ever: 0x9d7a5e 000240[0] 1776881224.389709 trunc from 0x9c707c to 0x9d7a5d (0x109e2 bytes) 000243[1] 1776881224.417934 read 0x883db2 thru 0x888cae (0x4efd bytes) 000245[1] 1776881224.448457 mapwrite 0x1c1d14 thru 0x1ca6ad (0x899a bytes) 000247[1] 1776881224.511505 mapread 0x8135d7 thru 0x81d4b9 (0x9ee3 bytes) 000248[0] 1776881224.533604 mapread 0x5a68a2 thru 0x5b432b (0xda8a bytes) 000249[1] 1776881224.562651 trunc from 0x9d7a5e to 0x541025 (0x496a38 bytes) 000251[0] 1776881224.598855 write 0x6a309b thru 0x6aae6a (0x7dd0 bytes) 000252[1] 1776881224.613660 mapread 0x336968 thru 0x341e78 (0xb511 bytes) 000256[0] 1776881224.672740 write 0x297547 thru 0x29f43d (0x7ef7 bytes) 000257[0] 1776881224.683716 write 0x166c7f thru 0x172347 (0xb6c9 bytes) 000258[1] 1776881224.690294 write 0x265b0c thru 0x26943c (0x3931 bytes) 000261[1] 1776881224.731695 read 0x2549e9 thru 0x257138 (0x2750 bytes) 000262[1] 1776881224.743910 read 0x444c68 thru 0x44957b (0x4914 bytes) 000263[1] 1776881224.757299 mapread 0x6317c1 thru 0x635832 (0x4072 bytes) 000266[0] 1776881224.798678 read 0x2e5e24 thru 0x2e8d48 (0x2f25 bytes) 000270[1] 1776881224.815521 write 0x10d5ee thru 0x11be02 (0xe815 bytes) 000271[0] 1776881224.826820 write 0x062eda thru 0x06ff36 (0xd05d bytes) 000272[0] 1776881224.851654 mapread 0x460b8d thru 0x467c5e (0x70d2 bytes) 000273[0] 1776881224.871421 trunc from 0x6aae6b to 0x12c5be (0x57e8ac bytes) 000275[1] 1776881224.889799 write 0x580f9e thru 0x58d41e (0xc481 bytes) 000276[0] 1776881224.908573 write 0x785f20 thru 0x7861f4 (0x2d5 bytes) 000279[0] 1776881224.939068 mapwrite 0x4013cd thru 0x405ca3 (0x48d7 bytes) 000282[0] 1776881225.003556 trunc from 0x7861f5 to 0x2c9457 (0x4bcd9d bytes) 000285[1] 1776881225.035688 write 0x098d40 thru 0x09a19f (0x1460 bytes) 000288[1] 1776881225.075556 trunc from 0x2c9458 to 0x97cfca (0x6b3b73 bytes) 000289[0] 1776881225.102036 read 0x1da3b0 thru 0x1dabf9 (0x84a bytes) 000290[0] 1776881225.117702 read 0x65834e thru 0x65e38a (0x603d bytes) 000292[1] 1776881225.128047 mapread 0x0e065b thru 0x0ed5d9 (0xcf7f bytes) 000293[1] 1776881225.142132 mapread 0x3fdec1 thru 0x404ad8 (0x6c18 bytes) 000294[1] 1776881225.156531 write 0x554566 thru 0x55f69a (0xb135 bytes) 000296[0] 1776881225.168318 write 0x47b8df thru 0x489a59 (0xe17b bytes) 000300[1] 1776881225.209789 mapread 0x3b6389 thru 0x3bef54 (0x8bcc bytes) 000301[0] 1776881225.232802 read 0x668087 thru 0x66a0a7 (0x2021 bytes) 000302[0] 1776881225.236598 mapwrite 0x1d8586 thru 0x1dcdcc (0x4847 bytes) 000303[1] 1776881225.280461 mapwrite 0x55a313 thru 0x55a442 (0x130 bytes) 000304[0] 1776881225.329573 mapwrite 0x76bae1 thru 0x76e26b (0x278b bytes) 000308[0] 1776881225.401502 trunc from 0x97cfcb to 0x7b36d2 (0x1c98f8 bytes) 000314[0] 1776881225.469875 trunc from 0x7b36d3 to 0x2cdc1a (0x4e5ab8 bytes) 000316[1] 1776881225.481371 read 0x09fd53 thru 0x0a9fcc (0xa27a bytes) 000317[1] 1776881225.500863 trunc from 0x2cdc1b to 0x2153a7 (0xb8873 bytes) 000320[0] 1776881225.541908 trunc from 0x2153a8 to 0x8da607 (0x6c5260 bytes) 000322[1] 1776881225.580766 read 0x4d3f73 thru 0x4e1d2b (0xddb9 bytes) 000323[0] 1776881225.594870 mapread 0x456ce6 thru 0x4570b4 (0x3cf bytes) 000325[0] 1776881225.609543 trunc from 0x8da608 to 0x4270f6 (0x4b3511 bytes) 000329[0] 1776881225.654566 mapwrite 0x80582c thru 0x809b20 (0x42f5 bytes) 000331[0] 1776881225.719808 trunc from 0x809b21 to 0x86c5f2 (0x62ad2 bytes) 000332[1] 1776881225.730291 write 0x7bd71a thru 0x7c5f42 (0x8829 bytes) 000339[1] 1776881225.804615 mapwrite 0x9b69c7 thru 0x9bfdf5 (0x942f bytes) 000340[0] 1776881225.865883 write 0x9ba0e4 thru 0x9c147e (0x739b bytes) 000344[0] 1776881225.909605 mapwrite 0x15e0da thru 0x1669c2 (0x88e9 bytes) 000346[0] 1776881225.967800 trunc from 0x9c147f to 0x30b99b (0x6b5ae3 bytes) 000347[1] 1776881225.987207 write 0x086f80 thru 0x08917f (0x2200 bytes) 000348[0] 1776881226.007613 write 0x28820c thru 0x291994 (0x9789 bytes) 000349[1] 1776881226.020992 trunc from 0x30b99c to 0x08a631 (0x28136a bytes) 000350[1] 1776881226.043115 mapread 0x03dca3 thru 0x04d57f (0xf8dd bytes) 000353[0] 1776881226.054625 read 0x065fcf thru 0x0711f1 (0xb223 bytes) 000357[0] 1776881226.091579 read 0x06c265 thru 0x074f5b (0x8cf7 bytes) 000358[1] 1776881226.099915 trunc from 0x08a632 to 0x94642e (0x8bbdfd bytes) 000359[1] 1776881226.128155 trunc from 0x94642f to 0x600f92 (0x34549c bytes) 000361[0] 1776881226.139438 mapread 0x576102 thru 0x58472b (0xe62a bytes) 000364[0] 1776881226.204137 write 0x3d9da6 thru 0x3dfe3e (0x6099 bytes) 000365[1] 1776881226.245116 mapread 0x4300dd thru 0x43c84d (0xc771 bytes) 000366[1] 1776881226.262375 trunc from 0x600f93 to 0x7c2b52 (0x1c1bc0 bytes) 000368[1] 1776881226.294913 mapread 0x2d1278 thru 0x2d8bf2 (0x797b bytes) 000369[0] 1776881226.306722 read 0x69a6f3 thru 0x6a0bb9 (0x64c7 bytes) 000370[0] 1776881226.327507 mapwrite 0x2f8319 thru 0x307f7e (0xfc66 bytes) 000373[0] 1776881226.392658 read 0x181acc thru 0x188cd6 (0x720b bytes) 000378[0] 1776881226.424631 write 0x2e2a36 thru 0x2ea2a9 (0x7874 bytes) 000382[0] 1776881226.448891 write 0x9d1f56 thru 0x9d6ed1 (0x4f7c bytes) 000387[1] 1776881226.469960 trunc from 0x9d6ed2 to 0x43d19f (0x599d32 bytes) 000388[1] 1776881226.494631 trunc from 0x43d1a0 to 0x1aa624 (0x292b7b bytes) 000390[0] 1776881226.514716 write 0x900dd1 thru 0x90aca2 (0x9ed2 bytes) 000391[1] 1776881226.531657 trunc from 0x90aca3 to 0x722518 (0x1e878a bytes) 000392[0] 1776881226.550959 trunc from 0x722519 to 0x54915b (0x1d93bd bytes) 000393[0] 1776881226.577822 mapread 0x30116a thru 0x30ac70 (0x9b07 bytes) 000396[0] 1776881226.600677 trunc from 0x54915c to 0x622d66 (0xd9c0b bytes) 000398[0] 1776881226.629691 mapwrite 0x39bab8 thru 0x3a472c (0x8c75 bytes) 000399[1] 1776881226.676827 mapread 0x02576b thru 0x03483a (0xf0d0 bytes) 000400[1] 1776881226.706340 mapwrite 0x6a42f0 thru 0x6a4eab (0xbbc bytes) 000405[1] 1776881226.792735 read 0x1b2bdd thru 0x1bd5b9 (0xa9dd bytes) 000407[1] 1776881226.798545 trunc from 0x6a4eac to 0x0180b2 (0x68cdf9 bytes) 000413[0] 1776881226.862726 write 0x8fa312 thru 0x901e03 (0x7af2 bytes) 000415[0] 1776881226.888584 read 0x168189 thru 0x16b5ff (0x3477 bytes) 000416[0] 1776881226.900724 trunc from 0x901e04 to 0x081352 (0x880ab1 bytes) 000418[0] 1776881226.916868 write 0x484534 thru 0x48afc5 (0x6a92 bytes) 000420[0] 1776881226.931628 trunc from 0x48afc6 to 0x878ee0 (0x3edf1b bytes) 000421[1] 1776881226.951700 read 0x098d8e thru 0x099777 (0x9ea bytes) 000423[0] 1776881226.972665 write 0x384559 thru 0x38a566 (0x600e bytes) 000425[1] 1776881226.985066 trunc from 0x878ee1 to 0x407056 (0x471e8a bytes) 000426[0] 1776881227.021172 trunc from 0x407057 to 0x01cc58 (0x3ea3fe bytes) 000432[0] 1776881227.061501 write 0x98f666 thru 0x99571c (0x60b7 bytes) 000438[1] 1776881227.084188 read 0x55968d thru 0x559bb1 (0x525 bytes) 000439[0] 1776881227.098941 mapread 0x0dbee2 thru 0x0e5a27 (0x9b46 bytes) 000441[1] 1776881227.126087 trunc from 0x99571d to 0x4bd717 (0x4d8005 bytes) 000450[1] 1776881227.200941 read 0x328b97 thru 0x329316 (0x780 bytes) 000452[1] 1776881227.222941 mapwrite 0x1e7a15 thru 0x1e836a (0x956 bytes) 000458[0] 1776881227.283402 trunc from 0x4bd718 to 0x606673 (0x148f5c bytes) 000459[0] 1776881227.311769 trunc from 0x606674 to 0x3bd58e (0x2490e5 bytes) 000461[0] 1776881227.322629 mapread 0x2fdddb thru 0x2ff2ae (0x14d4 bytes) 000463[0] 1776881227.354733 mapwrite 0x03997a thru 0x03a733 (0xdba bytes) 000464[0] 1776881227.384151 mapread 0x1eb4b0 thru 0x1efd2e (0x487f bytes) 000467[1] 1776881227.413631 write 0x830caa thru 0x831769 (0xac0 bytes) 000469[0] 1776881227.437353 mapread 0x0686f9 thru 0x070333 (0x7c3b bytes) 000470[1] 1776881227.468741 read 0x2b309b thru 0x2b7547 (0x44ad bytes) 000472[0] 1776881227.474308 read 0x5432ae thru 0x5464ec (0x323f bytes) 000478[0] 1776881227.505729 trunc from 0x83176a to 0x6f7fe3 (0x139786 bytes) 000479[0] 1776881227.521097 trunc from 0x6f7fe4 to 0x264e52 (0x493191 bytes) 000481[1] 1776881227.540707 mapwrite 0x44aa15 thru 0x45772c (0xcd18 bytes) 000483[1] 1776881227.615799 write 0x7fbf2e thru 0x80ae82 (0xef55 bytes) 000488[0] 1776881227.638491 read 0x1c6d7d thru 0x1c8317 (0x159b bytes) 000489[1] 1776881227.659636 mapread 0x22336d thru 0x22d445 (0xa0d9 bytes) 000493[1] 1776881227.674699 write 0x494cad thru 0x49c56b (0x78bf bytes) 000494[0] 1776881227.690498 read 0x7260b8 thru 0x73593e (0xf887 bytes) 000500[0] 1776881227.710464 mapread 0x030aa0 thru 0x035df3 (0x5354 bytes) All operations completed A-OK! Chance of close/open is 1 in 50 Seed set to 1227 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000001[1] 1776881228.052276 mapwrite 0x460000 thru 0x46db39 (0xdb3a bytes) 000002[1] 1776881228.113179 mapread 0x379000 thru 0x37bb7d (0x2b7e bytes) truncating to largest ever: 0x9ea65 000005[0] 1776881228.137730 trunc from 0x46db3a to 0x09ea64 (0x3cf0d5 bytes) truncating to largest ever: 0x211696 000006[1] 1776881228.160776 trunc from 0x09ea65 to 0x211695 (0x172c31 bytes) 000007[1] 1776881228.172174 read_OD 0x10b000 thru 0x10bfff (0x1000 bytes) 000008[0] 1776881228.179818 read 0x1f7000 thru 0x1f7efb (0xefc bytes) 000009[0] 1776881228.185224 read 0x02f000 thru 0x0331ba (0x41bb bytes) truncating to largest ever: 0x7c0230 000010[1] 1776881228.204710 trunc from 0x211696 to 0x7c022f (0x5aeb9a bytes) 000013[1] 1776881228.240525 write_OD 0x60b000 thru 0x60cfff (0x2000 bytes) 000021[1] 1776881228.271528 write_OD 0x902000 thru 0x905fff (0x4000 bytes) 000022[1] 1776881228.298011 mapread 0x5f2000 thru 0x5ffe26 (0xde27 bytes) 000024[0] 1776881228.315353 mapread 0x872000 thru 0x8780d1 (0x60d2 bytes) 000025[0] 1776881228.367673 trunc from 0x906000 to 0x75a8fd (0x1ab702 bytes) 000026[1] 1776881228.406419 mapwrite 0x10d000 thru 0x11ca3e (0xfa3f bytes) 000027[0] 1776881228.458756 mapread 0x337000 thru 0x343b8c (0xcb8d bytes) 000028[1] 1776881228.508359 write_OD 0x8c5000 thru 0x8d1fff (0xd000 bytes) 000030[1] 1776881228.540988 mapread 0x7f6000 thru 0x7f8f09 (0x2f0a bytes) 000031[0] 1776881228.553959 read 0x875000 thru 0x87f9e1 (0xa9e2 bytes) 000033[0] 1776881228.559406 write 0x67a000 thru 0x6833bc (0x93bd bytes) 000036[1] 1776881228.577594 trunc from 0x8d2000 to 0x5c8199 (0x309e66 bytes) 000037[1] 1776881228.596211 write_OD 0x01b000 thru 0x01cfff (0x2000 bytes) 000039[1] 1776881228.622586 read_OD 0x2f5000 thru 0x2fafff (0x6000 bytes) 000040[1] 1776881228.632427 mapread 0x1b0000 thru 0x1b6078 (0x6079 bytes) 000041[0] 1776881228.643867 write 0x894000 thru 0x89a49f (0x64a0 bytes) 000042[0] 1776881228.659834 write 0x8ea000 thru 0x8ecaff (0x2b00 bytes) 000044[0] 1776881228.670112 write 0x781000 thru 0x78dfc0 (0xcfc1 bytes) 000049[1] 1776881228.700086 mapwrite 0x237000 thru 0x23e698 (0x7699 bytes) 000050[0] 1776881228.753587 trunc from 0x8ecb00 to 0x6e2e72 (0x209c8d bytes) 000052[1] 1776881228.767407 mapread 0x473000 thru 0x481050 (0xe051 bytes) 000053[1] 1776881228.793168 read_OD 0x628000 thru 0x62dfff (0x6000 bytes) 000056[0] 1776881228.801483 mapread 0x5cc000 thru 0x5d9e15 (0xde16 bytes) 000057[1] 1776881228.817846 read_OD 0x01b000 thru 0x01efff (0x4000 bytes) 000059[0] 1776881228.821696 mapread 0x3ba000 thru 0x3be253 (0x4254 bytes) 000060[1] 1776881228.832480 mapread 0x0de000 thru 0x0ea87e (0xc87f bytes) 000063[0] 1776881228.848764 mapread 0x532000 thru 0x53d368 (0xb369 bytes) 000064[1] 1776881228.864555 read_OD 0x581000 thru 0x58bfff (0xb000 bytes) 000067[1] 1776881228.870815 mapwrite 0x943000 thru 0x946cde (0x3cdf bytes) 000069[1] 1776881228.922846 write_OD 0x8c9000 thru 0x8cffff (0x7000 bytes) 000070[0] 1776881228.949711 mapwrite 0x240000 thru 0x241ced (0x1cee bytes) 000073[0] 1776881228.997013 mapread 0x556000 thru 0x55b130 (0x5131 bytes) 000074[0] 1776881229.015519 mapwrite 0x213000 thru 0x215be7 (0x2be8 bytes) 000076[0] 1776881229.064515 trunc from 0x946cdf to 0x2397c3 (0x70d51b bytes) 000077[0] 1776881229.074175 mapread 0x0f7000 thru 0x0fe1ec (0x71ed bytes) 000080[0] 1776881229.096156 mapread 0x1ad000 thru 0x1b5f06 (0x8f07 bytes) 000081[0] 1776881229.108594 mapread 0x0ab000 thru 0x0adca3 (0x2ca4 bytes) 000083[1] 1776881229.131984 write_OD 0x82e000 thru 0x835fff (0x8000 bytes) 000084[1] 1776881229.163802 read_OD 0x70b000 thru 0x711fff (0x7000 bytes) 000085[0] 1776881229.170801 mapread 0x5e5000 thru 0x5eff0a (0xaf0b bytes) 000087[0] 1776881229.182298 write 0x382000 thru 0x3847b1 (0x27b2 bytes) 000088[1] 1776881229.191085 trunc from 0x836000 to 0x30ac48 (0x52b3b7 bytes) 000092[1] 1776881229.228226 mapread 0x001000 thru 0x010b23 (0xfb24 bytes) 000093[0] 1776881229.238939 mapwrite 0x2b6000 thru 0x2bcf8f (0x6f90 bytes) 000094[1] 1776881229.282694 trunc from 0x30ac49 to 0x677157 (0x36c50f bytes) 000096[0] 1776881229.298580 mapwrite 0x247000 thru 0x256621 (0xf622 bytes) 000099[1] 1776881229.363694 read_OD 0x322000 thru 0x32afff (0x9000 bytes) 000105[1] 1776881229.383282 trunc from 0x677158 to 0x1502e0 (0x526e77 bytes) 000107[0] 1776881229.398565 mapread 0x144000 thru 0x14956d (0x556e bytes) 000108[1] 1776881229.420404 trunc from 0x1502e1 to 0x5c24a5 (0x4721c5 bytes) 000109[1] 1776881229.436081 mapwrite 0x778000 thru 0x77bdea (0x3deb bytes) 000110[0] 1776881229.486514 write 0x5ab000 thru 0x5b672b (0xb72c bytes) 000112[0] 1776881229.513701 mapread 0x6db000 thru 0x6e6bfd (0xbbfe bytes) 000113[0] 1776881229.528766 mapwrite 0x569000 thru 0x56966e (0x66f bytes) 000115[0] 1776881229.565279 write 0x8f5000 thru 0x8f8316 (0x3317 bytes) 000119[1] 1776881229.591381 write_OD 0x586000 thru 0x594fff (0xf000 bytes) 000120[0] 1776881229.622121 trunc from 0x8f8317 to 0x264216 (0x694100 bytes) 000121[1] 1776881229.637017 write_OD 0x7e1000 thru 0x7e1fff (0x1000 bytes) 000123[0] 1776881229.664658 mapwrite 0x659000 thru 0x65b5e2 (0x25e3 bytes) 000126[1] 1776881229.703326 mapwrite 0x218000 thru 0x21eb65 (0x6b66 bytes) 000127[0] 1776881229.751992 write 0x9e2000 thru 0x9e4fa9 (0x2faa bytes) 000131[1] 1776881229.777329 write_OD 0x202000 thru 0x207fff (0x6000 bytes) 000133[0] 1776881229.811691 trunc from 0x9e4faa to 0x31cc95 (0x6c8314 bytes) 000134[0] 1776881229.827136 read 0x227000 thru 0x231b9a (0xab9b bytes) truncating to largest ever: 0x8233f9 000135[0] 1776881229.859579 trunc from 0x31cc96 to 0x8233f8 (0x506763 bytes) 000137[0] 1776881229.881510 read 0x211000 thru 0x21889d (0x789e bytes) 000139[1] 1776881229.882601 read_OD 0x020000 thru 0x02afff (0xb000 bytes) 000140[1] 1776881229.900647 mapread 0x67e000 thru 0x68bad6 (0xdad7 bytes) 000141[1] 1776881229.919428 read_OD 0x306000 thru 0x30bfff (0x6000 bytes) 000143[0] 1776881229.924667 mapread 0x37a000 thru 0x385b07 (0xbb08 bytes) 000144[1] 1776881229.940741 read_OD 0x3bb000 thru 0x3bbfff (0x1000 bytes) 000146[1] 1776881229.944620 mapwrite 0x03b000 thru 0x03e763 (0x3764 bytes) 000147[1] 1776881230.008926 mapread 0x144000 thru 0x1448b1 (0x8b2 bytes) 000148[1] 1776881230.012586 trunc from 0x8233f9 to 0x21fd9d (0x60365b bytes) 000149[0] 1776881230.024135 mapwrite 0x739000 thru 0x73e450 (0x5451 bytes) 000151[0] 1776881230.093100 trunc from 0x73e451 to 0x49ee11 (0x29f63f bytes) 000152[1] 1776881230.101030 trunc from 0x49ee12 to 0x057810 (0x447601 bytes) truncating to largest ever: 0x9ac4db 000153[0] 1776881230.119183 trunc from 0x057811 to 0x9ac4da (0x954cca bytes) 000154[1] 1776881230.133556 mapwrite 0x17a000 thru 0x1825bd (0x85be bytes) 000155[1] 1776881230.168341 read_OD 0x67e000 thru 0x683fff (0x6000 bytes) 000161[1] 1776881230.178563 read_OD 0x1e9000 thru 0x1f1fff (0x9000 bytes) 000162[0] 1776881230.180988 mapwrite 0x32a000 thru 0x32af7a (0xf7b bytes) 000163[0] 1776881230.197042 mapwrite 0x256000 thru 0x258264 (0x2265 bytes) 000166[0] 1776881230.217165 read 0x464000 thru 0x46e359 (0xa35a bytes) 000173[1] 1776881230.244716 trunc from 0x9ac4db to 0x6231f3 (0x3892e7 bytes) 000174[0] 1776881230.260389 mapwrite 0x271000 thru 0x27f29f (0xe2a0 bytes) 000176[1] 1776881230.321956 mapwrite 0x1be000 thru 0x1c9cbb (0xbcbc bytes) 000177[0] 1776881230.377644 write 0x22b000 thru 0x2397fc (0xe7fd bytes) 000179[0] 1776881230.390966 mapwrite 0x767000 thru 0x76ac6a (0x3c6b bytes) 000182[0] 1776881230.438367 read 0x673000 thru 0x67a0e9 (0x70ea bytes) 000184[0] 1776881230.462529 mapwrite 0x5a9000 thru 0x5b36f1 (0xa6f2 bytes) 000187[1] 1776881230.523506 read_OD 0x033000 thru 0x03efff (0xc000 bytes) 000190[1] 1776881230.540642 mapwrite 0x5a5000 thru 0x5b24e4 (0xd4e5 bytes) 000191[0] 1776881230.584961 mapwrite 0x02a000 thru 0x02a290 (0x291 bytes) 000194[0] 1776881230.630541 write 0x50a000 thru 0x515632 (0xb633 bytes) 000195[0] 1776881230.642505 mapwrite 0x359000 thru 0x35f401 (0x6402 bytes) 000196[1] 1776881230.680360 mapread 0x0e6000 thru 0x0f5bc8 (0xfbc9 bytes) 000198[1] 1776881230.704890 read_OD 0x14f000 thru 0x157fff (0x9000 bytes) 000199[0] 1776881230.711887 trunc from 0x76ac6b to 0x3c3782 (0x3a74e8 bytes) 000200[0] 1776881230.726995 write 0x2c3000 thru 0x2d25a1 (0xf5a2 bytes) 000201[1] 1776881230.741490 read_OD 0x0a3000 thru 0x0a7fff (0x5000 bytes) 000203[0] 1776881230.761353 trunc from 0x3c3783 to 0x576ef7 (0x1b3775 bytes) 000204[0] 1776881230.783480 trunc from 0x576ef8 to 0x0e35b6 (0x493941 bytes) 000208[0] 1776881230.804438 mapwrite 0x599000 thru 0x5a8df4 (0xfdf5 bytes) 000209[1] 1776881230.857075 mapread 0x123000 thru 0x12c3bb (0x93bc bytes) 000210[0] 1776881230.896578 read 0x41c000 thru 0x42b590 (0xf591 bytes) 000211[1] 1776881230.905498 mapwrite 0x37d000 thru 0x37d25b (0x25c bytes) 000212[1] 1776881230.954914 trunc from 0x5a8df5 to 0x38e02a (0x21adca bytes) 000215[0] 1776881230.976222 mapread 0x0b4000 thru 0x0c2937 (0xe938 bytes) 000217[0] 1776881231.000262 mapread 0x024000 thru 0x02cf24 (0x8f25 bytes) 000220[1] 1776881231.009037 mapread 0x277000 thru 0x281327 (0xa328 bytes) 000223[1] 1776881231.040262 trunc from 0x38e02b to 0x057718 (0x336912 bytes) 000227[1] 1776881231.080944 read_OD 0x030000 thru 0x033fff (0x4000 bytes) 000230[0] 1776881231.090084 mapwrite 0x631000 thru 0x6376f7 (0x66f8 bytes) 000232[0] 1776881231.145542 mapread 0x0dd000 thru 0x0ebcb8 (0xecb9 bytes) 000234[1] 1776881231.180524 read_OD 0x116000 thru 0x11cfff (0x7000 bytes) 000235[1] 1776881231.193994 trunc from 0x6376f8 to 0x09fab1 (0x597c46 bytes) 000237[0] 1776881231.225626 mapread 0x063000 thru 0x06e12a (0xb12b bytes) 000238[1] 1776881231.244074 trunc from 0x09fab2 to 0x1847b8 (0xe4d07 bytes) 000239[0] 1776881231.258106 mapread 0x13a000 thru 0x13b7d5 (0x17d6 bytes) 000241[0] 1776881231.277779 read 0x0ad000 thru 0x0bc760 (0xf761 bytes) 000242[1] 1776881231.291557 read_OD 0x089000 thru 0x08dfff (0x5000 bytes) 000245[1] 1776881231.297047 write_OD 0x047000 thru 0x054fff (0xe000 bytes) 000246[1] 1776881231.327036 mapread 0x0a0000 thru 0x0a12aa (0x12ab bytes) 000247[1] 1776881231.342464 mapwrite 0x0ea000 thru 0x0eff80 (0x5f81 bytes) 000248[0] 1776881231.403947 read 0x0a2000 thru 0x0adf6b (0xbf6c bytes) 000249[0] 1776881231.423632 write 0x069000 thru 0x076822 (0xd823 bytes) 000251[1] 1776881231.451434 read_OD 0x160000 thru 0x160fff (0x1000 bytes) 000253[1] 1776881231.469655 read_OD 0x16e000 thru 0x171fff (0x4000 bytes) 000262[1] 1776881231.476194 write_OD 0x63f000 thru 0x641fff (0x3000 bytes) 000264[1] 1776881231.506869 write_OD 0x93e000 thru 0x947fff (0xa000 bytes) 000265[1] 1776881231.535607 read_OD 0x7fa000 thru 0x7fafff (0x1000 bytes) 000266[0] 1776881231.538957 write 0x6a7000 thru 0x6b66ab (0xf6ac bytes) 000268[1] 1776881231.581631 write_OD 0x779000 thru 0x786fff (0xe000 bytes) 000269[0] 1776881231.622629 read 0x61f000 thru 0x62bbd7 (0xcbd8 bytes) 000274[0] 1776881231.631623 trunc from 0x948000 to 0x6c6ca7 (0x281358 bytes) 000275[0] 1776881231.646049 write 0x52b000 thru 0x52d2e8 (0x22e9 bytes) 000277[1] 1776881231.659614 trunc from 0x6c6ca8 to 0x5cab06 (0xfc1a1 bytes) 000278[0] 1776881231.685475 write 0x545000 thru 0x551788 (0xc789 bytes) 000279[0] 1776881231.704538 read 0x420000 thru 0x42892e (0x892f bytes) 000281[0] 1776881231.710136 trunc from 0x5cab07 to 0x2cc1f4 (0x2fe912 bytes) 000283[1] 1776881231.730044 trunc from 0x2cc1f5 to 0x14f3cd (0x17ce27 bytes) 000284[0] 1776881231.744057 trunc from 0x14f3ce to 0x11f578 (0x2fe55 bytes) 000285[0] 1776881231.765625 trunc from 0x11f579 to 0x18314c (0x63bd4 bytes) 000287[0] 1776881231.786287 mapread 0x04d000 thru 0x05875c (0xb75d bytes) 000288[1] 1776881231.805484 mapwrite 0x0b7000 thru 0x0baaad (0x3aae bytes) 000289[0] 1776881231.849273 read 0x0f0000 thru 0x0fd70c (0xd70d bytes) 000293[0] 1776881231.873008 write 0x3e0000 thru 0x3eeec6 (0xeec7 bytes) 000294[0] 1776881231.883310 mapread 0x1a7000 thru 0x1a88ce (0x18cf bytes) 000297[1] 1776881231.896806 mapwrite 0x6d8000 thru 0x6d85f8 (0x5f9 bytes) 000298[1] 1776881231.943664 read_OD 0x4ab000 thru 0x4b1fff (0x7000 bytes) 000300[0] 1776881231.953526 mapwrite 0x3ba000 thru 0x3c3e99 (0x9e9a bytes) 000301[1] 1776881232.020180 write_OD 0x634000 thru 0x642fff (0xf000 bytes) 000304[1] 1776881232.051967 write_OD 0x4b3000 thru 0x4b6fff (0x4000 bytes) 000306[1] 1776881232.078572 mapread 0x4dc000 thru 0x4e7100 (0xb101 bytes) 000307[1] 1776881232.096212 trunc from 0x6d85f9 to 0x92cdf9 (0x254801 bytes) 000308[0] 1776881232.119470 write 0x2f4000 thru 0x2fbf45 (0x7f46 bytes) 000309[0] 1776881232.138666 mapwrite 0x972000 thru 0x9751c1 (0x31c2 bytes) 000314[0] 1776881232.230919 trunc from 0x9751c2 to 0x7ddc01 (0x1975c0 bytes) 000315[1] 1776881232.244019 trunc from 0x7ddc02 to 0x7e1a4f (0x3e4e bytes) 000322[0] 1776881232.301892 trunc from 0x7e1a50 to 0x734fa1 (0xacaae bytes) 000325[0] 1776881232.339501 mapwrite 0x5c8000 thru 0x5c9929 (0x192a bytes) 000326[1] 1776881232.370010 mapread 0x06d000 thru 0x075d4d (0x8d4e bytes) 000331[1] 1776881232.394034 trunc from 0x734fa2 to 0x4dd877 (0x25772a bytes) 000335[0] 1776881232.442668 mapread 0x3b3000 thru 0x3bd71b (0xa71c bytes) 000336[1] 1776881232.483164 trunc from 0x4dd878 to 0x9ab916 (0x4ce09f bytes) 000338[0] 1776881232.499476 mapwrite 0x591000 thru 0x593786 (0x2787 bytes) 000339[0] 1776881232.547491 write 0x95e000 thru 0x9662b1 (0x82b2 bytes) 000343[0] 1776881232.571454 mapwrite 0x0b0000 thru 0x0be6d3 (0xe6d4 bytes) 000346[0] 1776881232.650942 read 0x925000 thru 0x92a5cb (0x55cc bytes) 000347[1] 1776881232.655667 trunc from 0x9ab917 to 0x119353 (0x8925c3 bytes) 000351[1] 1776881232.681936 read_OD 0x083000 thru 0x08afff (0x8000 bytes) 000352[1] 1776881232.691481 write_OD 0x1a1000 thru 0x1a4fff (0x4000 bytes) 000353[1] 1776881232.717446 read_OD 0x183000 thru 0x188fff (0x6000 bytes) 000356[0] 1776881232.724774 mapread 0x014000 thru 0x017c07 (0x3c08 bytes) 000361[1] 1776881232.737716 mapwrite 0x6b4000 thru 0x6bd503 (0x9504 bytes) 000364[1] 1776881232.846458 mapread 0x020000 thru 0x027662 (0x7663 bytes) 000365[0] 1776881232.848013 mapwrite 0x98b000 thru 0x992f35 (0x7f36 bytes) 000367[1] 1776881232.922097 trunc from 0x992f36 to 0x39f4fa (0x5f3a3b bytes) 000369[1] 1776881232.951479 read_OD 0x26a000 thru 0x26afff (0x1000 bytes) 000370[0] 1776881232.961407 write 0x603000 thru 0x606b75 (0x3b76 bytes) 000371[1] 1776881232.976587 write_OD 0x0b0000 thru 0x0bcfff (0xd000 bytes) 000372[1] 1776881233.007759 read_OD 0x425000 thru 0x430fff (0xc000 bytes) 000374[1] 1776881233.013140 read_OD 0x336000 thru 0x337fff (0x2000 bytes) 000376[1] 1776881233.018980 mapread 0x0c4000 thru 0x0d1e9d (0xde9e bytes) 000379[1] 1776881233.035522 mapwrite 0x731000 thru 0x736047 (0x5048 bytes) 000380[1] 1776881233.093589 trunc from 0x736048 to 0x72a3b2 (0xbc95 bytes) 000383[0] 1776881233.113563 trunc from 0x72a3b3 to 0x6e9155 (0x4125d bytes) 000390[1] 1776881233.173484 mapwrite 0x41a000 thru 0x420746 (0x6747 bytes) 000392[0] 1776881233.235586 mapread 0x282000 thru 0x28ee3b (0xce3c bytes) 000393[0] 1776881233.258733 write 0x8ae000 thru 0x8b5c56 (0x7c57 bytes) 000395[0] 1776881233.281443 mapread 0x4a0000 thru 0x4ab921 (0xb922 bytes) 000396[0] 1776881233.305837 read 0x331000 thru 0x33206e (0x106f bytes) 000397[1] 1776881233.309320 mapread 0x31e000 thru 0x320b13 (0x2b14 bytes) 000398[1] 1776881233.334703 mapwrite 0x1ef000 thru 0x1f25de (0x35df bytes) 000400[0] 1776881233.384169 mapwrite 0x55a000 thru 0x560b27 (0x6b28 bytes) 000401[1] 1776881233.437579 mapwrite 0x3b6000 thru 0x3c4306 (0xe307 bytes) 000403[1] 1776881233.506475 mapread 0x89f000 thru 0x8a847c (0x947d bytes) 000404[1] 1776881233.517851 mapread 0x441000 thru 0x441da6 (0xda7 bytes) 000406[0] 1776881233.537563 write 0x21e000 thru 0x2221c2 (0x41c3 bytes) 000407[0] 1776881233.546124 read 0x7c7000 thru 0x7cccdd (0x5cde bytes) 000411[1] 1776881233.568464 write_OD 0x422000 thru 0x430fff (0xf000 bytes) 000412[1] 1776881233.600765 mapwrite 0x074000 thru 0x07646e (0x246f bytes) 000413[1] 1776881233.649625 trunc from 0x8b5c57 to 0x7c3ea4 (0xf1db2 bytes) 000414[0] 1776881233.667352 trunc from 0x7c3ea5 to 0x5c5f09 (0x1fdf9b bytes) 000415[1] 1776881233.687361 trunc from 0x5c5f0a to 0x65e879 (0x98970 bytes) 000416[0] 1776881233.699960 mapread 0x31b000 thru 0x322684 (0x7685 bytes) 000417[1] 1776881233.721422 mapwrite 0x4b4000 thru 0x4b727e (0x327f bytes) 000418[1] 1776881233.762027 mapread 0x498000 thru 0x49c051 (0x4052 bytes) 000419[1] 1776881233.778625 mapread 0x515000 thru 0x5244e1 (0xf4e2 bytes) 000421[1] 1776881233.788762 mapwrite 0x72d000 thru 0x72fdcf (0x2dd0 bytes) 000422[1] 1776881233.843822 trunc from 0x72fdd0 to 0x569d1c (0x1c60b3 bytes) 000424[0] 1776881233.869438 write 0x731000 thru 0x732367 (0x1368 bytes) 000426[0] 1776881233.885799 mapread 0x227000 thru 0x230b28 (0x9b29 bytes) 000428[1] 1776881233.895732 read_OD 0x666000 thru 0x673fff (0xe000 bytes) 000430[0] 1776881233.912515 mapread 0x0db000 thru 0x0df73d (0x473e bytes) 000431[1] 1776881233.929991 mapwrite 0x1d4000 thru 0x1d5be1 (0x1be2 bytes) 000434[1] 1776881233.976610 mapwrite 0x28c000 thru 0x296375 (0xa376 bytes) 000437[1] 1776881234.030485 trunc from 0x732368 to 0x32c15f (0x406208 bytes) 000440[1] 1776881234.064462 trunc from 0x32c160 to 0x3005d8 (0x2bb87 bytes) 000441[1] 1776881234.074126 mapwrite 0x0e2000 thru 0x0ea43d (0x843e bytes) 000443[0] 1776881234.117522 mapread 0x1bf000 thru 0x1c9e75 (0xae76 bytes) 000444[0] 1776881234.141903 mapread 0x0a4000 thru 0x0b145c (0xd45d bytes) 000446[0] 1776881234.161466 mapwrite 0x12c000 thru 0x137a08 (0xba09 bytes) 000447[1] 1776881234.207841 read_OD 0x2b5000 thru 0x2bbfff (0x7000 bytes) 000448[1] 1776881234.222896 mapread 0x216000 thru 0x223a32 (0xda33 bytes) 000450[1] 1776881234.248042 write_OD 0x930000 thru 0x933fff (0x4000 bytes) 000451[1] 1776881234.281617 write_OD 0x86d000 thru 0x871fff (0x5000 bytes) 000452[1] 1776881234.308657 write_OD 0x703000 thru 0x708fff (0x6000 bytes) 000453[1] 1776881234.335958 read_OD 0x786000 thru 0x78afff (0x5000 bytes) 000454[1] 1776881234.342792 write_OD 0x63c000 thru 0x647fff (0xc000 bytes) 000455[0] 1776881234.370897 read 0x7e6000 thru 0x7edcaa (0x7cab bytes) 000456[1] 1776881234.377750 trunc from 0x934000 to 0x23ccc6 (0x6f7339 bytes) 000457[1] 1776881234.413063 mapread 0x0ea000 thru 0x0eef31 (0x4f32 bytes) 000459[0] 1776881234.423889 write 0x56b000 thru 0x578522 (0xd523 bytes) 000462[0] 1776881234.453469 write 0x39c000 thru 0x3a472c (0x872d bytes) 000464[1] 1776881234.465915 read_OD 0x1eb000 thru 0x1ecfff (0x2000 bytes) 000465[0] 1776881234.483755 mapwrite 0x932000 thru 0x93571f (0x3720 bytes) 000466[0] 1776881234.543522 write 0x7c6000 thru 0x7ca3ca (0x43cb bytes) 000467[1] 1776881234.555587 trunc from 0x935720 to 0x3ef67f (0x5460a0 bytes) 000468[0] 1776881234.580307 read 0x3e1000 thru 0x3ee54b (0xd54c bytes) 000470[1] 1776881234.613481 write_OD 0x008000 thru 0x016fff (0xf000 bytes) 000473[0] 1776881234.654973 mapread 0x252000 thru 0x25d2a5 (0xb2a6 bytes) 000477[0] 1776881234.691392 read 0x34a000 thru 0x357cb4 (0xdcb5 bytes) truncating to largest ever: 0x9ee3c6 000478[1] 1776881234.719916 trunc from 0x3ef680 to 0x9ee3c5 (0x5fed46 bytes) 000480[1] 1776881234.762386 write_OD 0x777000 thru 0x77bfff (0x5000 bytes) 000481[1] 1776881234.792800 write_OD 0x797000 thru 0x79efff (0x8000 bytes) 000482[0] 1776881234.820866 mapwrite 0x327000 thru 0x33387c (0xc87d bytes) 000483[0] 1776881234.907609 trunc from 0x9ee3c6 to 0x4ecd17 (0x5016ae bytes) 000484[1] 1776881234.926463 mapread 0x226000 thru 0x227fb5 (0x1fb6 bytes) 000485[1] 1776881234.941864 mapread 0x4ab000 thru 0x4b6c2a (0xbc2b bytes) 000486[0] 1776881234.961834 mapread 0x4dc000 thru 0x4dd75c (0x175d bytes) 000487[0] 1776881234.972623 read 0x3fc000 thru 0x40a321 (0xe322 bytes) 000488[0] 1776881234.976535 trunc from 0x4ecd18 to 0x0faaac (0x3f226b bytes) 000489[1] 1776881235.002442 trunc from 0x0faaad to 0x0c5575 (0x35537 bytes) 000494[0] 1776881235.035498 mapwrite 0x948000 thru 0x951737 (0x9738 bytes) 000495[1] 1776881235.096842 mapwrite 0x876000 thru 0x88038f (0xa390 bytes) 000498[0] 1776881235.163085 trunc from 0x951738 to 0x0a1d07 (0x8afa30 bytes) 000500[0] 1776881235.190506 mapwrite 0x8a9000 thru 0x8b57e7 (0xc7e8 bytes) All operations completed A-OK! PASS 16a (19s) == sanityn test 16b: 500 iterations of dual-mount fsx at small size ========================================================== 14:07:17 (1776881237) fallocate on zfs doesn't consume space fallocate not supported 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.136657 s, 61.4 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.1096 s, 76.5 MB/s Chance of close/open is 1 in 50 Seed set to 1237 fd 0: /mnt/lustre/f16b.sanityn fd 1: /mnt/lustre2/f16b.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read truncating to largest ever: 0x8fc9dd 000004[0] 1776881238.185868 trunc from 00000000 to 0x8fc9dc (0x8fc9dd bytes) 000006[0] 1776881238.216923 trunc from 0x8fc9dd to 0x49c205 (0x4607d7 bytes) 000008[1] 1776881238.247847 mapwrite 0x4e149a thru 0x4e22dc (0xe43 bytes) 000009[1] 1776881238.303619 trunc from 0x4e22dd to 0x140114 (0x3a21c8 bytes) 000013[1] 1776881238.328212 mapwrite 0x01fe5d thru 0x020013 (0x1b7 bytes) 000014[1] 1776881238.355459 trunc from 0x140115 to 0x552e74 (0x412d60 bytes) 000015[0] 1776881238.375655 read 0x223988 thru 0x223aa2 (0x11b bytes) 000019[0] 1776881238.413404 mapread 0x4afc8b thru 0x4b18c4 (0x1c3a bytes) 000020[1] 1776881238.433728 write 0x005e15 thru 0x007325 (0x1511 bytes) 000027[1] 1776881238.463729 mapwrite 0x6a4ecc thru 0x6a5597 (0x6cc bytes) 000028[1] 1776881238.505333 read 0x2edbce thru 0x2efae6 (0x1f19 bytes) 000030[1] 1776881238.529347 mapwrite 0x12e96b thru 0x12f2df (0x975 bytes) 000031[1] 1776881238.558008 mapread 0x4e6127 thru 0x4e74a4 (0x137e bytes) 000033[1] 1776881238.565232 write 0x12a37f thru 0x12bbf1 (0x1873 bytes) 000034[1] 1776881238.571418 write 0x28c622 thru 0x28d425 (0xe04 bytes) 000035[0] 1776881238.585784 mapwrite 0x31c05d thru 0x31dc6d (0x1c11 bytes) 000036[1] 1776881238.632453 mapread 0x662537 thru 0x6635f1 (0x10bb bytes) 000037[1] 1776881238.650540 trunc from 0x6a5598 to 0x3bbd19 (0x2e987e bytes) 000040[1] 1776881238.677436 write 0x1891c1 thru 0x18a13b (0xf7b bytes) 000042[0] 1776881238.684010 read 0x2e4838 thru 0x2e568c (0xe55 bytes) 000043[0] 1776881238.739622 mapwrite 0x34937c thru 0x349938 (0x5bd bytes) 000045[0] 1776881238.809338 mapread 0x1d3dbf thru 0x1d453b (0x77d bytes) 000046[1] 1776881238.827904 read 0x30e9e2 thru 0x30edf0 (0x40f bytes) 000047[1] 1776881238.882431 mapread 0x0007ad thru 0x000c36 (0x48a bytes) 000049[0] 1776881238.900070 read 0x1feebb thru 0x200dad (0x1ef3 bytes) 000050[1] 1776881238.948334 mapread 0x25e32d thru 0x25ebb0 (0x884 bytes) 000052[0] 1776881238.967214 read 0x0ad1a6 thru 0x0ae240 (0x109b bytes) 000053[0] 1776881238.975597 mapread 0x1fbe5b thru 0x1fcef8 (0x109e bytes) 000055[1] 1776881238.989455 read 0x043397 thru 0x04340d (0x77 bytes) 000057[1] 1776881239.007430 mapwrite 0x4412f5 thru 0x4426d2 (0x13de bytes) 000058[1] 1776881239.116875 trunc from 0x4426d3 to 0x8d318f (0x490abd bytes) 000059[1] 1776881239.138764 mapwrite 0x2180ed thru 0x2187f0 (0x704 bytes) 000060[1] 1776881239.172831 write 0x71f345 thru 0x720dad (0x1a69 bytes) 000062[1] 1776881239.226593 trunc from 0x8d3190 to 0x789216 (0x149f79 bytes) 000063[1] 1776881239.247861 trunc from 0x789217 to 0x1a87d4 (0x5e0a42 bytes) 000067[1] 1776881239.282410 mapread 0x13c866 thru 0x13c8a8 (0x43 bytes) 000068[1] 1776881239.283526 write 0x596148 thru 0x596c41 (0xafa bytes) 000069[0] 1776881239.286357 write 0x0e8812 thru 0x0e9a01 (0x11f0 bytes) 000073[0] 1776881239.349546 mapwrite 0x5b3068 thru 0x5b4f56 (0x1eef bytes) 000076[0] 1776881239.464817 read 0x0144e0 thru 0x0160c9 (0x1bea bytes) 000077[1] 1776881239.468120 trunc from 0x5b4f57 to 0x3c3565 (0x1f19f1 bytes) 000078[1] 1776881239.508782 mapread 0x246d4d thru 0x2487af (0x1a63 bytes) 000079[1] 1776881239.520489 read 0x22147e thru 0x222d8b (0x190e bytes) 000080[1] 1776881239.569280 mapread 0x36b949 thru 0x36d2bd (0x1975 bytes) 000081[0] 1776881239.570360 read 0x00b5ee thru 0x00b76a (0x17d bytes) 000082[1] 1776881239.624778 read 0x173dfb thru 0x174873 (0xa79 bytes) 000083[0] 1776881239.671588 trunc from 0x3c3566 to 0x614838 (0x2512d3 bytes) 000086[1] 1776881239.728627 trunc from 0x614839 to 0x87ab23 (0x2662eb bytes) 000087[0] 1776881239.756713 read 0x1c3baa thru 0x1c50a0 (0x14f7 bytes) 000090[0] 1776881239.786603 mapread 0x23d36b thru 0x23f069 (0x1cff bytes) 000091[0] 1776881239.805198 trunc from 0x87ab24 to 0x89433a (0x19817 bytes) 000094[1] 1776881239.838482 write 0x9bde34 thru 0x9bf9ab (0x1b78 bytes) 000095[0] 1776881239.854425 mapwrite 0x9f3035 thru 0x9f4c90 (0x1c5c bytes) truncating to largest ever: 0x94585e 000097[0] 1776881239.921459 trunc from 0x9f4c91 to 0x94585d (0xaf433 bytes) 000099[0] 1776881239.950634 trunc from 0x94585e to 0x22fb5f (0x715cfe bytes) 000100[0] 1776881239.969896 mapwrite 0x0e497b thru 0x0e573a (0xdc0 bytes) 000103[1] 1776881240.017505 write 0x6185f3 thru 0x619762 (0x1170 bytes) 000104[1] 1776881240.032593 mapwrite 0x7568e0 thru 0x757d95 (0x14b6 bytes) 000105[1] 1776881240.129734 mapread 0x106570 thru 0x107d47 (0x17d8 bytes) 000107[1] 1776881240.139289 read 0x1a0fa5 thru 0x1a299b (0x19f7 bytes) 000108[0] 1776881240.143347 read 0x43288f thru 0x434438 (0x1baa bytes) 000109[1] 1776881240.165519 mapwrite 0x87681d thru 0x877bc3 (0x13a7 bytes) 000110[1] 1776881240.220754 mapread 0x68f112 thru 0x68f57a (0x469 bytes) truncating to largest ever: 0x9cd374 000111[1] 1776881240.226801 trunc from 0x877bc4 to 0x9cd373 (0x1557b0 bytes) 000114[0] 1776881240.259798 trunc from 0x9cd374 to 0x951827 (0x7bb4c bytes) 000115[0] 1776881240.280199 mapwrite 0x0c2565 thru 0x0c3937 (0x13d3 bytes) 000116[0] 1776881240.320406 mapread 0x68486c thru 0x6862bb (0x1a50 bytes) 000118[0] 1776881240.332482 mapread 0x0f3915 thru 0x0f4fa6 (0x1692 bytes) 000120[1] 1776881240.349222 read 0x842e80 thru 0x843bed (0xd6e bytes) 000127[1] 1776881240.388512 mapread 0x76379c thru 0x7653c2 (0x1c27 bytes) 000128[1] 1776881240.406441 trunc from 0x951828 to 0x543c20 (0x40dc07 bytes) 000130[0] 1776881240.445616 write 0x12ff60 thru 0x130d26 (0xdc7 bytes) 000131[0] 1776881240.464112 mapread 0x3e8f60 thru 0x3e9032 (0xd3 bytes) 000133[0] 1776881240.494492 read 0x0ee1f8 thru 0x0efacd (0x18d6 bytes) 000134[0] 1776881240.544484 mapwrite 0x071761 thru 0x0735ff (0x1e9f bytes) 000138[0] 1776881240.608369 mapwrite 0x59f1e5 thru 0x5a0419 (0x1235 bytes) 000139[0] 1776881240.673925 mapwrite 0x254426 thru 0x255c03 (0x17de bytes) 000142[1] 1776881240.709682 mapwrite 0x918084 thru 0x918410 (0x38d bytes) 000143[0] 1776881240.767372 trunc from 0x918411 to 0x91e554 (0x6144 bytes) 000145[0] 1776881240.787524 mapread 0x429eb9 thru 0x42b113 (0x125b bytes) 000146[1] 1776881240.808743 trunc from 0x91e555 to 0x478e5f (0x4a56f5 bytes) 000147[1] 1776881240.837728 mapwrite 0x06f7be thru 0x0700ea (0x92d bytes) 000148[1] 1776881240.875215 mapread 0x40357e thru 0x404bd9 (0x165c bytes) 000149[0] 1776881240.890424 write 0x8ff762 thru 0x90159f (0x1e3e bytes) 000152[1] 1776881240.924620 trunc from 0x9015a0 to 0x07f28c (0x882313 bytes) 000154[1] 1776881240.954487 read 0x017985 thru 0x019353 (0x19cf bytes) 000155[1] 1776881240.972802 mapread 0x07bef9 thru 0x07c9e7 (0xaef bytes) 000156[0] 1776881240.981983 write 0x1635ee thru 0x1647fb (0x120e bytes) 000158[0] 1776881241.005679 read 0x010eb9 thru 0x0112c1 (0x409 bytes) 000161[1] 1776881241.046433 trunc from 0x1647fc to 0x029cb4 (0x13ab47 bytes) 000166[1] 1776881241.081090 write 0x1c1b0a thru 0x1c2690 (0xb87 bytes) 000169[1] 1776881241.102503 write 0x340d91 thru 0x34165c (0x8cc bytes) 000171[1] 1776881241.112957 trunc from 0x34165d to 0x28b56c (0xb60f0 bytes) 000173[0] 1776881241.135743 mapread 0x13c29b thru 0x13d04a (0xdb0 bytes) 000174[0] 1776881241.169029 mapread 0x0ad967 thru 0x0aed36 (0x13d0 bytes) 000175[1] 1776881241.179743 mapread 0x20fb16 thru 0x2112db (0x17c6 bytes) 000177[0] 1776881241.205475 mapread 0x03dd23 thru 0x03f3a8 (0x1686 bytes) 000179[1] 1776881241.229145 mapwrite 0x8a2083 thru 0x8a2ed6 (0xe54 bytes) 000181[1] 1776881241.329746 read 0x22682a thru 0x227fcd (0x17a4 bytes) 000182[1] 1776881241.342789 mapwrite 0x7f0ba6 thru 0x7f20f4 (0x154f bytes) 000183[0] 1776881241.386736 read 0x0526a2 thru 0x054504 (0x1e63 bytes) 000188[0] 1776881241.427104 write 0x7976f0 thru 0x797df2 (0x703 bytes) 000192[0] 1776881241.457369 trunc from 0x8a2ed7 to 0x399fbc (0x508f1a bytes) 000193[0] 1776881241.482545 mapwrite 0x1ed1c0 thru 0x1ee3de (0x121f bytes) 000195[0] 1776881241.514700 trunc from 0x399fbd to 0x396d57 (0x3265 bytes) 000201[1] 1776881241.562407 mapread 0x3852ab thru 0x3862ae (0x1004 bytes) 000203[0] 1776881241.580123 trunc from 0x396d58 to 0x449e9f (0xb3148 bytes) 000204[0] 1776881241.597196 read 0x019242 thru 0x01a1e5 (0xfa4 bytes) 000207[1] 1776881241.607986 read 0x3976fc thru 0x3989fb (0x1300 bytes) 000212[0] 1776881241.643426 mapread 0x35c86c thru 0x35e084 (0x1819 bytes) 000213[0] 1776881241.659050 trunc from 0x449ea0 to 0x8ee17a (0x4a42db bytes) 000214[0] 1776881241.670592 trunc from 0x8ee17b to 0x6ec1ed (0x201f8d bytes) 000217[0] 1776881241.691978 mapwrite 0x08f7c7 thru 0x090cf5 (0x152f bytes) 000219[1] 1776881241.725466 mapwrite 0x447ec9 thru 0x448093 (0x1cb bytes) 000220[1] 1776881241.764574 mapwrite 0x447f17 thru 0x4492b2 (0x139c bytes) 000221[0] 1776881241.796236 write 0x55c21a thru 0x55c48e (0x275 bytes) 000224[0] 1776881241.820426 read 0x4bb3d1 thru 0x4bc342 (0xf72 bytes) 000225[0] 1776881241.831571 read 0x640491 thru 0x6423aa (0x1f1a bytes) 000227[1] 1776881241.843366 read 0x28c849 thru 0x28ddb0 (0x1568 bytes) 000229[0] 1776881241.852142 mapread 0x655f5c thru 0x656e59 (0xefe bytes) 000230[0] 1776881241.859558 mapwrite 0x9c2816 thru 0x9c3d4e (0x1539 bytes) 000231[1] 1776881241.901165 mapwrite 0x31dff2 thru 0x31f246 (0x1255 bytes) 000232[1] 1776881241.962453 trunc from 0x9c3d4f to 0x457f41 (0x56be0d bytes) 000235[1] 1776881241.981759 mapread 0x43b3ee thru 0x43bd6d (0x980 bytes) 000237[1] 1776881241.998151 read 0x1b8611 thru 0x1ba234 (0x1c24 bytes) 000240[0] 1776881242.012226 mapwrite 0x61c82e thru 0x61df5c (0x172f bytes) 000243[0] 1776881242.076525 mapread 0x2c5e44 thru 0x2c5f29 (0xe6 bytes) 000247[0] 1776881242.091494 trunc from 0x61df5d to 0x25578f (0x3c87cd bytes) 000248[0] 1776881242.102302 mapwrite 0x43708a thru 0x438282 (0x11f9 bytes) 000251[0] 1776881242.163823 write 0x7c1b23 thru 0x7c3aba (0x1f98 bytes) 000252[0] 1776881242.166208 mapwrite 0x87ec73 thru 0x88037b (0x1709 bytes) 000255[1] 1776881242.224362 mapread 0x3dd6ea thru 0x3de49b (0xdb2 bytes) 000256[0] 1776881242.249371 mapread 0x394740 thru 0x396296 (0x1b57 bytes) 000257[0] 1776881242.262908 mapwrite 0x2fb43b thru 0x2fcffc (0x1bc2 bytes) 000258[1] 1776881242.296838 mapread 0x5a3c20 thru 0x5a4a8b (0xe6c bytes) 000260[0] 1776881242.326426 trunc from 0x88037c to 0x20b3ae (0x674fcd bytes) 000261[1] 1776881242.342525 mapwrite 0x1e5895 thru 0x1e6ab6 (0x1222 bytes) 000264[1] 1776881242.409445 mapread 0x07764e thru 0x078150 (0xb03 bytes) 000266[0] 1776881242.437452 mapread 0x0e6319 thru 0x0e730d (0xff5 bytes) 000267[1] 1776881242.453045 mapwrite 0x2a27bd thru 0x2a3c7f (0x14c3 bytes) 000268[1] 1776881242.511458 mapwrite 0x255d41 thru 0x256743 (0xa03 bytes) 000270[0] 1776881242.552901 write 0x69a889 thru 0x69b1c2 (0x93a bytes) 000271[0] 1776881242.567439 read 0x2f0b60 thru 0x2f161e (0xabf bytes) 000272[0] 1776881242.578142 trunc from 0x69b1c3 to 0x84b1af (0x1affed bytes) 000273[1] 1776881242.593269 mapwrite 0x810998 thru 0x810ae8 (0x151 bytes) 000274[0] 1776881242.640305 trunc from 0x84b1b0 to 0x399d04 (0x4b14ab bytes) 000275[1] 1776881242.655426 write 0x08419c thru 0x0857d4 (0x1639 bytes) 000280[1] 1776881242.712488 trunc from 0x399d05 to 0x2f0af5 (0xa920f bytes) 000281[1] 1776881242.736803 write 0x32205f thru 0x322129 (0xcb bytes) 000283[0] 1776881242.740683 trunc from 0x32212a to 0x8c67e9 (0x5a46c0 bytes) 000285[1] 1776881242.764077 trunc from 0x8c67ea to 0x414540 (0x4b22a9 bytes) 000286[1] 1776881242.792495 write 0x339871 thru 0x33b09b (0x182b bytes) 000290[1] 1776881242.829786 read 0x004359 thru 0x005d0b (0x19b3 bytes) 000295[0] 1776881242.873386 mapwrite 0x48ad34 thru 0x48c8db (0x1ba8 bytes) 000299[0] 1776881242.943390 write 0x77502d thru 0x775a12 (0x9e6 bytes) 000302[1] 1776881242.973349 trunc from 0x775a13 to 0x0d38cb (0x6a2147 bytes) 000304[0] 1776881242.995928 read 0x08336c thru 0x084bea (0x187f bytes) 000305[1] 1776881243.006817 read 0x0a2312 thru 0x0a2a63 (0x752 bytes) 000307[1] 1776881243.020051 mapwrite 0x347094 thru 0x34899c (0x1909 bytes) 000308[1] 1776881243.098633 read 0x230230 thru 0x23050c (0x2dd bytes) 000310[1] 1776881243.117633 write 0x3d6dc9 thru 0x3d7442 (0x67a bytes) 000311[0] 1776881243.127773 trunc from 0x3d7443 to 0x664e23 (0x28d9e1 bytes) 000312[1] 1776881243.152115 trunc from 0x664e24 to 0x022400 (0x642a23 bytes) 000313[1] 1776881243.180542 read 0x00e46b thru 0x00f92b (0x14c1 bytes) 000314[1] 1776881243.188271 mapread 0x016642 thru 0x017a06 (0x13c5 bytes) 000315[0] 1776881243.189225 write 0x978ce0 thru 0x9799ea (0xd0b bytes) 000325[1] 1776881243.286414 write 0x96f2f3 thru 0x971002 (0x1d10 bytes) 000327[0] 1776881243.370219 trunc from 0x9799eb to 0x982ef0 (0x9506 bytes) 000328[1] 1776881243.407982 read 0x323471 thru 0x324d15 (0x18a5 bytes) 000329[0] 1776881243.416614 write 0x87abac thru 0x87bee2 (0x1337 bytes) 000331[1] 1776881243.441894 mapread 0x07fdd4 thru 0x0808c3 (0xaf0 bytes) 000332[1] 1776881243.458398 trunc from 0x982ef1 to 0x39ec6a (0x5e4286 bytes) 000334[1] 1776881243.483753 trunc from 0x39ec6b to 0x06abe5 (0x334085 bytes) 000337[0] 1776881243.504744 trunc from 0x06abe6 to 0x1ede83 (0x18329e bytes) 000338[1] 1776881243.529055 write 0x134f80 thru 0x13645a (0x14db bytes) 000339[1] 1776881243.545162 mapwrite 0x784db6 thru 0x7864e8 (0x1733 bytes) 000345[1] 1776881243.627854 trunc from 0x7864e9 to 0x804cb6 (0x7e7ce bytes) 000346[1] 1776881243.643780 write 0x92a082 thru 0x92b59a (0x1519 bytes) 000350[1] 1776881243.655296 mapwrite 0x9c0f44 thru 0x9c144d (0x50a bytes) 000352[1] 1776881243.715002 trunc from 0x9c144e to 0x8ff9d0 (0xc1a7d bytes) 000353[1] 1776881243.734460 read 0x8363e5 thru 0x838289 (0x1ea5 bytes) 000356[1] 1776881243.759913 read 0x36846a thru 0x369387 (0xf1e bytes) 000358[0] 1776881243.772933 mapread 0x36d338 thru 0x36e85b (0x1524 bytes) 000360[1] 1776881243.799504 trunc from 0x8ff9d1 to 0x3d836d (0x527663 bytes) 000363[0] 1776881243.836562 mapwrite 0x398b7c thru 0x399129 (0x5ae bytes) 000364[1] 1776881243.882858 mapwrite 0x387dd8 thru 0x388eda (0x1103 bytes) 000370[0] 1776881243.949909 mapread 0x2eeeff thru 0x2f0c60 (0x1d62 bytes) 000371[1] 1776881243.972944 mapread 0x1b3b2d thru 0x1b5033 (0x1507 bytes) 000372[0] 1776881243.995103 trunc from 0x3d836e to 0x2906d2 (0x147c9b bytes) 000377[1] 1776881244.038519 read 0x1cd3e0 thru 0x1cd67e (0x29f bytes) 000380[1] 1776881244.075965 mapread 0x228909 thru 0x22a583 (0x1c7b bytes) 000381[0] 1776881244.090368 write 0x093609 thru 0x09363b (0x33 bytes) 000382[1] 1776881244.103075 mapread 0x172326 thru 0x1739fa (0x16d5 bytes) 000388[1] 1776881244.158299 mapread 0x1b6632 thru 0x1b6ed1 (0x8a0 bytes) 000389[0] 1776881244.173660 mapread 0x23eedd thru 0x240b71 (0x1c95 bytes) 000392[0] 1776881244.211377 read 0x070b4f thru 0x072646 (0x1af8 bytes) 000394[1] 1776881244.241541 read 0x01b19f thru 0x01bd64 (0xbc6 bytes) 000395[1] 1776881244.276826 mapread 0x21974f thru 0x21b259 (0x1b0b bytes) 000397[0] 1776881244.291501 mapwrite 0x6ffdeb thru 0x700725 (0x93b bytes) 000398[0] 1776881244.398331 mapread 0x46001f thru 0x4612c3 (0x12a5 bytes) 000399[1] 1776881244.411215 read 0x326637 thru 0x326be2 (0x5ac bytes) 000403[1] 1776881244.434525 trunc from 0x700726 to 0x1f95d3 (0x507152 bytes) 000408[1] 1776881244.492524 mapwrite 0x8566f7 thru 0x856894 (0x19e bytes) 000409[0] 1776881244.541474 trunc from 0x856895 to 0x711cda (0x144bba bytes) 000414[0] 1776881244.580423 mapread 0x1745e7 thru 0x1746c4 (0xde bytes) 000417[0] 1776881244.603674 mapwrite 0x9a516d thru 0x9a6190 (0x1024 bytes) 000420[0] 1776881244.681456 trunc from 0x9a6191 to 0x504755 (0x4a1a3b bytes) 000422[0] 1776881244.696754 mapwrite 0x930059 thru 0x9317a9 (0x1751 bytes) 000426[1] 1776881244.758503 mapwrite 0x4da225 thru 0x4da660 (0x43c bytes) 000431[1] 1776881244.838329 mapwrite 0x54198e thru 0x543053 (0x16c6 bytes) 000432[0] 1776881244.884273 write 0x9acb25 thru 0x9ad89b (0xd77 bytes) 000433[0] 1776881244.894119 write 0x868992 thru 0x86a454 (0x1ac3 bytes) 000434[1] 1776881244.908341 read 0x484232 thru 0x486214 (0x1fe3 bytes) 000435[1] 1776881244.920265 write 0x47e0a7 thru 0x480077 (0x1fd1 bytes) 000437[0] 1776881244.942438 mapwrite 0x35521e thru 0x35651d (0x1300 bytes) 000438[1] 1776881244.986973 write 0x472e23 thru 0x473cec (0xeca bytes) 000442[0] 1776881245.026762 trunc from 0x9ad89c to 0x10908d (0x8a480e bytes) 000443[1] 1776881245.056318 mapread 0x04e951 thru 0x04fa2f (0x10df bytes) 000447[0] 1776881245.104087 mapwrite 0x97e00d thru 0x97f9ba (0x19ae bytes) 000449[0] 1776881245.156050 mapwrite 0x0d1d81 thru 0x0d2f7a (0x11fa bytes) 000450[0] 1776881245.186112 read 0x70779f thru 0x708773 (0xfd5 bytes) 000451[0] 1776881245.201505 mapwrite 0x3689bd thru 0x369681 (0xcc5 bytes) 000452[1] 1776881245.230653 mapwrite 0x1162ba thru 0x117911 (0x1658 bytes) 000454[0] 1776881245.293821 trunc from 0x97f9bb to 0x746e86 (0x238b34 bytes) 000457[1] 1776881245.311658 write 0x33ed24 thru 0x340bd1 (0x1eae bytes) 000458[0] 1776881245.332653 write 0x1524f8 thru 0x154318 (0x1e21 bytes) 000459[1] 1776881245.356102 read 0x6f6d22 thru 0x6f7eb5 (0x1194 bytes) 000460[1] 1776881245.372467 read 0x609e2f thru 0x60a18e (0x360 bytes) 000465[1] 1776881245.414385 mapread 0x186ac2 thru 0x1889e0 (0x1f1f bytes) 000468[1] 1776881245.440563 trunc from 0x746e87 to 0x4c7e26 (0x27f060 bytes) 000471[1] 1776881245.472340 mapwrite 0x8d04c7 thru 0x8d1abb (0x15f5 bytes) 000475[0] 1776881245.535003 mapread 0x1721cb thru 0x173caf (0x1ae5 bytes) 000479[0] 1776881245.562065 mapread 0x23dc3b thru 0x23ee1c (0x11e2 bytes) 000482[0] 1776881245.581790 read 0x5fbbb2 thru 0x5fc5e3 (0xa32 bytes) 000483[1] 1776881245.590363 mapwrite 0x4ab67f thru 0x4ac49e (0xe20 bytes) 000484[1] 1776881245.630188 read 0x5aa910 thru 0x5ab532 (0xc23 bytes) 000486[1] 1776881245.643059 mapwrite 0x9112ab thru 0x911ac5 (0x81b bytes) 000489[0] 1776881245.694700 mapwrite 0x805242 thru 0x806e3d (0x1bfc bytes) 000494[0] 1776881245.767327 mapread 0x10946a thru 0x10a95d (0x14f4 bytes) 000498[1] 1776881245.813041 read 0x3586a0 thru 0x35a64a (0x1fab bytes) 000499[0] 1776881245.821961 mapwrite 0x9c7aa4 thru 0x9c84f0 (0xa4d bytes) 000500[0] 1776881245.880702 read 0x1a4bed thru 0x1a5959 (0xd6d bytes) All operations completed A-OK! PASS 16b (11s) == sanityn test 16c: verify data consistency on ldiskfs with cache disabled (b=17397) ========================================================== 14:07:28 (1776881248) SKIP: sanityn test_16c dio on ldiskfs only SKIP 16c (2s) == sanityn test 16d: Verify DIO and buffer IO with two clients ========================================================== 14:07:30 (1776881250) 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.186576 s, 45.0 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.13924 s, 60.2 MB/s ldlm.namespaces.MGC192.168.204.138@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff9c56c2fa4800.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff9c56c682c000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff9c56c2fa4800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff9c56c682c000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff9c56c2fa4800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff9c56c682c000.lru_size=clear 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.02403 s, 26.1 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 1.22095 s, 85.9 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 0.728217 s, 144 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 2.29507 s, 45.7 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 2.90389 s, 36.1 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 1.20102 s, 87.3 MB/s PASS 16d (17s) == sanityn test 16e: Verify size consistency for O_DIRECT write ========================================================== 14:07:47 (1776881267) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.147456 s, 71.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0502333 s, 20.9 MB/s PASS 16e (3s) == sanityn test 16f: rw sequential consistency vs drop_caches ========================================================== 14:07:50 (1776881270) TIMEOUT PASS 16f (24s) == sanityn test 16g: mmap rw sequential consistency vs drop_caches ========================================================== 14:08:14 (1776881294) TIMEOUT PASS 16g (23s) == sanityn test 16h: mmap read after truncate file ======= 14:08:37 (1776881317) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16h (3s) == sanityn test 16i: read after truncate file ============ 14:08:40 (1776881320) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16i (3s) == sanityn test 16j: race dio with buffered i/o ========== 14:08:43 (1776881323) 1+0 records in 1+0 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.84272 s, 31.1 MB/s bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.616551 s, 42.5 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.98476 s, 26.6 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.48339 s, 54.2 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.598997 s, 43.8 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.596309 s, 44.0 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.746468 s, 35.1 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.543914 s, 48.2 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.623091 s, 42.1 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.856096 s, 30.6 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.990082 s, 26.5 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.602558 s, 43.5 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.634872 s, 41.3 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.751972 s, 34.9 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.782683 s, 33.5 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.65902 s, 39.8 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.723568 s, 36.2 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.851097 s, 30.8 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.870671 s, 30.1 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK PASS 16j (13s) == sanityn test 16k: Parallel FSX and drop caches should not panic ========================================================== 14:08:56 (1776881336) + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.1 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.2 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.3 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.4 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.5 -l 67108864 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read truncating to largest ever: 0x1a7062c truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! + fsx 1 done: rc=0 All operations completed A-OK! Wed Apr 22 14:09:06 EDT 2026 + fsx 2 done: rc=0 Wed Apr 22 14:09:06 EDT 2026 + fsx 3 done: rc=0 Wed Apr 22 14:09:06 EDT 2026 All operations completed A-OK! + fsx 4 done: rc=0 Wed Apr 22 14:09:06 EDT 2026 + fsx 5 done: rc=0 Wed Apr 22 14:09:06 EDT 2026 PASS 16k (13s) == sanityn test 17: resource creation/LVB creation race ========================================================================= 14:09:09 (1776881349) fail_loc=0x8000030a PASS 17 (6s) == sanityn test 18: mmap sanity check =========================================================================================== 14:09:15 (1776881355) mmap test1: basic mmap operation (PASS, 0.026005s) mmap test2: MAP_PRIVATE not write back (PASS, 0.016723s) mmap test3: concurrent mmap ops on two nodes (PASS, 2.2235s) mmap test4: c1 write to f1 from mmapped f2, c2 write to f1 from mmapped f1 (PASS, 4.1455s) mmap test5: read/write file to/from the buffer which mmapped to just this file (PASS, 0.063676s) mmap test6: check mmap write/read content on two nodes (PASS, 0.0657s) mmap test7: file i/o with an unmapped buffer (SKIPPED, 0s) mmap test8: SIGBUS for beyond file size (SKIPPED, 0s) mmap test9: SIGBUS for negative file offset (SKIPPED, 0s) mmap test10: multi-thread mmap access (PASS, 0.4887s) mmap test11: multi-thread shared mmap (PASS, 0.50129s) mmap test10: mtime not change for readonly mmap access (PASS, 1.0306s) PASS 18 (13s) == sanityn test 19: test concurrent uncached read races ========================================================================= 14:09:28 (1776881368) oleg438-server: error: get_param: param_path 'osd-*/lustre-OST*/read_cache_enable': No such file or directory pdsh@oleg438-client: oleg438-server: ssh exited with exit code 2 SKIP: sanityn test_19 not cache-capable obdfilter SKIP 19 (2s) == sanityn test 20: test extra readahead page left in cache ============================================================== 14:09:30 (1776881370) PASS 20 (3s) == sanityn test 21: Try to remove mountpoint on another dir ============================================================== 14:09:33 (1776881373) rmdir: removing directory, '/mnt/lustre/d21.sanityn' rmdir: failed to remove '/mnt/lustre/d21.sanityn': Device or resource busy rmdir: removing directory, '/mnt/lustre2/d21.sanityn' Removed mounted directory from another mountpoint, needs to be fixed PASS 21 (3s) == sanityn test 23: others should see updated atime while another read============================================================== 14:09:36 (1776881376) now is 1776881377 starting reads multiop /mnt/lustre/f23.sanityn vor20_c TMPPIPE=/tmp/multiop_open_wait_pipe.7529 new atime is 1776881438 PASS 23 (65s) == sanityn test 24a: lfs df [-ih] [path] test =================================================================================== 14:10:41 (1776881441) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2048 2206592 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 6144 3763200 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2048 2206592 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 6144 3763200 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre2 UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.9K 386 524.5K 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 115.3K 402 114.9K 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 114.5K 399 114.1K 1% /mnt/lustre[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.9K 386 524.5K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.3K 402 114.9K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 399 114.1K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre2 UUID bytes Used Available Use% Mounted on lustre-MDT0000_UUID 2.1G 2.0M 2.1G 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3.6G 6.0M 3.6G 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3.6G 32.0M 3.6G 1% /mnt/lustre[OST:1] filesystem_summary: 7.2G 38.0M 7.2G 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537447 386 537061 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118066 402 117664 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117231 399 116832 1% /mnt/lustre2[OST:1] filesystem_summary: 234882 386 234496 1% /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2048 2206592 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 6144 3763200 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.9K 386 524.5K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.3K 402 114.9K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 399 114.1K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537447 386 537061 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 118066 402 117664 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 117231 399 116832 1% /mnt/lustre[OST:1] filesystem_summary: 234882 386 234496 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537447 386 537061 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118066 402 117664 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117231 399 116832 1% /mnt/lustre2[OST:1] filesystem_summary: 234882 386 234496 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2048 2206592 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 6144 3763200 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2048 2206592 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 6144 3763200 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre2 PASS 24a (3s) == sanityn test 24b: lfs df should show both filesystems ========================================================================= 14:10:44 (1776881444) PASS 24b (3s) == sanityn test 25a: change ACL on one mountpoint be seen on another ============================================================= 14:10:47 (1776881447) running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] PASS 25a (3s) == sanityn test 25b: change ACL under remote dir on one mountpoint be seen on another ========================================================== 14:10:50 (1776881450) SKIP: sanityn test_25b needs >= 2 MDTs SKIP 25b (2s) == sanityn test 26a: allow mtime to get older ============ 14:10:52 (1776881452) utime: good mknod times 1776881451 <= 1776881452 <= 1776881452 for /mnt/lustre/f26a utime: good mknod times 1776881451 <= 1776881452 <= 1776881452 for /mnt/lustre2/f26a utime: good utime mtimes 100000, atime 200000 utime: updated times for /mnt/lustre2/f26a PASS 26a (4s) == sanityn test 26b: sync mtime between ost and mds ====== 14:10:56 (1776881456) PASS 26b (5s) == sanityn test 26c: set-in-past on open file is not lost on close ========================================================== 14:11:01 (1776881461) multiop /mnt/lustre/f26c.sanityn vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7529 File: /mnt/lustre/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502398 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-04-22 14:11:02.000000000 -0400 Birth: 2026-04-22 14:11:02.000000000 -0400 File: /mnt/lustre2/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502398 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-04-22 14:11:02.000000000 -0400 Birth: 2026-04-22 14:11:02.000000000 -0400 PASS 26c (4s) SKIP: sanityn test_27 skipping excluded test 27 == sanityn test 30: recreate file race =================== 14:11:06 (1776881466) 0 PASS 30 (5s) == sanityn test 31a: voluntary cancel / blocking ast race======================================================================== 14:11:11 (1776881471) fail_loc=0x314 PASS 31a (4s) == sanityn test 31b: voluntary OST cancel / blocking ast race======================================================================== 14:11:15 (1776881475) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg438-server mds-ost sync done. sleep 5 for ZFS MDS Waiting for MDT destroys to complete fail_loc=0x314 fail_loc=0x316 fail_loc=0 fail_loc=0 Filesystem 1K-blocks Used Available Use% Mounted on 192.168.204.138@tcp:/lustre 7542784 39936 7496704 1% /mnt/lustre2 PASS 31b (13s) == sanityn test 31r: open-rename(replace) race =========== 14:11:28 (1776881488) -rw-r--r-- 1 root root 0 Apr 22 14:11 /mnt/lustre/f31r.sanityn.target fail_loc=0x1419 fail_val=3 mv: failed to access '/mnt/lustre2/f31r.sanityn.target': Input/output error PASS 31r (6s) == sanityn test 31s: open should not revalidate invalid dentry ========================================================== 14:11:34 (1776881494) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 1207: 42793 Terminated tail -f $tfile PASS 31s (3s) == sanityn test 31t: getattr should not revalidate invalid dentry ========================================================== 14:11:37 (1776881497) PASS 31t (4s) SKIP: sanityn test_33a skipping SLOW test 33a == sanityn test 33b: COS: cross create/delete, 2 clients, benchmark under remote dir ========================================================== 14:11:42 (1776881502) SKIP: sanityn test_33b Need two or more clients, have 1 SKIP 33b (2s) == sanityn test 33c: Cancel cross-MDT lock should trigger Sync-on-Lock-Cancel ========================================================== 14:11:44 (1776881504) SKIP: sanityn test_33c needs >= 2 MDTs SKIP 33c (1s) == sanityn test 33d: dependent transactions should trigger COS ========================================================== 14:11:46 (1776881506) SKIP: sanityn test_33d needs >= 2 MDTs SKIP 33d (2s) == sanityn test 33e: independent transactions shouldn't trigger COS ========================================================== 14:11:47 (1776881507) SKIP: sanityn test_33e needs >= 2 MDTs SKIP 33e (1s) == sanityn test 34: no lock timeout under IO ============= 14:11:48 (1776881508) fail_loc=0x512 fail_loc=0x512 lock should not expire writing on client1 reading on client2 success fail_loc=0x511 fail_loc=0x511 lock should expire writing on client1 reading on client2 success oleg438-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9c56c2fa4800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9c56c2fa4800.ost_server_uuid in FULL state after 0 sec oleg438-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9c56c2fa4800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9c56c2fa4800.ost_server_uuid in IDLE state after 0 sec oleg438-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9c56c2fa4800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9c56c2fa4800.ost_server_uuid in FULL state after 0 sec oleg438-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9c56c2fa4800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9c56c2fa4800.ost_server_uuid in IDLE state after 0 sec oleg438-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9c56c2fa4800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9c56c2fa4800.ost_server_uuid in FULL state after 0 sec oleg438-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9c56c2fa4800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9c56c2fa4800.ost_server_uuid in IDLE state after 0 sec PASS 34 (100s) == sanityn test 35: -EINTR cp_ast vs. bl_ast race does not evict client ========================================================== 14:13:28 (1776881608) fail_loc=0x80000317 Race attempt 0 Wait for 48841 48951 for 60 sec... - open/close 2540 (time 1776881621.60 total 10.00 last 253.99) total: 4000 open/close in 16.56 seconds: 241.54 ops/second fail_loc=0x0 Filesystem Size Used Avail Use% Mounted on 192.168.204.138@tcp:/lustre 7.2G 39M 7.2G 1% /mnt/lustre 192.168.204.138@tcp:/lustre 7.2G 39M 7.2G 1% /mnt/lustre2 PASS 35 (66s) == sanityn test 36: handle ESTALE/open-unlink correctly == 14:14:34 (1776881674) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg438-server mds-ost sync done. Waiting for MDT destroys to complete 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.460468 s, 114 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7529 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(0) *** before(7498752) after_dd(7447552) after(7498752) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.511027 s, 103 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7529 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(1) *** before(7498752) after_dd(7447552) after(7498752) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.48856 s, 107 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7529 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(2) *** before(7498752) after_dd(7447552) after(7498752) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.387602 s, 135 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7529 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(3) *** before(7498752) after_dd(7445504) after(7496704) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.46595 s, 113 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7529 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(4) *** before(7496704) after_dd(7447552) after(7498752) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.455803 s, 115 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7529 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(5) *** before(7498752) after_dd(7447552) after(7498752) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.524798 s, 99.9 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7529 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(6) *** before(7498752) after_dd(7447552) after(7498752) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.50098 s, 105 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7529 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(7) *** before(7498752) after_dd(7447552) after(7498752) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.407826 s, 129 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7529 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(8) *** before(7498752) after_dd(7447552) after(7498752) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.581537 s, 90.2 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7529 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(9) *** before(7498752) after_dd(7447552) after(7498752) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.428034 s, 122 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7529 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(10) *** before(7498752) after_dd(7447552) after(7498752) PASS 36 (236s) == sanityn test 37: check i_size is not updated for directory on close (bug 18695) ======================================================================== 14:18:30 (1776881910) multiop /mnt/lustre/d37.sanityn vD_c TMPPIPE=/tmp/multiop_open_wait_pipe.7529 - create 4944 (time 1776881922.33 total 10.00 last 494.31) - create 9255 (time 1776881932.33 total 20.00 last 431.04) total: 10000 create in 21.73 seconds: 460.14 ops/second PASS 37 (27s) == sanityn test 39a: file mtime does not change after rename ========================================================== 14:18:57 (1776881937) repeat after cancel_lru_locks PASS 39a (3s) == sanityn test 39b: file mtime the same on clients with/out lock ========================================================== 14:19:00 (1776881940) repeat after cancel_lru_locks PASS 39b (4s) == sanityn test 39c: check truncate mtime update ================================================================================ 14:19:04 (1776881944) repeat after cancel_lru_locks PASS 39c (5s) == sanityn test 39d: sync write should update mtime ====== 14:19:09 (1776881949) fail_loc=0x411 fail_loc=0 PASS 39d (3s) SKIP: sanityn test_40a skipping ALWAYS excluded test 40a == sanityn test 40b: pdirops: open|create and others ======================================================================== 14:19:12 (1776881952) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40b (10s) == sanityn test 40c: pdirops: link and others ======================================================================== 14:19:22 (1776881962) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40c (10s) == sanityn test 40d: pdirops: unlink and others ======================================================================== 14:19:32 (1776881972) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40d (9s) == sanityn test 40e: pdirops: rename and others ======================================================================== 14:19:42 (1776881982) No conflict No conflict No conflict No conflict No conflict No conflict PASS 40e (9s) == sanityn test 41a: pdirops: create vs mkdir ======================================================================== 14:19:51 (1776881991) mkdir: cannot create directory '/mnt/lustre2/f41a.sanityn': File exists Conflict PASS 41a (6s) == sanityn test 41b: pdirops: create vs create ======================================================================== 14:19:58 (1776881998) open: File exists Conflict PASS 41b (7s) == sanityn test 41c: pdirops: create vs link ======================================================================== 14:20:04 (1776882004) link: cannot create link '/mnt/lustre2/f41c.sanityn' to '/mnt/lustre2/f41c.sanityn-2': File exists Conflict PASS 41c (7s) == sanityn test 41d: pdirops: create vs unlink ======================================================================== 14:20:11 (1776882011) Conflict PASS 41d (7s) == sanityn test 41e: pdirops: create and rename (tgt) ======================================================================== 14:20:18 (1776882018) Conflict PASS 41e (7s) == sanityn test 41f: pdirops: create and rename (src) ======================================================================== 14:20:25 (1776882025) Conflict PASS 41f (7s) == sanityn test 41g: pdirops: create vs getattr ======================================================================== 14:20:32 (1776882032) Conflict PASS 41g (7s) == sanityn test 41h: pdirops: create vs readdir ======================================================================== 14:20:39 (1776882039) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 2318: lctl set_param -n fail_loc=0 2>/dev/null || true: No such file or directory Conflict PASS 41h (6s) == sanityn test 41i: reint_open: create vs create ======== 14:20:45 (1776882045) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 41i (767s) == sanityn test 42a: pdirops: mkdir vs mkdir ======================================================================== 14:33:32 (1776882812) mkdir: cannot create directory '/mnt/lustre2/f42a.sanityn': File exists Conflict PASS 42a (7s) == sanityn test 42b: pdirops: mkdir vs create ======================================================================== 14:33:39 (1776882819) open: File exists Conflict PASS 42b (6s) == sanityn test 42c: pdirops: mkdir vs link ======================================================================== 14:33:45 (1776882825) link: cannot create link '/mnt/lustre2/f42c.sanityn' to '/mnt/lustre2/f42c.sanityn-2': File exists Conflict PASS 42c (7s) == sanityn test 42d: pdirops: mkdir vs unlink ======================================================================== 14:33:52 (1776882832) Conflict PASS 42d (7s) == sanityn test 42e: pdirops: mkdir and rename (tgt) ======================================================================== 14:33:59 (1776882839) mv: cannot overwrite directory '/mnt/lustre2/f42e.sanityn' with non-directory Conflict PASS 42e (7s) == sanityn test 42f: pdirops: mkdir and rename (src) ======================================================================== 14:34:06 (1776882846) Conflict PASS 42f (7s) == sanityn test 42g: pdirops: mkdir vs getattr ======================================================================== 14:34:13 (1776882853) Conflict PASS 42g (7s) == sanityn test 42h: pdirops: mkdir vs readdir ======================================================================== 14:34:20 (1776882860) Conflict PASS 42h (7s) == sanityn test 43a: rmdir,mkdir doesn't return -EEXIST ======================================================================== 14:34:27 (1776882867) PASS 43a (27s) == sanityn test 43b: pdirops: unlink vs create ======================================================================== 14:34:55 (1776882895) Conflict PASS 43b (6s) == sanityn test 43c: pdirops: unlink vs link ======================================================================== 14:35:01 (1776882901) Conflict PASS 43c (8s) == sanityn test 43d: pdirops: unlink vs unlink ======================================================================== 14:35:09 (1776882909) rm: cannot remove '/mnt/lustre2/f43d.sanityn': No such file or directory Conflict PASS 43d (7s) == sanityn test 43e: pdirops: unlink and rename (tgt) ======================================================================== 14:35:16 (1776882916) Conflict PASS 43e (7s) == sanityn test 43f: pdirops: unlink and rename (src) ======================================================================== 14:35:23 (1776882923) mv: cannot stat '/mnt/lustre2/f43f.sanityn': No such file or directory Conflict PASS 43f (7s) == sanityn test 43g: pdirops: unlink vs getattr ======================================================================== 14:35:30 (1776882930) stat: cannot statx '/mnt/lustre2/f43g.sanityn': No such file or directory Conflict PASS 43g (7s) == sanityn test 43h: pdirops: unlink vs readdir ======================================================================== 14:35:37 (1776882937) Conflict PASS 43h (7s) == sanityn test 43i: pdirops: unlink vs remote mkdir ===== 14:35:44 (1776882944) SKIP: sanityn test_43i needs >= 2 MDTs SKIP 43i (2s) == sanityn test 43j: racy mkdir return EEXIST ======================================================================== 14:35:46 (1776882946) mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists PASS 43j (56s) == sanityn test 43k: unlink vs create ==================== 14:36:42 (1776883002) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 43k (519s) == sanityn test 44a: pdirops: rename tgt vs mkdir ======================================================================== 14:45:21 (1776883521) mkdir: cannot create directory '/mnt/lustre2/f44a.sanityn': File exists Conflict Wed Apr 22 14:45:25 EDT 2026 PASS 44a (5s) == sanityn test 44b: pdirops: rename tgt vs create ======================================================================== 14:45:26 (1776883526) open: File exists Conflict PASS 44b (5s) == sanityn test 44c: pdirops: rename tgt vs link ======================================================================== 14:45:31 (1776883531) link: cannot create link '/mnt/lustre2/f44c.sanityn' to '/mnt/lustre2/f44c.sanityn-3': File exists Conflict PASS 44c (4s) == sanityn test 44d: pdirops: rename tgt vs unlink ======================================================================== 14:45:35 (1776883535) Conflict PASS 44d (5s) == sanityn test 44e: pdirops: rename tgt and rename (tgt) ======================================================================== 14:45:40 (1776883540) Conflict PASS 44e (6s) == sanityn test 44f: pdirops: rename tgt and rename (src) ======================================================================== 14:45:46 (1776883546) Conflict PASS 44f (7s) == sanityn test 44g: pdirops: rename tgt vs getattr ======================================================================== 14:45:53 (1776883553) Conflict PASS 44g (7s) == sanityn test 44h: pdirops: rename tgt vs readdir ======================================================================== 14:46:00 (1776883560) Conflict PASS 44h (7s) == sanityn test 44i: pdirops: rename tgt vs remote mkdir ========================================================== 14:46:07 (1776883567) SKIP: sanityn test_44i needs >= 2 MDTs SKIP 44i (1s) == sanityn test 45a: rename,mkdir doesn't return -EEXIST ======================================================================== 14:46:08 (1776883568) PASS 45a (47s) == sanityn test 45b: pdirops: rename src vs create ======================================================================== 14:46:55 (1776883615) Conflict PASS 45b (6s) == sanityn test 45c: pdirops: rename src vs link ======================================================================== 14:47:01 (1776883621) Conflict PASS 45c (7s) == sanityn test 45d: pdirops: rename src vs unlink ======================================================================== 14:47:08 (1776883628) rm: cannot remove '/mnt/lustre2/f45d.sanityn': No such file or directory Conflict PASS 45d (6s) == sanityn test 45e: pdirops: rename src and rename (tgt) ======================================================================== 14:47:15 (1776883635) Conflict PASS 45e (6s) == sanityn test 45f: pdirops: rename src and rename (src) ======================================================================== 14:47:21 (1776883641) mv: cannot stat '/mnt/lustre2/f45f.sanityn': No such file or directory Conflict PASS 45f (7s) == sanityn test 45g: pdirops: rename src vs getattr ======================================================================== 14:47:28 (1776883648) stat: cannot statx '/mnt/lustre2/f45g.sanityn': No such file or directory Conflict PASS 45g (6s) == sanityn test 45h: pdirops: unlink vs readdir ======================================================================== 14:47:34 (1776883654) Conflict PASS 45h (5s) == sanityn test 45i: pdirops: rename src vs remote mkdir ========================================================== 14:47:39 (1776883659) SKIP: sanityn test_45i needs >= 2 MDTs SKIP 45i (1s) == sanityn test 45j: read vs rename ====================== 14:47:40 (1776883660) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 45j (593s) == sanityn test 46a: pdirops: link vs mkdir ======================================================================== 14:57:33 (1776884253) mkdir: cannot create directory '/mnt/lustre2/f46a.sanityn': File exists Conflict PASS 46a (7s) == sanityn test 46b: pdirops: link vs create ======================================================================== 14:57:40 (1776884260) open: File exists Conflict PASS 46b (7s) == sanityn test 46c: pdirops: link vs link ======================================================================== 14:57:47 (1776884267) link: cannot create link '/mnt/lustre2/f46c.sanityn' to '/mnt/lustre2/f46c.sanityn': File exists Conflict PASS 46c (7s) == sanityn test 46d: pdirops: link vs unlink ======================================================================== 14:57:54 (1776884274) Conflict PASS 46d (7s) == sanityn test 46e: pdirops: link and rename (tgt) ======================================================================== 14:58:01 (1776884281) Conflict PASS 46e (7s) == sanityn test 46f: pdirops: link and rename (src) ======================================================================== 14:58:08 (1776884288) Conflict PASS 46f (7s) == sanityn test 46g: pdirops: link vs getattr ======================================================================== 14:58:15 (1776884295) Conflict PASS 46g (7s) == sanityn test 46h: pdirops: link vs readdir ======================================================================== 14:58:22 (1776884302) Conflict PASS 46h (8s) == sanityn test 46i: pdirops: link vs remote mkdir ======= 14:58:30 (1776884310) SKIP: sanityn test_46i needs >= 2 MDTs SKIP 46i (1s) == sanityn test 47a: pdirops: remote mkdir vs mkdir ====== 14:58:31 (1776884311) SKIP: sanityn test_47a needs >= 2 MDTs SKIP 47a (2s) == sanityn test 47b: pdirops: remote mkdir vs create ===== 14:58:33 (1776884313) SKIP: sanityn test_47b needs >= 2 MDTs SKIP 47b (1s) == sanityn test 47c: pdirops: remote mkdir vs link ======= 14:58:35 (1776884315) SKIP: sanityn test_47c needs >= 2 MDTs SKIP 47c (1s) == sanityn test 47d: pdirops: remote mkdir vs unlink ===== 14:58:36 (1776884316) SKIP: sanityn test_47d needs >= 2 MDTs SKIP 47d (2s) == sanityn test 47e: pdirops: remote mkdir and rename (tgt) ========================================================== 14:58:38 (1776884318) SKIP: sanityn test_47e needs >= 2 MDTs SKIP 47e (1s) == sanityn test 47f: pdirops: remote mkdir and rename (src) ========================================================== 14:58:40 (1776884320) SKIP: sanityn test_47f needs >= 2 MDTs SKIP 47f (2s) == sanityn test 47g: pdirops: remote mkdir vs getattr ==== 14:58:41 (1776884321) SKIP: sanityn test_47g needs >= 2 MDTs SKIP 47g (2s) == sanityn test 50: osc lvb attrs: enqueue vs. CP AST ======================================================================== 14:58:43 (1776884323) 10+0 records in 10+0 records out 10240 bytes (10 kB, 10 KiB) copied, 0.0334169 s, 306 kB/s fail_loc=0x410 fail_loc=0x0 PASS 50 (8s) == sanityn test 51a: layout lock: refresh layout should work ========================================================== 14:58:51 (1776884331) 0+1 records in 0+1 records out 158 bytes copied, 0.0107221 s, 14.7 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 3581: kill: (216527) - No such process PASS 51a (5s) == sanityn test 51b: layout lock: glimpse should be able to restart if layout changed ========================================================== 14:58:57 (1776884337) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00524938 s, 195 kB/s fail_loc=0x1404 fail_val=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0307757 s, 34.1 MB/s 3145728 PASS 51b (16s) == sanityn test 51c: layout lock: IT_LAYOUT blocked and correct layout can be returned ========================================================== 14:59:12 (1776884352) fail_loc=0x172 Setting layout to have 2 stripes ... 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00832318 s, 123 kB/s PASS 51c (8s) == sanityn test 51d: layout lock: losing layout lock should clean up memory map region ========================================================== 14:59:20 (1776884360) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.024583 s, 42.7 MB/s Before revoking layout lock: 1024 KB mapped PASS 51d (4s) == sanityn test 51e: lfs getstripe does not break leases, part 2 ========================================================== 14:59:24 (1776884364) /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 2454 0x996 0x240000400 write lease(2) has applied. /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 2454 0x996 0x240000400 read lease(1) has applied. PASS 51e (6s) == sanityn test 54: rename locking ======================= 14:59:30 (1776884370) llite.lustre-ffff9c56c2fa4800.xattr_cache=1 llite.lustre-ffff9c56c682c000.xattr_cache=1 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 512 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505011 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-04-22 14:59:31.000000000 -0400 Modify: 2026-04-22 14:59:31.000000000 -0400 Change: 2026-04-22 14:59:31.000000000 -0400 Birth: 2026-04-22 14:59:31.000000000 -0400 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000154 File: /mnt/lustre/d1/d2 Size: 7680 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505014 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-04-22 14:59:36.000000000 -0400 Modify: 2026-04-22 14:59:37.000000000 -0400 Change: 2026-04-22 14:59:37.000000000 -0400 Birth: 2026-04-22 14:59:36.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 7680 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505017 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-04-22 14:59:42.000000000 -0400 Modify: 2026-04-22 14:59:42.000000000 -0400 Change: 2026-04-22 14:59:42.000000000 -0400 Birth: 2026-04-22 14:59:42.000000000 -0400 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1 Size: 7680 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505016 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-04-22 14:59:42.000000000 -0400 Modify: 2026-04-22 14:59:42.000000000 -0400 Change: 2026-04-22 14:59:42.000000000 -0400 Birth: 2026-04-22 14:59:42.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000154 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1/d2 Size: 7680 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505020 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-04-22 14:59:48.000000000 -0400 Modify: 2026-04-22 14:59:48.000000000 -0400 Change: 2026-04-22 14:59:48.000000000 -0400 Birth: 2026-04-22 14:59:48.000000000 -0400 File: /mnt/lustre/d1 Size: 7680 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505019 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-04-22 14:59:48.000000000 -0400 Modify: 2026-04-22 14:59:48.000000000 -0400 Change: 2026-04-22 14:59:48.000000000 -0400 Birth: 2026-04-22 14:59:48.000000000 -0400 PASS 54 (26s) == sanityn test 55a: rename vs unlink target dir ========= 14:59:56 (1776884396) fail_loc=0x80000156 mv: cannot move '/mnt/lustre/d55a.sanityn/d1/d2' to '/mnt/lustre/d55a.sanityn/d3/d2': No such file or directory PASS 55a (9s) == sanityn test 55b: rename vs unlink source dir ========= 15:00:05 (1776884405) fail_loc=0x80000156 mv: cannot stat '/mnt/lustre/d55b.sanityn/d1/d2': No such file or directory PASS 55b (8s) == sanityn test 55c: rename vs unlink orphan target dir == 15:00:13 (1776884413) fail_loc=0x156 mv: cannot move '/mnt/lustre/d55c.sanityn/d1/d2' to '/mnt/lustre/d55c.sanityn/d3/d2': No such file or directory PASS 55c (13s) == sanityn test 55d: rename file vs link ================= 15:00:26 (1776884426) fail_loc=0x155 mv: '/mnt/lustre/d55d.sanityn/f1' and '/mnt/lustre/d55d.sanityn/d55d.sanityn/f1' are the same file PASS 55d (9s) == sanityn test 55e: rename race AB/BA under the same parent dir ========================================================== 15:00:35 (1776884435) SKIP: sanityn test_55e needs >= 2 MDTs SKIP 55e (1s) == sanityn test 55f: rename: (P1/A -> P2/B) race with (P2/B -> P1/A) ========================================================== 15:00:36 (1776884436) fail_loc=0x155 We survived after P1A_P2B/P2B_P1A race test fail_loc=0x0 PASS 55f (14s) == sanityn test 55g: rename: race with trylock =========== 15:00:50 (1776884450) mdt.lustre-MDT0000.enable_rename_trylock=1 fail_loc=0x155 mv: cannot move '/mnt/lustre2/d55g.sanityn/b1/b2' to a subdirectory of itself, '/mnt/lustre2/d55g.sanityn/a1/a2/a3/a4/b2' We survived race with trylock test fail_loc=0x0 PASS 55g (16s) == sanityn test 56a: test llverdev with single large stripe ========================================================== 15:01:06 (1776884466) run_llverdev /mnt/lustre/f56a.sanityn -p -s 16000000 -c 4k llverdev: /mnt/lustre/f56a.sanityn is 16777216000000 bytes (15625 GB) in size timestamp: 1776884467 chunksize: 4096 size: 16777216000000 write offset: 16384000000kB 1.969 MB/s read offset: 16384000000kB inf MB/s llverdev: data verified successfully PASS 56a (34s) == sanityn test 56b: test llverdev and partial verify of wide stripe file ========================================================== 15:01:40 (1776884500) run_llverdev /mnt/lustre/f56b.sanityn -p -s 32000000 -c 4k llverdev: /mnt/lustre/f56b.sanityn is 33554432000000 bytes (31250 GB) in size timestamp: 1776884502 chunksize: 4096 size: 33554432000000 write offset: 32768000000kB 3.39095 MB/s read offset: 32768000000kB inf MB/s llverdev: data verified successfully PASS 56b (42s) == sanityn test 60: Verify data_version behaviour ======== 15:02:22 (1776884542) PASS 60 (3s) == sanityn test 70a: cd directory && rm directory ======== 15:02:25 (1776884545) PASS 70a (3s) == sanityn test 70b: remove files after calling rm_entry ========================================================== 15:02:28 (1776884548) PASS 70b (3s) == sanityn test 71a: correct file map just after write operation is finished ========================================================== 15:02:31 (1776884551) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.003826 s, 17.1 MB/s SKIP: sanityn test_71a ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71a (1s) == sanityn test 71b: check fiemap support for stripecount > 1 ========================================================== 15:02:32 (1776884552) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.00456167 s, 14.4 MB/s SKIP: sanityn test_71b ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71b (1s) == sanityn test 71c: check FIEMAP_EXTENT_LAST flag with different extents number ========================================================== 15:02:33 (1776884553) SKIP: sanityn test_71c support only ldiskfs ost SKIP 71c (2s) == sanityn test 71d: fiemap corruption test with fm_extent_count=0 ========================================================== 15:02:35 (1776884555) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.00899379 s, 7.3 MB/s SKIP: sanityn test_71d ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71d (1s) == sanityn test 72: getxattr/setxattr cache should be consistent between nodes ========================================================== 15:02:36 (1776884556) llite.lustre-ffff9c56c2fa4800.xattr_cache=1 llite.lustre-ffff9c56c682c000.xattr_cache=1 getfattr: Removing leading '/' from absolute path names user.attr1="value1" getfattr: Removing leading '/' from absolute path names user.attr1="value2" getfattr: Removing leading '/' from absolute path names getfattr: Removing leading '/' from absolute path names b362740ad371eae580e7f85452e8830b - 870e072da1ebfc20ea8c0cd3d6bdf3b6 - PASS 72 (4s) == sanityn test 73: getxattr should not cause xattr lock cancellation ========================================================== 15:02:40 (1776884560) llite.lustre-ffff9c56c2fa4800.xattr_cache=1 llite.lustre-ffff9c56c682c000.xattr_cache=1 getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" PASS 73 (3s) == sanityn test 74: flock deadlock: different mounts ======================================================================== 15:02:43 (1776884563) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00467534 s, 219 kB/s 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00657647 s, 156 kB/s 229313: get lock2 229314: get lock1 229313: done 229313: get lock1 229314: done 229314: get lock2 fcntl cmd 7 failed: Resource deadlock avoided 229313: cannot get lock1: Resource deadlock avoided 229314: done 229314: exit rc=0 229313: done 229313: exit rc=0 PASS 74 (7s) == sanityn test 75: osc: upcall after unuse lock============================================================================= 15:02:50 (1776884570) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0199305 s, 105 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013585 s, 77.2 MB/s fail_loc=0x31d /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 4264: 229894 Killed stat -c %s $DIR1/$tfile fail_loc=0 PASS 75 (8s) == sanityn test 76: Verify MDT open_files listing ======== 15:02:58 (1776884578) opening files: ........ closing files: ........ PASS 76 (26s) == sanityn test 77a: check FIFO NRS policy =============== 15:03:24 (1776884604) ost.OSS.ost.nrs_policies=fifo ost.OSS.ost_create.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_out.nrs_policies=fifo ost.OSS.ost_seq.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.217586 s, 77.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0558914 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0665271 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0638907 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0760626 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0853581 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0715866 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0925536 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.089877 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0807068 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144635 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149216 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151387 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159851 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143055 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176012 s, 6.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9c56c682c000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9c56c682c000.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158858 s, 6.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0407407 s, 25.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0334662 s, 31.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0315857 s, 33.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0392959 s, 26.7 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9c56c682c000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.040987 s, 25.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0268541 s, 39.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0260349 s, 40.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0229954 s, 45.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0408839 s, 25.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0360375 s, 29.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0378327 s, 27.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0370031 s, 28.3 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0314621 s, 33.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0348944 s, 30.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0466538 s, 22.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0274843 s, 38.2 MB/s removed '/mnt/lustre/d77a.sanityn/nrs_r_oleg438-client.virtnet' removed '/mnt/lustre/d77a.sanityn/nrs_w_oleg438-client.virtnet' removed directory '/mnt/lustre/d77a.sanityn' PASS 77a (4s) == sanityn test 77b: check CRR-N NRS policy ============== 15:03:28 (1776884608) ost.OSS.ost.nrs_policies=crrn ost.OSS.ost_create.nrs_policies=crrn ost.OSS.ost_io.nrs_policies=crrn ost.OSS.ost_out.nrs_policies=crrn ost.OSS.ost_seq.nrs_policies=crrn ost.OSS.ost.nrs_crrn_quantum=1 ost.OSS.ost_create.nrs_crrn_quantum=1 ost.OSS.ost_io.nrs_crrn_quantum=1 ost.OSS.ost_out.nrs_crrn_quantum=1 ost.OSS.ost_seq.nrs_crrn_quantum=1 policy: crr-n, crrn_quantum 1 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.13442 s, 125 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0452312 s, 23.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0916595 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0876709 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115539 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106148 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0870066 s, 12.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126991 s, 8.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112411 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111287 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11723 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134782 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110451 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159946 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132264 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120554 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126285 s, 8.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0258044 s, 40.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0213845 s, 49.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0222574 s, 47.1 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0223992 s, 46.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0283833 s, 36.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0211313 s, 49.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0105208 s, 99.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156209 s, 67.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157944 s, 66.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00681309 s, 154 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0252231 s, 41.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0076495 s, 137 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00549867 s, 191 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0181391 s, 57.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0261246 s, 40.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0205117 s, 51.1 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg438-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg438-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost.nrs_crrn_quantum=64 ost.OSS.ost_create.nrs_crrn_quantum=64 ost.OSS.ost_io.nrs_crrn_quantum=64 ost.OSS.ost_out.nrs_crrn_quantum=64 ost.OSS.ost_seq.nrs_crrn_quantum=64 policy: crr-n, crrn_quantum 64 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.124666 s, 135 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0265654 s, 39.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0270342 s, 38.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0510768 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0504655 s, 20.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.055777 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0321978 s, 32.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.049925 s, 21.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0725559 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.055966 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0738324 s, 14.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0924117 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102189 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105703 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0968629 s, 10.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0937386 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116567 s, 9.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162827 s, 64.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0236107 s, 44.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0190141 s, 55.1 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167271 s, 62.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0175258 s, 59.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0175062 s, 59.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0256466 s, 40.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0281062 s, 37.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0260841 s, 40.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130715 s, 80.2 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9c56c682c000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144951 s, 72.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0210224 s, 49.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0314917 s, 33.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00320149 s, 328 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0301677 s, 34.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0074671 s, 140 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg438-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg438-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77b (5s) == sanityn test 77c: check ORR NRS policy ================ 15:03:33 (1776884613) ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_orr_quantum=1 ost.OSS.ost_io.nrs_orr_offset_type=physical ost.OSS.ost_io.nrs_orr_supported=reads policy: orr, orr_quantum 1, orr_offset_type physical, orr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.176812 s, 94.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0281981 s, 37.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.036346 s, 28.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0377651 s, 27.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.05425 s, 19.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0455606 s, 23.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0612514 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0504159 s, 20.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0648595 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0730019 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0929619 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0795925 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0898678 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106129 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115224 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111653 s, 9.4 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9c56c682c000.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1334 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136648 s, 76.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161981 s, 64.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160076 s, 65.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.018265 s, 57.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0196132 s, 53.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0191307 s, 54.8 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.025536 s, 41.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0085697 s, 122 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00812945 s, 129 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00945797 s, 111 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0176104 s, 59.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00588735 s, 178 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.02446 s, 42.9 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00592267 s, 177 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00933249 s, 112 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159586 s, 65.7 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg438-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg438-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=writes ost.OSS.ost_io.nrs_orr_quantum=64 policy: orr, orr_quantum 64, orr_offset_type physical, orr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.215638 s, 77.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0239753 s, 43.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0284261 s, 36.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0403747 s, 26.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104412 s, 10.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110183 s, 9.5 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116599 s, 9.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109645 s, 9.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106175 s, 9.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.091869 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126502 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103716 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0984915 s, 10.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121637 s, 8.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11848 s, 8.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114885 s, 9.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9c56c682c000.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff9c56c682c000.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107151 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151217 s, 69.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151702 s, 69.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0202155 s, 51.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160216 s, 65.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0240845 s, 43.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016399 s, 63.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00424305 s, 247 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00881569 s, 119 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0242844 s, 43.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152361 s, 68.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0121582 s, 86.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0242118 s, 43.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00354664 s, 296 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00871975 s, 120 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9c56c682c000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00527029 s, 199 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0021685 s, 484 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg438-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg438-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=reads_and_writes ost.OSS.ost_io.nrs_orr_offset_type=logical policy: orr, orr_quantum 64, orr_offset_type logical, orr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.233001 s, 72.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0411292 s, 25.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0344232 s, 30.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104748 s, 10.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9c56c682c000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106898 s, 9.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10819 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0923331 s, 11.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103312 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108169 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101459 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109642 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0935302 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0998468 s, 10.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104101 s, 10.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108905 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123919 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122043 s, 8.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0209441 s, 50.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0111002 s, 94.5 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016688 s, 62.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00375249 s, 279 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170298 s, 61.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0210152 s, 49.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173342 s, 60.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.017358 s, 60.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0293094 s, 35.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0079164 s, 132 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167959 s, 62.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.025903 s, 40.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0251341 s, 41.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00992551 s, 106 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0171145 s, 61.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0053618 s, 196 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9c56c682c000.lock_unused_count=2 removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg438-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg438-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77c (6s) == sanityn test 77d: check TRR nrs policy ================ 15:03:39 (1776884619) ost.OSS.ost_io.nrs_policies=trr ost.OSS.ost_io.nrs_trr_quantum=1 ost.OSS.ost_io.nrs_trr_offset_type=physical ost.OSS.ost_io.nrs_trr_supported=reads policy: trr, trr_quantum 1, trr_offset_type physical, trr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.159357 s, 105 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0238966 s, 43.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0333928 s, 31.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0617253 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.041472 s, 25.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0638762 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0612944 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0515459 s, 20.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0743228 s, 14.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9c56c682c000.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0897783 s, 11.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103076 s, 10.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120239 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103563 s, 10.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132966 s, 7.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12877 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132089 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157681 s, 66.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140357 s, 74.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0254423 s, 41.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0282657 s, 37.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00468201 s, 224 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0296838 s, 35.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0251236 s, 41.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0192166 s, 54.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0227966 s, 46.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00844088 s, 124 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0243827 s, 43.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0227929 s, 46.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0294493 s, 35.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0242207 s, 43.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0282107 s, 37.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0277135 s, 37.8 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9c56c682c000.lock_unused_count=9 removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg438-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg438-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=writes ost.OSS.ost_io.nrs_trr_quantum=64 policy: trr, trr_quantum 64, trr_offset_type physical, trr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.2017 s, 83.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0302149 s, 34.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0460755 s, 22.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0417532 s, 25.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.075091 s, 14.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0944409 s, 11.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0764784 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0836501 s, 12.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0944537 s, 11.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109949 s, 9.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0927045 s, 11.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0901453 s, 11.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101367 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0881475 s, 11.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0966038 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0919678 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115284 s, 9.1 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9c56c682c000.lock_unused_count=6 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159705 s, 65.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016405 s, 63.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165941 s, 63.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016668 s, 62.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0204182 s, 51.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00908965 s, 115 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0248909 s, 42.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183528 s, 57.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00402447 s, 261 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00360705 s, 291 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0229512 s, 45.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0234216 s, 44.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0111942 s, 93.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00729071 s, 144 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0263285 s, 39.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159118 s, 65.9 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg438-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg438-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=reads_and_writes ost.OSS.ost_io.nrs_trr_offset_type=logical policy: trr, trr_quantum 64, trr_offset_type logical, trr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.110231 s, 152 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0494632 s, 21.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0749512 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136569 s, 7.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135311 s, 7.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151314 s, 6.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123183 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142469 s, 7.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141154 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161383 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12245 s, 8.6 MB/s 1+0 records in 1+0 records out ldlm.namespaces.lustre-OST0000-osc-ffff9c56c682c000.lock_unused_count=3 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123704 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149275 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143864 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151424 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149622 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144361 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016183 s, 64.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0124787 s, 84.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169031 s, 62.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151204 s, 69.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0267894 s, 39.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0206758 s, 50.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0207367 s, 50.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0213598 s, 49.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0331495 s, 31.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0184968 s, 56.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131113 s, 80.0 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00695552 s, 151 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0248634 s, 42.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160416 s, 65.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0388075 s, 27.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158548 s, 66.1 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg438-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg438-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77d (7s) == sanityn test 77e: check TBF NID nrs policy ============ 15:03:46 (1776884626) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_tbf_rule=start localhost nid={0@lo} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start clients nid={192.168.204.38@tcp} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start others nid={*.*.*.*@tcp} rate=50 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.197332 s, 85.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0409873 s, 25.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0372664 s, 28.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132738 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119643 s, 8.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142976 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150039 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159549 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162668 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159306 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178949 s, 5.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156605 s, 6.7 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165746 s, 6.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154712 s, 6.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147792 s, 7.1 MB/s 1+0 records in 1+0 records out ldlm.namespaces.lustre-OST0000-osc-ffff9c56c682c000.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff9c56c682c000.lock_unused_count=7 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173256 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161187 s, 6.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201582 s, 5.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20021 s, 5.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213594 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201466 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198695 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196347 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202659 s, 5.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201074 s, 5.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201866 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21236 s, 4.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199826 s, 5.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210947 s, 5.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209143 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216146 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.214799 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207894 s, 5.0 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg438-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg438-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change localhost rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change clients rate=101 ost.OSS.ost_io.nrs_tbf_rule=change others rate=51 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.191315 s, 87.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0303464 s, 34.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0768154 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106628 s, 9.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124209 s, 8.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127413 s, 8.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119667 s, 8.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0979889 s, 10.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128541 s, 8.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135621 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117551 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143896 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131183 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134479 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148405 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13667 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119637 s, 8.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9c56c682c000.lock_unused_count=3 ldlm.namespaces.lustre-OST0001-osc-ffff9c56c682c000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.235362 s, 4.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237679 s, 4.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.230499 s, 4.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23369 s, 4.5 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.241087 s, 4.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.232887 s, 4.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.233273 s, 4.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.235073 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23332 s, 4.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237982 s, 4.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.24519 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.243827 s, 4.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.240919 s, 4.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.235968 s, 4.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.254091 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.239604 s, 4.4 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg438-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg438-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop localhost ost.OSS.ost_io.nrs_tbf_rule=stop clients ost.OSS.ost_io.nrs_tbf_rule=stop others 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.190702 s, 88.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0301984 s, 34.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0472431 s, 22.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0482897 s, 21.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.051249 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0359101 s, 29.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0525346 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0760428 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0822603 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0909596 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0789362 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0922386 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1027 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0873828 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0919962 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102484 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104822 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0081867 s, 128 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0110185 s, 95.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0186254 s, 56.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0103303 s, 102 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0214636 s, 48.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0206527 s, 50.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0241089 s, 43.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0176987 s, 59.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0254646 s, 41.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142627 s, 73.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0113684 s, 92.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00980867 s, 107 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00377966 s, 277 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01905 s, 55.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00578137 s, 181 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0073139 s, 143 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9c56c682c000.lock_unused_count=1 removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg438-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg438-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.185308 s, 90.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.024082 s, 43.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0447124 s, 23.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0486305 s, 21.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0603862 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0409378 s, 25.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0609949 s, 17.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0655663 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0690959 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0663535 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0934445 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0828978 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105501 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11249 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12163 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121106 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111503 s, 9.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00500768 s, 209 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0291864 s, 35.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0124417 s, 84.3 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00408528 s, 257 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9c56c682c000.lock_unused_count=8 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0204441 s, 51.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0192642 s, 54.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0211796 s, 49.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00321908 s, 326 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0207437 s, 50.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0314561 s, 33.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155335 s, 67.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01104 s, 95.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0359445 s, 29.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00306611 s, 342 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013217 s, 79.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00367832 s, 285 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg438-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg438-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' PASS 77e (10s) == sanityn test 77f: check TBF JobID nrs policy ========== 15:03:56 (1776884636) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start runas jobid={iozone.500 dd.500 tiotest.500} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start iozone_runas jobid={iozone.500} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg438-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.311562 s, 53.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0267476 s, 39.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0817646 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0780006 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0923066 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0688766 s, 15.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9c56c682c000.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0782981 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0912487 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0865457 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0889214 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0799412 s, 13.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0648269 s, 16.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0868417 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0925856 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0962508 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110109 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115066 s, 9.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied ldlm.namespaces.lustre-OST0000-osc-ffff9c56c682c000.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0229506 s, 45.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0288925 s, 36.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0255338 s, 41.1 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00673471 s, 156 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0284472 s, 36.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152516 s, 68.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.032117 s, 32.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0212064 s, 49.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0239193 s, 43.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0248182 s, 42.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0106677 s, 98.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0265551 s, 39.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0258999 s, 40.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0349483 s, 30.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0191806 s, 54.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0421235 s, 24.9 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg438-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg438-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change runas rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change iozone_runas rate=101 ost.OSS.ost_io.nrs_tbf_rule=change dd_runas rate=51 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg438-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.249513 s, 67.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0430222 s, 24.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0313025 s, 33.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0359529 s, 29.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0861538 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0668025 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0773796 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118474 s, 8.9 MB/s 1+0 records in 1+0 records out running as uid/gid/euid/egid 500/500/500/500, groups:1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0963306 s, 10.9 MB/s 500 [declare] [-a] [pids_r] 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0925302 s, 11.3 MB/s execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14563 s, 7.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114625 s, 9.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108133 s, 9.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123853 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130355 s, 8.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12892 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133678 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0113204 s, 92.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out ldlm.namespaces.lustre-OST0000-osc-ffff9c56c682c000.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffff9c56c682c000.lock_unused_count=6 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.010884 s, 96.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00405151 s, 259 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0257275 s, 40.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0123102 s, 85.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0210267 s, 49.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00767401 s, 137 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00992596 s, 106 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0387815 s, 27.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0296841 s, 35.3 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0201984 s, 51.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00393008 s, 267 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00974236 s, 108 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00805652 s, 130 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00367466 s, 285 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00494427 s, 212 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg438-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg438-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop runas ost.OSS.ost_io.nrs_tbf_rule=stop iozone_runas ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg438-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.158079 s, 106 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.033333 s, 31.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0519294 s, 20.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0583716 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0846344 s, 12.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0904161 s, 11.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107284 s, 9.8 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0992621 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.084155 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111656 s, 9.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0918639 s, 11.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0830347 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.090749 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115455 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105878 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0989468 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105569 s, 9.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0266587 s, 39.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0186287 s, 56.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0205799 s, 51.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161352 s, 65.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013655 s, 76.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138772 s, 75.6 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.023119 s, 45.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0249883 s, 42.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0219603 s, 47.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0269372 s, 38.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153507 s, 68.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148304 s, 70.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0115756 s, 90.6 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9c56c682c000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0221062 s, 47.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00554251 s, 189 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166662 s, 62.9 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg438-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg438-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_policies=fifo running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg438-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.152433 s, 110 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0458483 s, 22.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0596046 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0506776 s, 20.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0606658 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0565004 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0585285 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0710683 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0658708 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0568768 s, 18.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9c56c682c000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0789662 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0921136 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0992705 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11895 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100262 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0842268 s, 12.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119117 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130566 s, 80.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0123038 s, 85.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183547 s, 57.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165703 s, 63.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173843 s, 60.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00639676 s, 164 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0109335 s, 95.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0204307 s, 51.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0198643 s, 52.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00668765 s, 157 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00363898 s, 288 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0269144 s, 39.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0228276 s, 45.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0227133 s, 46.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0239002 s, 43.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0239242 s, 43.8 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg438-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg438-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' PASS 77f (10s) == sanityn test 77g: Change TBF type directly ============ 15:04:06 (1776884646) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 ost.OSS.ost_io.nrs_policies=fifo PASS 77g (5s) == sanityn test 77h: Wrong policy name should report error, not LBUG ========================================================== 15:04:11 (1776884651) oleg438-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=abc: No such file or directory oleg438-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='abc': No such file or directory pdsh@oleg438-client: oleg438-server: ssh exited with exit code 2 oleg438-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc: Invalid argument oleg438-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc': Invalid argument pdsh@oleg438-client: oleg438-server: ssh exited with exit code 22 oleg438-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf reg abc: Invalid argument oleg438-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf reg abc': Invalid argument pdsh@oleg438-client: oleg438-server: ssh exited with exit code 22 oleg438-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc efg: Invalid argument oleg438-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc efg': Invalid argument pdsh@oleg438-client: oleg438-server: ssh exited with exit code 22 PASS 77h (5s) == sanityn test 77i: Change rank of TBF rule ============= 15:04:16 (1776884656) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start before jobid={jobid} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start after jobid={jobid} rate=1000 rank=default ost.OSS.ost_io.nrs_tbf_rule=start target jobid={jobid} rate=1000 rank=after Move before itself ost.OSS.ost_io.nrs_tbf_rule=change target rank=target Move to higher rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=before Move to lower rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=after Move before default ost.OSS.ost_io.nrs_tbf_rule=change target rank=default ost.OSS.ost_io.nrs_policies=fifo PASS 77i (10s) == sanityn test 77j: check TBF-OPCode NRS policy ========= 15:04:26 (1776884666) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf opcode ost.OSS.ost_io.nrs_tbf_rule=start ost_r opcode={ost_read} rate=5 ost.OSS.ost_io.nrs_tbf_rule=start ost_w opcode={ost_write} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.188118 s, 89.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0188434 s, 55.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0627993 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0777179 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0788423 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0877767 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0843166 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0679949 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0883849 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0841484 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.092154 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0905793 s, 11.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0825629 s, 12.7 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0887539 s, 11.8 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0774254 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.079814 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0859626 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0191546 s, 54.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.020343 s, 51.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0196342 s, 53.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0113594 s, 92.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.034757 s, 30.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00975427 s, 107 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0177412 s, 59.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0268027 s, 39.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0292343 s, 35.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0103373 s, 101 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00376985 s, 278 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0278488 s, 37.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00392282 s, 267 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0284625 s, 36.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0113739 s, 92.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00746669 s, 140 MB/s removed '/mnt/lustre/d77j.sanityn/nrs_r_oleg438-client.virtnet' removed '/mnt/lustre/d77j.sanityn/nrs_w_oleg438-client.virtnet' removed directory '/mnt/lustre/d77j.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 5 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.88168 s, 21.5 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.2758 s, 5.4 MB/s Read runtime is 20 s, speed is 5.000000 IOPS cleanup_tbf /mnt/lustre/d77j.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_r ost.OSS.ost_io.nrs_tbf_rule=stop ost_w ost.OSS.ost_io.nrs_policies=fifo PASS 77j (53s) == sanityn test 77ja: check TBF-UID/GID NRS policy ======= 15:05:19 (1776884719) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg438-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg438-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg438-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg438-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg438-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.22412 s, 13.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0854562 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0961792 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0667561 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.059204 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0965615 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0839461 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0793718 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.083842 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118897 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119357 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100066 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10767 s, 9.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0990025 s, 10.6 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141105 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140099 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130173 s, 8.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00574891 s, 182 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.024127 s, 43.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0223806 s, 46.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0256788 s, 40.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.027201 s, 38.5 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0193495 s, 54.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0299636 s, 35.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0199425 s, 52.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0320464 s, 32.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0269391 s, 38.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00649394 s, 161 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0357781 s, 29.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0264261 s, 39.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0271252 s, 38.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00384145 s, 273 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00446982 s, 235 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg438-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg438-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.4642 s, 5.4 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9091 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg438-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg438-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg438-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg438-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg438-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.2222 s, 13.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.030698 s, 34.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.06429 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0578798 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0833977 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0975624 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0913721 s, 11.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114263 s, 9.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123182 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0929999 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109055 s, 9.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145026 s, 7.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145126 s, 7.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130499 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106875 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115072 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130008 s, 8.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0209836 s, 50.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0245754 s, 42.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0266028 s, 39.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0210106 s, 49.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0244721 s, 42.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00329381 s, 318 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0238415 s, 44.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0307445 s, 34.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0241021 s, 43.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00468432 s, 224 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0248302 s, 42.2 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0232184 s, 45.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.027709 s, 37.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00392428 s, 267 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00563036 s, 186 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00170265 s, 616 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg438-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg438-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.4816 s, 5.4 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.89 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77ja (133s) == sanityn test 77jb: check TBF-UID/GID NRS policy on files that don't belong to us ========================================================== 15:07:32 (1776884852) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg438-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg438-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg438-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg438-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg438-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.46177 s, 36.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0712346 s, 14.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0718268 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0772272 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0619161 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0894142 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0950164 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0832186 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0920035 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0872954 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0939282 s, 11.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0984463 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0901326 s, 11.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103878 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0968723 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10313 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0988182 s, 10.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0227722 s, 46.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0175807 s, 59.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0190104 s, 55.2 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0232539 s, 45.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0176142 s, 59.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0282775 s, 37.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0108409 s, 96.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0234655 s, 44.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0222022 s, 47.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0198346 s, 52.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0220719 s, 47.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00438089 s, 239 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00472644 s, 222 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00392034 s, 267 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00377799 s, 278 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00195154 s, 537 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg438-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg438-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.4923 s, 5.4 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9038 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg438-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg438-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg438-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg438-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg438-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.22038 s, 13.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0871098 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0868015 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0722425 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0752055 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0959212 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.078811 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0917193 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0816994 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12667 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126646 s, 8.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121502 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125403 s, 8.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121736 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146097 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139002 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124162 s, 8.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0248492 s, 42.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0220273 s, 47.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183556 s, 57.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.026686 s, 39.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0264964 s, 39.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0247282 s, 42.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159158 s, 65.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0311862 s, 33.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.020075 s, 52.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0265947 s, 39.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00537374 s, 195 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133016 s, 78.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00909886 s, 115 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00514867 s, 204 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0285772 s, 36.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00432201 s, 243 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg438-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg438-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.5074 s, 5.4 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8986 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77jb (138s) == sanityn test 77k: check TBF policy with UID/GID/JobID/OPCode expression ========================================================== 15:09:50 (1776884990) ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg438-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.284814 s, 58.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0268528 s, 39.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0506149 s, 20.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0494521 s, 21.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0585854 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.057036 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0340901 s, 30.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.052153 s, 20.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0735445 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0832759 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0663659 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0784523 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0670487 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0891591 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0898685 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100589 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0951165 s, 11.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167112 s, 62.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0209387 s, 50.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0255055 s, 41.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0265303 s, 39.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0187028 s, 56.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0229946 s, 45.6 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.025671 s, 40.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0248962 s, 42.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0232688 s, 45.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0283617 s, 37.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0066939 s, 157 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0187785 s, 55.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0177277 s, 59.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00679919 s, 154 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0103846 s, 101 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00463148 s, 226 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg438-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg438-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.85759 s, 21.6 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.71279 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.204.38@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.204.38@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.226188 s, 74.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.026153 s, 40.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0474183 s, 22.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0359909 s, 29.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.067394 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0495307 s, 21.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0719257 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0498445 s, 21.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0499264 s, 21.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0650924 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0845146 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0964305 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0934124 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100683 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107245 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112841 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106609 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0174026 s, 60.3 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.017265 s, 60.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0230967 s, 45.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0199387 s, 52.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.029497 s, 35.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0371764 s, 28.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0215451 s, 48.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.028019 s, 37.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0261501 s, 40.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0189973 s, 55.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144416 s, 72.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135658 s, 77.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00740644 s, 142 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00245569 s, 427 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00978607 s, 107 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.011707 s, 89.6 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg438-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg438-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.90193 s, 21.4 MB/s Write runtime is 5 s, speed is 20.000000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.68293 s, 10.8 MB/s Read runtime is 10 s, speed is 10.000000 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.204.38@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg438-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.220435 s, 76.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0377386 s, 27.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0324226 s, 32.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0425494 s, 24.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.06362 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0538775 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0684193 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0850541 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0898624 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0965961 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110602 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0948476 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0966137 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107984 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127182 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104742 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108758 s, 9.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0190692 s, 55.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0188748 s, 55.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138148 s, 75.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155176 s, 67.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0122221 s, 85.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0205041 s, 51.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0213176 s, 49.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0304985 s, 34.4 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0290227 s, 36.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0265443 s, 39.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00617021 s, 170 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0225686 s, 46.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00749148 s, 140 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00324918 s, 323 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00318067 s, 330 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00368044 s, 285 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg438-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg438-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.87387 s, 21.5 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.86403 s, 21.6 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg438-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.234996 s, 71.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0509773 s, 20.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112669 s, 9.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133239 s, 7.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135258 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156234 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145637 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116937 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157016 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139203 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14449 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142256 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14101 s, 7.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142836 s, 7.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146117 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.29503 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.298574 s, 3.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0232453 s, 45.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0350807 s, 29.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0293647 s, 35.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.02844 s, 36.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159947 s, 65.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012728 s, 82.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0314852 s, 33.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0266193 s, 39.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00739532 s, 142 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0348653 s, 30.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0244179 s, 42.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154785 s, 67.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00936066 s, 112 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0283887 s, 36.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0258171 s, 40.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015365 s, 68.2 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg438-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg438-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.69915 s, 10.8 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.71453 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.8476 s, 21.6 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.65229 s, 10.9 MB/s Read runtime is 10 s, speed is 10.000000 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg438-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.2183 s, 76.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0468941 s, 22.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0580412 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0828847 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0689326 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0949619 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0746688 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0741998 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0694571 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0799343 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105283 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146439 s, 7.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129961 s, 8.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154922 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137378 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15833 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133575 s, 7.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0323265 s, 32.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0296259 s, 35.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.036208 s, 29.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0185188 s, 56.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0222798 s, 47.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00943823 s, 111 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0219591 s, 47.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0208548 s, 50.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0305281 s, 34.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0207886 s, 50.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183402 s, 57.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0190823 s, 55.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0379049 s, 27.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0384172 s, 27.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0191518 s, 54.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130084 s, 80.6 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg438-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg438-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.2981 s, 5.4 MB/s Write runtime is 20 s, speed is 5.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.4145 s, 5.4 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_ug ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg438-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.233827 s, 71.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0467823 s, 22.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0368264 s, 28.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113065 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103754 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146072 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160304 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143636 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156213 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15223 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172663 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161668 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141098 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145417 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151918 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151924 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152052 s, 6.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139161 s, 75.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0209896 s, 50.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0269729 s, 38.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0265332 s, 39.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.04528 s, 23.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0214114 s, 49.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0351075 s, 29.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183767 s, 57.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138019 s, 76.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0259281 s, 40.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0355318 s, 29.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.028402 s, 36.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161851 s, 64.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145205 s, 72.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0118491 s, 88.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0220788 s, 47.5 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg438-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg438-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.88759 s, 21.5 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.71506 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg438-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.243581 s, 68.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0518982 s, 20.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0758015 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0345271 s, 30.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0587954 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0692066 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100424 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.067439 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0734171 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112535 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108662 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144573 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127077 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.29594 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.314558 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.312197 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.322515 s, 3.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0206927 s, 50.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0248734 s, 42.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0309067 s, 33.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.021161 s, 49.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0253118 s, 41.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014265 s, 73.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0212564 s, 49.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0356419 s, 29.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0322564 s, 32.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0334353 s, 31.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0107244 s, 97.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164778 s, 63.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173568 s, 60.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0176864 s, 59.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00418797 s, 250 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140822 s, 74.5 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg438-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg438-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.71402 s, 10.8 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.71446 s, 10.8 MB/s Read runtime is 10 s, speed is 10.000000 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.71019 s, 10.8 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.62035 s, 10.9 MB/s Read runtime is 10 s, speed is 10.000000 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_policies=fifo PASS 77k (359s) == sanityn test 77kb: Check different granular TBF type combination: jobid+opcode ========================================================== 15:15:49 (1776885349) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/nrs_r_oleg438-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.265098 s, 63.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0668243 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0596801 s, 17.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100154 s, 10.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.09564 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102486 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106529 s, 9.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124788 s, 8.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120103 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137756 s, 7.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135451 s, 7.7 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13042 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143506 s, 7.3 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166701 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17127 s, 6.1 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145823 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132836 s, 7.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0233346 s, 44.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0337946 s, 31.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0397799 s, 26.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0228443 s, 45.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0331537 s, 31.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0198421 s, 52.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0190283 s, 55.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0290052 s, 36.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0364701 s, 28.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0348723 s, 30.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00974323 s, 108 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132518 s, 79.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0171458 s, 61.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0199403 s, 52.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.020114 s, 52.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127418 s, 82.3 MB/s removed '/mnt/lustre/d77kb.sanityn/nrs_w_oleg438-client.virtnet' removed '/mnt/lustre/d77kb.sanityn/nrs_r_oleg438-client.virtnet' removed directory '/mnt/lustre/d77kb.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.87118 s, 21.5 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.71408 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kb (43s) == sanityn test 77kc: Check different granular TBF type combination: nid+opcode ========================================================== 15:16:32 (1776885392) ost.OSS.ost_io.nrs_policies=tbf nid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.204.38@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.204.38@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.276602 s, 60.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0556786 s, 18.8 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0755801 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10217 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0937729 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105043 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113045 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131343 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125406 s, 8.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139622 s, 7.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161359 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126732 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12109 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167665 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193745 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216817 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.247518 s, 4.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0202632 s, 51.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0301553 s, 34.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0292736 s, 35.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0343204 s, 30.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0247023 s, 42.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0264303 s, 39.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0214319 s, 48.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0217412 s, 48.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00366557 s, 286 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0322028 s, 32.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00829138 s, 126 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0260362 s, 40.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0290342 s, 36.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0041081 s, 255 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00252179 s, 416 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00249422 s, 420 MB/s removed '/mnt/lustre/d77kc.sanityn/nrs_r_oleg438-client.virtnet' removed '/mnt/lustre/d77kc.sanityn/nrs_w_oleg438-client.virtnet' removed directory '/mnt/lustre/d77kc.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.863 s, 21.6 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.6871 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kc.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kc (39s) == sanityn test 77kd: Check different granular TBF type combination: nid+jobid ========================================================== 15:17:12 (1776885432) ost.OSS.ost_io.nrs_policies=tbf nid+jobid ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.204.38@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/nrs_r_oleg438-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.251251 s, 66.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0464012 s, 22.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0624612 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0856828 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0741688 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0651108 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0740943 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0996796 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114743 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114633 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101388 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126269 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142471 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139753 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160419 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152853 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1668 s, 6.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0225349 s, 46.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0213983 s, 49.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0216537 s, 48.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0253991 s, 41.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0241619 s, 43.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0240448 s, 43.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0310052 s, 33.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00650479 s, 161 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156755 s, 66.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0360202 s, 29.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0185348 s, 56.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0103672 s, 101 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0103376 s, 101 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0221033 s, 47.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0193183 s, 54.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148663 s, 70.5 MB/s removed '/mnt/lustre/d77kd.sanityn/nrs_w_oleg438-client.virtnet' removed '/mnt/lustre/d77kd.sanityn/nrs_r_oleg438-client.virtnet' removed directory '/mnt/lustre/d77kd.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.89942 s, 21.4 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kd.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.89752 s, 21.4 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kd.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kd (35s) == sanityn test 77ke: Check different granular TBF type combination: jobid+opcode ========================================================== 15:17:47 (1776885467) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/nrs_r_oleg438-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.442543 s, 37.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0319625 s, 32.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0634057 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0306967 s, 34.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0553527 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.088726 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103687 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100499 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0858461 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0902786 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0852895 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0969273 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107766 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165091 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137782 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193689 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206391 s, 5.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0181113 s, 57.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0213337 s, 49.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0286286 s, 36.6 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0237051 s, 44.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0239098 s, 43.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0235923 s, 44.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0190227 s, 55.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0241531 s, 43.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0176655 s, 59.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136132 s, 77.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00844281 s, 124 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0238891 s, 43.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0040564 s, 258 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00575612 s, 182 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00776026 s, 135 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00452176 s, 232 MB/s removed '/mnt/lustre/d77ke.sanityn/nrs_w_oleg438-client.virtnet' removed '/mnt/lustre/d77ke.sanityn/nrs_r_oleg438-client.virtnet' removed directory '/mnt/lustre/d77ke.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.72592 s, 10.8 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ke.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.71396 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.85012 s, 21.6 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.69901 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77ke (81s) == sanityn test 77kf: Check different granular TBF type combination: uid+gid ========================================================== 15:19:08 (1776885548) ost.OSS.ost_io.nrs_policies=tbf uid+gid ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/nrs_r_oleg438-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.841196 s, 19.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0501434 s, 20.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0973758 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0803281 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0704353 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124882 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124969 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.095751 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0944945 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10775 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131075 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154709 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147511 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166619 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180376 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200818 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178676 s, 5.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0216941 s, 48.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164405 s, 63.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0239165 s, 43.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0205696 s, 51.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0252935 s, 41.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0274204 s, 38.2 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016701 s, 62.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0235806 s, 44.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00927598 s, 113 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016774 s, 62.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016381 s, 64.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0350582 s, 29.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0375764 s, 27.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0338258 s, 31.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0211042 s, 49.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00916024 s, 114 MB/s removed '/mnt/lustre/d77kf.sanityn/nrs_w_oleg438-client.virtnet' removed '/mnt/lustre/d77kf.sanityn/nrs_r_oleg438-client.virtnet' removed directory '/mnt/lustre/d77kf.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.3281 s, 5.4 MB/s Write runtime is 20 s, speed is 5.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kf.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9047 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77kf.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kf (68s) == sanityn test 77kg: check different granular TBF type combination: uid+opcode ========================================================== 15:20:16 (1776885616) ost.OSS.ost_io.nrs_policies=tbf uid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg438-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.269453 s, 62.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0507644 s, 20.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0434624 s, 24.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133336 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135998 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155093 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165174 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168437 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169517 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190667 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166523 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211494 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188068 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197843 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215265 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196237 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186013 s, 5.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0227665 s, 46.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0276676 s, 37.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0280177 s, 37.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0317329 s, 33.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0285323 s, 36.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0353381 s, 29.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0387086 s, 27.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150297 s, 69.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0248655 s, 42.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0377749 s, 27.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0343237 s, 30.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0120867 s, 86.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0114432 s, 91.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0234771 s, 44.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0181825 s, 57.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00874462 s, 120 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg438-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg438-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.86716 s, 21.5 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.71378 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg438-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.23872 s, 70.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0714084 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0704363 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0754782 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0520994 s, 20.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0638272 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0714605 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0779105 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0959514 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0561633 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103657 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102128 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128634 s, 8.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.30773 s, 3.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.317597 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.362044 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.347027 s, 3.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0179035 s, 58.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0215304 s, 48.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.033231 s, 31.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.033791 s, 31.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0354264 s, 29.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0229942 s, 45.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0190246 s, 55.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0195485 s, 53.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.019601 s, 53.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0283646 s, 37.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163193 s, 64.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0247635 s, 42.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0268594 s, 39.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0191166 s, 54.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0359292 s, 29.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0403032 s, 26.0 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg438-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg438-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.66135 s, 10.9 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.71426 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.72624 s, 10.8 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.62451 s, 10.9 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kg (123s) == sanityn test 77kh: Verify that Project ID support for NRS TBF rule ========================================================== 15:22:19 (1776885739) using SAVE_PROJECT_SUPPORTED=0 Stopping clients: oleg438-client.virtnet /mnt/lustre (opts:) Stopping client oleg438-client.virtnet /mnt/lustre opts: Stopping clients: oleg438-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg438-client.virtnet /mnt/lustre2 opts: Stopping /mnt/lustre-mds1 (opts:-f) on oleg438-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg438-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg438-server sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime) proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) devtmpfs on /dev type devtmpfs (rw,nosuid,mode=755) securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime) tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev) devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000) tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755) tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755) cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd) pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime) bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700) cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb) cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event) cgroup on /sys/fs/cgroup/misc type cgroup (rw,nosuid,nodev,noexec,relatime,misc) cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio) cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids) cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer) cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices) cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma) cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct) cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory) cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio) cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset) none on /sys/kernel/tracing type tracefs (rw,relatime) configfs on /sys/kernel/config type configfs (rw,relatime) /dev/nbd0 on / type ext4 (ro,relatime,stripe=32) rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) mqueue on /dev/mqueue type mqueue (rw,relatime) debugfs on /sys/kernel/debug type debugfs (rw,relatime) hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M) none on /mnt type ramfs (rw,relatime) /dev/vda on /home/green/git/lustre-release type squashfs (ro,relatime) none on /var/lib/stateless/writable type tmpfs (rw,relatime) none on /var/cache/man type tmpfs (rw,relatime) none on /var/log type tmpfs (rw,relatime) none on /var/lib/dbus type tmpfs (rw,relatime) none on /tmp type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) none on /var/tmp type tmpfs (rw,relatime) none on /var/lib/NetworkManager type tmpfs (rw,relatime) none on /var/lib/systemd/random-seed type tmpfs (rw,relatime) none on /var/spool type tmpfs (rw,relatime) none on /var/lib/nfs type tmpfs (rw,relatime) none on /var/lib/gssproxy type tmpfs (rw,relatime) none on /var/lib/logrotate type tmpfs (rw,relatime) none on /etc type tmpfs (rw,relatime) none on /var/lib/rsyslog type tmpfs (rw,relatime) none on /root/.ssh type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) 192.168.200.253:/exports/state/oleg438-client.virtnet on /var/lib/stateless/state type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.204.38,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg438-client.virtnet/boot on /boot type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.204.38,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg438-client.virtnet/etc/kdump.conf on /etc/kdump.conf type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.204.38,local_lock=none,addr=192.168.200.253) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=378180k,mode=700) 192.168.200.253:/exports/testreports/62963/testresults/sanityn-zfs-rocky8.10_x86_64-rocky8.10_x86_64 on /tmp/testlogs type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.204.38,local_lock=none,addr=192.168.200.253) /dev/vda on /usr/sbin/mount.lustre type squashfs (ro,relatime) Checking servers environments Checking clients oleg438-client.virtnet environments /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg438-server' oleg438-server: oleg438-server.virtnet: executing load_modules_local oleg438-server: Loading modules from /home/green/git/lustre-release/lustre oleg438-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg438-server: detected 4 online CPUs by sysfs oleg438-server: MODOPTS_LIBCFS= oleg438-server: Force libcfs to create 2 CPU partitions Setup mgs, mdt, osts Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg438-server: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg438-client: oleg438-server: ssh exited with exit code 1 Started lustre-MDT0000 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg438-server: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg438-client: oleg438-server: ssh exited with exit code 1 Started lustre-OST0000 Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg438-server: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg438-client: oleg438-server: ssh exited with exit code 1 Started lustre-OST0001 Starting client: oleg438-client.virtnet: -o user_xattr,flock 192.168.204.138@tcp:/lustre /mnt/lustre Starting client oleg438-client.virtnet: -o user_xattr,flock 192.168.204.138@tcp:/lustre /mnt/lustre Started clients oleg438-client.virtnet: 192.168.204.138@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) Starting client: oleg438-client.virtnet: -o user_xattr,flock 192.168.204.138@tcp:/lustre /mnt/lustre2 Starting client oleg438-client.virtnet: -o user_xattr,flock 192.168.204.138@tcp:/lustre /mnt/lustre2 Started clients oleg438-client.virtnet: 192.168.204.138@tcp:/lustre on /mnt/lustre2 type lustre (rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9c56c31dc000.idle_timeout=debug osc.lustre-OST0000-osc-ffff9c56c61a4000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9c56c31dc000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9c56c61a4000.idle_timeout=debug disable quota as required ost.OSS.ost_io.nrs_policies=tbf projid ost.OSS.ost_io.nrs_tbf_rule=start ext_proj_rw projid={100} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.239936 s, 69.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0577944 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0551452 s, 19.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0384441 s, 27.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0877106 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0827058 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0585093 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0813826 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0699705 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0895385 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103487 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136866 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122746 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.238695 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.245006 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.271393 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.282629 s, 3.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0344954 s, 30.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0223738 s, 46.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0215439 s, 48.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.031544 s, 33.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.026241 s, 40.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0286767 s, 36.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0407675 s, 25.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0340274 s, 30.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0334909 s, 31.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0277788 s, 37.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0276366 s, 37.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0286345 s, 36.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0302188 s, 34.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0315005 s, 33.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0326486 s, 32.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0381709 s, 27.5 MB/s removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg438-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg438-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.99738 s, 17.5 MB/s Write runtime is 7 s, speed is 14.285714 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.8836 s, 21.5 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_proj_rw ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf projid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w projid={100}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r projid={100}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.228468 s, 73.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0228816 s, 45.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0408986 s, 25.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0492356 s, 21.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0878577 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.099352 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0948925 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0783477 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0779261 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0986057 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0907106 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0909496 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0952639 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103687 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0918016 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.234049 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.234175 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135868 s, 77.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163882 s, 64.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170418 s, 61.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0220455 s, 47.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0120661 s, 86.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0243367 s, 43.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0235776 s, 44.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0070397 s, 149 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0176934 s, 59.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0231369 s, 45.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0078551 s, 133 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00483076 s, 217 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0261909 s, 40.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00588327 s, 178 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0298108 s, 35.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0297202 s, 35.3 MB/s removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg438-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg438-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.86849 s, 21.5 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.62456 s, 10.9 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77kh (145s) == sanityn test 77ki: Add rule with unsupported TBF type should fail for generic TBF ========================================================== 15:24:44 (1776885884) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 oleg438-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_fail projid={100}&opcode={ost_write} rate=20: Invalid argument oleg438-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_fail projid={100}&opcode={ost_write} rate=20': Invalid argument pdsh@oleg438-client: oleg438-server: ssh exited with exit code 22 ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77ki (7s) == sanityn test 77l: check the output of NRS policies for generic TBF ========================================================== 15:24:51 (1776885891) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_policies=fifo PASS 77l (3s) == sanityn test 77m: check NRS Delay slows write RPC processing ========================================================== 15:24:55 (1776885895) ost.OSS.ost_io.nrs_policies=delay ost.OSS.ost_io.nrs_delay_min=4 ost.OSS.ost_io.nrs_delay_max=4 ost.OSS.ost_io.nrs_delay_pct=100 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg438-client.virtnet] [bs=1M] [count=1] [oflag=direct] [conv=fdatasync] 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 3.96216 s, 265 kB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg438-client.virtnet] [bs=1M] [count=10] [oflag=direct] [conv=fdatasync] 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 40.0069 s, 262 kB/s ost.OSS.ost_io.nrs_policies=fifo PASS 77m (50s) == sanityn test 77n: check wildcard support for TBF JobID NRS policy ========================================================== 15:25:46 (1776885946) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={*.500} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.131799 s, 127 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0264312 s, 39.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0338496 s, 31.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0744375 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0903457 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0859109 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0804456 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106144 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0957606 s, 10.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116713 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124915 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121225 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101513 s, 10.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123376 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126337 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129929 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124557 s, 8.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0111414 s, 94.1 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0174777 s, 60.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.02069 s, 50.7 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0119041 s, 88.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134212 s, 78.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.017372 s, 60.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167674 s, 62.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0068909 s, 152 MB/s 1+0 records in 1+0 records out ldlm.namespaces.lustre-OST0000-osc-ffff9c56c61a4000.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffff9c56c61a4000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00428843 s, 245 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0261847 s, 40.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0234951 s, 44.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00603247 s, 174 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145872 s, 71.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0178108 s, 58.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0104075 s, 101 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00356167 s, 294 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg438-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg438-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77n.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.86699 s, 21.5 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77n.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.95105 s, 21.2 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.*} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.224124 s, 74.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0365876 s, 28.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0689359 s, 15.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0687511 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0823991 s, 12.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0643403 s, 16.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0896587 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0777328 s, 13.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.071979 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0738108 s, 14.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0876106 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0685354 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0714848 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.241875 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.249548 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.291588 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.296808 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0198933 s, 52.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0182175 s, 57.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0179203 s, 58.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0177793 s, 59.0 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0229691 s, 45.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.020737 s, 50.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0242814 s, 43.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0277767 s, 37.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0227598 s, 46.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0206699 s, 50.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0199356 s, 52.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0194763 s, 53.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0251907 s, 41.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0243888 s, 43.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0244847 s, 42.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0261823 s, 40.0 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg438-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg438-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.83526 s, 21.7 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.93934 s, 21.2 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_policies=fifo PASS 77n (66s) == sanityn test 77o: Changing rank should not panic ====== 15:26:52 (1776886012) mds.MDS.mdt.nrs_policies=tbf nid mds.MDS.mdt.nrs_tbf_rule=start name nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=start name1 nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=change name1 rank=name mds.MDS.mdt.nrs_tbf_rule=stop name mds.MDS.mdt.nrs_policies=fifo PASS 77o (3s) == sanityn test 77q: Parallel TBF rule definitions should not panic ========================================================== 15:26:55 (1776886015) mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_policies=fifo PASS 77q (37s) == sanityn test 77p: Check validity of rule names for TBF policies ========================================================== 15:27:32 (1776886052) mds.MDS.mdt.nrs_policies=tbf oleg438-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_overflo uid={500} rate=500: File name too long oleg438-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_overflo uid={500} rate=500': File name too long pdsh@oleg438-client: oleg438-server: ssh exited with exit code 36 oleg438-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start uid={500} rate=500: Invalid argument oleg438-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start uid={500} rate=500': Invalid argument pdsh@oleg438-client: oleg438-server: ssh exited with exit code 22 oleg438-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_empty: Invalid argument oleg438-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_empty': Invalid argument pdsh@oleg438-client: oleg438-server: ssh exited with exit code 22 oleg438-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p@spec uid={500} rate=500: Invalid argument oleg438-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p@spec uid={500} rate=500': Invalid argument pdsh@oleg438-client: oleg438-server: ssh exited with exit code 22 oleg438-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p.spec uid={500} rate=500: Invalid argument oleg438-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p.spec uid={500} rate=500': Invalid argument pdsh@oleg438-client: oleg438-server: ssh exited with exit code 22 oleg438-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p~spec uid={500} rate=500: Invalid argument oleg438-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p~spec uid={500} rate=500': Invalid argument pdsh@oleg438-client: oleg438-server: ssh exited with exit code 22 oleg438-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p#spec uid={500} rate=500: Invalid argument oleg438-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p#spec uid={500} rate=500': Invalid argument pdsh@oleg438-client: oleg438-server: ssh exited with exit code 22 oleg438-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p/spec uid={500} rate=500: Invalid argument oleg438-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p/spec uid={500} rate=500': Invalid argument pdsh@oleg438-client: oleg438-server: ssh exited with exit code 22 oleg438-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p^spec uid={500} rate=500: Invalid argument oleg438-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p^spec uid={500} rate=500': Invalid argument pdsh@oleg438-client: oleg438-server: ssh exited with exit code 22 oleg438-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p%spec uid={500} rate=500: Invalid argument oleg438-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p%spec uid={500} rate=500': Invalid argument pdsh@oleg438-client: oleg438-server: ssh exited with exit code 22 oleg438-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p*spec uid={500} rate=500: Invalid argument oleg438-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p*spec uid={500} rate=500': Invalid argument pdsh@oleg438-client: oleg438-server: ssh exited with exit code 22 oleg438-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p;spec uid={500} rate=500: Invalid argument oleg438-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p;spec uid={500} rate=500': Invalid argument pdsh@oleg438-client: oleg438-server: ssh exited with exit code 22 oleg438-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p,spec uid={500} rate=500: Invalid argument oleg438-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p,spec uid={500} rate=500': Invalid argument pdsh@oleg438-client: oleg438-server: ssh exited with exit code 22 oleg438-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p?spec uid={500} rate=500: Invalid argument oleg438-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p?spec uid={500} rate=500': Invalid argument pdsh@oleg438-client: oleg438-server: ssh exited with exit code 22 oleg438-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77pspec uid={500} rate=500: Invalid argument oleg438-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p>spec uid={500} rate=500': Invalid argument pdsh@oleg438-client: oleg438-server: ssh exited with exit code 22 oleg438-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p:spec uid={500} rate=500: Invalid argument oleg438-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p:spec uid={500} rate=500': Invalid argument pdsh@oleg438-client: oleg438-server: ssh exited with exit code 22 oleg438-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p+spec uid={500} rate=500: Invalid argument oleg438-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p+spec uid={500} rate=500': Invalid argument pdsh@oleg438-client: oleg438-server: ssh exited with exit code 22 oleg438-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p=spec uid={500} rate=500: Invalid argument oleg438-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p=spec uid={500} rate=500': Invalid argument pdsh@oleg438-client: oleg438-server: ssh exited with exit code 22 oleg438-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p)spec uid={500} rate=500: Invalid argument oleg438-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p)spec uid={500} rate=500': Invalid argument pdsh@oleg438-client: oleg438-server: ssh exited with exit code 22 oleg438-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p(spec uid={500} rate=500: Invalid argument oleg438-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p(spec uid={500} rate=500': Invalid argument pdsh@oleg438-client: oleg438-server: ssh exited with exit code 22 oleg438-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p{spec uid={500} rate=500: Invalid argument oleg438-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p{spec uid={500} rate=500': Invalid argument pdsh@oleg438-client: oleg438-server: ssh exited with exit code 22 oleg438-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p}spec uid={500} rate=500: Invalid argument oleg438-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p}spec uid={500} rate=500': Invalid argument pdsh@oleg438-client: oleg438-server: ssh exited with exit code 22 oleg438-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p|spec uid={500} rate=500: Invalid argument oleg438-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p|spec uid={500} rate=500': Invalid argument pdsh@oleg438-client: oleg438-server: ssh exited with exit code 22 oleg438-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p[spec uid={500} rate=500: Invalid argument oleg438-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p[spec uid={500} rate=500': Invalid argument pdsh@oleg438-client: oleg438-server: ssh exited with exit code 22 oleg438-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p]spec uid={500} rate=500: Invalid argument oleg438-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p]spec uid={500} rate=500': Invalid argument pdsh@oleg438-client: oleg438-server: ssh exited with exit code 22 oleg438-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p!spec uid={500} rate=500: Invalid argument oleg438-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p!spec uid={500} rate=500': Invalid argument pdsh@oleg438-client: oleg438-server: ssh exited with exit code 22 oleg438-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p&spec uid={500} rate=500: Invalid argument oleg438-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p&spec uid={500} rate=500': Invalid argument pdsh@oleg438-client: oleg438-server: ssh exited with exit code 22 oleg438-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p$spec uid={500} rate=500: Invalid argument oleg438-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p$spec uid={500} rate=500': Invalid argument pdsh@oleg438-client: oleg438-server: ssh exited with exit code 22 oleg438-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p`spec uid={500} rate=500: Invalid argument oleg438-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p`spec uid={500} rate=500': Invalid argument pdsh@oleg438-client: oleg438-server: ssh exited with exit code 22 oleg438-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p\spec uid={500} rate=500: Invalid argument oleg438-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p\spec uid={500} rate=500': Invalid argument pdsh@oleg438-client: oleg438-server: ssh exited with exit code 22 mds.MDS.mdt.nrs_policies=fifo PASS 77p (12s) == sanityn test 77r: Change type of tbf policy at run time ========================================================== 15:27:44 (1776886064) mds.MDS.mdt.nrs_policies=tbf uid mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf gid mds.MDS.mdt.nrs_tbf_rule=start md_rule gid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start md_rule jobid={*.500} rate=1 mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=fifo check the number of requests in queue: fifo: 0 crrn: 0 tbf: 0 fairshare: 0 delay: 0 fifo: 0 crrn: 0 tbf: 0 fairshare: 0 delay: 0 cleanup 77r 269847 kill md_thread xargs PASS 77r (39s) == sanityn test 77s: Check TBF LRU shrinker ============== 15:28:23 (1776886103) jobid_name=%e.%p jobid_var=TEST_77s_JOBVAR mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start touch jobid={touch.*} rate=1000 mds.MDS.mdt.nrs_tbf_rule=start unlink jobid={unlink.*} rate=1000 touch jobid={touch.*} 1000, ref 10 touch jobid={touch.*} 1000, ref 0 unlink jobid={unlink.*} 1000, ref 10 unlink jobid={unlink.*} 1000, ref 0 mds.MDS.mdt.nrs_policies=fifo jobid_var=procname_uid jobid_name=%e.%u PASS 77s (12s) == sanityn test 77t: check TBF minrate parsing and output ========================================================== 15:28:36 (1776886116) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start fio jobid={fio.*} minrate=5 rate=20 realtime=1 oleg438-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=change fio rate=4: Invalid argument oleg438-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='change fio rate=4': Invalid argument pdsh@oleg438-client: oleg438-server: ssh exited with exit code 22 ost.OSS.ost_io.nrs_tbf_rule=change fio minrate=10 oleg438-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=change fio minrate=21: Invalid argument oleg438-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='change fio minrate=21': Invalid argument pdsh@oleg438-client: oleg438-server: ssh exited with exit code 22 ost.OSS.ost_io.nrs_tbf_rule=change fio rate=30 ost.OSS.ost_io.nrs_tbf_rule=change fio minrate=0 oleg438-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start bad jobid={bad.*} minrate=31 rate=30: Invalid argument oleg438-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start bad jobid={bad.*} minrate=31 rate=30': Invalid argument pdsh@oleg438-client: oleg438-server: ssh exited with exit code 22 ost.OSS.ost_io.nrs_tbf_rule=stop fio ost.OSS.ost_io.nrs_policies=fifo PASS 77t (10s) == sanityn test 77u: check TBF minrate on NID and generic rules ========================================================== 15:28:45 (1776886125) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_tbf_rule=start nid_min nid={0@lo 192.168.204.38@tcp} minrate=5 rate=20 ost.OSS.ost_io.nrs_tbf_rule=stop nid_min ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start combo_min jobid={dd.500}&opcode={ost_write} minrate=5 rate=20 ost.OSS.ost_io.nrs_tbf_rule=change combo_min minrate=0 ost.OSS.ost_io.nrs_tbf_rule=stop combo_min ost.OSS.ost_io.nrs_policies=fifo PASS 77u (9s) == sanityn test 77v: verify minimum RPC rate under congestion ========================================================== 15:28:54 (1776886134) SKIP: sanityn test_77v need OSTs and MDTs on separate nodes SKIP 77v (1s) == sanityn test 77w: check TBF cost_model inference from rate format ========================================================== 15:28:55 (1776886135) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_tbf_rule=start cm_nid nid={0@lo 192.168.204.38@tcp} rate=2M ost.OSS.ost_io.nrs_tbf_rule=change cm_nid rate=4M ost.OSS.ost_io.nrs_tbf_rule=change cm_nid rate=8MiB/s ost.OSS.ost_io.nrs_tbf_rule=change cm_nid rate=500MB/s ost.OSS.ost_io.nrs_tbf_rule=change cm_nid rate=1.5GiB/s ost.OSS.ost_io.nrs_tbf_rule=change cm_nid minrate=0 rate=5000 ost.OSS.ost_io.nrs_tbf_rule=start cm_both nid={0@lo} minrate=100K rate=1M oleg438-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start cm_bad nid={0@lo} rate=4G minrate=100: Invalid argument oleg438-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start cm_bad nid={0@lo} rate=4G minrate=100': Invalid argument pdsh@oleg438-client: oleg438-server: ssh exited with exit code 22 oleg438-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start cm_bad2 nid={0@lo} rate=5000 minrate=1M: Invalid argument oleg438-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start cm_bad2 nid={0@lo} rate=5000 minrate=1M': Invalid argument pdsh@oleg438-client: oleg438-server: ssh exited with exit code 22 mds.MDS.mdt.nrs_policies=tbf nid oleg438-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start cm_mdt nid={0@lo} rate=1G: Invalid argument oleg438-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start cm_mdt nid={0@lo} rate=1G': Invalid argument pdsh@oleg438-client: oleg438-server: ssh exited with exit code 22 mds.MDS.mdt.nrs_policies=fifo ost.OSS.ost_io.nrs_tbf_rule=stop cm_both ost.OSS.ost_io.nrs_tbf_rule=stop cm_nid ost.OSS.ost_io.nrs_policies=fifo PASS 77w (11s) == sanityn test 77x: check TBF bandwidth suffix rate parsing ========================================================== 15:29:06 (1776886146) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_tbf_rule=start bw_nid nid={0@lo 192.168.204.38@tcp} rate=4G ost.OSS.ost_io.nrs_tbf_rule=change bw_nid rate=512M ost.OSS.ost_io.nrs_tbf_rule=change bw_nid rate=256MiB/s ost.OSS.ost_io.nrs_tbf_rule=change bw_nid rate=1.5GiB/s ost.OSS.ost_io.nrs_tbf_rule=change bw_nid rate=2G minrate=1G ost.OSS.ost_io.nrs_tbf_rule=change bw_nid rate=4GiB/s minrate=512MiB/s ost.OSS.ost_io.nrs_tbf_rule=change bw_nid minrate=0 rate=5000 ost.OSS.ost_io.nrs_tbf_rule=change bw_nid rate=5000 minrate=100 oleg438-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=change bw_nid rate=1G: Invalid argument oleg438-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='change bw_nid rate=1G': Invalid argument pdsh@oleg438-client: oleg438-server: ssh exited with exit code 22 ost.OSS.ost_io.nrs_tbf_rule=change bw_nid rate=2G minrate=1G oleg438-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=change bw_nid rate=4X: Invalid argument oleg438-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='change bw_nid rate=4X': Invalid argument pdsh@oleg438-client: oleg438-server: ssh exited with exit code 22 ost.OSS.ost_io.nrs_tbf_rule=stop bw_nid ost.OSS.ost_io.nrs_policies=fifo PASS 77x (11s) == sanityn test 77y: verify page-weighted cost_model under congestion ========================================================== 15:29:17 (1776886157) SKIP: sanityn test_77y need OSTs and MDTs on separate nodes SKIP 77y (1s) == sanityn test 77z: check TBF global rate mode and per-rule stats ========================================================== 15:29:18 (1776886158) tbf_global_rate=1 ncpts=1 ost.OSS.ost_io.nrs_policies=tbf opcode ost.OSS.ost_io.nrs_tbf_rule=start ost_w opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ost_r opcode={ost_read} rate=20 Verify write rate under TBF global rate control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.86841 s, 21.5 MB/s Write runtime is 5 s, throughput is 20.000000 MiB/s Verify read rate under TBF global rate control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.85871 s, 21.6 MB/s Read runtime is 5 s, throughput is 20.000000 MiB/s Verify TBF rule stats are populated after I/O ost.OSS.ost_io.nrs_tbf_rule: regular_requests: - cpt: 0 rules: - name: ost_r conditions: "opcode={ost_read}" rate: 20 ref: 1 enqueue: 100 dispatch: 100 exhausted: 97 min_used: 0 steal_attempts: 0 tokens_stolen: 0 elapsed_ms: 10124 - name: ost_w conditions: "opcode={ost_write}" rate: 20 ref: 1 enqueue: 100 dispatch: 100 exhausted: 96 min_used: 0 steal_attempts: 0 tokens_stolen: 0 elapsed_ms: 10125 - name: default conditions: "*" rate: 10000 ref: 0 enqueue: 0 dispatch: 0 exhausted: 0 min_used: 0 steal_attempts: 0 tokens_stolen: 0 elapsed_ms: 10125 high_priority_requests: - cpt: 0 rules: - name: ost_r conditions: "opcode={ost_read}" rate: 20 ref: 0 enqueue: 0 dispatch: 0 exhausted: 0 min_used: 0 steal_attempts: 0 tokens_stolen: 0 elapsed_ms: 10124 - name: ost_w conditions: "opcode={ost_write}" rate: 20 ref: 0 enqueue: 0 dispatch: 0 exhausted: 0 min_used: 0 steal_attempts: 0 tokens_stolen: 0 elapsed_ms: 10125 - name: default conditions: "*" rate: 10000 ref: 0 enqueue: 0 dispatch: 0 exhausted: 0 min_used: 0 steal_attempts: 0 tokens_stolen: 0 elapsed_ms: 10125 ost_w dispatch count: 100 ost_r dispatch count: 100 ost_w enqueue count: 100 ost.OSS.ost_io.nrs_tbf_rule=stop ost_r ost.OSS.ost_io.nrs_tbf_rule=stop ost_w ost.OSS.ost_io.nrs_policies=fifo PASS 77z (14s) == sanityn test 77aa: check NRS fairshare basic smoke (jobid default) ========================================================== 15:29:32 (1776886172) ost.OSS.ost_io.nrs_policies=fairshare 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.127977 s, 131 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0201899 s, 51.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0387977 s, 27.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0331308 s, 31.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0626071 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0789182 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0760177 s, 13.8 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9c56c61a4000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0512332 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102268 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102449 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0907078 s, 11.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103744 s, 10.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0983047 s, 10.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.080713 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107444 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0995407 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107247 s, 9.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9c56c61a4000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9c56c61a4000.lock_unused_count=2 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0179626 s, 58.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162326 s, 64.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0174183 s, 60.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0186819 s, 56.1 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0188156 s, 55.7 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00493984 s, 212 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0178002 s, 58.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.018542 s, 56.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0082182 s, 128 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014088 s, 74.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0227422 s, 46.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0118909 s, 88.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00575803 s, 182 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0181162 s, 57.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00837493 s, 125 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00473372 s, 222 MB/s removed '/mnt/lustre/d77aa.sanityn/nrs_w_oleg438-client.virtnet' removed '/mnt/lustre/d77aa.sanityn/nrs_r_oleg438-client.virtnet' removed directory '/mnt/lustre/d77aa.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77aa (5s) == sanityn test 77ab: check NRS fairshare mode switching and tunables ========================================================== 15:29:37 (1776886177) ost.OSS.ost_io.nrs_policies=fairshare nid_fair ost.OSS.ost_io.nrs_fairshare_opp_threshold=32 ost.OSS.ost_io.nrs_fairshare_opp_threshold=8 ost.OSS.ost_io.nrs_fairshare_delta_ms=250 oleg438-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_fairshare_delta_ms=999999: Numerical result out of range oleg438-server: error: set_param: setting 'ost/OSS/ost_io/nrs_fairshare_delta_ms'='999999': Numerical result out of range pdsh@oleg438-client: oleg438-server: ssh exited with exit code 34 ost.OSS.ost_io.nrs_fairshare_delta_ms=100 ost.OSS.ost_io.nrs_policies=fairshare uid_fair oleg438-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=fairshare bogus_mode: Invalid argument oleg438-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='fairshare bogus_mode': Invalid argument pdsh@oleg438-client: oleg438-server: ssh exited with exit code 22 ost.OSS.ost_io.nrs_policies=fifo PASS 77ab (6s) == sanityn test 77ac: check NRS fairshare rule parse / start / change / stop ========================================================== 15:29:43 (1776886183) ost.OSS.ost_io.nrs_policies=fairshare uid_fair ost.OSS.ost_io.nrs_fairshare_rule=start r1 uid={500} rate=100 ost.OSS.ost_io.nrs_fairshare_rule=start r2 uid={501,502} rate=500 minrate=10 oleg438-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_fairshare_rule=start r1 uid={503} rate=1: File exists oleg438-server: error: set_param: setting 'ost/OSS/ost_io/nrs_fairshare_rule'='start r1 uid={503} rate=1': File exists pdsh@oleg438-client: oleg438-server: ssh exited with exit code 17 ost.OSS.ost_io.nrs_fairshare_rule=change r1 rate=200 oleg438-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_fairshare_rule=change nope rate=1: No such file or directory oleg438-server: error: set_param: setting 'ost/OSS/ost_io/nrs_fairshare_rule'='change nope rate=1': No such file or directory pdsh@oleg438-client: oleg438-server: ssh exited with exit code 2 oleg438-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_fairshare_rule=garbage r1 rate=1: Invalid argument oleg438-server: error: set_param: setting 'ost/OSS/ost_io/nrs_fairshare_rule'='garbage r1 rate=1': Invalid argument pdsh@oleg438-client: oleg438-server: ssh exited with exit code 22 ost.OSS.ost_io.nrs_fairshare_rule=stop r1 ost.OSS.ost_io.nrs_fairshare_rule=stop ost.OSS.ost_io.nrs_policies=fifo PASS 77ac (7s) == sanityn test 77ad: check NRS fairshare rate ceiling dispatches under load ========================================================== 15:29:50 (1776886190) ost.OSS.ost_io.nrs_policies=fairshare uid_fair ost.OSS.ost_io.nrs_fairshare_rule=start rcap uid={0} rate=25600 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.116385 s, 144 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0366381 s, 28.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0419596 s, 25.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0539041 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0565518 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0611253 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0791511 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.075787 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0762811 s, 13.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.073598 s, 14.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0685277 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0770483 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0863998 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0788018 s, 13.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9c56c61a4000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0829107 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0681212 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0880516 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158814 s, 66.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0195165 s, 53.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0243743 s, 43.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0226518 s, 46.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0203225 s, 51.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0198559 s, 52.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00749354 s, 140 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0235627 s, 44.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00390457 s, 269 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173144 s, 60.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125957 s, 83.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0285649 s, 36.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0323351 s, 32.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134994 s, 77.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168919 s, 62.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0264201 s, 39.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9c56c61a4000.lock_unused_count=1 removed '/mnt/lustre/d77ad.sanityn/nrs_w_oleg438-client.virtnet' removed '/mnt/lustre/d77ad.sanityn/nrs_r_oleg438-client.virtnet' removed directory '/mnt/lustre/d77ad.sanityn' ost.OSS.ost_io.nrs_fairshare_rule=stop ost.OSS.ost_io.nrs_policies=fifo PASS 77ad (4s) == sanityn test 77ae: check NRS fairshare composite (A_then_B) mode ========================================================== 15:29:54 (1776886194) ost.OSS.ost_io.nrs_policies=fairshare uid_then_jobid_fair ost.OSS.ost_io.nrs_policies=fairshare uid_then_jobid_fair 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.140121 s, 120 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0265696 s, 39.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0391703 s, 26.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0594828 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0732172 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0567573 s, 18.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0751583 s, 14.0 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9c56c61a4000.lock_unused_count=1 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.067697 s, 15.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0660835 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0711845 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0916779 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0713384 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0894268 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0816664 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0908553 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102584 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100671 s, 10.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00470287 s, 223 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183084 s, 57.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.018582 s, 56.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0216295 s, 48.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00950391 s, 110 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0205226 s, 51.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0284861 s, 36.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0274323 s, 38.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0249381 s, 42.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9c56c61a4000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9c56c61a4000.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0181369 s, 57.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0197335 s, 53.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0358198 s, 29.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0263231 s, 39.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0247611 s, 42.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0246206 s, 42.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0250255 s, 41.9 MB/s removed '/mnt/lustre/d77ae.sanityn/nrs_w_oleg438-client.virtnet' removed '/mnt/lustre/d77ae.sanityn/nrs_r_oleg438-client.virtnet' removed directory '/mnt/lustre/d77ae.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77ae (5s) == sanityn test 77af: check NRS fairshare delta kthread and delta_ms round-trip ========================================================== 15:29:59 (1776886199) ost.OSS.ost_io.nrs_policies=fairshare ost.OSS.ost_io.nrs_fairshare_delta_ms=10 ost.OSS.ost_io.nrs_fairshare_delta_ms=100 ost.OSS.ost_io.nrs_fairshare_delta_ms=1000 ost.OSS.ost_io.nrs_fairshare_delta_ms=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.143228 s, 117 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0329755 s, 31.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0297078 s, 35.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0841305 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0815619 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.084704 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100403 s, 10.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12281 s, 8.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105925 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126582 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0915751 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115023 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108785 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128675 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12438 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115433 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126791 s, 8.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9c56c61a4000.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffff9c56c61a4000.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0181521 s, 57.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0172086 s, 60.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0205141 s, 51.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0213935 s, 49.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0268936 s, 39.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0251651 s, 41.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0226673 s, 46.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00562414 s, 186 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.017153 s, 61.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00751413 s, 140 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015348 s, 68.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0219946 s, 47.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00588211 s, 178 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132047 s, 79.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151779 s, 69.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125018 s, 83.9 MB/s removed '/mnt/lustre/d77af.sanityn/nrs_w_oleg438-client.virtnet' removed '/mnt/lustre/d77af.sanityn/nrs_r_oleg438-client.virtnet' removed directory '/mnt/lustre/d77af.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77af (6s) == sanityn test 77ag: check NRS fairshare locality tunable and BRW ordering ========================================================== 15:30:05 (1776886205) ost.OSS.ost_io.nrs_policies=fairshare ost.OSS.ost_io.nrs_fairshare_locality=off ost.OSS.ost_io.nrs_fairshare_locality=on ost.OSS.ost_io.nrs_fairshare_locality=auto ost.OSS.ost_io.nrs_fairshare_locality=on 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.142556 s, 118 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159726 s, 65.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0414515 s, 25.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0240243 s, 43.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0396613 s, 26.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0532239 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0710753 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0786833 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0901075 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0770484 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0995704 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0844323 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0597506 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0986735 s, 10.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9c56c61a4000.lock_unused_count=5 ldlm.namespaces.lustre-OST0001-osc-ffff9c56c61a4000.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104729 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0900059 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106612 s, 9.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012431 s, 84.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00736447 s, 142 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012797 s, 81.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00360036 s, 291 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183206 s, 57.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0284451 s, 36.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00923171 s, 114 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0207596 s, 50.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00402922 s, 260 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0218073 s, 48.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00616432 s, 170 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149683 s, 70.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015322 s, 68.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0238143 s, 44.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00501019 s, 209 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9c56c61a4000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131775 s, 79.6 MB/s removed '/mnt/lustre/d77ag.sanityn/nrs_r_oleg438-client.virtnet' removed '/mnt/lustre/d77ag.sanityn/nrs_w_oleg438-client.virtnet' removed directory '/mnt/lustre/d77ag.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77ag (6s) == sanityn test 77ah: verify NRS fairshare minrate floor under congestion ========================================================== 15:30:11 (1776886211) SKIP: sanityn test_77ah need OSTs and MDTs on separate nodes SKIP 77ah (1s) == sanityn test 78: Enable policy and specify tunings right away ========================================================== 15:30:12 (1776886212) ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_orr_quantum=1 ost.OSS.ost_io.nrs_policies=fifo PASS 78 (3s) == sanityn test 79: xattr: intent error ================== 15:30:15 (1776886215) File: /mnt/lustre/d79.sanityn Size: 512 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115238826936149 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-04-22 15:30:16.000000000 -0400 Modify: 2026-04-22 15:30:16.000000000 -0400 Change: 2026-04-22 15:30:16.000000000 -0400 Birth: 2026-04-22 15:30:16.000000000 -0400 fail_val=0 fail_loc=0x80000160 fail_val=0 fail_loc=0x80000131 # file: mnt/lustre/d79.sanityn trusted.name1="value1" PASS 79 (13s) == sanityn test 80a: migrate directory when some children is being opened ========================================================== 15:30:28 (1776886228) SKIP: sanityn test_80a needs >= 2 MDTs SKIP 80a (0s) == sanityn test 80b: Accessing directory during migration ========================================================== 15:30:28 (1776886228) SKIP: sanityn test_80b needs >= 2 MDTs SKIP 80b (1s) == sanityn test 81a: rename and stat under striped directory ========================================================== 15:30:29 (1776886229) SKIP: sanityn test_81a needs >= 2 MDTs SKIP 81a (1s) == sanityn test 81b: rename under striped directory doesn't deadlock ========================================================== 15:30:30 (1776886230) SKIP: sanityn test_81b We need at least 2 MDTs for this test SKIP 81b (1s) == sanityn test 81c: rename revoke LOOKUP lock for remote object ========================================================== 15:30:31 (1776886231) SKIP: sanityn test_81c needs >= 4 MDTs SKIP 81c (1s) == sanityn test 81d: parallel rename file cross-dir on same MDT ========================================================== 15:30:32 (1776886232) total: 2500 open/close in 4.83 seconds: 517.48 ops/second snapshot_time 1776886252.540379125 secs.nsecs start_time 1776886233.521864038 secs.nsecs elapsed_time 19.018515087 secs.nsecs open 2505 samples [usecs] 68 1162 868385 320813561 close 2504 samples [usecs] 57 252 228991 22162489 mknod 2501 samples [usecs] 192 1110 776348 257720680 unlink 2500 samples [usecs] 304 18855 1542455 2960939875 mkdir 2 samples [usecs] 698 780 1478 1095604 rmdir 2 samples [usecs] 1451 4276 5727 20389577 rename 2500 samples [usecs] 260 9902 2887830 4704978418 getattr 3762 samples [usecs] 6 4832 73418 32475324 parallel_rename_file 2500 samples [usecs] 260 9902 2887830 4704978418 crossdir_rename 2500 samples [usecs] 260 9902 2887830 4704978418 PASS 81d (21s) == sanityn test 82: fsetxattr and fgetxattr on orphan files ========================================================== 15:30:53 (1776886253) multiop /mnt/lustre/f82.sanityn vO_ac TMPPIPE=/tmp/multiop_open_wait_pipe.7529 multiop /mnt/lustre2/f82.sanityn vo_Ac TMPPIPE=/tmp/multiop_open_wait_pipe.7529 PASS 82 (3s) == sanityn test 83: access striped directory while it is being created/unlinked ========================================================== 15:30:56 (1776886256) SKIP: sanityn test_83 needs >= 2 MDTs SKIP 83 (1s) == sanityn test 84: 0-nlink race in lu_object_find() ===== 15:30:57 (1776886257) fail_loc=0x8000060b PASS 84 (7s) == sanityn test 85: Lustre API root cache race =========== 15:31:04 (1776886264) Starting: llapi_root_test /mnt/lustre/f85.sanityn /mnt/lustre2/f85.sanityn Starting test test1 at 1776886264 Finishing test test1 at 1776886265 PASS 85 (3s) == sanityn test 90: open/create and unlink striped directory ========================================================== 15:31:07 (1776886267) SKIP: sanityn test_90 needs >= 2 MDTs SKIP 90 (1s) == sanityn test 91: chmod and unlink striped directory === 15:31:08 (1776886268) SKIP: sanityn test_91 needs >= 2 MDTs SKIP 91 (1s) == sanityn test 92: create remote directory under orphan directory ========================================================== 15:31:09 (1776886269) SKIP: sanityn test_92 needs >= 2 MDTs SKIP 92 (1s) == sanityn test 93: alloc_rr should not allocate on same ost ========================================================== 15:31:10 (1776886270) fail_loc=0x00000163 fail_loc=0x0 /mnt/lustre/f93.sanityn-1/file1 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 6289 0x1891 0x240000400 1 6254 0x186e 0x280000400 /mnt/lustre2/f93.sanityn-2/file2 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 6290 0x1892 0x240000400 1 6255 0x186f 0x280000400 rc1=0 and rc2=0 PASS 93 (9s) == sanityn test 94: signal vs CP callback race =========== 15:31:19 (1776886279) osc.lustre-OST0000-osc-ffff9c56c31dc000.idle_timeout=0 osc.lustre-OST0000-osc-ffff9c56c61a4000.idle_timeout=0 osc.lustre-OST0001-osc-ffff9c56c31dc000.idle_timeout=0 osc.lustre-OST0001-osc-ffff9c56c61a4000.idle_timeout=0 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0435003 s, 188 kB/s fail_val=5 fail_loc=0x80000312 fail_val=6 fail_loc=0x80000329 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 3.09141 s, 1.3 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 7545: 301569 Alarm clock dd if=/dev/zero of=$DIR/$tfile conv=notrunc oflag=append bs=4k count=1 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0133293 s, 307 kB/s osc.lustre-OST0000-osc-ffff9c56c31dc000.idle_timeout=debug osc.lustre-OST0000-osc-ffff9c56c61a4000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9c56c31dc000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9c56c61a4000.idle_timeout=debug PASS 94 (11s) == sanityn test 95a: Check readpage() on a page that was removed from page cache ========================================================== 15:31:30 (1776886290) llite.lustre-ffff9c56c31dc000.fast_read=0 llite.lustre-ffff9c56c61a4000.fast_read=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0184639 s, 114 MB/s fail_loc=0x80001421 fail_val=7 fail_loc=0x80001422 fail_val=10 llite.lustre-ffff9c56c31dc000.fast_read=1 llite.lustre-ffff9c56c61a4000.fast_read=1 PASS 95a (14s) == sanityn test 95b: Check readpage() on a page that is no longer uptodate ========================================================== 15:31:44 (1776886304) llite.lustre-ffff9c56c31dc000.fast_read=0 llite.lustre-ffff9c56c61a4000.fast_read=0 1+0 records in 1+0 records out 12288 bytes (12 kB, 12 KiB) copied, 0.00169013 s, 7.3 MB/s fail_loc=0x80001424 fail_val=5 fail_loc=0 llite.lustre-ffff9c56c31dc000.fast_read=1 llite.lustre-ffff9c56c61a4000.fast_read=1 PASS 95b (8s) == sanityn test 100a: DoM: glimpse RPCs for stat without IO lock (DoM only file) ========================================================== 15:31:53 (1776886313) SKIP: sanityn test_100a Reserved for glimpse-ahead SKIP 100a (1s) == sanityn test 100b: DoM: no glimpse RPC for stat with IO lock (DoM only file) ========================================================== 15:31:54 (1776886314) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00145259 s, 2.8 MB/s /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK PASS 100b (2s) == sanityn test 100c: DoM: write vs stat without IO lock (combined file) ========================================================== 15:31:56 (1776886316) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0448738 s, 46.7 MB/s /mnt/lustre/d100c.sanityn/dom has type file OK /mnt/lustre/d100c.sanityn/dom has size 2097152 OK PASS 100c (2s) == sanityn test 100d: DoM: write+truncate vs stat without IO lock (combined file) ========================================================== 15:31:58 (1776886318) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0390422 s, 53.7 MB/s /mnt/lustre/d100d.sanityn/dom has type file OK /mnt/lustre/d100d.sanityn/dom has size 4096 OK PASS 100d (3s) == sanityn test 100e: DoM: read on open and file size ==== 15:32:01 (1776886321) 1+0 records in 1+0 records out 12000 bytes (12 kB, 12 KiB) copied, 0.00188712 s, 6.4 MB/s PASS 100e (2s) == sanityn test 101a: Discard DoM data on unlink ========= 15:32:03 (1776886323) /mnt/lustre/f101a.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00171859 s, 2.4 MB/s PASS 101a (2s) == sanityn test 101b: Discard DoM data on rename ========= 15:32:05 (1776886325) /mnt/lustre/f101b.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0013491 s, 3.0 MB/s PASS 101b (2s) == sanityn test 101c: Discard DoM data on close-unlink === 15:32:07 (1776886327) /mnt/lustre/f101c.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00144937 s, 2.8 MB/s PASS 101c (3s) SKIP: sanityn test_102 skipping ALWAYS excluded test 102 == sanityn test 103: Test size correctness with lockahead ========================================================== 15:32:11 (1776886331) fail_loc=0x415 fail_loc=0x415 fail_val=2 Incorrect size expected (no glimpse fix): Starting test test23 at 1776886332 lockahead_test: lockahead_test.c:1171: test23: assertion 'sb.st_size == sb2.st_size' failed: size on /mnt/lustre/d103.sanityn/f103.sanityn and /mnt/lustre2/d103.sanityn/f103.sanityn differs: 0 vs 1048576 fail_loc=0 fail_loc=0x214 fail_val=2 Starting test test23 at 1776886334 Finishing test test23 at 1776886335 PASS 103 (7s) == sanityn test 104: Verify that MDS stores atime/mtime/ctime during close ========================================================== 15:32:18 (1776886338) SKIP: sanityn test_104 ldiskfs only test SKIP 104 (1s) == sanityn test 105: Glimpse and lock cancel race ======== 15:32:19 (1776886339) fail_loc=0x416 PASS 105 (18s) == sanityn test 106a: Verify the btime via statx() ======= 15:32:37 (1776886357) SKIP: sanityn test_106a Test only for ldiskfs and statx() supported SKIP 106a (1s) == sanityn test 106b: Glimpse RPCs test for statx ======== 15:32:38 (1776886358) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00306934 s, 334 kB/s File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238826938684 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-04-22 15:32:38.000000000 -0400 Modify: 2026-04-22 15:32:38.000000000 -0400 Change: 2026-04-22 15:32:38.000000000 -0400 Birth: 2026-04-22 15:32:39.000000000 -0400 /mnt/lustre/f106b.sanityn 144115238826938684 -rw-r--r-- 1 0 0 1776886359 1776886358 1776886358 File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238826938684 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-04-22 15:32:38.000000000 -0400 Modify: 2026-04-22 15:32:38.000000000 -0400 Change: 2026-04-22 15:32:38.000000000 -0400 Birth: 2026-04-22 15:32:39.000000000 -0400 1776886358 1024 1 PASS 106b (2s) == sanityn test 106c: Verify statx attributes mask ======= 15:32:40 (1776886360) PASS 106c (2s) == sanityn test 107a: Basic grouplock conflict =========== 15:32:42 (1776886362) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.0934277 s, 112 MB/s /mnt/lustre/f107a.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 6261 0x1875 0x280000400 multiop /mnt/lustre/f107a.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7529 multiop /mnt/lustre2/f107a.sanityn vO_G16022000r10g16022000c TMPPIPE=/tmp/multiop_open_wait_pipe.7529 First grouplock blocks second one PASS 107a (5s) == sanityn test 107b: Grouplock is added to the head of waiting list ========================================================== 15:32:47 (1776886367) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.0984578 s, 107 MB/s /mnt/lustre/f107b.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 6299 0x189b 0x240000400 multiop /mnt/lustre/f107b.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7529 Grouplock blocks IO First grouplock blocks second one Second grouplock blocks IO PASS 107b (8s) == sanityn test 108a: lseek: parallel updates ============ 15:32:55 (1776886375) 1+0 records in 1+0 records out 10000 bytes (10 kB, 9.8 KiB) copied, 0.00168918 s, 5.9 MB/s llite.lustre-ffff9c56c31dc000.hybrid_io=0 llite.lustre-ffff9c56c61a4000.hybrid_io=0 fail_val=4 fail_loc=0x80000414 1+0 records in 1+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 4.21768 s, 2.0 MB/s PASS 108a (6s) == sanityn test 109: Race with several mount instances on 1 node ========================================================== 15:33:01 (1776886381) 192.168.204.138@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg438-client.virtnet /mnt/lustre (opts:) 192.168.204.138@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg438-client.virtnet /mnt/lustre2 (opts:) Starting race between client mounts (duration 30) Iteration 0 Starting client: oleg438-client.virtnet: -o user_xattr,flock 192.168.204.138@tcp:/lustre /mnt/lustre Starting client: oleg438-client.virtnet: -o user_xattr,flock 192.168.204.138@tcp:/lustre /mnt/lustre2 192.168.204.138@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.138@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg438-client.virtnet /mnt/lustre (opts:) Stopping client oleg438-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg438-server' oleg438-server: oleg438-server.virtnet: executing load_modules_local oleg438-server: Loading modules from /home/green/git/lustre-release/lustre oleg438-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg438-server: detected 4 online CPUs by sysfs oleg438-server: MODOPTS_LIBCFS= oleg438-server: Force libcfs to create 2 CPU partitions Iteration 1 Starting client: oleg438-client.virtnet: -o user_xattr,flock 192.168.204.138@tcp:/lustre /mnt/lustre Starting client: oleg438-client.virtnet: -o user_xattr,flock 192.168.204.138@tcp:/lustre /mnt/lustre2 192.168.204.138@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.138@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg438-client.virtnet /mnt/lustre (opts:) Stopping client oleg438-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg438-server' oleg438-server: oleg438-server.virtnet: executing load_modules_local oleg438-server: Loading modules from /home/green/git/lustre-release/lustre oleg438-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg438-server: detected 4 online CPUs by sysfs oleg438-server: MODOPTS_LIBCFS= oleg438-server: Force libcfs to create 2 CPU partitions Iteration 2 Starting client: oleg438-client.virtnet: -o user_xattr,flock 192.168.204.138@tcp:/lustre /mnt/lustre Starting client: oleg438-client.virtnet: -o user_xattr,flock 192.168.204.138@tcp:/lustre /mnt/lustre2 192.168.204.138@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.138@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg438-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg438-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg438-server' oleg438-server: oleg438-server.virtnet: executing load_modules_local oleg438-server: Loading modules from /home/green/git/lustre-release/lustre oleg438-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg438-server: detected 4 online CPUs by sysfs oleg438-server: MODOPTS_LIBCFS= oleg438-server: Force libcfs to create 2 CPU partitions Starting client: oleg438-client.virtnet: -o user_xattr,flock 192.168.204.138@tcp:/lustre /mnt/lustre Starting client: oleg438-client.virtnet: -o user_xattr,flock 192.168.204.138@tcp:/lustre /mnt/lustre2 PASS 109 (35s) == sanityn test 110: do not grant another lock on resend ========================================================== 15:33:36 (1776886416) fail_loc=0x525 fail_val=3 fail_loc=0x534 File: /mnt/lustre/f110.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238860488707 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-04-22 15:33:36.000000000 -0400 Modify: 2026-04-22 15:33:36.000000000 -0400 Change: 2026-04-22 15:33:36.000000000 -0400 Birth: 2026-04-22 15:33:36.000000000 -0400 fail_loc=0 fail_val=0 Connected clients: oleg438-client.virtnet oleg438-client.virtnet PASS 110 (168s) == sanityn test 111: A racy rename/link an open file should not cause fs corruption ========================================================== 15:36:24 (1776886584) SKIP: sanityn test_111 needs >= 2 MDTs SKIP 111 (1s) == sanityn test 112: update max-inherit in default LMV === 15:36:25 (1776886585) SKIP: sanityn test_112 We need at least 2 MDTs for this test SKIP 112 (1s) == sanityn test 113: check servers of specified fs ======= 15:36:26 (1776886586) lustre-OST0000-osc-ffff9c56c38dd000 active. lustre-OST0001-osc-ffff9c56c38dd000 active. lustre-OST0000-osc-ffff9c56c38dd000 active. lustre-OST0001-osc-ffff9c56c38dd000 active. MGC192.168.204.138@tcp active. PASS 113 (2s) == sanityn test 114: implicit default LMV inherit ======== 15:36:28 (1776886588) SKIP: sanityn test_114 We need at least 2 MDTs for this test SKIP 114 (1s) == sanityn test 115: ldiskfs doesn't check direntry for uniqueness ========================================================== 15:36:29 (1776886589) SKIP: sanityn test_115 ldiskfs only test SKIP 115 (1s) == sanityn test 116: DNE: Set default LMV layout from a remote client ========================================================== 15:36:30 (1776886590) SKIP: sanityn test_116 needs >= 2 MDTs SKIP 116 (1s) == sanityn test 121: trunc append race =================== 15:36:31 (1776886591) 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0030318 s, 6.8 MB/s fail_loc=0x80001436 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0120246 s, 1.7 MB/s 0000000 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0005000 0000 0000 0000 0000 0000 0000 0000 0000 * 000c800 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0011800 PASS 121 (4s) == sanityn test 200: service remains healthy while able to process request ========================================================== 15:36:35 (1776886595) at_max=10 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=9 fail_loc=0x1e1 fail_val=20 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0879 s, 0.2 kB/s4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0861 s, 0.2 kB/s 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0838 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0873 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0812 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0926 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0949 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1444 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1385 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1468 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1442 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1519 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1572 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1572 s, 0.1 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2303 s, 0.1 kB/s 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2324 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2285 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2263 s, 0.1 kB/s at_max=600 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=116 oleg438-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9c56c38dd000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9c56c38dd000.ost_server_uuid in FULL state after 0 sec PASS 200 (66s) cleanup: ====================================================== == sanityn test complete, duration 5705 sec ============== 15:37:42 (1776886662) === sanityn: start cleanup 15:37:42 (1776886662) === Stopping clients: oleg438-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg438-client.virtnet /mnt/lustre2 opts: === sanityn: finish cleanup 15:38:29 (1776886709) ===