-----============= acceptance-small: sanityn ============----- Thu Sep 25 14:08:46 EDT 2025 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg220-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanityn.*ex': No such file or directory excepting tests: 27 40a 102 skipping tests SLOW=no: 33a === sanityn: start setup 14:08:53 (1758823733) === Starting client oleg220-client.virtnet: -o user_xattr,flock 192.168.202.120@tcp:/lustre /mnt/lustre2 Started clients oleg220-client.virtnet: 192.168.202.120@tcp:/lustre on /mnt/lustre2 type lustre (rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) oleg220-client.virtnet: executing check_config_client /mnt/lustre oleg220-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg220-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8c2889525800.idle_timeout=debug osc.lustre-OST0000-osc-ffff8c2898bad800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8c2889525800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8c2898bad800.idle_timeout=debug disable quota as required oleg220-server: oleg220-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanityn: finish setup 14:09:03 (1758823743) === 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0207084 s, 50.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7559] == sanityn test 0: do_node_vp() and do_facet_vp() do the right thing ========================================================== 14:09:04 (1758823744) PASS 0 (2s) == sanityn test 1: Check attribute updates on 2 mount points ========================================================== 14:09:06 (1758823746) /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0777 OK /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0666 OK /mnt/lustre/f1.sanityn: absent OK PASS 1 (2s) == sanityn test 2a: check cached attribute updates on 2 mtpt's ================================================================== 14:09:08 (1758823748) -rw-r--r-- 1 root root 0 Sep 25 14:09 /mnt/lustre2/f2a /mnt/lustre/f2a has type file OK /mnt/lustre/f2a has perms 0777 OK PASS 2a (2s) == sanityn test 2b: check cached attribute updates on 2 mtpt's ================================================================== 14:09:10 (1758823750) -rw-r--r-- 1 root root 0 Sep 25 14:09 /mnt/lustre2/f2b /mnt/lustre2/f2b has type file OK /mnt/lustre2/f2b has perms 0777 OK PASS 2b (3s) == sanityn test 2c: check cached attribute updates on 2 mtpt's root ============================================================= 14:09:13 (1758823753) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0777 OK PASS 2c (2s) == sanityn test 2d: check cached attribute updates on 2 mtpt's root ============================================================= 14:09:15 (1758823755) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0755 OK PASS 2d (2s) == sanityn test 2e: check chmod on root is propagated to others ========================================================== 14:09:17 (1758823757) total 1 -rwxrwxrwx 1 root root 0 Sep 25 14:09 f2a -rwxrwxrwx 1 root root 0 Sep 25 14:09 f2b total 1 -rwxrwxrwx 1 root root 0 Sep 25 14:09 f2a -rwxrwxrwx 1 root root 0 Sep 25 14:09 f2b running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre2/f2e.sanityn] [count=1] 1+0 records in 1+0 records out 512 bytes copied, 0.00268888 s, 190 kB/s PASS 2e (2s) == sanityn test 2f: check attr/owner updates on DNE with 2 mtpt's ========================================================== 14:09:19 (1758823759) SKIP: sanityn test_2f needs >= 2 MDTs SKIP 2f (1s) == sanityn test 2g: check blocks update on sync write ==== 14:09:20 (1758823760) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0769058 s, 27.3 MB/s /mnt/lustre/f2g.sanityn has 4103 blocks /mnt/lustre2/f2g.sanityn has 4103 blocks PASS 2g (2s) == sanityn test 3: symlink on one mtpt, readlink on another ===================================================================== 14:09:22 (1758823762) PASS 3 (2s) == sanityn test 4: fstat validation on multiple mount points ==================================================================== 14:09:24 (1758823764) PASS 4 (3s) == sanityn test 5: create a file on one mount, truncate it on the other ========================================================== 14:09:27 (1758823767) /mnt/lustre/f5 has type file OK /mnt/lustre/f5 has size 100 OK PASS 5 (2s) == sanityn test 6: remove of open file on other node ============================================================================ 14:09:29 (1758823769) opening writing unlinking /mnt/lustre2/f6.sanityn accessing (1) seeking (1) accessing (2) fstat... reading comparing data truncating seeking (2) writing again seeking (3) reading again comparing data again closing SUCCESS - goto beer PASS 6 (3s) == sanityn test 7: remove of open directory on other node ======================================================================= 14:09:32 (1758823772) creating directory /mnt/lustre/d7 opening directory unlinking /mnt/lustre/d7 Ok, everything goes well. PASS 7 (2s) == sanityn test 8: remove of open special file on other node ==================================================================== 14:09:34 (1758823774) creating special file /mnt/lustre/f8.sanityn opening file unlinking /mnt/lustre/f8.sanityn Ok, everything goes well. PASS 8 (2s) == sanityn test 9a: append of file with sub-page size on multiple mounts ========================================================== 14:09:36 (1758823776) PASS 9a (2s) == sanityn test 9b: append to striped sparse file ======== 14:09:38 (1758823778) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0205778 s, 51.0 MB/s 3+0 records in 3+0 records out 3 bytes copied, 0.0103094 s, 0.3 kB/s Data read (expecting 'foo'): foo PASS 9b (2s) == sanityn test 10a: write of file with sub-page size on multiple mounts ========================================================== 14:09:40 (1758823780) 1+0 records in 1+0 records out 1 byte copied, 0.00201556 s, 0.5 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.002804 s, 0.4 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00290111 s, 0.3 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00316119 s, 0.3 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00298349 s, 0.3 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00310989 s, 0.3 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00249568 s, 0.4 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0027921 s, 0.4 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00293493 s, 0.3 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00268587 s, 0.4 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0028986 s, 0.3 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00279154 s, 0.4 kB/s PASS 10a (3s) == sanityn test 10b: write of file with sub-page size on multiple mounts ========================================================== 14:09:43 (1758823783) 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.00285006 s, 1.1 MB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00737278 s, 556 kB/s 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.000159425 s, 19.3 MB/s PASS 10b (2s) == sanityn test 11: execution of file opened for write should return error ============================================================== 14:09:45 (1758823785) multiop /mnt/lustre/d11/f vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7559 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 300: /mnt/lustre2/d11/f: Text file busy PASS 11 (2s) == sanityn test 12: test lock ordering (link, stat, unlink) ========================================================== 14:09:47 (1758823787) warning: '-runtime' deprecated, use '-t runtime' instead - link 2258 (time 1758823798.07 total 10.01 last 225.67) - link 4283 (time 1758823808.07 total 20.01 last 202.47) - link 6335 (time 1758823818.08 total 30.01 last 205.07) - link 8349 (time 1758823828.08 total 40.02 last 201.27) - link 10000 (time 1758823836.25 total 48.19 last 202.16) - link 12185 (time 1758823846.25 total 58.19 last 218.46) - link 14329 (time 1758823856.25 total 68.19 last 214.38) - link 15153 (time 1758823866.26 total 78.19 last 82.37) - link 16018 (time 1758823876.26 total 88.20 last 86.45) - link 16511 (time 1758823886.28 total 98.22 last 49.19) total: 16561 link in 100.02 seconds: 165.58 ops/second using seed 3348634805 running for 100 seconds - stat 10000 (time 1758823796 ; total 7 ; last 7) - stat 20000 (time 1758823804 ; total 15 ; last 8) - stat 30000 (time 1758823813 ; total 24 ; last 9) - stat 40000 (time 1758823821 ; total 32 ; last 8) - stat 50000 (time 1758823829 ; total 40 ; last 8) - stat 60000 (time 1758823837 ; total 48 ; last 8) - stat 66386 (time 1758823848 ; total 59 ; last 11) - stat 70000 (time 1758823854 ; total 65 ; last 6) - stat 74613 (time 1758823865 ; total 76 ; last 11) - stat 80000 (time 1758823874 ; total 85 ; last 9) - stat 85532 (time 1758823885 ; total 96 ; last 11) total: 87241 stats in 100 seconds: 872.409973 stats/second - unlinked 0 (time 1758823839 ; total 0 ; last 0) - unlinked 10000 (time 1758823945 ; total 106 ; last 106) unlink(/mnt/lustre2/lockdir/lockfile16561) error: No such file or directory total: 16560 unlinks in 155 seconds: 106.838707 unlinks/second /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 77: kill: (21327) - No such process /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 78: kill: (21329) - No such process PASS 12 (209s) == sanityn test 13: test directory page revocation ======= 14:13:16 (1758823996) f13 f13 PASS 13 (4s) == sanityn test 14aa: execution of file open for write returns -ETXTBSY ========================================================== 14:13:20 (1758824000) multiop /mnt/lustre/d14aa.sanityn/f14aa.sanityn vOw_c TMPPIPE=/tmp/multiop_open_wait_pipe.7559 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 335: /mnt/lustre2/d14aa.sanityn/f14aa.sanityn: Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4975: 22524 Terminated $MULTIOP_PROG $FILE v$ARGS > $TMPPIPE (wd: ~) PASS 14aa (2s) == sanityn test 14ab: open(RDWR) of executing file returns -ETXTBSY ========================================================== 14:13:22 (1758824002) open(O_RDWR|O_CREAT): Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4975: 23103 Terminated $DIR1/$tdir/sleep 60 (wd: ~) PASS 14ab (3s) == sanityn test 14b: truncate of executing file returns -ETXTBSY ================================================================ 14:13:25 (1758824005) truncate: cannot truncate '/mnt/lustre2/d14b.sanityn/sleep' to length 60: Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 350: 23681 Terminated $DIR1/$tdir/sleep 60 PASS 14b (2s) == sanityn test 14c: open(O_TRUNC) of executing file return -ETXTBSY ============================================================ 14:13:27 (1758824007) cp: cannot create regular file '/mnt/lustre2/d14c.sanityn/sleep': Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 362: 24263 Terminated $DIR1/$tdir/sleep 60 PASS 14c (2s) == sanityn test 14d: chmod of executing file is still possible ================================================================== 14:13:29 (1758824009) chmod /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 373: 24844 Terminated $DIR1/$tdir/sleep 60 PASS 14d (4s) == sanityn test 15: test out-of-space with multiple writers ===================================================================== 14:13:33 (1758824013) PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin: mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky sleep 5 for ZFS MDS Waiting for MDT destroys to complete fallocate on zfs doesn't consume space STRIPECOUNT=2 ORIGFREE=7524352 MAXFREE=800000 SKIP: oos2 test_15 oos2.sh: 7524352KB free > 800000KB, increase MAXFREE (or reduce fs size) sleep 5 for ZFS MDS Waiting for MDT destroys to complete SKIP 15 (32s) == sanityn test 16a: 500 iterations of dual-mount fsx ==== 14:14:05 (1758824045) fallocate on zfs doesn't consume space fallocate not supported 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.262675 s, 31.9 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.196712 s, 42.6 MB/s Chance of close/open is 1 in 50 Seed set to 4047 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000001[1] 1758824048.116569 write 0x2106d6 thru 0x2167c0 (0x60eb bytes) 000002[0] 1758824048.137487 mapread 0x0d5356 thru 0x0e21cc (0xce77 bytes) 000003[0] 1758824048.198922 mapwrite 0x4696a0 thru 0x47845c (0xedbd bytes) truncating to largest ever: 0x2ed617 000006[0] 1758824048.338962 trunc from 0x47845d to 0x2ed616 (0x18ae46 bytes) 000007[0] 1758824048.371064 write 0x52f3b0 thru 0x53385c (0x44ad bytes) 000013[1] 1758824048.412198 write 0x8de1a4 thru 0x8de283 (0xe0 bytes) 000015[0] 1758824048.456271 mapread 0x570937 thru 0x576758 (0x5e22 bytes) 000018[0] 1758824048.587923 mapread 0x32031d thru 0x32dd14 (0xd9f8 bytes) 000021[1] 1758824048.657766 write 0x008e7f thru 0x016b1d (0xdc9f bytes) 000023[1] 1758824048.729103 mapread 0x1c6bd0 thru 0x1d6b93 (0xffc4 bytes) 000026[1] 1758824048.819206 write 0x28b395 thru 0x28ff64 (0x4bd0 bytes) 000028[0] 1758824048.854580 mapread 0x039f31 thru 0x043a7f (0x9b4f bytes) 000031[0] 1758824048.918693 mapread 0x18ebf3 thru 0x19b700 (0xcb0e bytes) 000032[1] 1758824048.964450 mapread 0x8c1576 thru 0x8ca8c9 (0x9354 bytes) 000033[1] 1758824049.009587 mapread 0x479084 thru 0x4888a1 (0xf81e bytes) truncating to largest ever: 0x9dee5b 000034[0] 1758824049.054701 trunc from 0x8de284 to 0x9dee5a (0x100bd7 bytes) 000035[0] 1758824049.098529 read 0x6b0c74 thru 0x6b233b (0x16c8 bytes) 000037[1] 1758824049.126902 trunc from 0x9dee5b to 0x4185ca (0x5c6890 bytes) 000038[1] 1758824049.168277 trunc from 0x4185cb to 0x2890c1 (0x18f509 bytes) 000040[0] 1758824049.215868 mapwrite 0x629fc1 thru 0x637d34 (0xdd74 bytes) 000042[1] 1758824049.309202 write 0x5af07e thru 0x5af1bf (0x142 bytes) 000045[1] 1758824049.375586 mapwrite 0x113050 thru 0x11e8a2 (0xb853 bytes) 000046[1] 1758824049.469713 read 0x323664 thru 0x32dddc (0xa779 bytes) 000049[0] 1758824049.490069 read 0x5503c4 thru 0x55d76f (0xd3ac bytes) 000050[1] 1758824049.523457 mapwrite 0x1d19c7 thru 0x1dbb19 (0xa153 bytes) 000052[1] 1758824049.628824 trunc from 0x637d35 to 0x9c16b8 (0x389984 bytes) 000053[0] 1758824049.673110 trunc from 0x9c16b9 to 0x3f4a74 (0x5ccc44 bytes) 000054[0] 1758824049.717254 mapread 0x3986c1 thru 0x39997f (0x12bf bytes) 000055[1] 1758824049.730231 trunc from 0x3f4a75 to 0x07af0f (0x379b65 bytes) 000059[1] 1758824049.803580 mapread 0x016948 thru 0x01985e (0x2f17 bytes) 000065[1] 1758824049.862188 mapwrite 0x09d4fb thru 0x0acf3c (0xfa42 bytes) 000069[0] 1758824049.962794 mapread 0x05bc9b thru 0x066852 (0xabb8 bytes) 000070[1] 1758824050.011389 trunc from 0x0acf3d to 0x0a2cdc (0xa260 bytes) 000073[0] 1758824050.038984 trunc from 0x0a2cdd to 0x87595d (0x7d2c81 bytes) 000075[1] 1758824050.096556 mapread 0x818ed4 thru 0x8192ca (0x3f7 bytes) 000076[0] 1758824050.128971 mapread 0x6bf512 thru 0x6c713b (0x7c2a bytes) 000078[0] 1758824050.157753 read 0x7abbb4 thru 0x7b865f (0xcaac bytes) 000079[0] 1758824050.180923 read 0x305466 thru 0x306728 (0x12c3 bytes) 000080[0] 1758824050.185224 read 0x827e32 thru 0x828ea1 (0x1070 bytes) 000081[0] 1758824050.189743 mapread 0x335d76 thru 0x344cb8 (0xef43 bytes) 000085[1] 1758824050.238788 read 0x811ad4 thru 0x821096 (0xf5c3 bytes) 000086[1] 1758824050.249118 mapread 0x2d14a1 thru 0x2de256 (0xcdb6 bytes) 000088[0] 1758824050.303404 mapread 0x2320e1 thru 0x236c6a (0x4b8a bytes) 000089[0] 1758824050.320935 write 0x47d24c thru 0x47f96a (0x271f bytes) 000090[0] 1758824050.332457 mapwrite 0x64c1d0 thru 0x6544ec (0x831d bytes) 000094[1] 1758824050.388527 mapread 0x62693e thru 0x633de7 (0xd4aa bytes) 000096[0] 1758824050.440514 trunc from 0x87595e to 0x3542d7 (0x521686 bytes) 000102[1] 1758824050.543188 mapread 0x31f442 thru 0x320f09 (0x1ac8 bytes) 000103[1] 1758824050.567611 mapread 0x1f6065 thru 0x1fe57f (0x851b bytes) 000104[1] 1758824050.598592 trunc from 0x3542d8 to 0x434fea (0xe0d13 bytes) 000106[1] 1758824050.667610 read 0x1dc981 thru 0x1e3432 (0x6ab2 bytes) 000107[0] 1758824050.700593 mapwrite 0x21af8c thru 0x2253fa (0xa46f bytes) 000108[0] 1758824050.820439 trunc from 0x434feb to 0x2406b5 (0x1f4935 bytes) 000109[0] 1758824050.847850 read 0x172fd7 thru 0x17f1dc (0xc206 bytes) 000110[1] 1758824050.896355 mapwrite 0x59cd4a thru 0x59dd97 (0x104e bytes) 000113[0] 1758824051.004387 write 0x9e1700 thru 0x9e587e (0x417f bytes) 000114[0] 1758824051.025445 read 0x69bda2 thru 0x6a64fc (0xa75b bytes) 000117[1] 1758824051.088753 read 0x320229 thru 0x3253cd (0x51a5 bytes) 000119[1] 1758824051.104926 write 0x7ca91a thru 0x7d13f7 (0x6ade bytes) 000121[0] 1758824051.130521 read 0x3de9bd thru 0x3e4d75 (0x63b9 bytes) 000125[1] 1758824051.164902 read 0x055ff0 thru 0x05d06a (0x707b bytes) 000127[1] 1758824051.171864 mapread 0x98aece thru 0x995405 (0xa538 bytes) 000130[1] 1758824051.192154 mapwrite 0x48e176 thru 0x490245 (0x20d0 bytes) 000133[1] 1758824051.285688 mapread 0x87e534 thru 0x88a52b (0xbff8 bytes) 000135[0] 1758824051.311462 write 0x50eb85 thru 0x514c2a (0x60a6 bytes) 000140[1] 1758824051.327227 trunc from 0x9e587f to 0x09dae9 (0x947d95 bytes) 000141[1] 1758824051.383427 write 0x305d00 thru 0x308363 (0x2664 bytes) 000142[1] 1758824051.407562 read 0x212a54 thru 0x21d607 (0xabb4 bytes) 000144[0] 1758824051.450050 read 0x0c0b52 thru 0x0ca7f0 (0x9c9f bytes) 000145[1] 1758824051.510756 trunc from 0x308364 to 0x7734c2 (0x46b15f bytes) 000147[0] 1758824051.580142 mapread 0x7126ba thru 0x71ded3 (0xb81a bytes) 000150[1] 1758824051.651870 trunc from 0x7734c3 to 0x7232b3 (0x5020f bytes) 000151[0] 1758824051.696527 trunc from 0x7232b4 to 0x543cd0 (0x1df5e3 bytes) 000154[0] 1758824051.748817 trunc from 0x543cd1 to 0x269b54 (0x2da17c bytes) 000155[0] 1758824051.770867 mapwrite 0x44b275 thru 0x454d79 (0x9b05 bytes) truncating to largest ever: 0x9eefa8 000158[0] 1758824051.856521 trunc from 0x454d7a to 0x9eefa7 (0x59a22e bytes) 000159[0] 1758824051.894238 trunc from 0x9eefa8 to 0x48423a (0x56ad6d bytes) 000160[0] 1758824051.926408 write 0x03c784 thru 0x03f956 (0x31d3 bytes) 000161[0] 1758824051.941426 mapwrite 0x7344d8 thru 0x73b4fe (0x7027 bytes) 000167[1] 1758824052.052928 mapread 0x078855 thru 0x078e61 (0x60d bytes) 000169[1] 1758824052.112338 trunc from 0x73b4ff to 0x37d651 (0x3bdead bytes) 000170[0] 1758824052.150351 mapwrite 0x638b5d thru 0x63a2f1 (0x1795 bytes) 000172[1] 1758824052.245685 trunc from 0x63a2f2 to 0x0ff29c (0x53b055 bytes) 000176[1] 1758824052.315849 write 0x451379 thru 0x459004 (0x7c8c bytes) 000179[1] 1758824052.333503 read 0x2ca8be thru 0x2caac5 (0x208 bytes) 000183[1] 1758824052.363213 read 0x00366e thru 0x00f606 (0xbf99 bytes) 000184[1] 1758824052.389096 read 0x1b8574 thru 0x1ba0f7 (0x1b84 bytes) 000187[1] 1758824052.423196 trunc from 0x459005 to 0x984476 (0x52b472 bytes) 000190[0] 1758824052.464359 mapread 0x394b57 thru 0x39ecfc (0xa1a6 bytes) 000191[1] 1758824052.564101 mapwrite 0x687e34 thru 0x68c1ae (0x437b bytes) 000193[1] 1758824052.647502 mapwrite 0x9cda5c thru 0x9dc964 (0xef09 bytes) 000195[0] 1758824052.771952 mapwrite 0x46d608 thru 0x47b95a (0xe353 bytes) 000198[1] 1758824052.957207 trunc from 0x9dc965 to 0x14dd39 (0x88ec2b bytes) 000200[0] 1758824053.005205 write 0x5ba5ba thru 0x5c0836 (0x627d bytes) 000202[1] 1758824053.048517 mapwrite 0x87dfa8 thru 0x882275 (0x42ce bytes) 000203[1] 1758824053.145475 mapread 0x0ea4c3 thru 0x0f8c11 (0xe74f bytes) 000204[1] 1758824053.182435 mapwrite 0x1df4f4 thru 0x1e21b1 (0x2cbe bytes) 000206[1] 1758824053.261920 trunc from 0x882276 to 0x89857e (0x16309 bytes) 000208[0] 1758824053.306006 read 0x5282ae thru 0x5378e7 (0xf63a bytes) 000209[1] 1758824053.322783 mapread 0x6272f8 thru 0x62900d (0x1d16 bytes) 000210[0] 1758824053.358570 read 0x783351 thru 0x78e2ae (0xaf5e bytes) 000211[1] 1758824053.390093 mapread 0x29a6d1 thru 0x2a2fbf (0x88ef bytes) 000212[0] 1758824053.436841 mapread 0x7b08c2 thru 0x7b6d72 (0x64b1 bytes) 000213[0] 1758824053.498221 write 0x3329df thru 0x336c48 (0x426a bytes) 000214[1] 1758824053.511934 mapread 0x51cccc thru 0x52024d (0x3582 bytes) 000216[1] 1758824053.584831 write 0x050bbb thru 0x053bde (0x3024 bytes) 000217[1] 1758824053.627910 trunc from 0x89857f to 0x5fb320 (0x29d25e bytes) 000219[1] 1758824053.691931 write 0x8ebf09 thru 0x8eee58 (0x2f50 bytes) 000220[0] 1758824053.694803 read 0x437047 thru 0x443545 (0xc4ff bytes) 000226[0] 1758824053.772125 mapread 0x034ead thru 0x043ed9 (0xf02d bytes) 000228[1] 1758824053.814463 trunc from 0x8eee59 to 0x8b89c5 (0x36493 bytes) 000231[1] 1758824053.886002 write 0x1513bd thru 0x15770e (0x6352 bytes) 000232[0] 1758824053.913596 read 0x46d13c thru 0x47198e (0x4853 bytes) 000233[1] 1758824053.936374 mapwrite 0x56b846 thru 0x57a5a5 (0xed60 bytes) 000234[1] 1758824054.022853 mapread 0x039bf9 thru 0x03d85b (0x3c63 bytes) 000235[0] 1758824054.047020 mapwrite 0x0d1726 thru 0x0d2aba (0x1395 bytes) 000236[1] 1758824054.129578 read 0x37f553 thru 0x381ba4 (0x2652 bytes) 000238[0] 1758824054.145134 trunc from 0x8b89c6 to 0x787ef5 (0x130ad0 bytes) 000246[1] 1758824054.250697 write 0x5ab413 thru 0x5b7b9f (0xc78d bytes) 000248[1] 1758824054.303539 read 0x44e053 thru 0x45af91 (0xcf3f bytes) 000249[1] 1758824054.325040 trunc from 0x787ef6 to 0x8540e0 (0xcc1eb bytes) 000259[1] 1758824054.445685 trunc from 0x8540e1 to 0x1de886 (0x67585a bytes) 000260[0] 1758824054.477729 mapwrite 0x7b224a thru 0x7beef1 (0xcca8 bytes) 000263[0] 1758824054.579917 mapwrite 0x14dfec thru 0x155779 (0x778e bytes) 000264[0] 1758824054.634006 mapread 0x6219c8 thru 0x62ee5f (0xd498 bytes) 000265[1] 1758824054.670066 mapread 0x576cb9 thru 0x58346d (0xc7b5 bytes) 000266[1] 1758824054.718548 read 0x70649c thru 0x706b07 (0x66c bytes) 000267[0] 1758824054.735147 read 0x68c9e3 thru 0x69adb7 (0xe3d5 bytes) 000270[0] 1758824054.759888 read 0x07935e thru 0x088d83 (0xfa26 bytes) 000271[0] 1758824054.777413 trunc from 0x7beef2 to 0x633ca4 (0x18b24d bytes) 000275[1] 1758824054.846907 mapwrite 0x798c2b thru 0x799416 (0x7ec bytes) 000276[1] 1758824054.917121 write 0x75057e thru 0x75ccd6 (0xc759 bytes) 000277[0] 1758824054.947637 read 0x250626 thru 0x25de9f (0xd87a bytes) 000279[0] 1758824054.969672 write 0x78f665 thru 0x790fd4 (0x1970 bytes) 000280[0] 1758824054.985055 write 0x5ba45a thru 0x5bf2f8 (0x4e9f bytes) 000281[1] 1758824055.004816 mapwrite 0x2e0d74 thru 0x2ea228 (0x94b5 bytes) 000282[1] 1758824055.079015 trunc from 0x799417 to 0x19a8ff (0x5feb17 bytes) 000284[0] 1758824055.124608 mapwrite 0x7f0e99 thru 0x7f4821 (0x3989 bytes) 000287[1] 1758824055.239542 write 0x72721b thru 0x7347aa (0xd590 bytes) 000288[1] 1758824055.275180 read 0x58a8a3 thru 0x58e817 (0x3f75 bytes) 000289[0] 1758824055.301235 mapwrite 0x2579bd thru 0x25bf7d (0x45c1 bytes) 000293[0] 1758824055.448477 read 0x53a168 thru 0x546937 (0xc7d0 bytes) 000294[1] 1758824055.470841 read 0x040398 thru 0x04bcfe (0xb967 bytes) 000295[0] 1758824055.483953 mapwrite 0x6a55a4 thru 0x6b0a8d (0xb4ea bytes) 000297[0] 1758824055.592096 read 0x7432ec thru 0x753229 (0xff3e bytes) 000298[0] 1758824055.616826 mapwrite 0x4f6005 thru 0x4f72bb (0x12b7 bytes) 000299[0] 1758824055.656199 write 0x5c47be thru 0x5d0f78 (0xc7bb bytes) 000301[0] 1758824055.687641 write 0x5d1adb thru 0x5dabe7 (0x910d bytes) 000304[0] 1758824055.730911 write 0x77e571 thru 0x785c43 (0x76d3 bytes) 000305[0] 1758824055.768231 trunc from 0x7f4822 to 0x03209a (0x7c2787 bytes) 000311[0] 1758824055.889987 mapread 0x025a68 thru 0x02649b (0xa34 bytes) 000312[1] 1758824055.921034 trunc from 0x03209b to 0x074d12 (0x42c78 bytes) 000314[1] 1758824055.981010 mapread 0x00c48d thru 0x00e6e1 (0x2255 bytes) 000315[1] 1758824055.994931 read 0x0622c8 thru 0x070e12 (0xeb4b bytes) 000316[0] 1758824056.025632 mapwrite 0x5e770a thru 0x5ea98f (0x3286 bytes) 000320[0] 1758824056.145528 trunc from 0x5ea990 to 0x8fcd31 (0x3123a2 bytes) 000322[0] 1758824056.170962 trunc from 0x8fcd32 to 0x3bf01a (0x53dd17 bytes) 000324[1] 1758824056.190691 trunc from 0x3bf01b to 0x791448 (0x3d242e bytes) 000327[1] 1758824056.299531 mapread 0x0d1587 thru 0x0da116 (0x8b90 bytes) 000330[1] 1758824056.376705 trunc from 0x791449 to 0x82489e (0x93456 bytes) 000331[1] 1758824056.388811 write 0x330473 thru 0x333a63 (0x35f1 bytes) 000335[0] 1758824056.447994 trunc from 0x82489f to 0x62f068 (0x1f5836 bytes) 000337[1] 1758824056.485826 read 0x440ab3 thru 0x449fde (0x952c bytes) 000341[1] 1758824056.549722 read 0x307b1f thru 0x31028f (0x8771 bytes) 000344[0] 1758824056.567345 write 0x947f68 thru 0x95356e (0xb607 bytes) 000346[1] 1758824056.590812 write 0x9c4991 thru 0x9d039c (0xba0c bytes) 000348[1] 1758824056.619474 read 0x4863bc thru 0x486fd5 (0xc1a bytes) 000349[1] 1758824056.624754 mapread 0x31ebeb thru 0x32e8db (0xfcf1 bytes) 000352[0] 1758824056.674134 mapread 0x3a2c4a thru 0x3aeac5 (0xbe7c bytes) 000357[0] 1758824056.728741 read 0x21e68c thru 0x22457f (0x5ef4 bytes) 000359[1] 1758824056.737293 write 0x449144 thru 0x4495d7 (0x494 bytes) 000360[0] 1758824056.753019 write 0x16f712 thru 0x1730a2 (0x3991 bytes) 000361[1] 1758824056.786642 trunc from 0x9d039d to 0x5c4a6a (0x40b932 bytes) 000362[1] 1758824056.822754 mapwrite 0x6d687d thru 0x6de782 (0x7f06 bytes) 000363[1] 1758824056.909520 write 0x997578 thru 0x99b92c (0x43b5 bytes) 000367[1] 1758824056.941818 mapread 0x63cc44 thru 0x649740 (0xcafd bytes) 000368[0] 1758824056.961963 mapwrite 0x998893 thru 0x999eeb (0x1659 bytes) 000369[0] 1758824057.039003 mapread 0x6e9b3b thru 0x6f3058 (0x951e bytes) 000370[1] 1758824057.105775 write 0x5fc353 thru 0x5fc7cf (0x47d bytes) 000372[0] 1758824057.147198 mapwrite 0x638886 thru 0x647c08 (0xf383 bytes) 000374[1] 1758824057.213270 read 0x510020 thru 0x51fa1e (0xf9ff bytes) 000375[1] 1758824057.232366 mapwrite 0x736a84 thru 0x740bb6 (0xa133 bytes) 000376[0] 1758824057.569615 mapread 0x4d72c2 thru 0x4dc42e (0x516d bytes) 000377[1] 1758824057.619626 read 0x619b4f thru 0x627001 (0xd4b3 bytes) 000378[0] 1758824057.653558 read 0x1731c8 thru 0x17ac23 (0x7a5c bytes) 000380[0] 1758824057.661534 mapwrite 0x06348a thru 0x06b524 (0x809b bytes) 000381[0] 1758824057.717952 write 0x6fdebc thru 0x704001 (0x6146 bytes) 000383[0] 1758824057.747563 mapread 0x2c368a thru 0x2d2ba5 (0xf51c bytes) 000385[1] 1758824057.767566 write 0x59aca0 thru 0x5a0d5f (0x60c0 bytes) 000390[0] 1758824057.866894 write 0x6d6279 thru 0x6e0331 (0xa0b9 bytes) 000392[0] 1758824057.919902 mapwrite 0x69eaef thru 0x6ab46a (0xc97c bytes) 000394[0] 1758824058.003222 write 0x434ae4 thru 0x437a88 (0x2fa5 bytes) 000395[1] 1758824058.029350 read 0x7160c8 thru 0x7174e5 (0x141e bytes) 000396[0] 1758824058.061992 mapwrite 0x26232b thru 0x268b3f (0x6815 bytes) 000399[0] 1758824058.167996 mapwrite 0x06794b thru 0x073176 (0xb82c bytes) 000400[0] 1758824058.240040 mapread 0x545461 thru 0x55410b (0xecab bytes) 000401[0] 1758824058.276856 trunc from 0x99b92d to 0x20a18d (0x79179f bytes) 000403[1] 1758824058.317537 mapwrite 0x2a1c70 thru 0x2aae0c (0x919d bytes) 000404[0] 1758824058.404001 mapwrite 0x8ddd4c thru 0x8e75c4 (0x9879 bytes) 000406[0] 1758824058.511267 trunc from 0x8e75c5 to 0x5be486 (0x32913e bytes) 000410[0] 1758824058.560628 mapwrite 0x9a419f thru 0x9a85d1 (0x4433 bytes) 000411[0] 1758824058.652042 trunc from 0x9a85d2 to 0x1c47e3 (0x7e3dee bytes) 000415[0] 1758824058.728277 mapwrite 0x2dda2e thru 0x2e7035 (0x9608 bytes) 000416[1] 1758824058.817227 read 0x166411 thru 0x16c360 (0x5f50 bytes) 000417[1] 1758824058.869751 mapread 0x0f3491 thru 0x0fa9f8 (0x7568 bytes) 000420[0] 1758824058.911617 write 0x015a65 thru 0x016513 (0xaaf bytes) 000425[0] 1758824058.983255 mapwrite 0x5bfd87 thru 0x5cf0e5 (0xf35f bytes) 000430[0] 1758824059.137264 mapwrite 0x1e7c00 thru 0x1eb20e (0x360f bytes) 000431[1] 1758824059.239001 write 0x717f67 thru 0x71c6bf (0x4759 bytes) 000432[1] 1758824059.265892 read 0x360e99 thru 0x36b9b1 (0xab19 bytes) 000433[1] 1758824059.305927 mapread 0x01886b thru 0x0278ee (0xf084 bytes) 000435[0] 1758824059.344295 write 0x5cd8a3 thru 0x5d31a6 (0x5904 bytes) 000436[1] 1758824059.383686 mapread 0x216ab7 thru 0x21ed1a (0x8264 bytes) 000438[1] 1758824059.417283 write 0x980f67 thru 0x986f43 (0x5fdd bytes) 000444[0] 1758824059.526748 mapread 0x1590bc thru 0x166a50 (0xd995 bytes) 000448[1] 1758824059.625849 mapwrite 0x774775 thru 0x778e9b (0x4727 bytes) 000453[1] 1758824059.718764 mapread 0x853487 thru 0x85ac6d (0x77e7 bytes) 000455[1] 1758824059.739616 write 0x78f78a thru 0x7929cf (0x3246 bytes) 000456[0] 1758824059.765558 mapwrite 0x354fdd thru 0x36299f (0xd9c3 bytes) 000459[0] 1758824059.886442 mapread 0x047424 thru 0x04b281 (0x3e5e bytes) 000460[0] 1758824059.936608 read 0x46ad59 thru 0x470725 (0x59cd bytes) 000461[1] 1758824059.966237 mapread 0x48877b thru 0x495a0d (0xd293 bytes) 000464[1] 1758824060.010779 write 0x9e6ca2 thru 0x9ecb26 (0x5e85 bytes) 000465[0] 1758824060.036646 read 0x444fde thru 0x4497ae (0x47d1 bytes) 000467[0] 1758824060.043797 read 0x1cb1f1 thru 0x1d18a0 (0x66b0 bytes) 000468[0] 1758824060.064019 write 0x1a41e4 thru 0x1b37d1 (0xf5ee bytes) 000469[1] 1758824060.104617 write 0x20aed4 thru 0x2178f2 (0xca1f bytes) 000473[1] 1758824060.171484 trunc from 0x9ecb27 to 0x94cd5c (0x9fdca bytes) 000476[0] 1758824060.280707 trunc from 0x94cd5d to 0x099017 (0x8b3d45 bytes) 000477[0] 1758824060.348339 write 0x401404 thru 0x40cc19 (0xb816 bytes) 000479[0] 1758824060.351034 trunc from 0x40cc1a to 0x80767a (0x3faa61 bytes) 000487[0] 1758824060.443855 read 0x6ab8cc thru 0x6adf94 (0x26c9 bytes) 000488[0] 1758824060.463745 trunc from 0x80767b to 0x479c7c (0x38d9fe bytes) 000490[1] 1758824060.518022 mapread 0x212e10 thru 0x21a2ad (0x749e bytes) 000492[1] 1758824060.574991 trunc from 0x479c7d to 0x848fbd (0x3cf341 bytes) 000497[1] 1758824060.637978 mapread 0x5d7e9e thru 0x5e1c4f (0x9db2 bytes) 000499[0] 1758824060.672630 mapread 0x04bc13 thru 0x055d3a (0xa128 bytes) 000500[0] 1758824060.716931 trunc from 0x848fbe to 0x80a2e5 (0x3ecd8 bytes) All operations completed A-OK! Chance of close/open is 1 in 50 Seed set to 4060 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported truncating to largest ever: 0x8993da 000001[0] 1758824061.242637 trunc from 00000000 to 0x8993d9 (0x8993da bytes) 000002[1] 1758824061.262230 read_OD 0x5d9000 thru 0x5e3fff (0xb000 bytes) 000003[0] 1758824061.290895 mapwrite 0x03e000 thru 0x04796f (0x9970 bytes) 000004[1] 1758824061.356786 mapwrite 0x02d000 thru 0x0362fe (0x92ff bytes) 000005[0] 1758824061.428057 mapwrite 0x20f000 thru 0x2117e1 (0x27e2 bytes) 000006[0] 1758824061.493754 mapread 0x0ce000 thru 0x0d1a83 (0x3a84 bytes) truncating to largest ever: 0x93c573 000008[0] 1758824061.518736 trunc from 0x8993da to 0x93c572 (0xa3199 bytes) 000010[1] 1758824061.575808 write_OD 0x4ec000 thru 0x4f0fff (0x5000 bytes) 000013[0] 1758824061.611191 mapwrite 0x15a000 thru 0x168f42 (0xef43 bytes) 000014[0] 1758824061.678189 read_OD 0x371000 thru 0x371fff (0x1000 bytes) 000016[0] 1758824061.693553 mapread 0x661000 thru 0x666b03 (0x5b04 bytes) 000021[1] 1758824061.720176 mapwrite 0x743000 thru 0x7469ab (0x39ac bytes) 000027[1] 1758824061.803959 mapread 0x4d9000 thru 0x4e570c (0xc70d bytes) 000029[1] 1758824061.821915 mapwrite 0x728000 thru 0x72ac1b (0x2c1c bytes) 000031[0] 1758824061.878947 mapwrite 0x661000 thru 0x66ea27 (0xda28 bytes) 000034[0] 1758824061.961043 mapread 0x3fb000 thru 0x3fbbdb (0xbdc bytes) 000035[0] 1758824061.965582 trunc from 0x93c573 to 0x2ec207 (0x65036b bytes) 000038[1] 1758824062.005255 read_OD 0x21b000 thru 0x229fff (0xf000 bytes) 000039[0] 1758824062.021409 write_OD 0x630000 thru 0x63afff (0xb000 bytes) 000041[1] 1758824062.067626 mapwrite 0x98b000 thru 0x99418e (0x918f bytes) 000044[0] 1758824062.231101 trunc from 0x99418f to 0x11db39 (0x876655 bytes) 000047[0] 1758824062.302818 write_OD 0x9a6000 thru 0x9a9fff (0x4000 bytes) 000051[0] 1758824062.346503 mapread 0x886000 thru 0x889abe (0x3abf bytes) 000052[1] 1758824062.367387 write_OD 0x926000 thru 0x928fff (0x3000 bytes) 000053[1] 1758824062.403470 mapread 0x39a000 thru 0x3a86a1 (0xe6a2 bytes) 000058[1] 1758824062.479338 trunc from 0x9aa000 to 0x117b65 (0x89249a bytes) 000061[1] 1758824062.525990 mapread 0x024000 thru 0x0260e9 (0x20ea bytes) 000062[1] 1758824062.539557 read_OD 0x0d5000 thru 0x0dcfff (0x8000 bytes) 000065[1] 1758824062.568138 read_OD 0x0c7000 thru 0x0cbfff (0x5000 bytes) 000067[1] 1758824062.578436 write_OD 0x38c000 thru 0x393fff (0x8000 bytes) 000071[1] 1758824062.626221 read_OD 0x311000 thru 0x316fff (0x6000 bytes) 000074[1] 1758824062.635386 read_OD 0x205000 thru 0x20ffff (0xb000 bytes) 000075[1] 1758824062.643945 write_OD 0x7cb000 thru 0x7cbfff (0x1000 bytes) 000076[1] 1758824062.691328 read_OD 0x048000 thru 0x04bfff (0x4000 bytes) 000080[0] 1758824062.700814 read_OD 0x66e000 thru 0x66ffff (0x2000 bytes) 000081[1] 1758824062.706961 mapwrite 0x230000 thru 0x23b735 (0xb736 bytes) 000084[1] 1758824062.789822 mapread 0x426000 thru 0x4287b6 (0x27b7 bytes) 000087[1] 1758824062.831894 mapwrite 0x181000 thru 0x18a8d7 (0x98d8 bytes) 000088[1] 1758824062.889203 write_OD 0x657000 thru 0x663fff (0xd000 bytes) 000091[1] 1758824062.927735 read_OD 0x6cb000 thru 0x6d8fff (0xe000 bytes) 000092[1] 1758824062.939555 read_OD 0x4d3000 thru 0x4dffff (0xd000 bytes) 000093[1] 1758824062.950193 trunc from 0x7cc000 to 0x81c969 (0x5096a bytes) 000095[1] 1758824062.997101 write_OD 0x249000 thru 0x24bfff (0x3000 bytes) 000098[0] 1758824063.039113 mapread 0x167000 thru 0x16baad (0x4aae bytes) 000101[0] 1758824063.054483 trunc from 0x81c96a to 0x1a5910 (0x677059 bytes) 000103[1] 1758824063.082120 mapwrite 0x1bb000 thru 0x1ca433 (0xf434 bytes) 000105[0] 1758824063.203067 mapread 0x0dd000 thru 0x0e2c3b (0x5c3c bytes) 000109[0] 1758824063.231136 write_OD 0x94b000 thru 0x955fff (0xb000 bytes) 000111[0] 1758824063.277348 trunc from 0x956000 to 0x30a8ff (0x64b700 bytes) 000113[1] 1758824063.322231 trunc from 0x30a900 to 0x2781a6 (0x92759 bytes) 000115[1] 1758824063.342866 read_OD 0x1dd000 thru 0x1ddfff (0x1000 bytes) 000117[1] 1758824063.354030 read_OD 0x066000 thru 0x071fff (0xc000 bytes) 000118[0] 1758824063.361775 mapread 0x1a4000 thru 0x1a7475 (0x3476 bytes) 000119[1] 1758824063.375841 mapwrite 0x1c9000 thru 0x1d71e6 (0xe1e7 bytes) 000121[0] 1758824063.501372 mapread 0x0cd000 thru 0x0d8e46 (0xbe47 bytes) 000129[0] 1758824063.547968 read_OD 0x253000 thru 0x25afff (0x8000 bytes) 000133[0] 1758824063.558888 write_OD 0x063000 thru 0x06afff (0x8000 bytes) 000135[0] 1758824063.607017 mapread 0x246000 thru 0x246190 (0x191 bytes) 000136[1] 1758824063.644746 read_OD 0x24e000 thru 0x25afff (0xd000 bytes) truncating to largest ever: 0x9461fe 000138[0] 1758824063.651946 trunc from 0x2781a7 to 0x9461fd (0x6ce057 bytes) 000139[1] 1758824063.697731 read_OD 0x265000 thru 0x26cfff (0x8000 bytes) 000140[1] 1758824063.722791 trunc from 0x9461fe to 0x5ead51 (0x35b4ac bytes) 000141[1] 1758824063.756118 mapwrite 0x10d000 thru 0x1155d0 (0x85d1 bytes) 000142[0] 1758824063.823401 trunc from 0x5ead52 to 0x646f13 (0x5c1c2 bytes) 000144[1] 1758824063.894742 mapread 0x468000 thru 0x46f7f4 (0x77f5 bytes) 000146[1] 1758824063.938698 mapwrite 0x8a4000 thru 0x8b111c (0xd11d bytes) 000147[1] 1758824064.044608 write_OD 0x96a000 thru 0x96cfff (0x3000 bytes) 000148[0] 1758824064.076226 write_OD 0x6dd000 thru 0x6defff (0x2000 bytes) 000149[0] 1758824064.114782 trunc from 0x96d000 to 0x3b3daf (0x5b9250 bytes) 000151[0] 1758824064.152558 mapread 0x27c000 thru 0x28747f (0xb480 bytes) 000152[0] 1758824064.173451 mapwrite 0x64d000 thru 0x65b58b (0xe58c bytes) 000153[0] 1758824064.282056 read_OD 0x129000 thru 0x134fff (0xc000 bytes) 000154[0] 1758824064.306285 trunc from 0x65b58c to 0x6e680d (0x8b282 bytes) 000156[0] 1758824064.357566 mapread 0x077000 thru 0x07f3a6 (0x83a7 bytes) 000157[0] 1758824064.419386 write_OD 0x3f1000 thru 0x3fafff (0xa000 bytes) 000158[1] 1758824064.460864 mapwrite 0x47c000 thru 0x48a975 (0xe976 bytes) 000159[0] 1758824064.535283 read_OD 0x61c000 thru 0x621fff (0x6000 bytes) 000164[1] 1758824064.563053 read_OD 0x295000 thru 0x2a1fff (0xd000 bytes) 000165[1] 1758824064.570532 trunc from 0x6e680e to 0x5e1a5a (0x104db3 bytes) 000167[0] 1758824064.622680 read_OD 0x379000 thru 0x37afff (0x2000 bytes) 000168[0] 1758824064.643071 read_OD 0x0b1000 thru 0x0bdfff (0xd000 bytes) 000170[1] 1758824064.653989 write_OD 0x090000 thru 0x09cfff (0xd000 bytes) 000173[0] 1758824064.712093 write_OD 0x44c000 thru 0x44dfff (0x2000 bytes) 000174[1] 1758824064.750836 mapwrite 0x5af000 thru 0x5af241 (0x242 bytes) 000178[1] 1758824064.870679 mapread 0x04d000 thru 0x050fab (0x3fac bytes) 000179[1] 1758824064.888182 read_OD 0x489000 thru 0x495fff (0xd000 bytes) 000181[0] 1758824064.908824 mapread 0x5c1000 thru 0x5c6bd7 (0x5bd8 bytes) 000183[0] 1758824064.940168 mapread 0x51e000 thru 0x521820 (0x3821 bytes) 000186[0] 1758824064.948552 mapwrite 0x3bd000 thru 0x3be60a (0x160b bytes) 000187[0] 1758824065.012137 read_OD 0x4d1000 thru 0x4d4fff (0x4000 bytes) 000189[1] 1758824065.031039 write_OD 0x02c000 thru 0x03afff (0xf000 bytes) 000190[0] 1758824065.078843 trunc from 0x5e1a5b to 0x3251ba (0x2bc8a0 bytes) 000193[0] 1758824065.141433 trunc from 0x3251bb to 0x70e5d5 (0x3e941b bytes) 000195[0] 1758824065.199435 mapwrite 0x595000 thru 0x5996fd (0x46fe bytes) 000196[1] 1758824065.249006 mapwrite 0x181000 thru 0x184afa (0x3afb bytes) 000197[1] 1758824065.324136 write_OD 0x2dc000 thru 0x2ddfff (0x2000 bytes) 000198[1] 1758824065.367580 read_OD 0x60a000 thru 0x611fff (0x8000 bytes) 000200[1] 1758824065.380645 trunc from 0x70e5d6 to 0x34dfee (0x3c05e7 bytes) 000201[0] 1758824065.398370 mapread 0x050000 thru 0x05ee3b (0xee3c bytes) 000203[0] 1758824065.443419 read_OD 0x26f000 thru 0x271fff (0x3000 bytes) 000205[0] 1758824065.448132 mapwrite 0x5c7000 thru 0x5d423e (0xd23f bytes) 000206[1] 1758824065.615882 mapread 0x303000 thru 0x30deae (0xaeaf bytes) 000207[0] 1758824065.657132 mapwrite 0x3c0000 thru 0x3c700d (0x700e bytes) 000209[1] 1758824065.748872 read_OD 0x231000 thru 0x232fff (0x2000 bytes) 000214[0] 1758824065.770847 mapwrite 0x7d7000 thru 0x7e5e93 (0xee94 bytes) 000220[0] 1758824065.941427 mapwrite 0x247000 thru 0x2513a7 (0xa3a8 bytes) 000221[1] 1758824066.037292 mapwrite 0x22a000 thru 0x234003 (0xa004 bytes) 000223[1] 1758824066.126264 write_OD 0x52b000 thru 0x539fff (0xf000 bytes) 000226[1] 1758824066.169962 read_OD 0x72c000 thru 0x734fff (0x9000 bytes) 000227[0] 1758824066.179058 mapread 0x35c000 thru 0x366af4 (0xaaf5 bytes) 000228[0] 1758824066.241762 read_OD 0x75a000 thru 0x75cfff (0x3000 bytes) 000229[0] 1758824066.248143 write_OD 0x29f000 thru 0x2a0fff (0x2000 bytes) 000232[0] 1758824066.300785 mapread 0x68c000 thru 0x698192 (0xc193 bytes) 000233[0] 1758824066.330887 mapwrite 0x633000 thru 0x640146 (0xd147 bytes) 000234[1] 1758824066.418743 mapread 0x771000 thru 0x77e472 (0xd473 bytes) 000235[1] 1758824066.456747 mapwrite 0x2ad000 thru 0x2b1a76 (0x4a77 bytes) 000236[1] 1758824066.507615 read_OD 0x760000 thru 0x761fff (0x2000 bytes) 000237[0] 1758824066.522452 mapread 0x204000 thru 0x2065bb (0x25bc bytes) 000239[1] 1758824066.549574 mapread 0x53b000 thru 0x548dea (0xddeb bytes) 000240[0] 1758824066.575411 write_OD 0x054000 thru 0x061fff (0xe000 bytes) 000241[0] 1758824066.608389 write_OD 0x431000 thru 0x439fff (0x9000 bytes) 000242[1] 1758824066.654425 mapread 0x7ca000 thru 0x7cc514 (0x2515 bytes) 000243[1] 1758824066.681232 read_OD 0x601000 thru 0x608fff (0x8000 bytes) 000247[0] 1758824066.689428 mapread 0x4e8000 thru 0x4ed691 (0x5692 bytes) 000252[0] 1758824066.707354 mapread 0x1c9000 thru 0x1d44c3 (0xb4c4 bytes) 000253[0] 1758824066.729368 write_OD 0x34f000 thru 0x35afff (0xc000 bytes) 000254[0] 1758824066.771535 write_OD 0x14d000 thru 0x14dfff (0x1000 bytes) truncating to largest ever: 0x9944d0 000256[1] 1758824066.824511 trunc from 0x7e5e94 to 0x9944cf (0x1ae63c bytes) 000258[0] 1758824066.868072 mapwrite 0x748000 thru 0x754bd7 (0xcbd8 bytes) 000259[0] 1758824066.945424 mapwrite 0x324000 thru 0x3312d1 (0xd2d2 bytes) truncating to largest ever: 0x9fbee8 000260[1] 1758824067.023432 trunc from 0x9944d0 to 0x9fbee7 (0x67a18 bytes) 000262[0] 1758824067.050333 mapwrite 0x206000 thru 0x2143e8 (0xe3e9 bytes) 000263[0] 1758824067.137384 write_OD 0x20c000 thru 0x21afff (0xf000 bytes) 000268[1] 1758824067.179057 write_OD 0x7d3000 thru 0x7dbfff (0x9000 bytes) 000271[1] 1758824067.224126 mapread 0x3f9000 thru 0x406f52 (0xdf53 bytes) 000273[0] 1758824067.245155 mapwrite 0x172000 thru 0x17ca35 (0xaa36 bytes) 000274[0] 1758824067.350059 mapwrite 0x71c000 thru 0x723923 (0x7924 bytes) 000276[0] 1758824067.467500 read_OD 0x3e3000 thru 0x3edfff (0xb000 bytes) 000277[1] 1758824067.481982 mapread 0x2b1000 thru 0x2b7c03 (0x6c04 bytes) 000279[1] 1758824067.498810 mapwrite 0x97b000 thru 0x98604a (0xb04b bytes) 000282[1] 1758824067.606964 mapwrite 0x32d000 thru 0x330759 (0x375a bytes) 000284[1] 1758824067.685935 mapread 0x908000 thru 0x91791d (0xf91e bytes) 000285[1] 1758824067.719497 read_OD 0x79d000 thru 0x7a2fff (0x6000 bytes) 000288[1] 1758824067.738742 mapwrite 0x74b000 thru 0x754c0b (0x9c0c bytes) 000293[0] 1758824067.843348 mapread 0x568000 thru 0x5716cf (0x96d0 bytes) 000295[0] 1758824067.882265 write_OD 0x527000 thru 0x528fff (0x2000 bytes) 000296[0] 1758824067.922907 mapwrite 0x670000 thru 0x6799de (0x99df bytes) 000299[0] 1758824068.016176 mapwrite 0x426000 thru 0x42d5ed (0x75ee bytes) 000301[1] 1758824068.119087 trunc from 0x9fbee8 to 0x757720 (0x2a47c7 bytes) 000302[1] 1758824068.140147 mapwrite 0x928000 thru 0x93512a (0xd12b bytes) 000303[0] 1758824068.235658 mapread 0x237000 thru 0x241fdf (0xafe0 bytes) 000305[0] 1758824068.283281 mapwrite 0x495000 thru 0x49bb69 (0x6b6a bytes) 000306[1] 1758824068.384073 trunc from 0x93512b to 0x49b803 (0x499927 bytes) 000315[0] 1758824068.464264 read_OD 0x0d6000 thru 0x0dffff (0xa000 bytes) 000317[0] 1758824068.497915 read_OD 0x038000 thru 0x043fff (0xc000 bytes) 000322[0] 1758824068.519350 mapread 0x0f0000 thru 0x0f4660 (0x4661 bytes) 000324[0] 1758824068.537951 read_OD 0x2a4000 thru 0x2aafff (0x7000 bytes) 000326[0] 1758824068.546868 trunc from 0x49b804 to 0x894ad0 (0x3f92cd bytes) 000327[0] 1758824068.586753 mapwrite 0x2c7000 thru 0x2ca840 (0x3841 bytes) 000332[1] 1758824068.711627 mapwrite 0x1a0000 thru 0x1a1390 (0x1391 bytes) 000336[1] 1758824068.794528 write_OD 0x03d000 thru 0x044fff (0x8000 bytes) 000338[1] 1758824068.841968 mapwrite 0x84b000 thru 0x856a6c (0xba6d bytes) 000339[1] 1758824068.928644 mapread 0x7c3000 thru 0x7c43b4 (0x13b5 bytes) 000346[0] 1758824068.988733 mapwrite 0x7c3000 thru 0x7cc5d0 (0x95d1 bytes) 000347[0] 1758824069.048047 read_OD 0x1d1000 thru 0x1d8fff (0x8000 bytes) 000348[1] 1758824069.059814 write_OD 0x0ee000 thru 0x0f6fff (0x9000 bytes) 000349[1] 1758824069.094485 write_OD 0x884000 thru 0x888fff (0x5000 bytes) 000353[1] 1758824069.136752 read_OD 0x23f000 thru 0x24afff (0xc000 bytes) 000356[1] 1758824069.147910 trunc from 0x894ad1 to 0x763fca (0x130b06 bytes) 000358[1] 1758824069.171647 mapread 0x739000 thru 0x73cdd4 (0x3dd5 bytes) 000362[1] 1758824069.233480 mapread 0x46b000 thru 0x46d5e5 (0x25e6 bytes) 000367[1] 1758824069.270247 trunc from 0x763fcb to 0x5a5f67 (0x1be063 bytes) 000370[0] 1758824069.340928 mapwrite 0x204000 thru 0x21327f (0xf280 bytes) 000372[1] 1758824069.450257 write_OD 0x013000 thru 0x019fff (0x7000 bytes) 000378[0] 1758824069.492917 mapwrite 0x1f8000 thru 0x1f899f (0x9a0 bytes) 000381[1] 1758824069.611861 read 0x44d000 thru 0x4551ec (0x81ed bytes) 000382[0] 1758824069.630778 trunc from 0x5a5f68 to 0x4e60ae (0xbfeb9 bytes) 000383[1] 1758824069.662887 write 0x562000 thru 0x565f6a (0x3f6b bytes) 000388[1] 1758824069.712404 mapwrite 0x1ad000 thru 0x1b612b (0x912c bytes) 000391[0] 1758824069.792735 read 0x53a000 thru 0x53ed15 (0x4d16 bytes) 000393[1] 1758824069.807531 read 0x085000 thru 0x08b124 (0x6125 bytes) 000395[0] 1758824069.814879 read 0x252000 thru 0x254ea3 (0x2ea4 bytes) 000396[1] 1758824069.819938 read 0x528000 thru 0x52afe1 (0x2fe2 bytes) 000398[0] 1758824069.826038 read 0x2f6000 thru 0x3026ef (0xc6f0 bytes) 000401[0] 1758824069.835047 mapread 0x171000 thru 0x17a3fa (0x93fb bytes) 000402[0] 1758824069.857916 trunc from 0x565f6b to 0x16e4bd (0x3f7aad bytes) 000406[0] 1758824069.912189 write 0x814000 thru 0x81fbc4 (0xbbc5 bytes) 000409[1] 1758824069.936979 mapwrite 0x22d000 thru 0x237a14 (0xaa15 bytes) 000410[1] 1758824070.010355 mapread 0x23a000 thru 0x23ab29 (0xb2a bytes) 000413[1] 1758824070.035649 read 0x48b000 thru 0x48d8f1 (0x28f2 bytes) 000415[1] 1758824070.049577 mapread 0x552000 thru 0x559fe2 (0x7fe3 bytes) 000416[0] 1758824070.080580 trunc from 0x81fbc5 to 0x4ccba7 (0x35301d bytes) 000419[0] 1758824070.117856 mapwrite 0x5e9000 thru 0x5ed0d1 (0x40d2 bytes) 000420[1] 1758824070.187479 mapwrite 0x35d000 thru 0x36af66 (0xdf67 bytes) 000421[0] 1758824070.272136 write 0x238000 thru 0x23caef (0x4af0 bytes) 000422[0] 1758824070.286938 mapwrite 0x480000 thru 0x48adb0 (0xadb1 bytes) 000423[0] 1758824070.342361 mapwrite 0x728000 thru 0x7283ab (0x3ac bytes) 000424[0] 1758824070.410440 trunc from 0x7283ac to 0x5ed136 (0x13b275 bytes) 000425[1] 1758824070.438089 read 0x3b1000 thru 0x3bc9e0 (0xb9e1 bytes) 000426[0] 1758824070.463208 mapread 0x3be000 thru 0x3cd065 (0xf066 bytes) 000431[1] 1758824070.502782 mapread 0x502000 thru 0x505822 (0x3823 bytes) 000433[0] 1758824070.520495 trunc from 0x5ed137 to 0x5e3cb6 (0x9480 bytes) 000434[0] 1758824070.545218 read 0x39a000 thru 0x3a5520 (0xb521 bytes) 000435[1] 1758824070.552695 read 0x430000 thru 0x436045 (0x6046 bytes) 000437[1] 1758824070.571559 write 0x8ec000 thru 0x8f80a5 (0xc0a6 bytes) 000443[1] 1758824070.604801 mapread 0x2e2000 thru 0x2e5055 (0x3056 bytes) 000444[1] 1758824070.632274 write 0x089000 thru 0x096a19 (0xda1a bytes) 000445[0] 1758824070.657212 mapread 0x54b000 thru 0x55ae46 (0xfe47 bytes) 000449[0] 1758824070.712119 trunc from 0x8f80a6 to 0x98e0df (0x9603a bytes) 000452[0] 1758824070.778260 read 0x41c000 thru 0x4219e8 (0x59e9 bytes) 000453[0] 1758824070.793121 write 0x0fa000 thru 0x10353e (0x953f bytes) 000454[1] 1758824070.799089 trunc from 0x98e0e0 to 0x568be1 (0x4254fe bytes) 000455[1] 1758824070.839446 mapwrite 0x6da000 thru 0x6db4f9 (0x14fa bytes) 000457[0] 1758824070.913085 trunc from 0x6db4fa to 0x60ab75 (0xd0984 bytes) 000462[0] 1758824070.956995 read 0x164000 thru 0x164f1d (0xf1e bytes) 000463[1] 1758824070.977755 write 0x544000 thru 0x54c1c0 (0x81c1 bytes) 000470[0] 1758824071.043204 trunc from 0x60ab76 to 0x0a7d64 (0x562e11 bytes) 000473[0] 1758824071.114459 read 0x025000 thru 0x034bdf (0xfbe0 bytes) 000475[0] 1758824071.151515 mapwrite 0x84e000 thru 0x84fd5b (0x1d5c bytes) 000476[1] 1758824071.283503 mapread 0x7ee000 thru 0x7f6d0e (0x8d0f bytes) 000482[0] 1758824071.318422 mapwrite 0x690000 thru 0x6911ee (0x11ef bytes) 000484[1] 1758824071.362552 write 0x355000 thru 0x35ac2d (0x5c2e bytes) 000485[0] 1758824071.373984 mapwrite 0x9f9000 thru 0x9fa945 (0x1946 bytes) 000486[0] 1758824071.459018 read 0x6f4000 thru 0x6f4def (0xdf0 bytes) 000487[1] 1758824071.468296 trunc from 0x9fa946 to 0x508563 (0x4f23e2 bytes) 000490[0] 1758824071.528295 trunc from 0x508564 to 0x38b5a9 (0x17cfba bytes) 000492[0] 1758824071.558450 mapread 0x0c6000 thru 0x0c843a (0x243b bytes) 000494[1] 1758824071.580947 write 0x810000 thru 0x81bfb7 (0xbfb8 bytes) 000496[0] 1758824071.613683 mapread 0x438000 thru 0x43f5d3 (0x75d4 bytes) 000497[0] 1758824071.663062 mapread 0x57a000 thru 0x588049 (0xe04a bytes) 000498[0] 1758824071.705993 mapread 0x814000 thru 0x815f4a (0x1f4b bytes) 000499[0] 1758824071.714486 mapwrite 0x25e000 thru 0x265255 (0x7256 bytes) All operations completed A-OK! PASS 16a (29s) == sanityn test 16b: 500 iterations of dual-mount fsx at small size ========================================================== 14:14:34 (1758824074) fallocate on zfs doesn't consume space fallocate not supported 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.189426 s, 44.3 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.147139 s, 57.0 MB/s Chance of close/open is 1 in 50 Seed set to 4075 fd 0: /mnt/lustre/f16b.sanityn fd 1: /mnt/lustre2/f16b.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000004[0] 1758824076.253733 mapwrite 0x799a1e thru 0x79b6e7 (0x1cca bytes) 000005[1] 1758824076.350028 mapread 0x35139a thru 0x351f6c (0xbd3 bytes) 000006[0] 1758824076.370397 write 0x055a24 thru 0x056159 (0x736 bytes) 000014[1] 1758824076.470975 mapread 0x530973 thru 0x53107c (0x70a bytes) 000015[0] 1758824076.496159 mapwrite 0x315494 thru 0x315dc3 (0x930 bytes) 000016[0] 1758824076.540842 read 0x465793 thru 0x4666b6 (0xf24 bytes) 000018[0] 1758824076.550714 mapread 0x4736ee thru 0x4741fe (0xb11 bytes) truncating to largest ever: 0x8a5e8d 000019[0] 1758824076.572331 trunc from 0x79b6e8 to 0x8a5e8c (0x10a7a5 bytes) 000022[1] 1758824076.635389 trunc from 0x8a5e8d to 0x1c2940 (0x6e354c bytes) 000024[1] 1758824076.713713 mapread 0x087c27 thru 0x08947d (0x1857 bytes) truncating to largest ever: 0x932c60 000025[0] 1758824076.739576 trunc from 0x1c2941 to 0x932c5f (0x77031f bytes) 000028[0] 1758824076.774431 trunc from 0x932c60 to 0x629445 (0x30981a bytes) 000030[1] 1758824076.787728 mapread 0x5ef3c7 thru 0x5f0bdb (0x1815 bytes) 000031[1] 1758824076.824950 read 0x1f4998 thru 0x1f5fef (0x1658 bytes) 000034[0] 1758824076.859689 mapread 0x291341 thru 0x2930c7 (0x1d87 bytes) 000036[0] 1758824076.881388 mapwrite 0x3a05ea thru 0x3a13ef (0xe06 bytes) 000039[1] 1758824076.973832 write 0x403298 thru 0x403c00 (0x969 bytes) 000041[0] 1758824077.028033 write 0x33458b thru 0x335b84 (0x15fa bytes) 000043[1] 1758824077.065124 mapread 0x3aa453 thru 0x3aaac3 (0x671 bytes) 000047[0] 1758824077.121082 read 0x1eb149 thru 0x1ec72f (0x15e7 bytes) 000049[0] 1758824077.137543 mapread 0x55565d thru 0x556442 (0xde6 bytes) 000050[1] 1758824077.165586 trunc from 0x629446 to 0x7dfda0 (0x1b695b bytes) 000051[1] 1758824077.200154 read 0x316fed thru 0x3187a9 (0x17bd bytes) 000052[1] 1758824077.227720 write 0x9bd8b8 thru 0x9bf497 (0x1be0 bytes) 000055[1] 1758824077.264246 mapwrite 0x1d6975 thru 0x1d8167 (0x17f3 bytes) 000057[0] 1758824077.341111 mapwrite 0x54b84b thru 0x54c5a8 (0xd5e bytes) 000060[0] 1758824077.452539 read 0x28b303 thru 0x28ca0d (0x170b bytes) 000061[1] 1758824077.469951 trunc from 0x9bf498 to 0x4c22f2 (0x4fd1a5 bytes) 000062[0] 1758824077.531373 trunc from 0x4c22f3 to 0x34473f (0x17dbb3 bytes) 000063[0] 1758824077.576955 mapread 0x2617cb thru 0x2633e0 (0x1c16 bytes) 000064[1] 1758824077.598688 mapwrite 0x630af9 thru 0x631761 (0xc69 bytes) 000065[0] 1758824077.707836 write 0x7079ae thru 0x7094bf (0x1b12 bytes) 000066[0] 1758824077.743806 mapwrite 0x63d707 thru 0x63e072 (0x96c bytes) 000068[1] 1758824077.811344 trunc from 0x7094c0 to 0x2fe462 (0x40b05d bytes) 000069[1] 1758824077.864943 mapwrite 0x25cdda thru 0x25e8cb (0x1af2 bytes) 000070[0] 1758824077.934881 mapread 0x2350f3 thru 0x2361e3 (0x10f1 bytes) 000071[0] 1758824077.981162 mapwrite 0x9f7dcf thru 0x9f9491 (0x16c3 bytes) 000072[0] 1758824078.088931 trunc from 0x9f9492 to 0x2fe1a8 (0x6fb2e9 bytes) 000073[0] 1758824078.106808 trunc from 0x2fe1a9 to 0x84ad29 (0x54cb81 bytes) 000080[1] 1758824078.173756 write 0x944528 thru 0x9453a8 (0xe81 bytes) 000081[1] 1758824078.196487 mapread 0x052e0b thru 0x053823 (0xa19 bytes) 000082[1] 1758824078.241921 write 0x673129 thru 0x6740cf (0xfa7 bytes) 000083[1] 1758824078.258416 read 0x783964 thru 0x785599 (0x1c36 bytes) 000084[1] 1758824078.262971 read 0x1c93a6 thru 0x1c9ede (0xb39 bytes) 000086[1] 1758824078.288629 mapwrite 0x216cbd thru 0x218c58 (0x1f9c bytes) 000088[0] 1758824078.366233 mapread 0x301e0e thru 0x3029d8 (0xbcb bytes) 000091[1] 1758824078.448383 mapread 0x8ebb21 thru 0x8ecc29 (0x1109 bytes) 000092[0] 1758824078.470699 trunc from 0x9453a9 to 0x675be0 (0x2cf7c8 bytes) 000093[0] 1758824078.508275 trunc from 0x675be1 to 0x1af02c (0x4c6bb4 bytes) 000094[1] 1758824078.535002 mapread 0x07f1bd thru 0x0805e0 (0x1424 bytes) 000095[0] 1758824078.578215 read 0x09e14f thru 0x09e4c7 (0x379 bytes) 000096[1] 1758824078.631943 mapread 0x04a1bd thru 0x04bf15 (0x1d59 bytes) 000097[1] 1758824078.648907 trunc from 0x1af02d to 0x8537bd (0x6a4791 bytes) 000099[0] 1758824078.711290 mapread 0x41fc53 thru 0x421677 (0x1a25 bytes) 000100[0] 1758824078.751289 mapread 0x2690ee thru 0x2696d6 (0x5e9 bytes) 000101[1] 1758824078.796230 write 0x7f57a5 thru 0x7f5a45 (0x2a1 bytes) 000104[1] 1758824078.817179 trunc from 0x8537be to 0x9095ca (0xb5e0d bytes) 000108[0] 1758824078.893250 mapwrite 0x67daec thru 0x67e68e (0xba3 bytes) 000109[0] 1758824078.989785 read 0x25815e thru 0x259428 (0x12cb bytes) 000111[1] 1758824079.023479 trunc from 0x9095cb to 0x13d548 (0x7cc082 bytes) 000113[0] 1758824079.054199 mapwrite 0x0d4739 thru 0x0d5b51 (0x1419 bytes) 000118[1] 1758824079.213644 read 0x11c87a thru 0x11dc9a (0x1421 bytes) 000121[0] 1758824079.271192 mapwrite 0x2d0e8a thru 0x2d258b (0x1702 bytes) 000122[1] 1758824079.372906 mapread 0x258fc9 thru 0x259ffe (0x1036 bytes) 000123[1] 1758824079.407475 trunc from 0x2d258c to 0x58e7a2 (0x2bc217 bytes) 000124[0] 1758824079.446673 mapwrite 0x18baf9 thru 0x18bbf9 (0x101 bytes) 000125[1] 1758824079.508700 mapread 0x25eb34 thru 0x25f462 (0x92f bytes) 000126[1] 1758824079.590730 write 0x894ef9 thru 0x8953bb (0x4c3 bytes) 000129[1] 1758824079.616417 mapwrite 0x6ace73 thru 0x6adffc (0x118a bytes) 000130[0] 1758824079.697195 mapread 0x6cf097 thru 0x6d08cd (0x1837 bytes) 000131[1] 1758824079.767955 trunc from 0x8953bc to 0x5715b6 (0x323e05 bytes) 000132[1] 1758824079.798209 mapwrite 0x282106 thru 0x2832f7 (0x11f2 bytes) 000133[1] 1758824079.859859 trunc from 0x5715b7 to 0x17a022 (0x3f7594 bytes) 000136[1] 1758824079.917046 read 0x08d773 thru 0x08efae (0x183c bytes) 000137[0] 1758824079.944581 trunc from 0x17a023 to 0x879642 (0x6ff620 bytes) 000139[1] 1758824080.017591 mapread 0x6553d7 thru 0x6560f7 (0xd21 bytes) 000140[0] 1758824080.050344 write 0x16dc6e thru 0x16fbc0 (0x1f53 bytes) 000148[0] 1758824080.184977 mapwrite 0x3a2ea8 thru 0x3a494c (0x1aa5 bytes) 000150[1] 1758824080.238141 write 0x403aa3 thru 0x404492 (0x9f0 bytes) 000153[1] 1758824080.295067 trunc from 0x879643 to 0x757d28 (0x12191a bytes) 000154[1] 1758824080.315885 write 0x34b4fb thru 0x34cb69 (0x166f bytes) 000158[1] 1758824080.355886 mapwrite 0x2c1195 thru 0x2c2106 (0xf72 bytes) truncating to largest ever: 0x9c38e3 000159[0] 1758824080.404726 trunc from 0x757d29 to 0x9c38e2 (0x26bbba bytes) 000163[1] 1758824080.483041 read 0x06e886 thru 0x06eae2 (0x25d bytes) 000164[1] 1758824080.500003 mapwrite 0x759ef9 thru 0x75a403 (0x50b bytes) 000167[1] 1758824080.587378 mapwrite 0x13a0fd thru 0x13a5e9 (0x4ed bytes) 000168[1] 1758824080.667648 trunc from 0x9c38e3 to 0x2e93a9 (0x6da539 bytes) 000169[1] 1758824080.696001 mapread 0x16323f thru 0x164f5f (0x1d21 bytes) 000171[0] 1758824080.710959 mapwrite 0x271c1b thru 0x272ba1 (0xf87 bytes) 000172[0] 1758824080.801244 trunc from 0x2e93aa to 0x1d4469 (0x114f40 bytes) 000175[1] 1758824080.851058 mapwrite 0x6d412b thru 0x6d576d (0x1643 bytes) 000176[1] 1758824080.944404 mapwrite 0x9bbb99 thru 0x9bc1e7 (0x64f bytes) 000179[0] 1758824081.064906 write 0x2dabf3 thru 0x2db546 (0x954 bytes) 000180[1] 1758824081.104898 mapread 0x0f4f49 thru 0x0f5fa7 (0x105f bytes) 000181[1] 1758824081.144610 write 0x2df77d thru 0x2e0845 (0x10c9 bytes) 000185[1] 1758824081.172505 trunc from 0x9bc1e8 to 0x1079f4 (0x8b47f3 bytes) 000186[0] 1758824081.210225 read 0x06291b thru 0x063d3f (0x1425 bytes) 000187[0] 1758824081.256835 write 0x304308 thru 0x3058f4 (0x15ed bytes) 000188[1] 1758824081.286587 trunc from 0x3058f5 to 0x2dc1eb (0x29709 bytes) 000190[0] 1758824081.365614 write 0x7bbd7a thru 0x7bd5c3 (0x184a bytes) 000192[1] 1758824081.411991 trunc from 0x7bd5c4 to 0x4165c7 (0x3a6ffc bytes) 000194[0] 1758824081.472991 trunc from 0x4165c8 to 0x226843 (0x1efd84 bytes) 000195[0] 1758824081.528926 trunc from 0x226844 to 0x001264 (0x2255df bytes) 000196[0] 1758824081.573050 mapread 0x00007b thru 0x000a86 (0xa0c bytes) 000198[1] 1758824081.591515 write 0x088d97 thru 0x089bd4 (0xe3e bytes) 000200[0] 1758824081.635965 read 0x02b604 thru 0x02d1cc (0x1bc9 bytes) 000202[1] 1758824081.679122 write 0x7b153f thru 0x7b16f9 (0x1bb bytes) 000203[1] 1758824081.695609 read 0x235dd0 thru 0x235e4a (0x7b bytes) 000205[1] 1758824081.701145 mapwrite 0x62ebb6 thru 0x62fce3 (0x112e bytes) 000208[1] 1758824081.789373 trunc from 0x7b16fa to 0x96c6f5 (0x1baffc bytes) 000210[1] 1758824081.817190 trunc from 0x96c6f6 to 0x097c8d (0x8d4a68 bytes) 000211[0] 1758824081.855198 mapwrite 0x678c5a thru 0x679c02 (0xfa9 bytes) 000217[0] 1758824082.037226 write 0x5e541e thru 0x5e5ec1 (0xaa4 bytes) 000218[0] 1758824082.058803 read 0x086222 thru 0x087f41 (0x1d20 bytes) 000219[1] 1758824082.064573 trunc from 0x679c03 to 0x87ec88 (0x205086 bytes) 000220[1] 1758824082.133127 trunc from 0x87ec89 to 0x4b96e0 (0x3c55a8 bytes) 000221[1] 1758824082.153704 mapread 0x03756e thru 0x037abf (0x552 bytes) 000222[0] 1758824082.207698 write 0x056994 thru 0x057b43 (0x11b0 bytes) 000224[0] 1758824082.263277 read 0x31eb02 thru 0x31ed7c (0x27b bytes) 000225[1] 1758824082.284844 trunc from 0x4b96e1 to 0x66a451 (0x1b0d71 bytes) 000227[0] 1758824082.348829 read 0x1c4a63 thru 0x1c5597 (0xb35 bytes) 000228[1] 1758824082.376028 read 0x187d2b thru 0x1894bf (0x1795 bytes) 000230[1] 1758824082.409192 write 0x9b6731 thru 0x9b7fd9 (0x18a9 bytes) 000236[0] 1758824082.527726 mapread 0x6ebc98 thru 0x6ec853 (0xbbc bytes) 000240[1] 1758824082.579554 write 0x5ec13d thru 0x5ec561 (0x425 bytes) 000241[0] 1758824082.604677 read 0x6da6ab thru 0x6dc11c (0x1a72 bytes) 000242[1] 1758824082.659782 mapread 0x2bbdda thru 0x2bdb2e (0x1d55 bytes) 000243[0] 1758824082.706712 write 0x82bc8d thru 0x82d80d (0x1b81 bytes) 000244[1] 1758824082.748172 mapwrite 0x774de7 thru 0x775d72 (0xf8c bytes) 000249[1] 1758824082.971238 mapread 0x12e685 thru 0x12f3e1 (0xd5d bytes) 000250[0] 1758824083.030079 read 0x4319a0 thru 0x4334e0 (0x1b41 bytes) 000251[0] 1758824083.044794 mapread 0x4a4454 thru 0x4a4d16 (0x8c3 bytes) 000252[0] 1758824083.049791 mapread 0x54b6cf thru 0x54b716 (0x48 bytes) 000254[0] 1758824083.053855 write 0x240a11 thru 0x2429b1 (0x1fa1 bytes) 000256[0] 1758824083.085705 read 0x3faa52 thru 0x3fb8b7 (0xe66 bytes) 000259[1] 1758824083.113093 mapwrite 0x1f4456 thru 0x1f5a17 (0x15c2 bytes) 000260[0] 1758824083.189338 mapread 0x9091a3 thru 0x90a51d (0x137b bytes) 000262[1] 1758824083.224695 mapwrite 0x58d40f thru 0x58f11d (0x1d0f bytes) 000263[1] 1758824083.300044 read 0x2340df thru 0x235ced (0x1c0f bytes) 000264[0] 1758824083.322711 read 0x020daf thru 0x021285 (0x4d7 bytes) 000265[0] 1758824083.352886 trunc from 0x9b7fda to 0x0f355a (0x8c4a7f bytes) 000266[0] 1758824083.441745 mapread 0x0d2ba7 thru 0x0d2ca3 (0xfd bytes) 000267[0] 1758824083.477042 write 0x00db77 thru 0x00ecde (0x1168 bytes) 000269[1] 1758824083.496570 trunc from 0x0f355b to 0x98d60c (0x89a0b2 bytes) 000270[1] 1758824083.582116 mapwrite 0x8d59a8 thru 0x8d6567 (0xbc0 bytes) 000273[0] 1758824083.671121 mapwrite 0x4f5e16 thru 0x4f611f (0x30a bytes) 000276[0] 1758824083.768740 mapwrite 0x1d92cd thru 0x1da8e9 (0x161d bytes) 000277[0] 1758824083.828356 mapread 0x52e651 thru 0x52e859 (0x209 bytes) 000281[1] 1758824083.860588 read 0x415b70 thru 0x417547 (0x19d8 bytes) 000282[0] 1758824083.893793 mapread 0x729001 thru 0x72a3dd (0x13dd bytes) 000284[0] 1758824083.959488 read 0x5a12ad thru 0x5a1e2a (0xb7e bytes) 000285[1] 1758824083.981912 mapread 0x240cab thru 0x2428b9 (0x1c0f bytes) 000292[0] 1758824084.037548 trunc from 0x98d60d to 0x0fac78 (0x892994 bytes) 000295[1] 1758824084.139395 trunc from 0x0fac79 to 0x4ddf79 (0x3e3301 bytes) 000296[1] 1758824084.196086 trunc from 0x4ddf7a to 0x535f67 (0x57fee bytes) 000297[0] 1758824084.213194 trunc from 0x535f68 to 0x5938dd (0x5d976 bytes) 000298[0] 1758824084.265751 mapread 0x3655a1 thru 0x367249 (0x1ca9 bytes) 000299[1] 1758824084.311424 mapwrite 0x7cfe5e thru 0x7d0fbb (0x115e bytes) 000301[1] 1758824084.429789 read 0x5ece0f thru 0x5edc12 (0xe04 bytes) 000302[0] 1758824084.455827 write 0x2d9fad thru 0x2db5cf (0x1623 bytes) 000305[0] 1758824084.536158 mapwrite 0x96869a thru 0x969d8f (0x16f6 bytes) 000306[1] 1758824084.614089 mapread 0x911d0b thru 0x912686 (0x97c bytes) 000307[1] 1758824084.663467 write 0x090a48 thru 0x0917f3 (0xdac bytes) 000310[1] 1758824084.740355 mapwrite 0x790d54 thru 0x7919d8 (0xc85 bytes) 000311[1] 1758824084.836788 trunc from 0x969d90 to 0x08b8cd (0x8de4c2 bytes) 000312[0] 1758824084.855167 write 0x09a6cd thru 0x09b16e (0xaa2 bytes) 000313[0] 1758824084.877471 mapwrite 0x46c114 thru 0x46c3ce (0x2bb bytes) 000317[1] 1758824084.984757 mapread 0x3cafbf thru 0x3cb965 (0x9a7 bytes) 000319[1] 1758824085.038890 trunc from 0x46c3cf to 0x448b83 (0x2384b bytes) 000321[1] 1758824085.120446 read 0x2098cf thru 0x20aefe (0x1630 bytes) 000323[0] 1758824085.161106 write 0x8f18c3 thru 0x8f2ddb (0x1519 bytes) 000324[1] 1758824085.192650 mapread 0x319fc6 thru 0x31a865 (0x8a0 bytes) 000325[1] 1758824085.239425 write 0x37905e thru 0x37a7e8 (0x178b bytes) 000327[1] 1758824085.297822 trunc from 0x8f2ddc to 0x0087ba (0x8ea621 bytes) 000330[0] 1758824085.332637 mapread 0x005920 thru 0x00744d (0x1b2e bytes) 000331[1] 1758824085.387146 mapwrite 0x7f9499 thru 0x7fb357 (0x1ebf bytes) 000332[0] 1758824085.528489 mapread 0x252768 thru 0x252b47 (0x3e0 bytes) 000335[0] 1758824085.606243 trunc from 0x7fb358 to 0x2d8a5e (0x5228f9 bytes) 000336[1] 1758824085.655220 write 0x91a36a thru 0x91afae (0xc45 bytes) 000337[0] 1758824085.698183 mapread 0x20c944 thru 0x20e617 (0x1cd4 bytes) 000339[1] 1758824085.752993 trunc from 0x91afaf to 0x0126af (0x9088ff bytes) 000341[1] 1758824085.792867 mapread 0x00e865 thru 0x00fada (0x1276 bytes) 000342[1] 1758824085.804169 mapread 0x00ea13 thru 0x00ece7 (0x2d5 bytes) 000347[0] 1758824085.838562 write 0x96a09e thru 0x96b7f6 (0x1759 bytes) 000349[0] 1758824085.890782 write 0x9ac8a0 thru 0x9ae555 (0x1cb6 bytes) 000350[1] 1758824085.910168 mapread 0x90773a thru 0x907a2c (0x2f3 bytes) 000355[0] 1758824085.989944 write 0x3ed006 thru 0x3ee9f9 (0x19f4 bytes) 000358[1] 1758824086.057631 read 0x927785 thru 0x928bb6 (0x1432 bytes) 000359[1] 1758824086.080964 mapread 0x31a25f thru 0x31b034 (0xdd6 bytes) 000360[0] 1758824086.098864 write 0x04c823 thru 0x04d820 (0xffe bytes) 000361[1] 1758824086.130286 read 0x45faa2 thru 0x460fc1 (0x1520 bytes) 000362[1] 1758824086.167409 mapread 0x6b7248 thru 0x6b805a (0xe13 bytes) 000364[0] 1758824086.231189 mapwrite 0x1af35f thru 0x1b08d8 (0x157a bytes) 000365[1] 1758824086.288218 read 0x2cc429 thru 0x2cdf57 (0x1b2f bytes) 000366[0] 1758824086.304867 read 0x57692b thru 0x577388 (0xa5e bytes) 000367[0] 1758824086.321961 read 0x0a5580 thru 0x0a7302 (0x1d83 bytes) 000368[1] 1758824086.326855 mapwrite 0x368a36 thru 0x368eda (0x4a5 bytes) 000369[1] 1758824086.378677 trunc from 0x9ae556 to 0x61441d (0x39a138 bytes) 000372[0] 1758824086.440222 read 0x4570fc thru 0x45750f (0x414 bytes) 000373[0] 1758824086.458110 read 0x04c05f thru 0x04c9a9 (0x94b bytes) 000376[0] 1758824086.481355 mapwrite 0x710fff thru 0x712df7 (0x1df9 bytes) 000377[0] 1758824086.586543 read 0x02147e thru 0x022a74 (0x15f7 bytes) 000379[0] 1758824086.615149 trunc from 0x712df8 to 0x240e9b (0x4d1f5c bytes) 000380[1] 1758824086.634611 trunc from 0x240e9c to 0x44923c (0x2083a1 bytes) 000381[0] 1758824086.677159 trunc from 0x44923d to 0x632c7d (0x1e9a41 bytes) 000386[0] 1758824086.736573 mapread 0x382461 thru 0x3834e3 (0x1083 bytes) 000387[1] 1758824086.750248 trunc from 0x632c7e to 0x422855 (0x210428 bytes) 000388[1] 1758824086.785721 write 0x253aa2 thru 0x254618 (0xb77 bytes) 000390[0] 1758824086.864265 read 0x1b336e thru 0x1b4d8a (0x1a1d bytes) 000392[1] 1758824086.896569 mapread 0x0ef62f thru 0x0efc10 (0x5e2 bytes) 000393[0] 1758824086.905293 read 0x154e32 thru 0x15520f (0x3de bytes) 000394[0] 1758824086.914265 trunc from 0x422856 to 0x708624 (0x2e5dcf bytes) 000397[1] 1758824086.968428 read 0x096a22 thru 0x09724a (0x829 bytes) 000401[1] 1758824087.022759 mapread 0x26c076 thru 0x26cf9f (0xf2a bytes) 000403[1] 1758824087.037016 read 0x312614 thru 0x313d89 (0x1776 bytes) 000404[1] 1758824087.058084 read 0x36a68b thru 0x36b075 (0x9eb bytes) 000405[1] 1758824087.062920 trunc from 0x708625 to 0x148e17 (0x5bf80d bytes) 000407[0] 1758824087.139187 mapwrite 0x0f1517 thru 0x0f1f51 (0xa3b bytes) 000411[0] 1758824087.237675 read 0x0788e9 thru 0x079a5a (0x1172 bytes) 000412[0] 1758824087.274663 write 0x191cd3 thru 0x19347b (0x17a9 bytes) 000415[0] 1758824087.318433 mapread 0x1461c7 thru 0x146c02 (0xa3c bytes) 000420[1] 1758824087.397609 write 0x79a69c thru 0x79c3dd (0x1d42 bytes) 000421[0] 1758824087.412869 trunc from 0x79c3de to 0x5c1c65 (0x1da778 bytes) 000423[0] 1758824087.436545 read 0x04c4b3 thru 0x04cc72 (0x7c0 bytes) 000426[1] 1758824087.453274 mapwrite 0x35cfa7 thru 0x35eeaa (0x1f04 bytes) 000429[0] 1758824087.577734 read 0x01168a thru 0x012f9d (0x1914 bytes) 000432[1] 1758824087.607117 mapwrite 0x9cdefe thru 0x9ce23a (0x33d bytes) 000433[1] 1758824087.687813 mapread 0x26434f thru 0x264714 (0x3c6 bytes) 000434[1] 1758824087.704367 trunc from 0x9ce23b to 0x8b510a (0x119130 bytes) 000437[1] 1758824087.744578 mapread 0x1288cb thru 0x128c83 (0x3b9 bytes) 000439[0] 1758824087.753802 trunc from 0x8b510b to 0x3fa4ee (0x4bac1c bytes) 000440[1] 1758824087.809332 mapread 0x24ebe3 thru 0x250423 (0x1841 bytes) 000441[0] 1758824087.855705 mapread 0x141ad7 thru 0x1426c0 (0xbea bytes) 000442[0] 1758824087.872262 read 0x3b180c thru 0x3b2204 (0x9f9 bytes) 000445[0] 1758824087.950892 read 0x016565 thru 0x0175c8 (0x1064 bytes) 000447[0] 1758824087.972697 mapread 0x158bb1 thru 0x159f11 (0x1361 bytes) 000448[1] 1758824087.973577 write 0x37c952 thru 0x37d1a1 (0x850 bytes) 000450[1] 1758824088.017934 read 0x2151c8 thru 0x215963 (0x79c bytes) 000451[1] 1758824088.077947 mapwrite 0x1dae04 thru 0x1dcc09 (0x1e06 bytes) 000453[1] 1758824088.160233 trunc from 0x3fa4ef to 0x229ee1 (0x1d060d bytes) 000455[1] 1758824088.215709 read 0x07d6dd thru 0x07f268 (0x1b8c bytes) 000456[1] 1758824088.229769 read 0x01c216 thru 0x01df57 (0x1d42 bytes) 000457[0] 1758824088.233713 read 0x21b59d thru 0x21c12d (0xb91 bytes) 000458[1] 1758824088.285145 mapread 0x077c7a thru 0x07840e (0x795 bytes) 000460[1] 1758824088.308380 mapread 0x09b72a thru 0x09c63d (0xf14 bytes) 000461[1] 1758824088.309142 mapwrite 0x42f335 thru 0x4303fb (0x10c7 bytes) 000462[0] 1758824088.418631 trunc from 0x4303fc to 0x844f8a (0x414b8f bytes) 000463[0] 1758824088.443829 mapread 0x5a9250 thru 0x5a97a2 (0x553 bytes) 000464[0] 1758824088.473072 mapwrite 0x6f1d7d thru 0x6f39ba (0x1c3e bytes) 000466[0] 1758824088.543197 mapread 0x0ffd4b thru 0x1010fb (0x13b1 bytes) 000470[0] 1758824088.603589 trunc from 0x844f8b to 0x268665 (0x5dc925 bytes) 000471[0] 1758824088.624014 trunc from 0x268666 to 0x2b2bac (0x4a547 bytes) 000472[1] 1758824088.636370 write 0x0269f5 thru 0x0282b0 (0x18bc bytes) 000479[1] 1758824088.749490 read 0x179f4c thru 0x17b5bf (0x1674 bytes) 000480[1] 1758824088.801795 trunc from 0x2b2bad to 0x2f0a82 (0x3ded6 bytes) 000481[1] 1758824088.839445 read 0x1c1bc4 thru 0x1c1bd1 (0xe bytes) 000484[1] 1758824088.855424 read 0x040e88 thru 0x041943 (0xabc bytes) 000485[0] 1758824088.855957 mapread 0x1fa240 thru 0x1fb9e9 (0x17aa bytes) 000489[0] 1758824088.965792 trunc from 0x2f0a83 to 0x4c4a24 (0x1d3fa2 bytes) 000492[1] 1758824089.048687 read 0x3b2948 thru 0x3b3942 (0xffb bytes) 000493[0] 1758824089.062228 write 0x2f9091 thru 0x2f9810 (0x780 bytes) 000496[1] 1758824089.083340 mapread 0x3117dc thru 0x313327 (0x1b4c bytes) 000498[1] 1758824089.117578 trunc from 0x4c4a25 to 0x4e6987 (0x21f63 bytes) All operations completed A-OK! PASS 16b (17s) == sanityn test 16c: verify data consistency on ldiskfs with cache disabled (b=17397) ========================================================== 14:14:51 (1758824091) SKIP: sanityn test_16c dio on ldiskfs only SKIP 16c (2s) == sanityn test 16d: Verify DIO and buffer IO with two clients ========================================================== 14:14:53 (1758824093) 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.195935 s, 42.8 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.168167 s, 49.9 MB/s ldlm.namespaces.MGC192.168.202.120@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8c2889525800.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8c2898bad800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8c2889525800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8c2898bad800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8c2889525800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8c2898bad800.lru_size=clear 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.47113 s, 19.2 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 1.59403 s, 65.8 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 1.24093 s, 84.5 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 2.96536 s, 35.4 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.45602 s, 23.5 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 1.74419 s, 60.1 MB/s PASS 16d (24s) == sanityn test 16e: Verify size consistency for O_DIRECT write ========================================================== 14:15:17 (1758824117) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.122691 s, 85.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0699506 s, 15.0 MB/s PASS 16e (3s) == sanityn test 16f: rw sequential consistency vs drop_caches ========================================================== 14:15:20 (1758824120) TIMEOUT PASS 16f (24s) == sanityn test 16g: mmap rw sequential consistency vs drop_caches ========================================================== 14:15:44 (1758824144) TIMEOUT PASS 16g (23s) == sanityn test 16h: mmap read after truncate file ======= 14:16:07 (1758824167) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16h (3s) == sanityn test 16i: read after truncate file ============ 14:16:10 (1758824170) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16i (2s) == sanityn test 16j: race dio with buffered i/o ========== 14:16:13 (1758824173) 1+0 records in 1+0 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.469332 s, 55.9 MB/s bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.484646 s, 54.1 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.892643 s, 29.4 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.583467 s, 44.9 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.769094 s, 34.1 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.780755 s, 33.6 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.08628 s, 24.1 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.848832 s, 30.9 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.890574 s, 29.4 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.17389 s, 22.3 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.20192 s, 21.8 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.962668 s, 27.2 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.04215 s, 25.2 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.25203 s, 20.9 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.32781 s, 19.7 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.19295 s, 22.0 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.40188 s, 18.7 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.53668 s, 17.1 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.5845 s, 16.5 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK PASS 16j (14s) == sanityn test 16k: Parallel FSX and drop caches should not panic ========================================================== 14:16:28 (1758824188) + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.1 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.2 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.3 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.4 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.5 -l 67108864 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read skipping zero size read truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x3c569db truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! + fsx 1 done: rc=0 Thu Sep 25 14:16:42 EDT 2025 + fsx 2 done: rc=0 Thu Sep 25 14:16:42 EDT 2025 + fsx 3 done: rc=0 Thu Sep 25 14:16:42 EDT 2025 All operations completed A-OK! + fsx 4 done: rc=0 Thu Sep 25 14:16:42 EDT 2025 + fsx 5 done: rc=0 Thu Sep 25 14:16:42 EDT 2025 PASS 16k (20s) == sanityn test 17: resource creation/LVB creation race ========================================================================= 14:16:47 (1758824207) fail_loc=0x8000030a PASS 17 (5s) == sanityn test 18: mmap sanity check =========================================================================================== 14:16:53 (1758824213) mmap test1: basic mmap operation (PASS, 0.055438s) mmap test2: MAP_PRIVATE not write back (PASS, 0.03592s) mmap test3: concurrent mmap ops on two nodes (PASS, 15.439s) mmap test4: c1 write to f1 from mmapped f2, c2 write to f1 from mmapped f1 (PASS, 4.2549s) mmap test5: read/write file to/from the buffer which mmapped to just this file (PASS, 0.095212s) mmap test6: check mmap write/read content on two nodes (PASS, 0.10634s) mmap test7: file i/o with an unmapped buffer (SKIPPED, 0s) mmap test8: SIGBUS for beyond file size (SKIPPED, 0s) mmap test9: SIGBUS for negative file offset (SKIPPED, 0s) mmap test10: multi-thread mmap access (PASS, 0.52369s) mmap test11: multi-thread shared mmap (PASS, 0.53764s) mmap test10: mtime not change for readonly mmap access (PASS, 1.0701s) PASS 18 (26s) == sanityn test 19: test concurrent uncached read races ========================================================================= 14:17:19 (1758824239) oleg220-server: error: get_param: param_path 'osd-*/lustre-OST*/read_cache_enable': No such file or directory pdsh@oleg220-client: oleg220-server: ssh exited with exit code 2 SKIP: sanityn test_19 not cache-capable obdfilter SKIP 19 (2s) == sanityn test 20: test extra readahead page left in cache ============================================================== 14:17:21 (1758824241) PASS 20 (3s) == sanityn test 21: Try to remove mountpoint on another dir ============================================================== 14:17:24 (1758824244) rmdir: removing directory, '/mnt/lustre/d21.sanityn' rmdir: failed to remove '/mnt/lustre/d21.sanityn': Device or resource busy rmdir: removing directory, '/mnt/lustre2/d21.sanityn' Removed mounted directory from another mountpoint, needs to be fixed PASS 21 (3s) == sanityn test 23: others should see updated atime while another read============================================================== 14:17:27 (1758824247) now is 1758824248 starting reads multiop /mnt/lustre/f23.sanityn vor20_c TMPPIPE=/tmp/multiop_open_wait_pipe.7559 new atime is 1758824309 PASS 23 (65s) == sanityn test 24a: lfs df [-ih] [path] test =================================================================================== 14:18:32 (1758824312) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 6144 3763200 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 6144 3763200 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre2 UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 490.6K 387 490.2K 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 115.3K 402 114.9K 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 114.5K 399 114.1K 1% /mnt/lustre[OST:1] filesystem_summary: 229.4K 387 229.0K 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 490.6K 387 490.2K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.3K 402 114.9K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 399 114.1K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.4K 387 229.0K 1% /mnt/lustre2 UUID bytes Used Available Use% Mounted on lustre-MDT0000_UUID 2.1G 3.2M 2.1G 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3.6G 6.0M 3.6G 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3.6G 32.0M 3.6G 1% /mnt/lustre[OST:1] filesystem_summary: 7.2G 38.0M 7.2G 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 502378 387 501991 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118066 402 117664 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117231 399 116832 1% /mnt/lustre2[OST:1] filesystem_summary: 234883 387 234496 1% /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 6144 3763200 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 490.6K 387 490.2K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.3K 402 114.9K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 399 114.1K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.4K 387 229.0K 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 502378 387 501991 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 118066 402 117664 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 117231 399 116832 1% /mnt/lustre[OST:1] filesystem_summary: 234883 387 234496 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 502378 387 501991 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118066 402 117664 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117231 399 116832 1% /mnt/lustre2[OST:1] filesystem_summary: 234883 387 234496 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 6144 3763200 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 6144 3763200 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre2 PASS 24a (2s) == sanityn test 24b: lfs df should show both filesystems ========================================================================= 14:18:34 (1758824314) PASS 24b (3s) == sanityn test 25a: change ACL on one mountpoint be seen on another ============================================================= 14:18:37 (1758824317) running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] PASS 25a (3s) == sanityn test 25b: change ACL under remote dir on one mountpoint be seen on another ========================================================== 14:18:40 (1758824320) SKIP: sanityn test_25b needs >= 2 MDTs SKIP 25b (1s) == sanityn test 26a: allow mtime to get older ============ 14:18:41 (1758824321) utime: good mknod times 1758824321 <= 1758824322 <= 1758824322 for /mnt/lustre/f26a utime: good mknod times 1758824321 <= 1758824322 <= 1758824322 for /mnt/lustre2/f26a utime: good utime mtimes 100000, atime 200000 utime: updated times for /mnt/lustre2/f26a PASS 26a (4s) == sanityn test 26b: sync mtime between ost and mds ====== 14:18:45 (1758824325) PASS 26b (4s) == sanityn test 26c: set-in-past on open file is not lost on close ========================================================== 14:18:49 (1758824329) multiop /mnt/lustre/f26c.sanityn vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7559 File: /mnt/lustre/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502398 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2025-09-25 14:18:50.000000000 -0400 Birth: 2025-09-25 14:18:50.000000000 -0400 File: /mnt/lustre2/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502398 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2025-09-25 14:18:50.000000000 -0400 Birth: 2025-09-25 14:18:50.000000000 -0400 PASS 26c (4s) SKIP: sanityn test_27 skipping excluded test 27 == sanityn test 30: recreate file race =================== 14:18:54 (1758824334) 0 PASS 30 (4s) == sanityn test 31a: voluntary cancel / blocking ast race======================================================================== 14:18:59 (1758824339) fail_loc=0x314 PASS 31a (4s) == sanityn test 31b: voluntary OST cancel / blocking ast race======================================================================== 14:19:02 (1758824342) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg220-server mds-ost sync done. sleep 5 for ZFS MDS Waiting for MDT destroys to complete fail_loc=0x314 fail_loc=0x316 fail_loc=0 fail_loc=0 Filesystem 1K-blocks Used Available Use% Mounted on 192.168.202.120@tcp:/lustre 7542784 39936 7496704 1% /mnt/lustre2 PASS 31b (13s) == sanityn test 31r: open-rename(replace) race =========== 14:19:15 (1758824355) -rw-r--r-- 1 root root 0 Sep 25 14:19 /mnt/lustre/f31r.sanityn.target fail_loc=0x1419 fail_val=3 mv: failed to access '/mnt/lustre2/f31r.sanityn.target': Input/output error PASS 31r (5s) == sanityn test 31s: open should not revalidate invalid dentry ========================================================== 14:19:20 (1758824360) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 1207: 42833 Terminated tail -f $tfile PASS 31s (4s) == sanityn test 31t: getattr should not revalidate invalid dentry ========================================================== 14:19:24 (1758824364) PASS 31t (3s) SKIP: sanityn test_33a skipping SLOW test 33a == sanityn test 33b: COS: cross create/delete, 2 clients, benchmark under remote dir ========================================================== 14:19:28 (1758824368) SKIP: sanityn test_33b Need two or more clients, have 1 SKIP 33b (2s) == sanityn test 33c: Cancel cross-MDT lock should trigger Sync-on-Lock-Cancel ========================================================== 14:19:30 (1758824370) SKIP: sanityn test_33c needs >= 2 MDTs SKIP 33c (1s) == sanityn test 33d: dependent transactions should trigger COS ========================================================== 14:19:31 (1758824371) SKIP: sanityn test_33d needs >= 2 MDTs SKIP 33d (1s) == sanityn test 33e: independent transactions shouldn't trigger COS ========================================================== 14:19:32 (1758824372) SKIP: sanityn test_33e needs >= 2 MDTs SKIP 33e (2s) == sanityn test 34: no lock timeout under IO ============= 14:19:34 (1758824374) fail_loc=0x512 fail_loc=0x512 lock should not expire writing on client1 reading on client2 success fail_loc=0x511 fail_loc=0x511 lock should expire writing on client1 reading on client2 success oleg220-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8c2889525800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8c2889525800.ost_server_uuid in FULL state after 0 sec oleg220-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8c2889525800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8c2889525800.ost_server_uuid in IDLE state after 0 sec oleg220-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8c2889525800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8c2889525800.ost_server_uuid in FULL state after 0 sec oleg220-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8c2889525800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8c2889525800.ost_server_uuid in IDLE state after 0 sec oleg220-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8c2889525800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8c2889525800.ost_server_uuid in IDLE state after 0 sec oleg220-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8c2889525800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8c2889525800.ost_server_uuid in IDLE state after 0 sec PASS 34 (95s) == sanityn test 35: -EINTR cp_ast vs. bl_ast race does not evict client ========================================================== 14:21:09 (1758824469) fail_loc=0x80000317 Race attempt 0 Wait for 48954 49084 for 60 sec... ls: reading directory '/mnt/lustre/d35.sanityn': Interrupted system call - open/close 2478 (time 1758824481.95 total 10.00 last 247.73) total: 4000 open/close in 17.56 seconds: 227.85 ops/second fail_loc=0x0 Filesystem Size Used Avail Use% Mounted on 192.168.202.120@tcp:/lustre 7.2G 40M 7.2G 1% /mnt/lustre 192.168.202.120@tcp:/lustre 7.2G 40M 7.2G 1% /mnt/lustre2 PASS 35 (66s) == sanityn test 36: handle ESTALE/open-unlink correctly == 14:22:15 (1758824535) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg220-server mds-ost sync done. Waiting for MDT destroys to complete 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.590585 s, 88.8 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7559 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(0) *** before(7497728) after_dd(7446528) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.484341 s, 108 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7559 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(1) *** before(7497728) after_dd(7444480) after(7495680) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.535716 s, 97.9 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7559 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(2) *** before(7495680) after_dd(7446528) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.51318 s, 102 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7559 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(3) *** before(7497728) after_dd(7446528) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.497493 s, 105 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7559 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(4) *** before(7497728) after_dd(7446528) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.499538 s, 105 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7559 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(5) *** before(7497728) after_dd(7446528) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.498727 s, 105 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7559 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(6) *** before(7497728) after_dd(7446528) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.481227 s, 109 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7559 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(7) *** before(7497728) after_dd(7446528) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.451463 s, 116 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7559 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(8) *** before(7497728) after_dd(7446528) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.450246 s, 116 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7559 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(9) *** before(7497728) after_dd(7446528) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.423369 s, 124 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7559 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(10) *** before(7497728) after_dd(7446528) after(7497728) PASS 36 (235s) == sanityn test 37: check i_size is not updated for directory on close (bug 18695) ======================================================================== 14:26:10 (1758824770) multiop /mnt/lustre/d37.sanityn vD_c TMPPIPE=/tmp/multiop_open_wait_pipe.7559 - create 8547 (time 1758824781.61 total 10.00 last 854.63) total: 10000 create in 11.68 seconds: 856.37 ops/second PASS 37 (15s) == sanityn test 39a: file mtime does not change after rename ========================================================== 14:26:25 (1758824785) repeat after cancel_lru_locks PASS 39a (3s) == sanityn test 39b: file mtime the same on clients with/out lock ========================================================== 14:26:28 (1758824788) repeat after cancel_lru_locks PASS 39b (3s) == sanityn test 39c: check truncate mtime update ================================================================================ 14:26:31 (1758824791) repeat after cancel_lru_locks PASS 39c (3s) == sanityn test 39d: sync write should update mtime ====== 14:26:34 (1758824794) fail_loc=0x411 fail_loc=0 PASS 39d (3s) SKIP: sanityn test_40a skipping ALWAYS excluded test 40a == sanityn test 40b: pdirops: open|create and others ======================================================================== 14:26:37 (1758824797) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40b (8s) == sanityn test 40c: pdirops: link and others ======================================================================== 14:26:45 (1758824805) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40c (9s) == sanityn test 40d: pdirops: unlink and others ======================================================================== 14:26:54 (1758824814) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40d (8s) == sanityn test 40e: pdirops: rename and others ======================================================================== 14:27:02 (1758824822) No conflict No conflict No conflict No conflict No conflict No conflict PASS 40e (8s) == sanityn test 41a: pdirops: create vs mkdir ======================================================================== 14:27:10 (1758824830) mkdir: cannot create directory '/mnt/lustre2/f41a.sanityn': File exists Conflict PASS 41a (5s) == sanityn test 41b: pdirops: create vs create ======================================================================== 14:27:15 (1758824835) open: File exists Conflict PASS 41b (5s) == sanityn test 41c: pdirops: create vs link ======================================================================== 14:27:21 (1758824841) link: cannot create link '/mnt/lustre2/f41c.sanityn' to '/mnt/lustre2/f41c.sanityn-2': File exists Conflict PASS 41c (6s) == sanityn test 41d: pdirops: create vs unlink ======================================================================== 14:27:26 (1758824846) Conflict PASS 41d (5s) == sanityn test 41e: pdirops: create and rename (tgt) ======================================================================== 14:27:31 (1758824851) Conflict PASS 41e (5s) == sanityn test 41f: pdirops: create and rename (src) ======================================================================== 14:27:36 (1758824856) Conflict PASS 41f (6s) == sanityn test 41g: pdirops: create vs getattr ======================================================================== 14:27:42 (1758824862) Conflict PASS 41g (5s) == sanityn test 41h: pdirops: create vs readdir ======================================================================== 14:27:47 (1758824867) Conflict PASS 41h (6s) == sanityn test 41i: reint_open: create vs create ======== 14:27:53 (1758824873) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 41i (727s) == sanityn test 42a: pdirops: mkdir vs mkdir ======================================================================== 14:40:00 (1758825600) mkdir: cannot create directory '/mnt/lustre2/f42a.sanityn': File exists Conflict PASS 42a (5s) == sanityn test 42b: pdirops: mkdir vs create ======================================================================== 14:40:05 (1758825605) open: File exists Conflict PASS 42b (5s) == sanityn test 42c: pdirops: mkdir vs link ======================================================================== 14:40:10 (1758825610) link: cannot create link '/mnt/lustre2/f42c.sanityn' to '/mnt/lustre2/f42c.sanityn-2': File exists Conflict PASS 42c (6s) == sanityn test 42d: pdirops: mkdir vs unlink ======================================================================== 14:40:16 (1758825616) Conflict PASS 42d (5s) == sanityn test 42e: pdirops: mkdir and rename (tgt) ======================================================================== 14:40:21 (1758825621) mv: cannot overwrite directory '/mnt/lustre2/f42e.sanityn' with non-directory Conflict PASS 42e (5s) == sanityn test 42f: pdirops: mkdir and rename (src) ======================================================================== 14:40:26 (1758825626) Conflict PASS 42f (6s) == sanityn test 42g: pdirops: mkdir vs getattr ======================================================================== 14:40:32 (1758825632) Conflict PASS 42g (5s) == sanityn test 42h: pdirops: mkdir vs readdir ======================================================================== 14:40:37 (1758825637) Conflict PASS 42h (6s) == sanityn test 43a: rmdir,mkdir doesn't return -EEXIST ======================================================================== 14:40:43 (1758825643) PASS 43a (18s) == sanityn test 43b: pdirops: unlink vs create ======================================================================== 14:41:01 (1758825661) Conflict PASS 43b (6s) == sanityn test 43c: pdirops: unlink vs link ======================================================================== 14:41:07 (1758825667) Conflict PASS 43c (5s) == sanityn test 43d: pdirops: unlink vs unlink ======================================================================== 14:41:12 (1758825672) rm: cannot remove '/mnt/lustre2/f43d.sanityn': No such file or directory Conflict PASS 43d (5s) == sanityn test 43e: pdirops: unlink and rename (tgt) ======================================================================== 14:41:17 (1758825677) Conflict PASS 43e (6s) == sanityn test 43f: pdirops: unlink and rename (src) ======================================================================== 14:41:23 (1758825683) mv: cannot stat '/mnt/lustre2/f43f.sanityn': No such file or directory Conflict PASS 43f (5s) == sanityn test 43g: pdirops: unlink vs getattr ======================================================================== 14:41:28 (1758825688) stat: cannot statx '/mnt/lustre2/f43g.sanityn': No such file or directory Conflict PASS 43g (5s) == sanityn test 43h: pdirops: unlink vs readdir ======================================================================== 14:41:33 (1758825693) Conflict PASS 43h (5s) == sanityn test 43i: pdirops: unlink vs remote mkdir ===== 14:41:38 (1758825698) SKIP: sanityn test_43i needs >= 2 MDTs SKIP 43i (1s) == sanityn test 43j: racy mkdir return EEXIST ======================================================================== 14:41:39 (1758825699) mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists PASS 43j (45s) == sanityn test 43k: unlink vs create ==================== 14:42:24 (1758825744) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 43k (471s) == sanityn test 44a: pdirops: rename tgt vs mkdir ======================================================================== 14:50:15 (1758826215) mkdir: cannot create directory '/mnt/lustre2/f44a.sanityn': File exists Conflict Thu Sep 25 14:50:19 EDT 2025 PASS 44a (5s) == sanityn test 44b: pdirops: rename tgt vs create ======================================================================== 14:50:20 (1758826220) open: File exists Conflict PASS 44b (5s) == sanityn test 44c: pdirops: rename tgt vs link ======================================================================== 14:50:25 (1758826225) link: cannot create link '/mnt/lustre2/f44c.sanityn' to '/mnt/lustre2/f44c.sanityn-3': File exists Conflict PASS 44c (6s) == sanityn test 44d: pdirops: rename tgt vs unlink ======================================================================== 14:50:31 (1758826231) Conflict PASS 44d (5s) == sanityn test 44e: pdirops: rename tgt and rename (tgt) ======================================================================== 14:50:36 (1758826236) Conflict PASS 44e (5s) == sanityn test 44f: pdirops: rename tgt and rename (src) ======================================================================== 14:50:41 (1758826241) Conflict PASS 44f (6s) == sanityn test 44g: pdirops: rename tgt vs getattr ======================================================================== 14:50:47 (1758826247) Conflict PASS 44g (5s) == sanityn test 44h: pdirops: rename tgt vs readdir ======================================================================== 14:50:52 (1758826252) Conflict PASS 44h (6s) == sanityn test 44i: pdirops: rename tgt vs remote mkdir ========================================================== 14:50:58 (1758826258) SKIP: sanityn test_44i needs >= 2 MDTs SKIP 44i (1s) == sanityn test 45a: rename,mkdir doesn't return -EEXIST ======================================================================== 14:50:59 (1758826259) PASS 45a (30s) == sanityn test 45b: pdirops: rename src vs create ======================================================================== 14:51:29 (1758826289) Conflict PASS 45b (5s) == sanityn test 45c: pdirops: rename src vs link ======================================================================== 14:51:34 (1758826294) Conflict PASS 45c (5s) == sanityn test 45d: pdirops: rename src vs unlink ======================================================================== 14:51:40 (1758826300) rm: cannot remove '/mnt/lustre2/f45d.sanityn': No such file or directory Conflict PASS 45d (5s) == sanityn test 45e: pdirops: rename src and rename (tgt) ======================================================================== 14:51:45 (1758826305) Conflict PASS 45e (5s) == sanityn test 45f: pdirops: rename src and rename (src) ======================================================================== 14:51:50 (1758826310) mv: cannot stat '/mnt/lustre2/f45f.sanityn': No such file or directory Conflict PASS 45f (6s) == sanityn test 45g: pdirops: rename src vs getattr ======================================================================== 14:51:56 (1758826316) stat: cannot statx '/mnt/lustre2/f45g.sanityn': No such file or directory Conflict PASS 45g (5s) == sanityn test 45h: pdirops: unlink vs readdir ======================================================================== 14:52:01 (1758826321) Conflict PASS 45h (5s) == sanityn test 45i: pdirops: rename src vs remote mkdir ========================================================== 14:52:06 (1758826326) SKIP: sanityn test_45i needs >= 2 MDTs SKIP 45i (1s) == sanityn test 45j: read vs rename ====================== 14:52:07 (1758826327) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 45j (466s) == sanityn test 46a: pdirops: link vs mkdir ======================================================================== 14:59:53 (1758826793) mkdir: cannot create directory '/mnt/lustre2/f46a.sanityn': File exists Conflict PASS 46a (5s) == sanityn test 46b: pdirops: link vs create ======================================================================== 14:59:58 (1758826798) open: File exists Conflict PASS 46b (6s) == sanityn test 46c: pdirops: link vs link ======================================================================== 15:00:04 (1758826804) link: cannot create link '/mnt/lustre2/f46c.sanityn' to '/mnt/lustre2/f46c.sanityn': File exists Conflict PASS 46c (6s) == sanityn test 46d: pdirops: link vs unlink ======================================================================== 15:00:10 (1758826810) Conflict PASS 46d (5s) == sanityn test 46e: pdirops: link and rename (tgt) ======================================================================== 15:00:15 (1758826815) Conflict PASS 46e (6s) == sanityn test 46f: pdirops: link and rename (src) ======================================================================== 15:00:21 (1758826821) Conflict PASS 46f (5s) == sanityn test 46g: pdirops: link vs getattr ======================================================================== 15:00:26 (1758826826) Conflict PASS 46g (5s) == sanityn test 46h: pdirops: link vs readdir ======================================================================== 15:00:32 (1758826832) Conflict PASS 46h (6s) == sanityn test 46i: pdirops: link vs remote mkdir ======= 15:00:37 (1758826837) SKIP: sanityn test_46i needs >= 2 MDTs SKIP 46i (1s) == sanityn test 47a: pdirops: remote mkdir vs mkdir ====== 15:00:38 (1758826838) SKIP: sanityn test_47a needs >= 2 MDTs SKIP 47a (1s) == sanityn test 47b: pdirops: remote mkdir vs create ===== 15:00:39 (1758826839) SKIP: sanityn test_47b needs >= 2 MDTs SKIP 47b (1s) == sanityn test 47c: pdirops: remote mkdir vs link ======= 15:00:40 (1758826840) SKIP: sanityn test_47c needs >= 2 MDTs SKIP 47c (1s) == sanityn test 47d: pdirops: remote mkdir vs unlink ===== 15:00:41 (1758826841) SKIP: sanityn test_47d needs >= 2 MDTs SKIP 47d (1s) == sanityn test 47e: pdirops: remote mkdir and rename (tgt) ========================================================== 15:00:42 (1758826842) SKIP: sanityn test_47e needs >= 2 MDTs SKIP 47e (2s) == sanityn test 47f: pdirops: remote mkdir and rename (src) ========================================================== 15:00:44 (1758826844) SKIP: sanityn test_47f needs >= 2 MDTs SKIP 47f (1s) == sanityn test 47g: pdirops: remote mkdir vs getattr ==== 15:00:45 (1758826845) SKIP: sanityn test_47g needs >= 2 MDTs SKIP 47g (1s) == sanityn test 50: osc lvb attrs: enqueue vs. CP AST ======================================================================== 15:00:46 (1758826846) 10+0 records in 10+0 records out 10240 bytes (10 kB, 10 KiB) copied, 0.0052147 s, 2.0 MB/s fail_loc=0x410 fail_loc=0x0 PASS 50 (7s) == sanityn test 51a: layout lock: refresh layout should work ========================================================== 15:00:53 (1758826853) 0+1 records in 0+1 records out 158 bytes copied, 0.00424696 s, 37.2 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 3510: kill: (260122) - No such process PASS 51a (4s) == sanityn test 51b: layout lock: glimpse should be able to restart if layout changed ========================================================== 15:00:57 (1758826857) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00331447 s, 309 kB/s fail_loc=0x1404 fail_val=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162377 s, 64.6 MB/s 3145728 PASS 51b (15s) == sanityn test 51c: layout lock: IT_LAYOUT blocked and correct layout can be returned ========================================================== 15:01:12 (1758826872) fail_loc=0x172 Setting layout to have 2 stripes ... 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00394765 s, 259 kB/s PASS 51c (6s) == sanityn test 51d: layout lock: losing layout lock should clean up memory map region ========================================================== 15:01:18 (1758826878) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145238 s, 72.2 MB/s Before revoking layout lock: 1024 KB mapped PASS 51d (3s) == sanityn test 51e: lfs getstripe does not break leases, part 2 ========================================================== 15:01:21 (1758826881) /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 2451 0x993 0x280000400 write lease(2) has applied. /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 2451 0x993 0x280000400 read lease(1) has applied. PASS 51e (4s) == sanityn test 54: rename locking ======================= 15:01:25 (1758826885) llite.lustre-ffff8c2889525800.xattr_cache=1 llite.lustre-ffff8c2898bad800.xattr_cache=1 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 512 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505011 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-09-25 15:01:26.000000000 -0400 Modify: 2025-09-25 15:01:26.000000000 -0400 Change: 2025-09-25 15:01:26.000000000 -0400 Birth: 2025-09-25 15:01:26.000000000 -0400 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000154 File: /mnt/lustre/d1/d2 Size: 11776 Blocks: 23 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505014 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-09-25 15:01:31.000000000 -0400 Modify: 2025-09-25 15:01:32.000000000 -0400 Change: 2025-09-25 15:01:32.000000000 -0400 Birth: 2025-09-25 15:01:31.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 512 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505017 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-09-25 15:01:37.000000000 -0400 Modify: 2025-09-25 15:01:37.000000000 -0400 Change: 2025-09-25 15:01:37.000000000 -0400 Birth: 2025-09-25 15:01:37.000000000 -0400 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1 Size: 11776 Blocks: 23 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505016 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-09-25 15:01:37.000000000 -0400 Modify: 2025-09-25 15:01:37.000000000 -0400 Change: 2025-09-25 15:01:37.000000000 -0400 Birth: 2025-09-25 15:01:37.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000154 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1/d2 Size: 11776 Blocks: 23 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505020 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-09-25 15:01:42.000000000 -0400 Modify: 2025-09-25 15:01:43.000000000 -0400 Change: 2025-09-25 15:01:43.000000000 -0400 Birth: 2025-09-25 15:01:42.000000000 -0400 File: /mnt/lustre/d1 Size: 11776 Blocks: 23 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505019 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-09-25 15:01:42.000000000 -0400 Modify: 2025-09-25 15:01:43.000000000 -0400 Change: 2025-09-25 15:01:43.000000000 -0400 Birth: 2025-09-25 15:01:42.000000000 -0400 PASS 54 (25s) == sanityn test 55a: rename vs unlink target dir ========= 15:01:50 (1758826910) fail_loc=0x80000156 mv: cannot move '/mnt/lustre/d55a.sanityn/d1/d2' to '/mnt/lustre/d55a.sanityn/d3/d2': No such file or directory PASS 55a (7s) == sanityn test 55b: rename vs unlink source dir ========= 15:01:57 (1758826917) fail_loc=0x80000156 mv: cannot stat '/mnt/lustre/d55b.sanityn/d1/d2': No such file or directory PASS 55b (8s) == sanityn test 55c: rename vs unlink orphan target dir == 15:02:05 (1758826925) fail_loc=0x156 mv: cannot move '/mnt/lustre/d55c.sanityn/d1/d2' to '/mnt/lustre/d55c.sanityn/d3/d2': No such file or directory PASS 55c (13s) == sanityn test 55d: rename file vs link ================= 15:02:18 (1758826938) fail_loc=0x155 mv: '/mnt/lustre/d55d.sanityn/f1' and '/mnt/lustre/d55d.sanityn/d55d.sanityn/f1' are the same file PASS 55d (10s) == sanityn test 55e: rename race AB/BA under the same parent dir ========================================================== 15:02:28 (1758826948) SKIP: sanityn test_55e needs >= 2 MDTs SKIP 55e (1s) == sanityn test 55f: rename: (P1/A -> P2/B) race with (P2/B -> P1/A) ========================================================== 15:02:29 (1758826949) fail_loc=0x155 We survived after P1A_P2B/P2B_P1A race test fail_loc=0x0 PASS 55f (13s) == sanityn test 55g: rename: race with trylock =========== 15:02:42 (1758826962) mdt.lustre-MDT0000.enable_rename_trylock=1 fail_loc=0x155 mv: cannot move '/mnt/lustre2/d55g.sanityn/b1/b2' to a subdirectory of itself, '/mnt/lustre2/d55g.sanityn/a1/a2/a3/a4/b2' We survived race with trylock test fail_loc=0x0 PASS 55g (14s) == sanityn test 56a: test llverdev with single large stripe ========================================================== 15:02:56 (1758826976) run_llverdev /mnt/lustre/f56a.sanityn -p -s 16000000 -c 4k llverdev: /mnt/lustre/f56a.sanityn is 16777216000000 bytes (15625 GB) in size timestamp: 1758826977 chunksize: 4096 size: 16777216000000 write offset: 16384000000kB 2.54329 MB/s read offset: 16384000000kB inf MB/s llverdev: data verified successfully PASS 56a (27s) == sanityn test 56b: test llverdev and partial verify of wide stripe file ========================================================== 15:03:23 (1758827003) run_llverdev /mnt/lustre/f56b.sanityn -p -s 32000000 -c 4k llverdev: /mnt/lustre/f56b.sanityn is 33554432000000 bytes (31250 GB) in size timestamp: 1758827005 chunksize: 4096 size: 33554432000000 write offset: 32768000000kB 3.59042 MB/s read offset: 32768000000kB inf MB/s llverdev: data verified successfully PASS 56b (39s) == sanityn test 60: Verify data_version behaviour ======== 15:04:02 (1758827042) PASS 60 (3s) == sanityn test 70a: cd directory && rm directory ======== 15:04:05 (1758827045) PASS 70a (2s) == sanityn test 70b: remove files after calling rm_entry ========================================================== 15:04:08 (1758827048) PASS 70b (3s) == sanityn test 71a: correct file map just after write operation is finished ========================================================== 15:04:11 (1758827051) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.00260216 s, 25.2 MB/s SKIP: sanityn test_71a ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71a (1s) == sanityn test 71b: check fiemap support for stripecount > 1 ========================================================== 15:04:12 (1758827052) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.00382147 s, 17.1 MB/s SKIP: sanityn test_71b ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71b (2s) == sanityn test 71c: check FIEMAP_EXTENT_LAST flag with different extents number ========================================================== 15:04:14 (1758827054) SKIP: sanityn test_71c support only ldiskfs ost SKIP 71c (1s) == sanityn test 71d: fiemap corruption test with fm_extent_count=0 ========================================================== 15:04:15 (1758827055) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.00449354 s, 14.6 MB/s SKIP: sanityn test_71d ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71d (1s) == sanityn test 72: getxattr/setxattr cache should be consistent between nodes ========================================================== 15:04:16 (1758827056) llite.lustre-ffff8c2889525800.xattr_cache=1 llite.lustre-ffff8c2898bad800.xattr_cache=1 getfattr: Removing leading '/' from absolute path names user.attr1="value1" getfattr: Removing leading '/' from absolute path names user.attr1="value2" getfattr: Removing leading '/' from absolute path names getfattr: Removing leading '/' from absolute path names b362740ad371eae580e7f85452e8830b - 870e072da1ebfc20ea8c0cd3d6bdf3b6 - PASS 72 (2s) == sanityn test 73: getxattr should not cause xattr lock cancellation ========================================================== 15:04:18 (1758827058) llite.lustre-ffff8c2889525800.xattr_cache=1 llite.lustre-ffff8c2898bad800.xattr_cache=1 getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" PASS 73 (3s) == sanityn test 74: flock deadlock: different mounts ======================================================================== 15:04:21 (1758827061) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0045863 s, 223 kB/s 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00350869 s, 292 kB/s 272907: get lock2 272908: get lock1 272907: done 272907: get lock1 272908: done 272908: get lock2 fcntl cmd 7 failed: Resource deadlock avoided 272907: cannot get lock1: Resource deadlock avoided fcntl cmd 7 failed: Resource deadlock avoided 272908: cannot get lock2: Resource deadlock avoided 272908: exit rc=0 272907: done 272907: exit rc=0 PASS 74 (7s) == sanityn test 75: osc: upcall after unuse lock============================================================================= 15:04:28 (1758827068) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0209087 s, 100 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0117144 s, 89.5 MB/s fail_loc=0x31d /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 4193: 273488 Killed stat -c %s $DIR1/$tfile fail_loc=0 PASS 75 (7s) == sanityn test 76: Verify MDT open_files listing ======== 15:04:35 (1758827075) opening files: ........ closing files: ........ PASS 76 (22s) == sanityn test 77a: check FIFO NRS policy =============== 15:04:57 (1758827097) ost.OSS.ost.nrs_policies=fifo ost.OSS.ost_create.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_out.nrs_policies=fifo ost.OSS.ost_seq.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.124869 s, 134 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0479505 s, 21.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0448924 s, 23.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144203 s, 7.3 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8c2889525800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170402 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163237 s, 6.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187483 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193892 s, 5.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173557 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186208 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173261 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177984 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168183 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173338 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198221 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169374 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194838 s, 5.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0182577 s, 57.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0225844 s, 46.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0226093 s, 46.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0253365 s, 41.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0317861 s, 33.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0267181 s, 39.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.021964 s, 47.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.028878 s, 36.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0266941 s, 39.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0270554 s, 38.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0362695 s, 28.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.033353 s, 31.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0340691 s, 30.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0375428 s, 27.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0376353 s, 27.9 MB/sldlm.namespaces.lustre-OST0001-osc-ffff8c2889525800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0369993 s, 28.3 MB/s removed '/mnt/lustre/d77a.sanityn/nrs_r_oleg220-client.virtnet' removed '/mnt/lustre/d77a.sanityn/nrs_w_oleg220-client.virtnet' removed directory '/mnt/lustre/d77a.sanityn' PASS 77a (3s) == sanityn test 77b: check CRR-N NRS policy ============== 15:05:00 (1758827100) ost.OSS.ost.nrs_policies=crrn ost.OSS.ost_create.nrs_policies=crrn ost.OSS.ost_io.nrs_policies=crrn ost.OSS.ost_out.nrs_policies=crrn ost.OSS.ost_seq.nrs_policies=crrn ost.OSS.ost.nrs_crrn_quantum=1 ost.OSS.ost_create.nrs_crrn_quantum=1 ost.OSS.ost_io.nrs_crrn_quantum=1 ost.OSS.ost_out.nrs_crrn_quantum=1 ost.OSS.ost_seq.nrs_crrn_quantum=1 policy: crr-n, crrn_quantum 1 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.157904 s, 106 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0467077 s, 22.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0441645 s, 23.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0569384 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110459 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0965973 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109681 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109619 s, 9.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112784 s, 9.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123752 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121369 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13946 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121312 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132037 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168323 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11919 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139873 s, 7.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8c2889525800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff8c2889525800.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0105066 s, 99.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0286562 s, 36.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0276247 s, 38.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0258183 s, 40.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168114 s, 62.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00928904 s, 113 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135154 s, 77.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165002 s, 63.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0354851 s, 29.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.021996 s, 47.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0259248 s, 40.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0310264 s, 33.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0420661 s, 24.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.024836 s, 42.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154293 s, 68.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0305973 s, 34.3 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg220-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg220-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost.nrs_crrn_quantum=64 ost.OSS.ost_create.nrs_crrn_quantum=64 ost.OSS.ost_io.nrs_crrn_quantum=64 ost.OSS.ost_out.nrs_crrn_quantum=64 ost.OSS.ost_seq.nrs_crrn_quantum=64 policy: crr-n, crrn_quantum 64 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.123279 s, 136 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0470048 s, 22.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0929071 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119899 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110154 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150315 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122712 s, 8.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16369 s, 6.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135171 s, 7.8 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148812 s, 7.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155645 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153461 s, 6.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15468 s, 6.8 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14731 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161049 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152781 s, 6.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8c2889525800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff8c2889525800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153592 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.032469 s, 32.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0317619 s, 33.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0491621 s, 21.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0333729 s, 31.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0347414 s, 30.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0358206 s, 29.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0365349 s, 28.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0357712 s, 29.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0375923 s, 27.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0381037 s, 27.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0379875 s, 27.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0437567 s, 24.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0468301 s, 22.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.048601 s, 21.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0451154 s, 23.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110754 s, 9.5 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg220-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg220-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77b (5s) == sanityn test 77c: check ORR NRS policy ================ 15:05:05 (1758827105) ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_orr_quantum=1 ost.OSS.ost_io.nrs_orr_offset_type=physical ost.OSS.ost_io.nrs_orr_supported=reads policy: orr, orr_quantum 1, orr_offset_type physical, orr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.267234 s, 62.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0336402 s, 31.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0504057 s, 20.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122654 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158726 s, 6.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15543 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164989 s, 6.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150573 s, 7.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137825 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145044 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158878 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167747 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147227 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158489 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160493 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166227 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170069 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0182123 s, 57.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0192867 s, 54.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132302 s, 79.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0187608 s, 55.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00538935 s, 195 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0214809 s, 48.8 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0300529 s, 34.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00576033 s, 182 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0176779 s, 59.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.028998 s, 36.2 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8c2889525800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0123783 s, 84.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0180779 s, 58.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00699126 s, 150 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139218 s, 75.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161394 s, 65.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0113745 s, 92.2 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg220-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg220-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=writes ost.OSS.ost_io.nrs_orr_quantum=64 policy: orr, orr_quantum 64, orr_offset_type physical, orr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.155979 s, 108 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0568722 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0560319 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.086913 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104959 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129332 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134471 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129809 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111061 s, 9.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138913 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13742 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136089 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150249 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144502 s, 7.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136427 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114573 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165954 s, 6.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00992953 s, 106 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0322126 s, 32.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0400971 s, 26.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0215557 s, 48.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0249944 s, 42.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.026868 s, 39.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0258329 s, 40.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0256139 s, 40.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0330951 s, 31.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0291572 s, 36.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0332091 s, 31.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0305424 s, 34.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00630434 s, 166 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0455571 s, 23.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0453311 s, 23.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8c2889525800.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff8c2889525800.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0492244 s, 21.3 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg220-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg220-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=reads_and_writes ost.OSS.ost_io.nrs_orr_offset_type=logical policy: orr, orr_quantum 64, orr_offset_type logical, orr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.186668 s, 89.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.058773 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0722499 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0940402 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0940738 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.090979 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0858465 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0809987 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103633 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102074 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0791185 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0988691 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0900334 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0957488 s, 11.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0905227 s, 11.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104271 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119318 s, 8.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139205 s, 75.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0184414 s, 56.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0184791 s, 56.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0222244 s, 47.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0216648 s, 48.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00643216 s, 163 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0245975 s, 42.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0229056 s, 45.8 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8c2889525800.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0107453 s, 97.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0193879 s, 54.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00924946 s, 113 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158039 s, 66.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0263432 s, 39.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0299368 s, 35.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0113421 s, 92.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0110275 s, 95.1 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg220-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg220-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77c (6s) == sanityn test 77d: check TRR nrs policy ================ 15:05:11 (1758827111) ost.OSS.ost_io.nrs_policies=trr ost.OSS.ost_io.nrs_trr_quantum=1 ost.OSS.ost_io.nrs_trr_offset_type=physical ost.OSS.ost_io.nrs_trr_supported=reads policy: trr, trr_quantum 1, trr_offset_type physical, trr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.201979 s, 83.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.022095 s, 47.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0543591 s, 19.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0675765 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0486998 s, 21.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0447236 s, 23.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0654185 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0431791 s, 24.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0764145 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0577562 s, 18.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8c2889525800.lock_unused_count=3 ldlm.namespaces.lustre-OST0001-osc-ffff8c2889525800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108697 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107491 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102837 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126702 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118026 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132115 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11902 s, 8.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0112678 s, 93.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0192404 s, 54.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01468 s, 71.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0174864 s, 60.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156001 s, 67.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144651 s, 72.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0180701 s, 58.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00652218 s, 161 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00988555 s, 106 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0186847 s, 56.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0329747 s, 31.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00578617 s, 181 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147244 s, 71.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167617 s, 62.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0111727 s, 93.9 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8c2889525800.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00523962 s, 200 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg220-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg220-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=writes ost.OSS.ost_io.nrs_trr_quantum=64 policy: trr, trr_quantum 64, trr_offset_type physical, trr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.168829 s, 99.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183133 s, 57.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0481696 s, 21.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0432844 s, 24.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0600568 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.055579 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0668264 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.035717 s, 29.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0613195 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0663859 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0797428 s, 13.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8c2889525800.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff8c2889525800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0746757 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.096012 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105002 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114997 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0993992 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131871 s, 8.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00997203 s, 105 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173736 s, 60.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0231243 s, 45.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0234647 s, 44.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0269595 s, 38.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00564365 s, 186 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.004278 s, 245 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0219135 s, 47.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0212827 s, 49.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.030199 s, 34.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0229574 s, 45.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130355 s, 80.4 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0261305 s, 40.1 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8c2889525800.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0035993 s, 291 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0210605 s, 49.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0218887 s, 47.9 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg220-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg220-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=reads_and_writes ost.OSS.ost_io.nrs_trr_offset_type=logical policy: trr, trr_quantum 64, trr_offset_type logical, trr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.164669 s, 102 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0262456 s, 40.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0348475 s, 30.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.035561 s, 29.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0564038 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0980894 s, 10.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0954915 s, 11.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0945405 s, 11.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8c2889525800.lock_unused_count=3 ldlm.namespaces.lustre-OST0001-osc-ffff8c2889525800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.095106 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.098164 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11293 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134438 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12346 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105219 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119012 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119806 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116336 s, 9.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8c2889525800.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff8c2889525800.lock_unused_count=2 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127904 s, 82.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127631 s, 82.2 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0206572 s, 50.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0218729 s, 47.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0212855 s, 49.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154168 s, 68.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0273199 s, 38.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160137 s, 65.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.023726 s, 44.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0240465 s, 43.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0107929 s, 97.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00997903 s, 105 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00637647 s, 164 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00428851 s, 245 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00613239 s, 171 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0018081 s, 580 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg220-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg220-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77d (5s) == sanityn test 77e: check TBF NID nrs policy ============ 15:05:16 (1758827116) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_tbf_rule=start localhost nid={0@lo} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start clients nid={192.168.202.20@tcp} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start others nid={*.*.*.*@tcp} rate=50 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.165824 s, 101 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0300991 s, 34.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0355495 s, 29.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117431 s, 8.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132882 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127169 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149466 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1409 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1356 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14571 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146382 s, 7.2 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160839 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145749 s, 7.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128869 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139416 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140197 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101867 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0906798 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0970994 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104593 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0984584 s, 10.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11283 s, 9.3 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0976474 s, 10.7 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106164 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117546 s, 8.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105209 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104113 s, 10.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108302 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104762 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104986 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106027 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111143 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10805 s, 9.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8c2889525800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff8c2889525800.lock_unused_count=4 removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg220-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg220-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change localhost rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change clients rate=101 ost.OSS.ost_io.nrs_tbf_rule=change others rate=51 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.130772 s, 128 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0370486 s, 28.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10185 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121966 s, 8.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119831 s, 8.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143473 s, 7.3 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133302 s, 7.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140718 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147374 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129978 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12837 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145898 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133749 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140583 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159075 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141889 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13521 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0992182 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105817 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0929728 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0995155 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111494 s, 9.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111365 s, 9.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11891 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119126 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111611 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112685 s, 9.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10835 s, 9.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10953 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104293 s, 10.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10854 s, 9.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112209 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111223 s, 9.4 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg220-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg220-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop localhost ost.OSS.ost_io.nrs_tbf_rule=stop clients ost.OSS.ost_io.nrs_tbf_rule=stop others 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.157292 s, 107 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0405018 s, 25.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0444315 s, 23.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0460634 s, 22.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0339217 s, 30.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0885791 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0731708 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10031 s, 10.5 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8c2889525800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0988649 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0964966 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106662 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104329 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117169 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0984461 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112937 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106373 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106662 s, 9.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0186298 s, 56.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173767 s, 60.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0213875 s, 49.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0262589 s, 39.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0196337 s, 53.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0249494 s, 42.0 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157018 s, 66.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00739185 s, 142 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0292196 s, 35.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0220844 s, 47.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0203005 s, 51.7 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8c2889525800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0293218 s, 35.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127445 s, 82.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0252598 s, 41.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00788315 s, 133 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0188485 s, 55.6 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg220-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg220-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.126178 s, 133 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0307387 s, 34.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0568551 s, 18.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0865717 s, 12.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0817217 s, 12.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8c2889525800.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0918899 s, 11.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0751733 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0939721 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0976026 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105969 s, 9.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100965 s, 10.4 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.092547 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118304 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117603 s, 8.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108711 s, 9.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10447 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125875 s, 8.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8c2889525800.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff8c2889525800.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0213073 s, 49.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0190953 s, 54.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0279021 s, 37.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.03045 s, 34.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136648 s, 76.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0243896 s, 43.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0218322 s, 48.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0212924 s, 49.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0252902 s, 41.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0207541 s, 50.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0225114 s, 46.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0213709 s, 49.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0276763 s, 37.9 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00943377 s, 111 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0338153 s, 31.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00232439 s, 451 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg220-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg220-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' PASS 77e (9s) == sanityn test 77f: check TBF JobID nrs policy ========== 15:05:25 (1758827125) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start runas jobid={iozone.500 dd.500 tiotest.500} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start iozone_runas jobid={iozone.500} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg220-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.184259 s, 91.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0314064 s, 33.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.035111 s, 29.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0812707 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0949653 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106241 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0955666 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100945 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111602 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0969975 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140603 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146217 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149738 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148044 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147748 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137567 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13844 s, 7.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8c2889525800.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffff8c2889525800.lock_unused_count=1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158857 s, 66.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0232031 s, 45.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.019397 s, 54.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00660814 s, 159 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.023571 s, 44.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014255 s, 73.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0107729 s, 97.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00687493 s, 153 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00601005 s, 174 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015136 s, 69.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00614651 s, 171 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125218 s, 83.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0120674 s, 86.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00944303 s, 111 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00904061 s, 116 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155826 s, 67.3 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg220-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg220-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change runas rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change iozone_runas rate=101 ost.OSS.ost_io.nrs_tbf_rule=change dd_runas rate=51 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg220-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.138765 s, 121 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0573229 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0604032 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123386 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127062 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147221 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135798 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148568 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158932 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130747 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1473 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14287 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156464 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146884 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158038 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150103 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147791 s, 7.1 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8c2889525800.lock_unused_count=2 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161592 s, 64.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0290662 s, 36.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0197609 s, 53.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00781741 s, 134 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0221894 s, 47.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0186939 s, 56.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.023981 s, 43.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.010708 s, 97.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0341792 s, 30.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0245838 s, 42.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0366314 s, 28.6 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146958 s, 71.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0192626 s, 54.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014933 s, 70.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0213832 s, 49.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0204349 s, 51.3 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg220-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg220-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop runas ost.OSS.ost_io.nrs_tbf_rule=stop iozone_runas ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg220-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.1308 s, 128 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0395019 s, 26.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0292049 s, 35.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0935492 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0859 s, 12.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118126 s, 8.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115505 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0915279 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121978 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119889 s, 8.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] ldlm.namespaces.lustre-OST0000-osc-ffff8c2889525800.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffff8c2889525800.lock_unused_count=4 execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121459 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129722 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146209 s, 7.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131395 s, 8.0 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122384 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139308 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132077 s, 7.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0184387 s, 56.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141309 s, 74.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0218021 s, 48.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160826 s, 65.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0212296 s, 49.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0212598 s, 49.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0206864 s, 50.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0223081 s, 47.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0228238 s, 45.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0328267 s, 31.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0310499 s, 33.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0262237 s, 40.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0245802 s, 42.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0260444 s, 40.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0275911 s, 38.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0265796 s, 39.5 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg220-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg220-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_policies=fifo running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg220-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.129603 s, 129 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0351469 s, 29.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0541535 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0668774 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0892914 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0818726 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0969057 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0928094 s, 11.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8c2889525800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101118 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0668622 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120394 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112798 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100054 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111903 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123004 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110372 s, 9.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14201 s, 7.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169276 s, 61.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0215969 s, 48.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0274966 s, 38.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0096336 s, 109 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0120492 s, 87.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0262403 s, 40.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.022948 s, 45.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0121043 s, 86.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0224518 s, 46.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.021952 s, 47.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0280261 s, 37.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0194471 s, 53.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00481351 s, 218 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0185972 s, 56.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0238035 s, 44.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0207187 s, 50.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8c2889525800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff8c2889525800.lock_unused_count=1 removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg220-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg220-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' PASS 77f (7s) == sanityn test 77g: Change TBF type directly ============ 15:05:32 (1758827132) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 ost.OSS.ost_io.nrs_policies=fifo PASS 77g (4s) == sanityn test 77h: Wrong policy name should report error, not LBUG ========================================================== 15:05:36 (1758827136) oleg220-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=abc: No such file or directory oleg220-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='abc': No such file or directory pdsh@oleg220-client: oleg220-server: ssh exited with exit code 2 oleg220-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc: Invalid argument oleg220-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc': Invalid argument pdsh@oleg220-client: oleg220-server: ssh exited with exit code 22 oleg220-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf reg abc: Invalid argument oleg220-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf reg abc': Invalid argument pdsh@oleg220-client: oleg220-server: ssh exited with exit code 22 oleg220-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc efg: Invalid argument oleg220-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc efg': Invalid argument pdsh@oleg220-client: oleg220-server: ssh exited with exit code 22 PASS 77h (4s) == sanityn test 77i: Change rank of TBF rule ============= 15:05:40 (1758827140) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start before jobid={jobid} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start after jobid={jobid} rate=1000 rank=default ost.OSS.ost_io.nrs_tbf_rule=start target jobid={jobid} rate=1000 rank=after Move before itself ost.OSS.ost_io.nrs_tbf_rule=change target rank=target Move to higher rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=before Move to lower rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=after Move before default ost.OSS.ost_io.nrs_tbf_rule=change target rank=default ost.OSS.ost_io.nrs_policies=fifo PASS 77i (10s) == sanityn test 77j: check TBF-OPCode NRS policy ========= 15:05:50 (1758827150) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf opcode ost.OSS.ost_io.nrs_tbf_rule=start ost_r opcode={ost_read} rate=5 ost.OSS.ost_io.nrs_tbf_rule=start ost_w opcode={ost_write} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.185452 s, 90.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.091602 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139818 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144614 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144441 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149976 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148534 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133373 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150666 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145229 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158065 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138673 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146617 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176643 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155288 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143636 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155647 s, 6.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0175824 s, 59.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0120119 s, 87.3 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166499 s, 63.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0204026 s, 51.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0246209 s, 42.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0220178 s, 47.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0237075 s, 44.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0276665 s, 37.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170223 s, 61.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0250203 s, 41.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00409172 s, 256 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00402928 s, 260 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0265735 s, 39.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00844382 s, 124 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00230498 s, 455 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00243138 s, 431 MB/s removed '/mnt/lustre/d77j.sanityn/nrs_r_oleg220-client.virtnet' removed '/mnt/lustre/d77j.sanityn/nrs_w_oleg220-client.virtnet' removed directory '/mnt/lustre/d77j.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 5 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.95984 s, 21.1 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.4464 s, 5.4 MB/s Read runtime is 20 s, speed is 5.000000 IOPS cleanup_tbf /mnt/lustre/d77j.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_r ost.OSS.ost_io.nrs_tbf_rule=stop ost_w ost.OSS.ost_io.nrs_policies=fifo PASS 77j (54s) == sanityn test 77ja: check TBF-UID/GID NRS policy ======= 15:06:44 (1758827204) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg220-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg220-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg220-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg220-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg220-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.23147 s, 13.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0987224 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0760799 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0930895 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0924507 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0853963 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0820085 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0873405 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113225 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129522 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130264 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103209 s, 10.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11533 s, 9.1 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128805 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134283 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153545 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128897 s, 8.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0237623 s, 44.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0239488 s, 43.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0250266 s, 41.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0320436 s, 32.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0227413 s, 46.1 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0359367 s, 29.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0336419 s, 31.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012139 s, 86.4 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0305056 s, 34.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00808364 s, 130 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.030158 s, 34.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0342147 s, 30.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0273541 s, 38.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0198152 s, 52.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0274849 s, 38.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0171465 s, 61.2 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg220-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg220-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.6794 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9177 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg220-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg220-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg220-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg220-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg220-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.22531 s, 13.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0507164 s, 20.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0619618 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0817512 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0816484 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0938174 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0908165 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0817608 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0865246 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152801 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208279 s, 5.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209021 s, 5.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208462 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.226312 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21644 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197216 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.214302 s, 4.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0289257 s, 36.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0294118 s, 35.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.029652 s, 35.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0286465 s, 36.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0262169 s, 40.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0296643 s, 35.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0343827 s, 30.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0187185 s, 56.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0116492 s, 90.0 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0269077 s, 39.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0340079 s, 30.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0028757 s, 365 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.030331 s, 34.6 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0356581 s, 29.4 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0392325 s, 26.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0266583 s, 39.3 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg220-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg220-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8379 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.955 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77ja (133s) == sanityn test 77jb: check TBF-UID/GID NRS policy on files that don't belong to us ========================================================== 15:08:57 (1758827337) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg220-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg220-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg220-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg220-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg220-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.22445 s, 13.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0497743 s, 21.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0798999 s, 13.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0881886 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0846194 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0948783 s, 11.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0917499 s, 11.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0864347 s, 12.1 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0862353 s, 12.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156619 s, 6.7 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156615 s, 6.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155095 s, 6.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143297 s, 7.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135085 s, 7.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154268 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165117 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194214 s, 5.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0261836 s, 40.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0210372 s, 49.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0295843 s, 35.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0351667 s, 29.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0258317 s, 40.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0362168 s, 29.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0307511 s, 34.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00473023 s, 222 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0326607 s, 32.1 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0193796 s, 54.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0357124 s, 29.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0209198 s, 50.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136357 s, 76.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0195817 s, 53.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0194294 s, 54.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00620631 s, 169 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg220-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg220-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8365 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9591 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg220-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg220-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg220-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg220-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg220-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.22755 s, 13.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0858113 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0741497 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0774603 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0722849 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0783284 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0677499 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0798263 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0750641 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145843 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146083 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189729 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193122 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202601 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220733 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213506 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208973 s, 5.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0236098 s, 44.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0245907 s, 42.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0238759 s, 43.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0257485 s, 40.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.028215 s, 37.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0307848 s, 34.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016975 s, 61.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.029674 s, 35.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0269925 s, 38.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0263314 s, 39.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170976 s, 61.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.011271 s, 93.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0374827 s, 28.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00641825 s, 163 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160263 s, 65.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162601 s, 64.5 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg220-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg220-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.7185 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9271 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77jb (135s) == sanityn test 77k: check TBF policy with UID/GID/JobID/OPCode expression ========================================================== 15:11:12 (1758827472) ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg220-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.264757 s, 63.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.031321 s, 33.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0376992 s, 27.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0350243 s, 29.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101374 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110627 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141918 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131132 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143202 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145012 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126933 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128961 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156612 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12893 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137754 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145825 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151677 s, 6.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166792 s, 62.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014246 s, 73.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151846 s, 69.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151283 s, 69.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154469 s, 67.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126162 s, 83.1 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00483595 s, 217 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0110643 s, 94.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149628 s, 70.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00767884 s, 137 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0178499 s, 58.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00487224 s, 215 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125327 s, 83.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00579988 s, 181 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168952 s, 62.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00970161 s, 108 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg220-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg220-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.04699 s, 20.8 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73943 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.202.20@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.202.20@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.125425 s, 134 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0222056 s, 47.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0334956 s, 31.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0577072 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0775858 s, 13.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0782887 s, 13.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0947868 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0908722 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0949737 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109599 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109872 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0917155 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0933192 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18022 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168595 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.226785 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.219359 s, 4.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146748 s, 71.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0194957 s, 53.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0214998 s, 48.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0258313 s, 40.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0197379 s, 53.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0219321 s, 47.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0188327 s, 55.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.021506 s, 48.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0190256 s, 55.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00628492 s, 167 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0263875 s, 39.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00582561 s, 180 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00682903 s, 154 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00998396 s, 105 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0201181 s, 52.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0195857 s, 53.5 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg220-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg220-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.94 s, 21.2 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.74879 s, 10.8 MB/s Read runtime is 10 s, speed is 10.000000 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.202.20@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg220-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.240553 s, 69.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0452867 s, 23.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0367429 s, 28.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137003 s, 7.7 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116395 s, 9.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170399 s, 6.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143504 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161713 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150122 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167315 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148021 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164424 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17792 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183345 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171699 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16292 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17043 s, 6.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0234558 s, 44.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0379721 s, 27.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0308045 s, 34.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0211716 s, 49.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0172168 s, 60.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0372233 s, 28.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0254164 s, 41.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0393711 s, 26.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00903906 s, 116 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0240926 s, 43.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0177075 s, 59.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0187528 s, 55.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0211321 s, 49.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0370904 s, 28.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.041492 s, 25.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0301893 s, 34.7 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg220-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg220-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.99429 s, 21.0 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.90352 s, 21.4 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg220-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.190639 s, 88.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.031293 s, 33.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.054175 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0574035 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0866391 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.076715 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0711864 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121001 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0935126 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111477 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118185 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127652 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131115 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15392 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174724 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.384727 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.373984 s, 2.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0206852 s, 50.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159307 s, 65.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0260018 s, 40.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0174025 s, 60.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.022317 s, 47.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0242932 s, 43.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0181248 s, 57.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0252731 s, 41.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00488759 s, 215 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0318245 s, 32.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0318612 s, 32.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0071077 s, 148 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0073488 s, 143 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0058789 s, 178 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00391091 s, 268 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00541979 s, 193 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg220-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg220-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.81579 s, 10.7 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.7466 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.95787 s, 21.1 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.75192 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg220-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.896119 s, 18.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0336565 s, 31.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0628207 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0858436 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0966722 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0949653 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117657 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135666 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118073 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0954921 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189612 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173757 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188171 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180624 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.217536 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209105 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191714 s, 5.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0176154 s, 59.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0179856 s, 58.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0219835 s, 47.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149565 s, 70.1 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152581 s, 68.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173383 s, 60.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0214188 s, 49.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0194215 s, 54.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.010783 s, 97.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00374014 s, 280 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00844096 s, 124 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0105049 s, 99.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00897826 s, 117 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00393656 s, 266 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00781944 s, 134 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00203983 s, 514 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg220-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg220-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.5216 s, 5.4 MB/s Write runtime is 20 s, speed is 5.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.4424 s, 5.4 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_ug ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg220-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.20865 s, 80.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0367458 s, 28.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0830498 s, 12.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112703 s, 9.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12964 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123942 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130375 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1262 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148136 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113506 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137356 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14281 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131146 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160136 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122604 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147412 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144395 s, 7.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142488 s, 73.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141834 s, 73.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0217048 s, 48.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0175951 s, 59.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0065021 s, 161 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0184602 s, 56.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0306637 s, 34.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00716707 s, 146 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00907849 s, 116 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.010009 s, 105 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143955 s, 72.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0188353 s, 55.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0344923 s, 30.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00604763 s, 173 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0212355 s, 49.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159847 s, 65.6 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg220-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg220-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.9662 s, 21.1 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.74433 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg220-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.149217 s, 112 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0249253 s, 42.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0756239 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10082 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.070887 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0870475 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128148 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138041 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132092 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130591 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126977 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143971 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119398 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138863 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125658 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192502 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.204375 s, 5.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.018347 s, 57.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0260753 s, 40.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0301579 s, 34.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0203757 s, 51.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0291378 s, 36.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0230196 s, 45.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0315773 s, 33.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.029992 s, 35.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0290794 s, 36.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00525746 s, 199 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0207674 s, 50.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.024732 s, 42.4 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0108992 s, 96.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0190391 s, 55.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00466324 s, 225 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0122102 s, 85.9 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg220-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg220-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.80111 s, 10.7 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73977 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.77582 s, 10.7 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.74844 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_policies=fifo PASS 77k (354s) == sanityn test 77kb: Check different granular TBF type combination: jobid+opcode ========================================================== 15:17:06 (1758827826) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/nrs_r_oleg220-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.25578 s, 65.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0379875 s, 27.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0646384 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0593521 s, 17.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0997266 s, 10.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.093138 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0910631 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0767564 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113649 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145224 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138599 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142191 s, 7.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155604 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155007 s, 6.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165501 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167203 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146007 s, 7.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156706 s, 66.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155192 s, 67.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0110645 s, 94.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0171853 s, 61.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169525 s, 61.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00489444 s, 214 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183789 s, 57.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01205 s, 87.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0246419 s, 42.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0189295 s, 55.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0219134 s, 47.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0050699 s, 207 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00788221 s, 133 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00846619 s, 124 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00896272 s, 117 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00918756 s, 114 MB/s removed '/mnt/lustre/d77kb.sanityn/nrs_r_oleg220-client.virtnet' removed '/mnt/lustre/d77kb.sanityn/nrs_w_oleg220-client.virtnet' removed directory '/mnt/lustre/d77kb.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.11755 s, 20.5 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.74262 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kb (40s) == sanityn test 77kc: Check different granular TBF type combination: nid+opcode ========================================================== 15:17:46 (1758827866) ost.OSS.ost_io.nrs_policies=tbf nid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.202.20@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.202.20@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.260895 s, 64.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0269652 s, 38.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0606567 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0946283 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0745884 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0782362 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106865 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0868687 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.082648 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101718 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0908408 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147682 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136053 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242838 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.235336 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.295095 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.305027 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0177842 s, 59.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166039 s, 63.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0184068 s, 57.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0234218 s, 44.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0222526 s, 47.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162839 s, 64.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0197919 s, 53.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00413509 s, 254 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170364 s, 61.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0234895 s, 44.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0113836 s, 92.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0237282 s, 44.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0228748 s, 45.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0246989 s, 42.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129914 s, 80.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0269152 s, 39.0 MB/s removed '/mnt/lustre/d77kc.sanityn/nrs_w_oleg220-client.virtnet' removed '/mnt/lustre/d77kc.sanityn/nrs_r_oleg220-client.virtnet' removed directory '/mnt/lustre/d77kc.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.95241 s, 21.2 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.74954 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kc.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kc (39s) == sanityn test 77kd: Check different granular TBF type combination: nid+jobid ========================================================== 15:18:25 (1758827905) ost.OSS.ost_io.nrs_policies=tbf nid+jobid ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.202.20@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/nrs_r_oleg220-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.297749 s, 56.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0516758 s, 20.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0977438 s, 10.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0984963 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11675 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0892486 s, 11.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0915979 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103877 s, 10.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0920969 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127197 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110355 s, 9.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104693 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103943 s, 10.1 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113531 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102497 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102384 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114783 s, 9.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0122105 s, 85.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152386 s, 68.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015771 s, 66.5 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0171892 s, 61.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147437 s, 71.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165847 s, 63.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0185006 s, 56.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00398256 s, 263 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00809149 s, 130 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0233741 s, 44.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00420648 s, 249 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00428686 s, 245 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0244053 s, 43.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00602069 s, 174 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0116172 s, 90.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00191601 s, 547 MB/s removed '/mnt/lustre/d77kd.sanityn/nrs_r_oleg220-client.virtnet' removed '/mnt/lustre/d77kd.sanityn/nrs_w_oleg220-client.virtnet' removed directory '/mnt/lustre/d77kd.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.01547 s, 20.9 MB/s Write runtime is 7 s, speed is 14.285714 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kd.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.93277 s, 21.3 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kd.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kd (34s) == sanityn test 77ke: Check different granular TBF type combination: jobid+opcode ========================================================== 15:18:59 (1758827939) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/nrs_r_oleg220-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.14332 s, 117 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0426111 s, 24.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0661643 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0703548 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101005 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122997 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148313 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133894 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156519 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160354 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139329 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131758 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152341 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165662 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16545 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20308 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211183 s, 5.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0244064 s, 43.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0226892 s, 46.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0196822 s, 53.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0240011 s, 43.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0266621 s, 39.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00923617 s, 114 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0357826 s, 29.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0193333 s, 54.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00801921 s, 131 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0288145 s, 36.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0193023 s, 54.3 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151695 s, 69.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0196388 s, 53.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173151 s, 60.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162537 s, 64.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156751 s, 66.9 MB/s removed '/mnt/lustre/d77ke.sanityn/nrs_r_oleg220-client.virtnet' removed '/mnt/lustre/d77ke.sanityn/nrs_w_oleg220-client.virtnet' removed directory '/mnt/lustre/d77ke.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.85148 s, 10.6 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ke.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.74324 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.96645 s, 21.1 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.74313 s, 10.8 MB/s Read runtime is 10 s, speed is 10.000000 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77ke (81s) == sanityn test 77kf: Check different granular TBF type combination: uid+gid ========================================================== 15:20:20 (1758828020) ost.OSS.ost_io.nrs_policies=tbf uid+gid ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/nrs_r_oleg220-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.878451 s, 19.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0801476 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0454273 s, 23.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0528523 s, 19.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0677527 s, 15.5 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0595311 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0709965 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0809084 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0661237 s, 15.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108354 s, 9.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111505 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133601 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147393 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137329 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126913 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172251 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153378 s, 6.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.018969 s, 55.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0241713 s, 43.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0200227 s, 52.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0284199 s, 36.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0265922 s, 39.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140183 s, 74.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.02738 s, 38.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0216583 s, 48.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0259106 s, 40.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0208626 s, 50.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165071 s, 63.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138286 s, 75.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0301604 s, 34.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00453761 s, 231 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00666506 s, 157 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00685744 s, 153 MB/s removed '/mnt/lustre/d77kf.sanityn/nrs_r_oleg220-client.virtnet' removed '/mnt/lustre/d77kf.sanityn/nrs_w_oleg220-client.virtnet' removed directory '/mnt/lustre/d77kf.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.6769 s, 5.3 MB/s Write runtime is 20 s, speed is 5.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kf.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9352 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77kf.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kf (66s) == sanityn test 77kg: check different granular TBF type combination: uid+opcode ========================================================== 15:21:26 (1758828086) ost.OSS.ost_io.nrs_policies=tbf uid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg220-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.246953 s, 67.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0213488 s, 49.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0566155 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0676718 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0844627 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0895731 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0775379 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0918461 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0915356 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0979664 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0832307 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0803134 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0856363 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100839 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104493 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101709 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112054 s, 9.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141908 s, 73.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0172028 s, 61.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0234032 s, 44.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0287011 s, 36.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0248622 s, 42.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125451 s, 83.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.021662 s, 48.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0285498 s, 36.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166474 s, 63.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147614 s, 71.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0373602 s, 28.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00667888 s, 157 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0189253 s, 55.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140483 s, 74.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134384 s, 78.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0075294 s, 139 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg220-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg220-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.93336 s, 21.3 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.72951 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg220-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.1283 s, 131 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0294687 s, 35.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0419793 s, 25.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0401659 s, 26.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0521164 s, 20.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0628062 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0510099 s, 20.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0605483 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0655324 s, 16.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0996734 s, 10.5 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.099781 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172761 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160542 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184368 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171584 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.268993 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.275873 s, 3.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161679 s, 64.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153561 s, 68.3 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0184955 s, 56.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150706 s, 69.6 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0105064 s, 99.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0177676 s, 59.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164767 s, 63.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0103793 s, 101 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0196009 s, 53.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0207434 s, 50.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0227698 s, 46.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00438276 s, 239 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00560499 s, 187 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00860626 s, 122 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00365315 s, 287 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0116545 s, 90.0 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg220-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg220-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.79218 s, 10.7 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73148 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.77378 s, 10.7 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73481 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kg (114s) == sanityn test 77l: check the output of NRS policies for generic TBF ========================================================== 15:23:20 (1758828200) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_policies=fifo PASS 77l (3s) == sanityn test 77m: check NRS Delay slows write RPC processing ========================================================== 15:23:23 (1758828203) ost.OSS.ost_io.nrs_policies=delay ost.OSS.ost_io.nrs_delay_min=4 ost.OSS.ost_io.nrs_delay_max=4 ost.OSS.ost_io.nrs_delay_pct=100 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg220-client.virtnet] [bs=1M] [count=1] [oflag=direct] [conv=fdatasync] 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 3.31133 s, 317 kB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg220-client.virtnet] [bs=1M] [count=10] [oflag=direct] [conv=fdatasync] 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 40.007 s, 262 kB/s ost.OSS.ost_io.nrs_policies=fifo PASS 77m (52s) == sanityn test 77n: check wildcard support for TBF JobID NRS policy ========================================================== 15:24:15 (1758828255) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={*.500} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.224607 s, 74.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0479512 s, 21.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0300929 s, 34.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0809563 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0921732 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109472 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133929 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139522 s, 7.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8c2889525800.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff8c2889525800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111211 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124261 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17384 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14175 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139993 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144672 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134874 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110264 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159285 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0188402 s, 55.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170242 s, 61.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0236393 s, 44.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173066 s, 60.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0119823 s, 87.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0242834 s, 43.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0216239 s, 48.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00855383 s, 123 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8c2889525800.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff8c2889525800.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0213394 s, 49.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150988 s, 69.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0251827 s, 41.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.019292 s, 54.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0112104 s, 93.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126934 s, 82.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00465463 s, 225 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0254042 s, 41.3 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg220-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg220-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77n.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.99363 s, 21.0 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77n.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.98109 s, 21.1 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.*} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.291113 s, 57.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0407611 s, 25.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0443593 s, 23.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0841688 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10271 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0987864 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110471 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0977429 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124336 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109648 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111662 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145379 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115377 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119572 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124346 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192132 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199655 s, 5.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0111418 s, 94.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0068603 s, 153 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139996 s, 74.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0302655 s, 34.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00748273 s, 140 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00759567 s, 138 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0123043 s, 85.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00768645 s, 136 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00630335 s, 166 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00635423 s, 165 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168802 s, 62.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0378647 s, 27.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0265204 s, 39.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00825885 s, 127 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0269317 s, 38.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.058808 s, 17.8 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg220-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg220-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.99066 s, 21.0 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.94308 s, 21.2 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_policies=fifo PASS 77n (67s) == sanityn test 77o: Changing rank should not panic ====== 15:25:22 (1758828322) mds.MDS.mdt.nrs_policies=tbf nid mds.MDS.mdt.nrs_tbf_rule=start name nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=start name1 nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=change name1 rank=name mds.MDS.mdt.nrs_tbf_rule=stop name mds.MDS.mdt.nrs_policies=fifo PASS 77o (5s) == sanityn test 77q: Parallel TBF rule definitions should not panic ========================================================== 15:25:27 (1758828327) mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_policies=fifo PASS 77q (49s) == sanityn test 77p: Check validity of rule names for TBF policies ========================================================== 15:26:16 (1758828376) mds.MDS.mdt.nrs_policies=tbf oleg220-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_overflo uid={500} rate=500: File name too long oleg220-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_overflo uid={500} rate=500': File name too long pdsh@oleg220-client: oleg220-server: ssh exited with exit code 36 oleg220-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start uid={500} rate=500: Invalid argument oleg220-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start uid={500} rate=500': Invalid argument pdsh@oleg220-client: oleg220-server: ssh exited with exit code 22 oleg220-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_empty: Invalid argument oleg220-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_empty': Invalid argument pdsh@oleg220-client: oleg220-server: ssh exited with exit code 22 oleg220-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p@spec uid={500} rate=500: Invalid argument oleg220-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p@spec uid={500} rate=500': Invalid argument pdsh@oleg220-client: oleg220-server: ssh exited with exit code 22 oleg220-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p.spec uid={500} rate=500: Invalid argument oleg220-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p.spec uid={500} rate=500': Invalid argument pdsh@oleg220-client: oleg220-server: ssh exited with exit code 22 oleg220-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p~spec uid={500} rate=500: Invalid argument oleg220-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p~spec uid={500} rate=500': Invalid argument pdsh@oleg220-client: oleg220-server: ssh exited with exit code 22 oleg220-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p#spec uid={500} rate=500: Invalid argument oleg220-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p#spec uid={500} rate=500': Invalid argument pdsh@oleg220-client: oleg220-server: ssh exited with exit code 22 oleg220-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p/spec uid={500} rate=500: Invalid argument oleg220-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p/spec uid={500} rate=500': Invalid argument pdsh@oleg220-client: oleg220-server: ssh exited with exit code 22 oleg220-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p^spec uid={500} rate=500: Invalid argument oleg220-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p^spec uid={500} rate=500': Invalid argument pdsh@oleg220-client: oleg220-server: ssh exited with exit code 22 oleg220-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p%spec uid={500} rate=500: Invalid argument oleg220-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p%spec uid={500} rate=500': Invalid argument pdsh@oleg220-client: oleg220-server: ssh exited with exit code 22 oleg220-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p*spec uid={500} rate=500: Invalid argument oleg220-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p*spec uid={500} rate=500': Invalid argument pdsh@oleg220-client: oleg220-server: ssh exited with exit code 22 oleg220-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p;spec uid={500} rate=500: Invalid argument oleg220-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p;spec uid={500} rate=500': Invalid argument pdsh@oleg220-client: oleg220-server: ssh exited with exit code 22 oleg220-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p,spec uid={500} rate=500: Invalid argument oleg220-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p,spec uid={500} rate=500': Invalid argument pdsh@oleg220-client: oleg220-server: ssh exited with exit code 22 oleg220-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p?spec uid={500} rate=500: Invalid argument oleg220-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p?spec uid={500} rate=500': Invalid argument pdsh@oleg220-client: oleg220-server: ssh exited with exit code 22 oleg220-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77pspec uid={500} rate=500: Invalid argument oleg220-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p>spec uid={500} rate=500': Invalid argument pdsh@oleg220-client: oleg220-server: ssh exited with exit code 22 oleg220-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p:spec uid={500} rate=500: Invalid argument oleg220-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p:spec uid={500} rate=500': Invalid argument pdsh@oleg220-client: oleg220-server: ssh exited with exit code 22 oleg220-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p+spec uid={500} rate=500: Invalid argument oleg220-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p+spec uid={500} rate=500': Invalid argument pdsh@oleg220-client: oleg220-server: ssh exited with exit code 22 oleg220-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p=spec uid={500} rate=500: Invalid argument oleg220-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p=spec uid={500} rate=500': Invalid argument pdsh@oleg220-client: oleg220-server: ssh exited with exit code 22 oleg220-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p)spec uid={500} rate=500: Invalid argument oleg220-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p)spec uid={500} rate=500': Invalid argument pdsh@oleg220-client: oleg220-server: ssh exited with exit code 22 oleg220-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p(spec uid={500} rate=500: Invalid argument oleg220-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p(spec uid={500} rate=500': Invalid argument pdsh@oleg220-client: oleg220-server: ssh exited with exit code 22 oleg220-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p{spec uid={500} rate=500: Invalid argument oleg220-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p{spec uid={500} rate=500': Invalid argument pdsh@oleg220-client: oleg220-server: ssh exited with exit code 22 oleg220-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p}spec uid={500} rate=500: Invalid argument oleg220-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p}spec uid={500} rate=500': Invalid argument pdsh@oleg220-client: oleg220-server: ssh exited with exit code 22 oleg220-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p|spec uid={500} rate=500: Invalid argument oleg220-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p|spec uid={500} rate=500': Invalid argument pdsh@oleg220-client: oleg220-server: ssh exited with exit code 22 oleg220-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p[spec uid={500} rate=500: Invalid argument oleg220-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p[spec uid={500} rate=500': Invalid argument pdsh@oleg220-client: oleg220-server: ssh exited with exit code 22 oleg220-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p]spec uid={500} rate=500: Invalid argument oleg220-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p]spec uid={500} rate=500': Invalid argument pdsh@oleg220-client: oleg220-server: ssh exited with exit code 22 oleg220-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p!spec uid={500} rate=500: Invalid argument oleg220-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p!spec uid={500} rate=500': Invalid argument pdsh@oleg220-client: oleg220-server: ssh exited with exit code 22 oleg220-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p&spec uid={500} rate=500: Invalid argument oleg220-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p&spec uid={500} rate=500': Invalid argument pdsh@oleg220-client: oleg220-server: ssh exited with exit code 22 oleg220-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p$spec uid={500} rate=500: Invalid argument oleg220-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p$spec uid={500} rate=500': Invalid argument pdsh@oleg220-client: oleg220-server: ssh exited with exit code 22 oleg220-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p`spec uid={500} rate=500: Invalid argument oleg220-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p`spec uid={500} rate=500': Invalid argument pdsh@oleg220-client: oleg220-server: ssh exited with exit code 22 oleg220-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p\spec uid={500} rate=500: Invalid argument oleg220-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p\spec uid={500} rate=500': Invalid argument pdsh@oleg220-client: oleg220-server: ssh exited with exit code 22 mds.MDS.mdt.nrs_policies=fifo PASS 77p (16s) == sanityn test 77r: Change type of tbf policy at run time ========================================================== 15:26:32 (1758828392) mds.MDS.mdt.nrs_policies=tbf uid mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf gid mds.MDS.mdt.nrs_tbf_rule=start md_rule gid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start md_rule jobid={*.500} rate=1 mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=fifo check the number of requests in queue: fifo: 0 crrn: 0 tbf: 0 delay: 0 fifo: 0 crrn: 0 tbf: 0 delay: 0 cleanup 77r 309033 kill md_thread xargs PASS 77r (42s) == sanityn test 78: Enable policy and specify tunings right away ========================================================== 15:27:14 (1758828434) oleg220-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_orr_quantum=1: No such device oleg220-server: error: set_param: setting 'ost/OSS/*/nrs_orr_quantum'='1': No such device pdsh@oleg220-client: oleg220-server: ssh exited with exit code 19 ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_policies=fifo PASS 78 (4s) == sanityn test 79: xattr: intent error ================== 15:27:18 (1758828438) File: /mnt/lustre/d79.sanityn Size: 512 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272506472 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-09-25 15:27:19.000000000 -0400 Modify: 2025-09-25 15:27:19.000000000 -0400 Change: 2025-09-25 15:27:19.000000000 -0400 Birth: 2025-09-25 15:27:19.000000000 -0400 fail_val=0 fail_loc=0x80000160 fail_val=0 fail_loc=0x80000131 # file: mnt/lustre/d79.sanityn trusted.name1="value1" PASS 79 (14s) == sanityn test 80a: migrate directory when some children is being opened ========================================================== 15:27:32 (1758828452) SKIP: sanityn test_80a needs >= 2 MDTs SKIP 80a (1s) == sanityn test 80b: Accessing directory during migration ========================================================== 15:27:33 (1758828453) SKIP: sanityn test_80b needs >= 2 MDTs SKIP 80b (2s) == sanityn test 81a: rename and stat under striped directory ========================================================== 15:27:35 (1758828455) SKIP: sanityn test_81a needs >= 2 MDTs SKIP 81a (1s) == sanityn test 81b: rename under striped directory doesn't deadlock ========================================================== 15:27:36 (1758828456) SKIP: sanityn test_81b We need at least 2 MDTs for this test SKIP 81b (1s) == sanityn test 81c: rename revoke LOOKUP lock for remote object ========================================================== 15:27:37 (1758828457) SKIP: sanityn test_81c needs >= 4 MDTs SKIP 81c (2s) == sanityn test 81d: parallel rename file cross-dir on same MDT ========================================================== 15:27:39 (1758828459) total: 2500 open/close in 9.01 seconds: 277.58 ops/second snapshot_time 1758828488.908221129 secs.nsecs start_time 1758828460.447090919 secs.nsecs elapsed_time 28.461130210 secs.nsecs open 2505 samples [usecs] 71 5259 1295709 779932067 close 2504 samples [usecs] 67 1154 420952 77910522 mknod 2501 samples [usecs] 181 5058 1127051 605643557 unlink 2500 samples [usecs] 276 11597 1127902 921205012 mkdir 2 samples [usecs] 1051 1747 2798 4156610 rmdir 2 samples [usecs] 1772 2778 4550 10857268 rename 2500 samples [usecs] 247 10589 3038531 5723952731 getattr 3760 samples [usecs] 8 4573 80158 24205758 parallel_rename_file 2500 samples [usecs] 247 10589 3038531 5723952731 crossdir_rename 2500 samples [usecs] 247 10589 3038531 5723952731 PASS 81d (31s) == sanityn test 82: fsetxattr and fgetxattr on orphan files ========================================================== 15:28:10 (1758828490) multiop /mnt/lustre/f82.sanityn vO_ac TMPPIPE=/tmp/multiop_open_wait_pipe.7559 multiop /mnt/lustre2/f82.sanityn vo_Ac TMPPIPE=/tmp/multiop_open_wait_pipe.7559 PASS 82 (3s) == sanityn test 83: access striped directory while it is being created/unlinked ========================================================== 15:28:13 (1758828493) SKIP: sanityn test_83 needs >= 2 MDTs SKIP 83 (1s) == sanityn test 84: 0-nlink race in lu_object_find() ===== 15:28:14 (1758828494) fail_loc=0x8000060b PASS 84 (8s) == sanityn test 85: Lustre API root cache race =========== 15:28:22 (1758828502) Starting: llapi_root_test /mnt/lustre/f85.sanityn /mnt/lustre2/f85.sanityn Starting test test1 at 1758828503 Finishing test test1 at 1758828503 PASS 85 (3s) == sanityn test 90: open/create and unlink striped directory ========================================================== 15:28:25 (1758828505) SKIP: sanityn test_90 needs >= 2 MDTs SKIP 90 (1s) == sanityn test 91: chmod and unlink striped directory === 15:28:26 (1758828506) SKIP: sanityn test_91 needs >= 2 MDTs SKIP 91 (1s) == sanityn test 92: create remote directory under orphan directory ========================================================== 15:28:27 (1758828507) SKIP: sanityn test_92 needs >= 2 MDTs SKIP 92 (1s) == sanityn test 93: alloc_rr should not allocate on same ost ========================================================== 15:28:28 (1758828508) fail_loc=0x00000163 fail_loc=0x0 /mnt/lustre/f93.sanityn-1/file1 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5810 0x16b2 0x240000400 1 5818 0x16ba 0x280000400 /mnt/lustre2/f93.sanityn-2/file2 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 5819 0x16bb 0x280000400 0 5811 0x16b3 0x240000400 rc1=0 and rc2=0 PASS 93 (10s) == sanityn test 94: signal vs CP callback race =========== 15:28:38 (1758828518) osc.lustre-OST0000-osc-ffff8c2889525800.idle_timeout=0 osc.lustre-OST0000-osc-ffff8c2898bad800.idle_timeout=0 osc.lustre-OST0001-osc-ffff8c2889525800.idle_timeout=0 osc.lustre-OST0001-osc-ffff8c2898bad800.idle_timeout=0 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0436089 s, 188 kB/s fail_val=5 fail_loc=0x80000312 fail_val=6 fail_loc=0x80000329 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 3.08603 s, 1.3 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 5988: 327445 Alarm clock dd if=/dev/zero of=$DIR/$tfile conv=notrunc oflag=append bs=4k count=1 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0231598 s, 177 kB/s osc.lustre-OST0000-osc-ffff8c2889525800.idle_timeout=debug osc.lustre-OST0000-osc-ffff8c2898bad800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8c2889525800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8c2898bad800.idle_timeout=debug PASS 94 (10s) == sanityn test 95a: Check readpage() on a page that was removed from page cache ========================================================== 15:28:48 (1758828528) llite.lustre-ffff8c2889525800.fast_read=0 llite.lustre-ffff8c2898bad800.fast_read=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0271195 s, 77.3 MB/s fail_loc=0x80001421 fail_val=7 fail_loc=0x80001422 fail_val=10 llite.lustre-ffff8c2889525800.fast_read=1 llite.lustre-ffff8c2898bad800.fast_read=1 PASS 95a (16s) == sanityn test 95b: Check readpage() on a page that is no longer uptodate ========================================================== 15:29:04 (1758828544) llite.lustre-ffff8c2889525800.fast_read=0 llite.lustre-ffff8c2898bad800.fast_read=0 1+0 records in 1+0 records out 12288 bytes (12 kB, 12 KiB) copied, 0.00275723 s, 4.5 MB/s fail_loc=0x80001424 fail_val=5 fail_loc=0 llite.lustre-ffff8c2889525800.fast_read=1 llite.lustre-ffff8c2898bad800.fast_read=1 PASS 95b (9s) == sanityn test 100a: DoM: glimpse RPCs for stat without IO lock (DoM only file) ========================================================== 15:29:13 (1758828553) SKIP: sanityn test_100a Reserved for glimpse-ahead SKIP 100a (1s) == sanityn test 100b: DoM: no glimpse RPC for stat with IO lock (DoM only file) ========================================================== 15:29:15 (1758828555) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00144092 s, 2.8 MB/s /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK PASS 100b (3s) == sanityn test 100c: DoM: write vs stat without IO lock (combined file) ========================================================== 15:29:17 (1758828557) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0769645 s, 27.2 MB/s /mnt/lustre/d100c.sanityn/dom has type file OK /mnt/lustre/d100c.sanityn/dom has size 2097152 OK PASS 100c (2s) == sanityn test 100d: DoM: write+truncate vs stat without IO lock (combined file) ========================================================== 15:29:19 (1758828559) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.04721 s, 44.4 MB/s /mnt/lustre/d100d.sanityn/dom has type file OK /mnt/lustre/d100d.sanityn/dom has size 4096 OK PASS 100d (3s) == sanityn test 100e: DoM: read on open and file size ==== 15:29:22 (1758828562) 1+0 records in 1+0 records out 12000 bytes (12 kB, 12 KiB) copied, 0.00190181 s, 6.3 MB/s PASS 100e (2s) == sanityn test 101a: Discard DoM data on unlink ========= 15:29:24 (1758828564) /mnt/lustre/f101a.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00106221 s, 3.9 MB/s PASS 101a (3s) == sanityn test 101b: Discard DoM data on rename ========= 15:29:27 (1758828567) /mnt/lustre/f101b.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.000551295 s, 7.4 MB/s PASS 101b (2s) == sanityn test 101c: Discard DoM data on close-unlink === 15:29:29 (1758828569) /mnt/lustre/f101c.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.000613098 s, 6.7 MB/s PASS 101c (4s) SKIP: sanityn test_102 skipping ALWAYS excluded test 102 == sanityn test 103: Test size correctness with lockahead ========================================================== 15:29:33 (1758828573) fail_loc=0x415 fail_loc=0x415 fail_val=2 Incorrect size expected (no glimpse fix): Starting test test23 at 1758828574 lockahead_test: lockahead_test.c:1171: test23: assertion 'sb.st_size == sb2.st_size' failed: size on /mnt/lustre/d103.sanityn/f103.sanityn and /mnt/lustre2/d103.sanityn/f103.sanityn differs: 0 vs 1048576 fail_loc=0 fail_loc=0x214 fail_val=2 Starting test test23 at 1758828577 Finishing test test23 at 1758828577 PASS 103 (8s) == sanityn test 104: Verify that MDS stores atime/mtime/ctime during close ========================================================== 15:29:41 (1758828581) SKIP: sanityn test_104 ldiskfs only test SKIP 104 (1s) == sanityn test 105: Glimpse and lock cancel race ======== 15:29:42 (1758828582) fail_loc=0x416 PASS 105 (23s) == sanityn test 106a: Verify the btime via statx() ======= 15:30:05 (1758828605) SKIP: sanityn test_106a Test only for ldiskfs and statx() supported SKIP 106a (1s) == sanityn test 106b: Glimpse RPCs test for statx ======== 15:30:06 (1758828606) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00364621 s, 281 kB/s File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205272509007 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-09-25 15:30:07.000000000 -0400 Modify: 2025-09-25 15:30:07.000000000 -0400 Change: 2025-09-25 15:30:07.000000000 -0400 Birth: 2025-09-25 15:30:07.000000000 -0400 /mnt/lustre/f106b.sanityn 144115205272509007 -rw-r--r-- 1 0 0 1758828607 1758828607 1758828607 File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205272509007 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-09-25 15:30:07.000000000 -0400 Modify: 2025-09-25 15:30:07.000000000 -0400 Change: 2025-09-25 15:30:07.000000000 -0400 Birth: 2025-09-25 15:30:07.000000000 -0400 1758828607 1024 1 PASS 106b (3s) == sanityn test 106c: Verify statx attributes mask ======= 15:30:09 (1758828609) PASS 106c (2s) == sanityn test 107a: Basic grouplock conflict =========== 15:30:11 (1758828611) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.0892311 s, 118 MB/s /mnt/lustre/f107a.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5819 0x16bb 0x240000400 multiop /mnt/lustre/f107a.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7559 multiop /mnt/lustre2/f107a.sanityn vO_G16022000r10g16022000c TMPPIPE=/tmp/multiop_open_wait_pipe.7559 First grouplock blocks second one PASS 107a (4s) == sanityn test 107b: Grouplock is added to the head of waiting list ========================================================== 15:30:15 (1758828615) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.0780871 s, 134 MB/s /mnt/lustre/f107b.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 5826 0x16c2 0x280000400 multiop /mnt/lustre/f107b.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7559 Grouplock blocks IO First grouplock blocks second one Second grouplock blocks IO PASS 107b (8s) == sanityn test 108a: lseek: parallel updates ============ 15:30:23 (1758828623) 1+0 records in 1+0 records out 10000 bytes (10 kB, 9.8 KiB) copied, 0.00456176 s, 2.2 MB/s llite.lustre-ffff8c2889525800.hybrid_io=0 llite.lustre-ffff8c2898bad800.hybrid_io=0 fail_val=4 fail_loc=0x80000414 1+0 records in 1+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 4.29588 s, 2.0 MB/s PASS 108a (7s) == sanityn test 109: Race with several mount instances on 1 node ========================================================== 15:30:30 (1758828630) 192.168.202.120@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg220-client.virtnet /mnt/lustre (opts:) 192.168.202.120@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg220-client.virtnet /mnt/lustre2 (opts:) Starting race between client mount instances (50 iterations): Iteration 1 Starting client: oleg220-client.virtnet: -o user_xattr,flock 192.168.202.120@tcp:/lustre /mnt/lustre Starting client: oleg220-client.virtnet: -o user_xattr,flock 192.168.202.120@tcp:/lustre /mnt/lustre2 192.168.202.120@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.120@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg220-client.virtnet /mnt/lustre (opts:) Stopping client oleg220-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg220-server' oleg220-server: oleg220-server.virtnet: executing load_modules_local oleg220-server: Loading modules from /home/green/git/lustre-release/lustre oleg220-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg220-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg220-server: detected 4 online CPUs by sysfs oleg220-server: MODOPTS_LIBCFS= oleg220-server: Force libcfs to create 2 CPU partitions Iteration 2 Starting client: oleg220-client.virtnet: -o user_xattr,flock 192.168.202.120@tcp:/lustre /mnt/lustre Starting client: oleg220-client.virtnet: -o user_xattr,flock 192.168.202.120@tcp:/lustre /mnt/lustre2 192.168.202.120@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.120@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg220-client.virtnet /mnt/lustre (opts:) Stopping client oleg220-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg220-server' oleg220-server: oleg220-server.virtnet: executing load_modules_local oleg220-server: Loading modules from /home/green/git/lustre-release/lustre oleg220-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg220-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg220-server: detected 4 online CPUs by sysfs oleg220-server: MODOPTS_LIBCFS= oleg220-server: Force libcfs to create 2 CPU partitions Iteration 3 Starting client: oleg220-client.virtnet: -o user_xattr,flock 192.168.202.120@tcp:/lustre /mnt/lustre Starting client: oleg220-client.virtnet: -o user_xattr,flock 192.168.202.120@tcp:/lustre /mnt/lustre2 192.168.202.120@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.120@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg220-client.virtnet /mnt/lustre (opts:) Stopping client oleg220-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg220-server' oleg220-server: oleg220-server.virtnet: executing load_modules_local oleg220-server: Loading modules from /home/green/git/lustre-release/lustre oleg220-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg220-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg220-server: detected 4 online CPUs by sysfs oleg220-server: MODOPTS_LIBCFS= oleg220-server: Force libcfs to create 2 CPU partitions Iteration 4 Starting client: oleg220-client.virtnet: -o user_xattr,flock 192.168.202.120@tcp:/lustre /mnt/lustre Starting client: oleg220-client.virtnet: -o user_xattr,flock 192.168.202.120@tcp:/lustre /mnt/lustre2 192.168.202.120@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.120@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg220-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg220-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg220-server' oleg220-server: oleg220-server.virtnet: executing load_modules_local oleg220-server: Loading modules from /home/green/git/lustre-release/lustre oleg220-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg220-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg220-server: detected 4 online CPUs by sysfs oleg220-server: MODOPTS_LIBCFS= oleg220-server: Force libcfs to create 2 CPU partitions Iteration 5 Starting client: oleg220-client.virtnet: -o user_xattr,flock 192.168.202.120@tcp:/lustre /mnt/lustre Starting client: oleg220-client.virtnet: -o user_xattr,flock 192.168.202.120@tcp:/lustre /mnt/lustre2 192.168.202.120@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.120@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg220-client.virtnet /mnt/lustre (opts:) Stopping client oleg220-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg220-server' oleg220-server: oleg220-server.virtnet: executing load_modules_local oleg220-server: Loading modules from /home/green/git/lustre-release/lustre oleg220-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg220-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg220-server: detected 4 online CPUs by sysfs oleg220-server: MODOPTS_LIBCFS= oleg220-server: Force libcfs to create 2 CPU partitions Iteration 6 Starting client: oleg220-client.virtnet: -o user_xattr,flock 192.168.202.120@tcp:/lustre /mnt/lustre2 Starting client: oleg220-client.virtnet: -o user_xattr,flock 192.168.202.120@tcp:/lustre /mnt/lustre 192.168.202.120@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.120@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg220-client.virtnet /mnt/lustre (opts:) Stopping client oleg220-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg220-server' oleg220-server: oleg220-server.virtnet: executing load_modules_local oleg220-server: Loading modules from /home/green/git/lustre-release/lustre oleg220-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg220-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg220-server: detected 4 online CPUs by sysfs oleg220-server: MODOPTS_LIBCFS= oleg220-server: Force libcfs to create 2 CPU partitions Iteration 7 Starting client: oleg220-client.virtnet: -o user_xattr,flock 192.168.202.120@tcp:/lustre /mnt/lustre2 Starting client: oleg220-client.virtnet: -o user_xattr,flock 192.168.202.120@tcp:/lustre /mnt/lustre 192.168.202.120@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.120@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg220-client.virtnet /mnt/lustre (opts:) Stopping client oleg220-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg220-server' oleg220-server: oleg220-server.virtnet: executing load_modules_local oleg220-server: Loading modules from /home/green/git/lustre-release/lustre oleg220-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg220-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg220-server: detected 4 online CPUs by sysfs oleg220-server: MODOPTS_LIBCFS= oleg220-server: Force libcfs to create 2 CPU partitions Iteration 8 Starting client: oleg220-client.virtnet: -o user_xattr,flock 192.168.202.120@tcp:/lustre /mnt/lustre2 Starting client: oleg220-client.virtnet: -o user_xattr,flock 192.168.202.120@tcp:/lustre /mnt/lustre 192.168.202.120@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.120@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg220-client.virtnet /mnt/lustre (opts:) Stopping client oleg220-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg220-server' oleg220-server: oleg220-server.virtnet: executing load_modules_local oleg220-server: Loading modules from /home/green/git/lustre-release/lustre oleg220-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg220-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg220-server: detected 4 online CPUs by sysfs oleg220-server: MODOPTS_LIBCFS= oleg220-server: Force libcfs to create 2 CPU partitions Iteration 9 Starting client: oleg220-client.virtnet: -o user_xattr,flock 192.168.202.120@tcp:/lustre /mnt/lustre Starting client: oleg220-client.virtnet: -o user_xattr,flock 192.168.202.120@tcp:/lustre /mnt/lustre2 192.168.202.120@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.120@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg220-client.virtnet /mnt/lustre (opts:) Stopping client oleg220-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg220-server' oleg220-server: oleg220-server.virtnet: executing load_modules_local oleg220-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg220-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg220-server: Loading modules from /home/green/git/lustre-release/lustre oleg220-server: detected 4 online CPUs by sysfs oleg220-server: MODOPTS_LIBCFS= oleg220-server: Force libcfs to create 2 CPU partitions Iteration 10 Starting client: oleg220-client.virtnet: -o user_xattr,flock 192.168.202.120@tcp:/lustre /mnt/lustre Starting client: oleg220-client.virtnet: -o user_xattr,flock 192.168.202.120@tcp:/lustre /mnt/lustre2 192.168.202.120@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.120@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg220-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg220-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg220-server' oleg220-server: oleg220-server.virtnet: executing load_modules_local oleg220-server: Loading modules from /home/green/git/lustre-release/lustre oleg220-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg220-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg220-server: detected 4 online CPUs by sysfs oleg220-server: MODOPTS_LIBCFS= oleg220-server: Force libcfs to create 2 CPU partitions Iteration 11 Starting client: oleg220-client.virtnet: -o user_xattr,flock 192.168.202.120@tcp:/lustre /mnt/lustre Starting client: oleg220-client.virtnet: -o user_xattr,flock 192.168.202.120@tcp:/lustre /mnt/lustre2 192.168.202.120@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.120@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg220-client.virtnet /mnt/lustre (opts:) Stopping client oleg220-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg220-server' oleg220-server: oleg220-server.virtnet: executing load_modules_local oleg220-server: Loading modules from /home/green/git/lustre-release/lustre oleg220-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg220-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg220-server: detected 4 online CPUs by sysfs oleg220-server: MODOPTS_LIBCFS= oleg220-server: Force libcfs to create 2 CPU partitions Iteration 12 Starting client: oleg220-client.virtnet: -o user_xattr,flock 192.168.202.120@tcp:/lustre /mnt/lustre Starting client: oleg220-client.virtnet: -o user_xattr,flock 192.168.202.120@tcp:/lustre /mnt/lustre2 192.168.202.120@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.120@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg220-client.virtnet /mnt/lustre (opts:) Stopping client oleg220-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg220-server' oleg220-server: oleg220-server.virtnet: executing load_modules_local oleg220-server: Loading modules from /home/green/git/lustre-release/lustre oleg220-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg220-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg220-server: detected 4 online CPUs by sysfs oleg220-server: MODOPTS_LIBCFS= oleg220-server: Force libcfs to create 2 CPU partitions Iteration 13 Starting client: oleg220-client.virtnet: -o user_xattr,flock 192.168.202.120@tcp:/lustre /mnt/lustre2 Starting client: oleg220-client.virtnet: -o user_xattr,flock 192.168.202.120@tcp:/lustre /mnt/lustre 192.168.202.120@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.120@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg220-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg220-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg220-server' oleg220-server: oleg220-server.virtnet: executing load_modules_local oleg220-server: Loading modules from /home/green/git/lustre-release/lustre oleg220-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg220-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg220-server: detected 4 online CPUs by sysfs oleg220-server: MODOPTS_LIBCFS= oleg220-server: Force libcfs to create 2 CPU partitions Iteration 14 Starting client: oleg220-client.virtnet: -o user_xattr,flock 192.168.202.120@tcp:/lustre /mnt/lustre Starting client: oleg220-client.virtnet: -o user_xattr,flock 192.168.202.120@tcp:/lustre /mnt/lustre2 192.168.202.120@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.120@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg220-client.virtnet /mnt/lustre (opts:) Stopping client oleg220-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg220-server' oleg220-server: oleg220-server.virtnet: executing load_modules_local oleg220-server: Loading modules from /home/green/git/lustre-release/lustre oleg220-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg220-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg220-server: detected 4 online CPUs by sysfs oleg220-server: MODOPTS_LIBCFS= oleg220-server: Force libcfs to create 2 CPU partitions Iteration 15 Starting client: oleg220-client.virtnet: -o user_xattr,flock 192.168.202.120@tcp:/lustre /mnt/lustre Starting client: oleg220-client.virtnet: -o user_xattr,flock 192.168.202.120@tcp:/lustre /mnt/lustre2 192.168.202.120@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.120@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg220-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg220-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg220-server' oleg220-server: oleg220-server.virtnet: executing load_modules_local oleg220-server: Loading modules from /home/green/git/lustre-release/lustre oleg220-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg220-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg220-server: detected 4 online CPUs by sysfs oleg220-server: MODOPTS_LIBCFS= oleg220-server: Force libcfs to create 2 CPU partitions Iteration 16 Starting client: oleg220-client.virtnet: -o user_xattr,flock 192.168.202.120@tcp:/lustre /mnt/lustre Starting client: oleg220-client.virtnet: -o user_xattr,flock 192.168.202.120@tcp:/lustre /mnt/lustre2 192.168.202.120@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.120@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg220-client.virtnet /mnt/lustre (opts:) Stopping client oleg220-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg220-server' oleg220-server: oleg220-server.virtnet: executing load_modules_local oleg220-server: Loading modules from /home/green/git/lustre-release/lustre oleg220-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg220-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg220-server: detected 4 online CPUs by sysfs oleg220-server: MODOPTS_LIBCFS= oleg220-server: Force libcfs to create 2 CPU partitions Iteration 17 Starting client: oleg220-client.virtnet: -o user_xattr,flock 192.168.202.120@tcp:/lustre /mnt/lustre Starting client: oleg220-client.virtnet: -o user_xattr,flock 192.168.202.120@tcp:/lustre /mnt/lustre2 192.168.202.120@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.120@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg220-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg220-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg220-server' oleg220-server: oleg220-server.virtnet: executing load_modules_local oleg220-server: Loading modules from /home/green/git/lustre-release/lustre oleg220-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg220-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg220-server: detected 4 online CPUs by sysfs oleg220-server: MODOPTS_LIBCFS= oleg220-server: Force libcfs to create 2 CPU partitions Iteration 18 Starting client: oleg220-client.virtnet: -o user_xattr,flock 192.168.202.120@tcp:/lustre /mnt/lustre Starting client: oleg220-client.virtnet: -o user_xattr,flock 192.168.202.120@tcp:/lustre /mnt/lustre2 192.168.202.120@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.120@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg220-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg220-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg220-server' oleg220-server: oleg220-server.virtnet: executing load_modules_local oleg220-server: Loading modules from /home/green/git/lustre-release/lustre oleg220-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg220-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg220-server: detected 4 online CPUs by sysfs oleg220-server: MODOPTS_LIBCFS= oleg220-server: Force libcfs to create 2 CPU partitions Iteration 19 Starting client: oleg220-client.virtnet: -o user_xattr,flock 192.168.202.120@tcp:/lustre /mnt/lustre Starting client: oleg220-client.virtnet: -o user_xattr,flock 192.168.202.120@tcp:/lustre /mnt/lustre2 192.168.202.120@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.120@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg220-client.virtnet /mnt/lustre (opts:) Stopping client oleg220-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg220-server' oleg220-server: oleg220-server.virtnet: executing load_modules_local oleg220-server: Loading modules from /home/green/git/lustre-release/lustre oleg220-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg220-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg220-server: detected 4 online CPUs by sysfs oleg220-server: MODOPTS_LIBCFS= oleg220-server: Force libcfs to create 2 CPU partitions Iteration 20 Starting client: oleg220-client.virtnet: -o user_xattr,flock 192.168.202.120@tcp:/lustre /mnt/lustre Starting client: oleg220-client.virtnet: -o user_xattr,flock 192.168.202.120@tcp:/lustre /mnt/lustre2 192.168.202.120@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.120@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg220-client.virtnet /mnt/lustre (opts:) Stopping client oleg220-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg220-server' oleg220-server: oleg220-server.virtnet: executing load_modules_local oleg220-server: Loading modules from /home/green/git/lustre-release/lustre oleg220-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg220-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg220-server: detected 4 online CPUs by sysfs oleg220-server: MODOPTS_LIBCFS= oleg220-server: Force libcfs to create 2 CPU partitions Iteration 21 Starting client: oleg220-client.virtnet: -o user_xattr,flock 192.168.202.120@tcp:/lustre /mnt/lustre Starting client: oleg220-client.virtnet: -o user_xattr,flock 192.168.202.120@tcp:/lustre /mnt/lustre2 192.168.202.120@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.120@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg220-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg220-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg220-server' oleg220-server: oleg220-server.virtnet: executing load_modules_local oleg220-server: Loading modules from /home/green/git/lustre-release/lustre oleg220-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg220-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg220-server: detected 4 online CPUs by sysfs oleg220-server: MODOPTS_LIBCFS= oleg220-server: Force libcfs to create 2 CPU partitions Iteration 22 Starting client: oleg220-client.virtnet: -o user_xattr,flock 192.168.202.120@tcp:/lustre /mnt/lustre2 Starting client: oleg220-client.virtnet: -o user_xattr,flock 192.168.202.120@tcp:/lustre /mnt/lustre 192.168.202.120@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.120@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg220-client.virtnet /mnt/lustre (opts:) Stopping client oleg220-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg220-server' oleg220-server: oleg220-server.virtnet: executing load_modules_local oleg220-server: Loading modules from /home/green/git/lustre-release/lustre oleg220-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg220-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg220-server: detected 4 online CPUs by sysfs oleg220-server: MODOPTS_LIBCFS= oleg220-server: Force libcfs to create 2 CPU partitions Iteration 23 Starting client: oleg220-client.virtnet: -o user_xattr,flock 192.168.202.120@tcp:/lustre /mnt/lustre Starting client: oleg220-client.virtnet: -o user_xattr,flock 192.168.202.120@tcp:/lustre /mnt/lustre2 192.168.202.120@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.120@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg220-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg220-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg220-server' oleg220-server: oleg220-server.virtnet: executing load_modules_local oleg220-server: Loading modules from /home/green/git/lustre-release/lustre oleg220-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg220-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg220-server: detected 4 online CPUs by sysfs oleg220-server: MODOPTS_LIBCFS= oleg220-server: Force libcfs to create 2 CPU partitions Iteration 24 Starting client: oleg220-client.virtnet: -o user_xattr,flock 192.168.202.120@tcp:/lustre /mnt/lustre Starting client: oleg220-client.virtnet: -o user_xattr,flock 192.168.202.120@tcp:/lustre /mnt/lustre2 192.168.202.120@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.120@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg220-client.virtnet /mnt/lustre (opts:) Stopping client oleg220-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg220-server' oleg220-server: oleg220-server.virtnet: executing load_modules_local oleg220-server: Loading modules from /home/green/git/lustre-release/lustre oleg220-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg220-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg220-server: detected 4 online CPUs by sysfs oleg220-server: MODOPTS_LIBCFS= oleg220-server: Force libcfs to create 2 CPU partitions Iteration 25 Starting client: oleg220-client.virtnet: -o user_xattr,flock 192.168.202.120@tcp:/lustre /mnt/lustre Starting client: oleg220-client.virtnet: -o user_xattr,flock 192.168.202.120@tcp:/lustre /mnt/lustre2 192.168.202.120@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.120@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg220-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg220-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg220-server' oleg220-server: oleg220-server.virtnet: executing load_modules_local oleg220-server: Loading modules from /home/green/git/lustre-release/lustre oleg220-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg220-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg220-server: detected 4 online CPUs by sysfs oleg220-server: MODOPTS_LIBCFS= oleg220-server: Force libcfs to create 2 CPU partitions Iteration 26 Starting client: oleg220-client.virtnet: -o user_xattr,flock 192.168.202.120@tcp:/lustre /mnt/lustre2 Starting client: oleg220-client.virtnet: -o user_xattr,flock 192.168.202.120@tcp:/lustre /mnt/lustre 192.168.202.120@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.120@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg220-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg220-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg220-server' oleg220-server: oleg220-server.virtnet: executing load_modules_local oleg220-server: Loading modules from /home/green/git/lustre-release/lustre oleg220-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg220-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg220-server: detected 4 online CPUs by sysfs oleg220-server: MODOPTS_LIBCFS= oleg220-server: Force libcfs to create 2 CPU partitions Iteration 27 Starting client: oleg220-client.virtnet: -o user_xattr,flock 192.168.202.120@tcp:/lustre /mnt/lustre Starting client: oleg220-client.virtnet: -o user_xattr,flock 192.168.202.120@tcp:/lustre /mnt/lustre2 192.168.202.120@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.120@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg220-client.virtnet /mnt/lustre (opts:) Stopping client oleg220-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg220-server' oleg220-server: oleg220-server.virtnet: executing load_modules_local oleg220-server: Loading modules from /home/green/git/lustre-release/lustre oleg220-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg220-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg220-server: detected 4 online CPUs by sysfs oleg220-server: MODOPTS_LIBCFS= oleg220-server: Force libcfs to create 2 CPU partitions Iteration 28 Starting client: oleg220-client.virtnet: -o user_xattr,flock 192.168.202.120@tcp:/lustre /mnt/lustre Starting client: oleg220-client.virtnet: -o user_xattr,flock 192.168.202.120@tcp:/lustre /mnt/lustre2 192.168.202.120@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.120@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg220-client.virtnet /mnt/lustre (opts:) Stopping client oleg220-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg220-server' oleg220-server: oleg220-server.virtnet: executing load_modules_local oleg220-server: Loading modules from /home/green/git/lustre-release/lustre oleg220-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg220-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg220-server: detected 4 online CPUs by sysfs oleg220-server: MODOPTS_LIBCFS= oleg220-server: Force libcfs to create 2 CPU partitions Iteration 29 Starting client: oleg220-client.virtnet: -o user_xattr,flock 192.168.202.120@tcp:/lustre /mnt/lustre2 Starting client: oleg220-client.virtnet: -o user_xattr,flock 192.168.202.120@tcp:/lustre /mnt/lustre 192.168.202.120@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.120@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg220-client.virtnet /mnt/lustre (opts:) Stopping client oleg220-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg220-server' oleg220-server: oleg220-server.virtnet: executing load_modules_local oleg220-server: Loading modules from /home/green/git/lustre-release/lustre oleg220-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg220-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg220-server: detected 4 online CPUs by sysfs oleg220-server: MODOPTS_LIBCFS= oleg220-server: Force libcfs to create 2 CPU partitions Iteration 30 Starting client: oleg220-client.virtnet: -o user_xattr,flock 192.168.202.120@tcp:/lustre /mnt/lustre Starting client: oleg220-client.virtnet: -o user_xattr,flock 192.168.202.120@tcp:/lustre /mnt/lustre2 192.168.202.120@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.120@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg220-client.virtnet /mnt/lustre (opts:) Stopping client oleg220-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg220-server' oleg220-server: oleg220-server.virtnet: executing load_modules_local oleg220-server: Loading modules from /home/green/git/lustre-release/lustre oleg220-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg220-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg220-server: detected 4 online CPUs by sysfs oleg220-server: MODOPTS_LIBCFS= oleg220-server: Force libcfs to create 2 CPU partitions Iteration 31 Starting client: oleg220-client.virtnet: -o user_xattr,flock 192.168.202.120@tcp:/lustre /mnt/lustre Starting client: oleg220-client.virtnet: -o user_xattr,flock 192.168.202.120@tcp:/lustre /mnt/lustre2 192.168.202.120@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.120@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg220-client.virtnet /mnt/lustre (opts:) Stopping client oleg220-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg220-server' oleg220-server: oleg220-server.virtnet: executing load_modules_local oleg220-server: Loading modules from /home/green/git/lustre-release/lustre oleg220-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg220-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg220-server: detected 4 online CPUs by sysfs oleg220-server: MODOPTS_LIBCFS= oleg220-server: Force libcfs to create 2 CPU partitions Iteration 32 Starting client: oleg220-client.virtnet: -o user_xattr,flock 192.168.202.120@tcp:/lustre /mnt/lustre Starting client: oleg220-client.virtnet: -o user_xattr,flock 192.168.202.120@tcp:/lustre /mnt/lustre2 192.168.202.120@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.120@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg220-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg220-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg220-server' oleg220-server: oleg220-server.virtnet: executing load_modules_local oleg220-server: Loading modules from /home/green/git/lustre-release/lustre oleg220-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg220-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg220-server: detected 4 online CPUs by sysfs oleg220-server: MODOPTS_LIBCFS= oleg220-server: Force libcfs to create 2 CPU partitions Iteration 33 Starting client: oleg220-client.virtnet: -o user_xattr,flock 192.168.202.120@tcp:/lustre /mnt/lustre Starting client: oleg220-client.virtnet: -o user_xattr,flock 192.168.202.120@tcp:/lustre /mnt/lustre2 192.168.202.120@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.120@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg220-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg220-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg220-server' oleg220-server: oleg220-server.virtnet: executing load_modules_local oleg220-server: Loading modules from /home/green/git/lustre-release/lustre oleg220-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg220-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg220-server: detected 4 online CPUs by sysfs oleg220-server: MODOPTS_LIBCFS= oleg220-server: Force libcfs to create 2 CPU partitions Iteration 34 Starting client: oleg220-client.virtnet: -o user_xattr,flock 192.168.202.120@tcp:/lustre /mnt/lustre Starting client: oleg220-client.virtnet: -o user_xattr,flock 192.168.202.120@tcp:/lustre /mnt/lustre2 192.168.202.120@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.120@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg220-client.virtnet /mnt/lustre (opts:) Stopping client oleg220-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg220-server' oleg220-server: oleg220-server.virtnet: executing load_modules_local oleg220-server: Loading modules from /home/green/git/lustre-release/lustre oleg220-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg220-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg220-server: detected 4 online CPUs by sysfs oleg220-server: MODOPTS_LIBCFS= oleg220-server: Force libcfs to create 2 CPU partitions Iteration 35 Starting client: oleg220-client.virtnet: -o user_xattr,flock 192.168.202.120@tcp:/lustre /mnt/lustre Starting client: oleg220-client.virtnet: -o user_xattr,flock 192.168.202.120@tcp:/lustre /mnt/lustre2 192.168.202.120@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.120@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg220-client.virtnet /mnt/lustre (opts:) Stopping client oleg220-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg220-server' oleg220-server: oleg220-server.virtnet: executing load_modules_local oleg220-server: Loading modules from /home/green/git/lustre-release/lustre oleg220-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg220-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg220-server: detected 4 online CPUs by sysfs oleg220-server: MODOPTS_LIBCFS= oleg220-server: Force libcfs to create 2 CPU partitions Iteration 36 Starting client: oleg220-client.virtnet: -o user_xattr,flock 192.168.202.120@tcp:/lustre /mnt/lustre Starting client: oleg220-client.virtnet: -o user_xattr,flock 192.168.202.120@tcp:/lustre /mnt/lustre2 192.168.202.120@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.120@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg220-client.virtnet /mnt/lustre (opts:) Stopping client oleg220-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg220-server' oleg220-server: oleg220-server.virtnet: executing load_modules_local oleg220-server: Loading modules from /home/green/git/lustre-release/lustre oleg220-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg220-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg220-server: detected 4 online CPUs by sysfs oleg220-server: MODOPTS_LIBCFS= oleg220-server: Force libcfs to create 2 CPU partitions Iteration 37 Starting client: oleg220-client.virtnet: -o user_xattr,flock 192.168.202.120@tcp:/lustre /mnt/lustre Starting client: oleg220-client.virtnet: -o user_xattr,flock 192.168.202.120@tcp:/lustre /mnt/lustre2 192.168.202.120@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.120@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg220-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg220-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg220-server' oleg220-server: oleg220-server.virtnet: executing load_modules_local oleg220-server: Loading modules from /home/green/git/lustre-release/lustre oleg220-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg220-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg220-server: detected 4 online CPUs by sysfs oleg220-server: MODOPTS_LIBCFS= oleg220-server: Force libcfs to create 2 CPU partitions Iteration 38 Starting client: oleg220-client.virtnet: -o user_xattr,flock 192.168.202.120@tcp:/lustre /mnt/lustre Starting client: oleg220-client.virtnet: -o user_xattr,flock 192.168.202.120@tcp:/lustre /mnt/lustre2 192.168.202.120@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.120@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg220-client.virtnet /mnt/lustre (opts:) Stopping client oleg220-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg220-server' oleg220-server: oleg220-server.virtnet: executing load_modules_local oleg220-server: Loading modules from /home/green/git/lustre-release/lustre oleg220-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg220-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg220-server: detected 4 online CPUs by sysfs oleg220-server: MODOPTS_LIBCFS= oleg220-server: Force libcfs to create 2 CPU partitions Iteration 39 Starting client: oleg220-client.virtnet: -o user_xattr,flock 192.168.202.120@tcp:/lustre /mnt/lustre Starting client: oleg220-client.virtnet: -o user_xattr,flock 192.168.202.120@tcp:/lustre /mnt/lustre2 192.168.202.120@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.120@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg220-client.virtnet /mnt/lustre (opts:) Stopping client oleg220-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg220-server' oleg220-server: oleg220-server.virtnet: executing load_modules_local oleg220-server: Loading modules from /home/green/git/lustre-release/lustre oleg220-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg220-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg220-server: detected 4 online CPUs by sysfs oleg220-server: MODOPTS_LIBCFS= oleg220-server: Force libcfs to create 2 CPU partitions Iteration 40 Starting client: oleg220-client.virtnet: -o user_xattr,flock 192.168.202.120@tcp:/lustre /mnt/lustre Starting client: oleg220-client.virtnet: -o user_xattr,flock 192.168.202.120@tcp:/lustre /mnt/lustre2 192.168.202.120@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.120@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg220-client.virtnet /mnt/lustre (opts:) Stopping client oleg220-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg220-server' oleg220-server: oleg220-server.virtnet: executing load_modules_local oleg220-server: Loading modules from /home/green/git/lustre-release/lustre oleg220-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg220-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg220-server: detected 4 online CPUs by sysfs oleg220-server: MODOPTS_LIBCFS= oleg220-server: Force libcfs to create 2 CPU partitions Iteration 41 Starting client: oleg220-client.virtnet: -o user_xattr,flock 192.168.202.120@tcp:/lustre /mnt/lustre Starting client: oleg220-client.virtnet: -o user_xattr,flock 192.168.202.120@tcp:/lustre /mnt/lustre2 192.168.202.120@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.120@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg220-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg220-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg220-server' oleg220-server: oleg220-server.virtnet: executing load_modules_local oleg220-server: Loading modules from /home/green/git/lustre-release/lustre oleg220-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg220-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg220-server: detected 4 online CPUs by sysfs oleg220-server: MODOPTS_LIBCFS= oleg220-server: Force libcfs to create 2 CPU partitions Iteration 42 Starting client: oleg220-client.virtnet: -o user_xattr,flock 192.168.202.120@tcp:/lustre /mnt/lustre Starting client: oleg220-client.virtnet: -o user_xattr,flock 192.168.202.120@tcp:/lustre /mnt/lustre2 192.168.202.120@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.120@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg220-client.virtnet /mnt/lustre (opts:) Stopping client oleg220-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg220-server' oleg220-server: oleg220-server.virtnet: executing load_modules_local oleg220-server: Loading modules from /home/green/git/lustre-release/lustre oleg220-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg220-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg220-server: detected 4 online CPUs by sysfs oleg220-server: MODOPTS_LIBCFS= oleg220-server: Force libcfs to create 2 CPU partitions Iteration 43 Starting client: oleg220-client.virtnet: -o user_xattr,flock 192.168.202.120@tcp:/lustre /mnt/lustre Starting client: oleg220-client.virtnet: -o user_xattr,flock 192.168.202.120@tcp:/lustre /mnt/lustre2 192.168.202.120@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.120@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg220-client.virtnet /mnt/lustre (opts:) Stopping client oleg220-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg220-server' oleg220-server: oleg220-server.virtnet: executing load_modules_local oleg220-server: Loading modules from /home/green/git/lustre-release/lustre oleg220-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg220-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg220-server: detected 4 online CPUs by sysfs oleg220-server: MODOPTS_LIBCFS= oleg220-server: Force libcfs to create 2 CPU partitions Iteration 44 Starting client: oleg220-client.virtnet: -o user_xattr,flock 192.168.202.120@tcp:/lustre /mnt/lustre Starting client: oleg220-client.virtnet: -o user_xattr,flock 192.168.202.120@tcp:/lustre /mnt/lustre2 192.168.202.120@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.120@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg220-client.virtnet /mnt/lustre (opts:) Stopping client oleg220-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg220-server' oleg220-server: oleg220-server.virtnet: executing load_modules_local oleg220-server: Loading modules from /home/green/git/lustre-release/lustre oleg220-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg220-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg220-server: detected 4 online CPUs by sysfs oleg220-server: MODOPTS_LIBCFS= oleg220-server: Force libcfs to create 2 CPU partitions Iteration 45 Starting client: oleg220-client.virtnet: -o user_xattr,flock 192.168.202.120@tcp:/lustre /mnt/lustre Starting client: oleg220-client.virtnet: -o user_xattr,flock 192.168.202.120@tcp:/lustre /mnt/lustre2 192.168.202.120@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.120@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg220-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg220-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg220-server' oleg220-server: oleg220-server.virtnet: executing load_modules_local oleg220-server: Loading modules from /home/green/git/lustre-release/lustre oleg220-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg220-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg220-server: detected 4 online CPUs by sysfs oleg220-server: MODOPTS_LIBCFS= oleg220-server: Force libcfs to create 2 CPU partitions Iteration 46 Starting client: oleg220-client.virtnet: -o user_xattr,flock 192.168.202.120@tcp:/lustre /mnt/lustre Starting client: oleg220-client.virtnet: -o user_xattr,flock 192.168.202.120@tcp:/lustre /mnt/lustre2 192.168.202.120@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.120@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg220-client.virtnet /mnt/lustre (opts:) Stopping client oleg220-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg220-server' oleg220-server: oleg220-server.virtnet: executing load_modules_local oleg220-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg220-server: Loading modules from /home/green/git/lustre-release/lustre oleg220-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg220-server: detected 4 online CPUs by sysfs oleg220-server: MODOPTS_LIBCFS= oleg220-server: Force libcfs to create 2 CPU partitions Iteration 47 Starting client: oleg220-client.virtnet: -o user_xattr,flock 192.168.202.120@tcp:/lustre /mnt/lustre2 Starting client: oleg220-client.virtnet: -o user_xattr,flock 192.168.202.120@tcp:/lustre /mnt/lustre 192.168.202.120@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.120@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg220-client.virtnet /mnt/lustre (opts:) Stopping client oleg220-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg220-server' oleg220-server: oleg220-server.virtnet: executing load_modules_local oleg220-server: Loading modules from /home/green/git/lustre-release/lustre oleg220-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg220-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg220-server: detected 4 online CPUs by sysfs oleg220-server: MODOPTS_LIBCFS= oleg220-server: Force libcfs to create 2 CPU partitions Iteration 48 Starting client: oleg220-client.virtnet: -o user_xattr,flock 192.168.202.120@tcp:/lustre /mnt/lustre Starting client: oleg220-client.virtnet: -o user_xattr,flock 192.168.202.120@tcp:/lustre /mnt/lustre2 192.168.202.120@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.120@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg220-client.virtnet /mnt/lustre (opts:) Stopping client oleg220-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg220-server' oleg220-server: oleg220-server.virtnet: executing load_modules_local oleg220-server: Loading modules from /home/green/git/lustre-release/lustre oleg220-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg220-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg220-server: detected 4 online CPUs by sysfs oleg220-server: MODOPTS_LIBCFS= oleg220-server: Force libcfs to create 2 CPU partitions Iteration 49 Starting client: oleg220-client.virtnet: -o user_xattr,flock 192.168.202.120@tcp:/lustre /mnt/lustre Starting client: oleg220-client.virtnet: -o user_xattr,flock 192.168.202.120@tcp:/lustre /mnt/lustre2 192.168.202.120@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.120@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg220-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg220-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg220-server' oleg220-server: oleg220-server.virtnet: executing load_modules_local oleg220-server: Loading modules from /home/green/git/lustre-release/lustre oleg220-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg220-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg220-server: detected 4 online CPUs by sysfs oleg220-server: MODOPTS_LIBCFS= oleg220-server: Force libcfs to create 2 CPU partitions Iteration 50 Starting client: oleg220-client.virtnet: -o user_xattr,flock 192.168.202.120@tcp:/lustre /mnt/lustre Starting client: oleg220-client.virtnet: -o user_xattr,flock 192.168.202.120@tcp:/lustre /mnt/lustre2 192.168.202.120@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.120@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg220-client.virtnet /mnt/lustre (opts:) Stopping client oleg220-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg220-server' oleg220-server: oleg220-server.virtnet: executing load_modules_local oleg220-server: Loading modules from /home/green/git/lustre-release/lustre oleg220-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg220-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg220-server: detected 4 online CPUs by sysfs oleg220-server: MODOPTS_LIBCFS= oleg220-server: Force libcfs to create 2 CPU partitions Starting client: oleg220-client.virtnet: -o user_xattr,flock 192.168.202.120@tcp:/lustre /mnt/lustre Starting client: oleg220-client.virtnet: -o user_xattr,flock 192.168.202.120@tcp:/lustre /mnt/lustre2 PASS 109 (506s) == sanityn test 110: do not grant another lock on resend ========================================================== 15:38:56 (1758829136) fail_loc=0x525 fail_val=3 fail_loc=0x534 File: /mnt/lustre/f110.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205306056707 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-09-25 15:38:57.000000000 -0400 Modify: 2025-09-25 15:38:57.000000000 -0400 Change: 2025-09-25 15:38:57.000000000 -0400 Birth: 2025-09-25 15:38:57.000000000 -0400 fail_loc=0 fail_val=0 Connected clients: oleg220-client.virtnet oleg220-client.virtnet PASS 110 (81s) == sanityn test 111: A racy rename/link an open file should not cause fs corruption ========================================================== 15:40:17 (1758829217) SKIP: sanityn test_111 needs >= 2 MDTs SKIP 111 (1s) == sanityn test 112: update max-inherit in default LMV === 15:40:19 (1758829219) SKIP: sanityn test_112 We need at least 2 MDTs for this test SKIP 112 (0s) == sanityn test 113: check servers of specified fs ======= 15:40:20 (1758829220) lustre-OST0000-osc-ffff8c2888b9a000 active. lustre-OST0001-osc-ffff8c2888b9a000 active. lustre-OST0000-osc-ffff8c2888b9a000 active. lustre-OST0001-osc-ffff8c2888b9a000 active. MGC192.168.202.120@tcp active. PASS 113 (3s) == sanityn test 114: implicit default LMV inherit ======== 15:40:22 (1758829222) SKIP: sanityn test_114 We need at least 2 MDTs for this test SKIP 114 (1s) == sanityn test 115: ldiskfs doesn't check direntry for uniqueness ========================================================== 15:40:23 (1758829223) SKIP: sanityn test_115 ldiskfs only test SKIP 115 (1s) == sanityn test 116: DNE: Set default LMV layout from a remote client ========================================================== 15:40:24 (1758829224) SKIP: sanityn test_116 needs >= 2 MDTs SKIP 116 (1s) == sanityn test 200: service remains healthy while able to process request ========================================================== 15:40:25 (1758829225) at_max=10 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=20 fail_loc=0x1e1 fail_val=20 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.084 s, 0.2 kB/s 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0867 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0899 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1017 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0876 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0859 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0868 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0928 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0902 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1471 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.167 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1317 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1706 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1467 s, 0.1 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1672 s, 0.1 kB/s4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1772 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1435 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1659 s, 0.1 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2019 s, 0.1 kB/s4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2476 s, 0.1 kB/s 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2001 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2441 s, 0.1 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2206 s, 0.1 kB/s 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2165 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2226 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2279 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2686 s, 0.1 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 80.2775 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 80.2757 s, 0.1 kB/s 4096 bytes (4.1 kB, 4.0 KiB) copied, 80.3204 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 80.3552 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 80.3226 s, 0.1 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 80.3246 s, 0.1 kB/s 4096 bytes (4.1 kB, 4.0 KiB) copied, 80.319 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 80.3616 s, 0.1 kB/s1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 80.3451 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 100.407 s, 0.0 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 100.388 s, 0.0 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 100.403 s, 0.0 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 100.417 s, 0.0 kB/s at_max=600 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=116 oleg220-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8c288317e800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8c288317e800.ost_server_uuid in IDLE state after 0 sec PASS 200 (106s) cleanup: ====================================================== == sanityn test complete, duration 5605 sec ============== 15:42:11 (1758829331) === sanityn: start cleanup 15:42:12 (1758829332) === Stopping clients: oleg220-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg220-client.virtnet /mnt/lustre2 opts: === sanityn: finish cleanup 15:43:00 (1758829380) ===