-----============= acceptance-small: sanityn ============----- Thu Jul 30 01:17:26 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg441-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanityn.*ex': No such file or directory excepting tests: 27 40a skipping tests SLOW=no: 33a === sanityn: start setup 01:18:05 (1785388685) === Starting client oleg441-client.virtnet: -o user_xattr,flock 192.168.204.141@tcp:/lustre /mnt/lustre2 Mount client oleg441-client.virtnet: mount -t lustre -o user_xattr,flock 192.168.204.141@tcp:/lustre /mnt/lustre2 Started clients oleg441-client.virtnet: 192.168.204.141@tcp:/lustre on /mnt/lustre2 type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) oleg441-client.virtnet: executing check_config_client /mnt/lustre oleg441-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg441-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8be707e69000.idle_timeout=debug osc.lustre-OST0000-osc-ffff8be710a2d000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8be707e69000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8be710a2d000.idle_timeout=debug disable quota as required oleg441-server: oleg441-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanityn: finish setup 01:18:54 (1785388734) === 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0705145 s, 14.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7625] == sanityn test 0: do_node_vp() and do_facet_vp() do the right thing ========================================================== 01:18:57 (1785388737) PASS 0 (10s) == sanityn test 1: Check attribute updates on 2 mount points ========================================================== 01:19:08 (1785388748) /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0777 OK /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0666 OK /mnt/lustre/f1.sanityn: absent OK PASS 1 (9s) == sanityn test 2a: check cached attribute updates on 2 mtpt's ================================================================== 01:19:17 (1785388757) -rw-r--r-- 1 root root 0 Jul 30 01:19 /mnt/lustre2/f2a /mnt/lustre/f2a has type file OK /mnt/lustre/f2a has perms 0777 OK PASS 2a (8s) == sanityn test 2b: check cached attribute updates on 2 mtpt's ================================================================== 01:19:24 (1785388764) -rw-r--r-- 1 root root 0 Jul 30 01:19 /mnt/lustre2/f2b /mnt/lustre2/f2b has type file OK /mnt/lustre2/f2b has perms 0777 OK PASS 2b (8s) == sanityn test 2c: check cached attribute updates on 2 mtpt's root ============================================================= 01:19:32 (1785388772) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0777 OK PASS 2c (8s) == sanityn test 2d: check cached attribute updates on 2 mtpt's root ============================================================= 01:19:41 (1785388781) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0755 OK PASS 2d (12s) == sanityn test 2e: check chmod on root is propagated to others ========================================================== 01:19:53 (1785388793) total 1 -rwxrwxrwx 1 root root 0 Jul 30 01:19 f2a -rwxrwxrwx 1 root root 0 Jul 30 01:19 f2b total 1 -rwxrwxrwx 1 root root 0 Jul 30 01:19 f2a -rwxrwxrwx 1 root root 0 Jul 30 01:19 f2b running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre2/f2e.sanityn] [count=1] 1+0 records in 1+0 records out 512 bytes copied, 0.0312921 s, 16.4 kB/s PASS 2e (9s) == sanityn test 2f: check attr/owner updates on DNE with 2 mtpt's ========================================================== 01:20:02 (1785388802) SKIP: sanityn test_2f needs >= 2 MDTs SKIP 2f (4s) == sanityn test 2g: check blocks update on sync write ==== 01:20:07 (1785388807) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.567978 s, 3.7 MB/s /mnt/lustre/f2g.sanityn has 4103 blocks /mnt/lustre2/f2g.sanityn has 4103 blocks PASS 2g (11s) == sanityn test 3: symlink on one mtpt, readlink on another ===================================================================== 01:20:18 (1785388818) PASS 3 (11s) == sanityn test 4: fstat validation on multiple mount points ==================================================================== 01:20:30 (1785388830) PASS 4 (12s) == sanityn test 5: create a file on one mount, truncate it on the other ========================================================== 01:20:42 (1785388842) /mnt/lustre/f5 has type file OK /mnt/lustre/f5 has size 100 OK PASS 5 (10s) == sanityn test 6: remove of open file on other node ============================================================================ 01:20:52 (1785388852) opening writing unlinking /mnt/lustre2/f6.sanityn accessing (1) seeking (1) accessing (2) fstat... reading comparing data truncating seeking (2) writing again seeking (3) reading again comparing data again closing SUCCESS - goto beer PASS 6 (9s) == sanityn test 7: remove of open directory on other node ======================================================================= 01:21:01 (1785388861) creating directory /mnt/lustre/d7 opening directory unlinking /mnt/lustre/d7 Ok, everything goes well. PASS 7 (8s) == sanityn test 8: remove of open special file on other node ==================================================================== 01:21:09 (1785388869) creating special file /mnt/lustre/f8.sanityn opening file unlinking /mnt/lustre/f8.sanityn Ok, everything goes well. PASS 8 (8s) == sanityn test 9a: append of file with sub-page size on multiple mounts ========================================================== 01:21:17 (1785388877) PASS 9a (11s) == sanityn test 9b: append to striped sparse file ======== 01:21:28 (1785388888) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0983905 s, 10.7 MB/s 3+0 records in 3+0 records out 3 bytes copied, 0.0375414 s, 0.1 kB/s Data read (expecting 'foo'): foo PASS 9b (9s) == sanityn test 10a: write of file with sub-page size on multiple mounts ========================================================== 01:21:38 (1785388898) 1+0 records in 1+0 records out 1 byte copied, 0.00919868 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0151174 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0246687 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0266259 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00693162 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0190929 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0198649 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0184424 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0177224 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0252688 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0273462 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0238563 s, 0.0 kB/s PASS 10a (9s) == sanityn test 10b: write of file with sub-page size on multiple mounts ========================================================== 01:21:47 (1785388907) 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.0191548 s, 160 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0742576 s, 55.2 kB/s 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.00022266 s, 13.8 MB/s PASS 10b (9s) == sanityn test 11: execution of file opened for write should return error ============================================================== 01:21:56 (1785388916) multiop /mnt/lustre/d11/f vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7625 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 298: /mnt/lustre2/d11/f: Text file busy PASS 11 (10s) == sanityn test 12: test lock ordering (link, stat, unlink) ========================================================== 01:22:07 (1785388927) warning: '-runtime' deprecated, use '-t runtime' instead - link 240 (time 1785388940.36 total 10.02 last 23.95) - link 558 (time 1785388950.38 total 20.04 last 31.73) - link 825 (time 1785388960.41 total 30.08 last 26.61) - link 1047 (time 1785388970.42 total 40.08 last 22.19) - link 1311 (time 1785388980.44 total 50.10 last 26.34) - link 1564 (time 1785388990.47 total 60.14 last 25.21) - link 1775 (time 1785389000.48 total 70.14 last 21.08) - link 2070 (time 1785389010.59 total 80.25 last 29.20) - link 2280 (time 1785389020.66 total 90.32 last 20.85) total: 2514 link in 100.01 seconds: 25.14 ops/second using seed 551376909 running for 100 seconds - stat 3115 (time 1785388942 ; total 11 ; last 11) - stat 8498 (time 1785388953 ; total 22 ; last 11) - stat 10000 (time 1785388958 ; total 27 ; last 5) - stat 13095 (time 1785388969 ; total 38 ; last 11) - stat 16368 (time 1785388980 ; total 49 ; last 11) - stat 18841 (time 1785388991 ; total 60 ; last 11) - stat 20000 (time 1785388998 ; total 67 ; last 7) - stat 21792 (time 1785389009 ; total 78 ; last 11) - stat 23310 (time 1785389020 ; total 89 ; last 11) - stat 24957 (time 1785389031 ; total 100 ; last 11) total: 24958 stats in 100 seconds: 249.580002 stats/second - unlinked 0 (time 1785388981 ; total 0 ; last 0) unlink(/mnt/lustre2/lockdir/lockfile2514) error: No such file or directory total: 2513 unlinks in 90 seconds: 27.922222 unlinks/second /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 77: kill: (21433) - No such process /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 78: kill: (21435) - No such process PASS 12 (155s) == sanityn test 13: test directory page revocation ======= 01:24:43 (1785389083) f13 f13 PASS 13 (19s) == sanityn test 14aa: execution of file open for write returns -ETXTBSY ========================================================== 01:25:03 (1785389103) multiop /mnt/lustre/d14aa.sanityn/f14aa.sanityn vOw_c TMPPIPE=/tmp/multiop_open_wait_pipe.7625 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 333: /mnt/lustre2/d14aa.sanityn/f14aa.sanityn: Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5070: 22628 Terminated $MULTIOP_PROG $FILE v$ARGS > $TMPPIPE (wd: ~) PASS 14aa (10s) == sanityn test 14ab: open(RDWR) of executing file returns -ETXTBSY ========================================================== 01:25:13 (1785389113) open(O_RDWR|O_CREAT): Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5070: 23209 Terminated $DIR1/$tdir/sleep 60 (wd: ~) PASS 14ab (11s) == sanityn test 14b: truncate of executing file returns -ETXTBSY ================================================================ 01:25:25 (1785389125) truncate: cannot truncate '/mnt/lustre2/d14b.sanityn/sleep' to length 60: Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 348: 23788 Terminated $DIR1/$tdir/sleep 60 PASS 14b (10s) == sanityn test 14c: open(O_TRUNC) of executing file return -ETXTBSY ============================================================ 01:25:35 (1785389135) cp: cannot create regular file '/mnt/lustre2/d14c.sanityn/sleep': Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 360: 24370 Terminated $DIR1/$tdir/sleep 60 PASS 14c (12s) == sanityn test 14d: chmod of executing file is still possible ================================================================== 01:25:47 (1785389147) chmod /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 371: 24955 Terminated $DIR1/$tdir/sleep 60 PASS 14d (13s) == sanityn test 15: test out-of-space with multiple writers ===================================================================== 01:26:00 (1785389160) PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin: MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky sleep 5 for ZFS MDS Waiting for MDT destroys to complete fallocate on zfs doesn't consume space STRIPECOUNT=2 ORIGFREE=7531520 MAXFREE=800000 SKIP: oos2 test_15 oos2.sh: 7531520KB free > 800000KB, increase MAXFREE (or reduce fs size) sleep 5 for ZFS MDS Waiting for MDT destroys to complete SKIP 15 (61s) == sanityn test 16a: 500 iterations of dual-mount fsx ==== 01:27:01 (1785389221) fallocate on zfs doesn't consume space fallocate not supported 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.551424 s, 15.2 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.312684 s, 26.8 MB/s Chance of close/open is 1 in 50 Seed set to 9226 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read truncating to largest ever: 0x83addb 000002[0] 1785389227.119643 trunc from 00000000 to 0x83adda (0x83addb bytes) 000003[0] 1785389227.243011 mapwrite 0x2d3a8b thru 0x2e1f82 (0xe4f8 bytes) 000005[0] 1785389227.446259 mapread 0x3a6f61 thru 0x3aee88 (0x7f28 bytes) 000006[0] 1785389227.480922 mapread 0x48bf86 thru 0x49810b (0xc186 bytes) 000013[1] 1785389227.760556 write 0x25efa4 thru 0x265512 (0x656f bytes) 000014[0] 1785389227.804992 mapread 0x71077e thru 0x720425 (0xfca8 bytes) 000017[1] 1785389227.888219 write 0x57f6c7 thru 0x58c89a (0xd1d4 bytes) 000018[0] 1785389227.922695 write 0x267c77 thru 0x268daf (0x1139 bytes) 000020[0] 1785389228.057078 mapread 0x45551e thru 0x457197 (0x1c7a bytes) 000022[1] 1785389228.322084 read 0x13cefa thru 0x14a83c (0xd943 bytes) 000023[0] 1785389228.358390 mapread 0x4e2ede thru 0x4ed1b0 (0xa2d3 bytes) 000024[0] 1785389228.381967 read 0x11dd43 thru 0x127bf9 (0x9eb7 bytes) 000026[1] 1785389228.394473 write 0x270d57 thru 0x2794b7 (0x8761 bytes) 000029[0] 1785389228.433232 read 0x05359a thru 0x05e915 (0xb37c bytes) 000032[1] 1785389228.509537 mapwrite 0x21a9a3 thru 0x228afd (0xe15b bytes) 000033[1] 1785389228.762618 trunc from 0x83addb to 0x242964 (0x5f8476 bytes) 000040[0] 1785389229.023104 mapwrite 0x5adaaf thru 0x5b331e (0x5870 bytes) 000042[0] 1785389229.433953 mapread 0x1f0325 thru 0x1ff42a (0xf106 bytes) 000043[0] 1785389229.513766 write 0x53b089 thru 0x545b48 (0xaac0 bytes) 000045[1] 1785389229.527167 mapwrite 0x96b060 thru 0x977341 (0xc2e2 bytes) 000046[0] 1785389229.863769 mapread 0x269fe3 thru 0x27864a (0xe668 bytes) 000050[1] 1785389230.092578 mapread 0x08fc43 thru 0x09ca9b (0xce59 bytes) 000055[0] 1785389230.213014 read 0x06481e thru 0x0660f3 (0x18d6 bytes) 000058[0] 1785389230.225242 mapread 0x8d8b30 thru 0x8df4d1 (0x69a2 bytes) 000059[1] 1785389230.300611 mapwrite 0x2056ef thru 0x20f904 (0xa216 bytes) 000060[1] 1785389230.501226 read 0x5ddd0f thru 0x5e413e (0x6430 bytes) 000062[1] 1785389230.607606 mapwrite 0x4ef20c thru 0x4f9cbd (0xaab2 bytes) 000066[1] 1785389230.921545 trunc from 0x977342 to 0x7c8fbf (0x1ae382 bytes) 000071[0] 1785389230.965205 trunc from 0x7c8fc0 to 0x218766 (0x5b0859 bytes) 000074[1] 1785389231.092023 write 0x4e6e72 thru 0x4e9a19 (0x2ba8 bytes) 000076[0] 1785389231.222444 write 0x011ff4 thru 0x012d23 (0xd30 bytes) 000080[1] 1785389231.330399 mapread 0x4e85dc thru 0x4e9a19 (0x143e bytes) 000083[1] 1785389231.445466 mapread 0x120c5e thru 0x12735c (0x66ff bytes) 000084[0] 1785389231.588405 write 0x9a2308 thru 0x9adfdc (0xbcd5 bytes) 000085[0] 1785389231.647034 trunc from 0x9adfdd to 0x242476 (0x76bb66 bytes) 000086[0] 1785389231.758759 mapread 0x043166 thru 0x04521b (0x20b6 bytes) 000088[1] 1785389231.822357 trunc from 0x242477 to 0x3474a5 (0x10502f bytes) 000091[0] 1785389232.063385 write 0x0c4a3b thru 0x0c5f25 (0x14eb bytes) 000092[1] 1785389232.117570 mapread 0x25259a thru 0x25dc2f (0xb696 bytes) 000094[1] 1785389232.256252 mapwrite 0x75ace6 thru 0x760603 (0x591e bytes) 000095[0] 1785389232.352704 read 0x47a0c0 thru 0x483640 (0x9581 bytes) 000096[0] 1785389232.374437 mapwrite 0x2c03c9 thru 0x2c9fa5 (0x9bdd bytes) 000097[1] 1785389232.443727 mapwrite 0x3cfb51 thru 0x3d8639 (0x8ae9 bytes) 000100[1] 1785389232.547828 read 0x263373 thru 0x2696db (0x6369 bytes) 000101[0] 1785389232.567201 read 0x5f33be thru 0x5f60f7 (0x2d3a bytes) 000104[0] 1785389232.619068 read 0x68ca33 thru 0x68f8c4 (0x2e92 bytes) 000106[0] 1785389232.640906 mapread 0x178c6b thru 0x184979 (0xbd0f bytes) 000112[0] 1785389232.708382 mapread 0x5e4e7d thru 0x5e8a86 (0x3c0a bytes) 000113[0] 1785389232.730797 mapwrite 0x462789 thru 0x4643ea (0x1c62 bytes) 000115[0] 1785389232.813820 trunc from 0x760604 to 0x32d964 (0x432c9f bytes) 000116[1] 1785389232.844922 mapwrite 0x346354 thru 0x352f4c (0xcbf9 bytes) 000117[1] 1785389232.927841 trunc from 0x352f4d to 0x032436 (0x320b16 bytes) 000119[1] 1785389232.962295 write 0x63303f thru 0x634c0b (0x1bcd bytes) 000120[0] 1785389232.966534 write 0x6838d6 thru 0x68df8b (0xa6b6 bytes) 000124[0] 1785389233.024605 write 0x66b975 thru 0x67b5d3 (0xfc5f bytes) 000126[0] 1785389233.058846 trunc from 0x68df8c to 0x7d272a (0x14479f bytes) 000127[1] 1785389233.088968 write 0x15028d thru 0x15b9c0 (0xb734 bytes) 000131[0] 1785389233.150870 mapwrite 0x0a3534 thru 0x0b1ff2 (0xeabf bytes) 000132[1] 1785389233.243213 mapwrite 0x6d0841 thru 0x6df5b7 (0xed77 bytes) 000133[1] 1785389233.383622 mapwrite 0x5c00b0 thru 0x5cb057 (0xafa8 bytes) 000134[1] 1785389233.485122 trunc from 0x7d272b to 0x67544c (0x15d2de bytes) 000136[0] 1785389233.582755 read 0x1f5e7d thru 0x1fe0d9 (0x825d bytes) 000139[0] 1785389233.629840 trunc from 0x67544d to 0x180e71 (0x4f45db bytes) 000140[0] 1785389233.679211 mapwrite 0x8c9159 thru 0x8cbbd6 (0x2a7e bytes) 000141[0] 1785389233.750320 mapwrite 0x6c029e thru 0x6cad63 (0xaac6 bytes) 000143[1] 1785389233.840292 write 0x300e24 thru 0x30f0bc (0xe299 bytes) 000145[0] 1785389233.881717 mapread 0x558bd7 thru 0x55f145 (0x656f bytes) truncating to largest ever: 0x83f096 000146[1] 1785389233.911306 trunc from 0x8cbbd7 to 0x83f095 (0x8cb41 bytes) 000147[1] 1785389233.939910 mapread 0x737516 thru 0x742477 (0xaf62 bytes) 000148[1] 1785389234.021653 read 0x372c75 thru 0x379b0f (0x6e9b bytes) 000152[1] 1785389234.224260 write 0x9de7d1 thru 0x9e95a7 (0xadd7 bytes) 000153[0] 1785389234.233544 write 0x53dae6 thru 0x53e3ba (0x8d5 bytes) 000159[0] 1785389234.410380 write 0x19689b thru 0x19e7cb (0x7f31 bytes) 000161[1] 1785389234.481204 read 0x55d027 thru 0x55ff18 (0x2ef2 bytes) 000163[1] 1785389234.524251 write 0x983365 thru 0x989fb8 (0x6c54 bytes) 000164[0] 1785389234.547822 read 0x458804 thru 0x45c072 (0x386f bytes) 000167[0] 1785389234.585831 read 0x2207e3 thru 0x2299c0 (0x91de bytes) 000168[1] 1785389234.612780 mapwrite 0x0769ec thru 0x07b5a6 (0x4bbb bytes) 000173[0] 1785389234.724525 trunc from 0x9e95a8 to 0x3eacef (0x5fe8b8 bytes) 000175[0] 1785389234.867768 read 0x392599 thru 0x3962a5 (0x3d0d bytes) 000179[1] 1785389235.044253 mapwrite 0x2be2ff thru 0x2c97bb (0xb4bd bytes) 000180[1] 1785389235.339372 mapwrite 0x7ca17c thru 0x7d4435 (0xa2ba bytes) 000181[0] 1785389235.693309 write 0x054071 thru 0x055399 (0x1329 bytes) 000183[0] 1785389235.822400 write 0x9d24f2 thru 0x9d457b (0x208a bytes) 000186[0] 1785389235.917248 trunc from 0x9d457c to 0x313d89 (0x6c07f2 bytes) 000189[1] 1785389236.007597 read 0x278e41 thru 0x28515c (0xc31c bytes) 000191[1] 1785389236.162886 read 0x02e035 thru 0x03680f (0x87db bytes) 000192[0] 1785389236.267495 trunc from 0x313d8a to 0x0d6ac6 (0x23d2c3 bytes) 000196[0] 1785389236.527009 trunc from 0x0d6ac7 to 0x235a16 (0x15ef50 bytes) 000197[1] 1785389236.633461 write 0x8cda33 thru 0x8d6c28 (0x91f6 bytes) 000200[0] 1785389236.773221 mapwrite 0x185c7d thru 0x1918b4 (0xbc38 bytes) 000205[1] 1785389237.214016 mapread 0x16de36 thru 0x17506a (0x7235 bytes) 000213[0] 1785389237.356400 read 0x746706 thru 0x754fed (0xe8e8 bytes) 000214[1] 1785389237.373298 write 0x9d5f80 thru 0x9dd0fc (0x717d bytes) 000215[0] 1785389237.404309 write 0x14cb0b thru 0x155749 (0x8c3f bytes) 000216[0] 1785389237.452757 mapread 0x98a68c thru 0x98d56f (0x2ee4 bytes) 000218[0] 1785389237.498804 mapwrite 0x3018d1 thru 0x311386 (0xfab6 bytes) 000219[1] 1785389237.587030 mapwrite 0x9406d4 thru 0x941f4c (0x1879 bytes) 000220[0] 1785389237.658667 read 0x9d8c9c thru 0x9dd0fc (0x4461 bytes) truncating to largest ever: 0x93012e 000221[0] 1785389237.679687 trunc from 0x9dd0fd to 0x93012d (0xacfcf bytes) 000222[0] 1785389237.706261 trunc from 0x93012e to 0x54dfeb (0x3e2142 bytes) 000223[0] 1785389237.721452 mapread 0x00b0b4 thru 0x016629 (0xb576 bytes) 000226[1] 1785389237.760279 trunc from 0x54dfec to 0x31c757 (0x231894 bytes) 000227[0] 1785389237.800740 write 0x2dc452 thru 0x2ea6d1 (0xe280 bytes) 000228[0] 1785389237.830683 write 0x77e7a5 thru 0x78c208 (0xda64 bytes) 000229[1] 1785389237.859884 write 0x2dc7a5 thru 0x2e6e9d (0xa6f9 bytes) 000232[0] 1785389237.915564 write 0x0c42fe thru 0x0d265e (0xe361 bytes) 000234[1] 1785389237.956685 write 0x059186 thru 0x059945 (0x7c0 bytes) 000235[1] 1785389237.986871 mapwrite 0x202251 thru 0x20bf9f (0x9d4f bytes) 000236[0] 1785389238.093345 mapwrite 0x58d636 thru 0x59382c (0x61f7 bytes) 000237[1] 1785389238.169013 read 0x4964b3 thru 0x4a0bd1 (0xa71f bytes) 000238[1] 1785389238.197583 read 0x262f79 thru 0x26eded (0xbe75 bytes) 000239[0] 1785389238.204380 read 0x669542 thru 0x669e08 (0x8c7 bytes) 000240[1] 1785389238.209636 mapwrite 0x8aeffe thru 0x8b68b6 (0x78b9 bytes) 000241[1] 1785389238.366823 mapwrite 0x79189a thru 0x79e168 (0xc8cf bytes) 000242[0] 1785389238.478762 read 0x80f4a7 thru 0x811725 (0x227f bytes) 000243[0] 1785389238.529860 read 0x5a0faf thru 0x5a7524 (0x6576 bytes) 000249[1] 1785389238.566529 write 0x5ce836 thru 0x5d591f (0x70ea bytes) 000252[0] 1785389238.609675 mapwrite 0x092b26 thru 0x09ba17 (0x8ef2 bytes) 000259[1] 1785389238.948932 mapread 0x8a6bf3 thru 0x8ab4c0 (0x48ce bytes) 000260[1] 1785389239.094990 trunc from 0x8b68b7 to 0x3b2a1d (0x503e99 bytes) 000263[1] 1785389239.241967 write 0x463427 thru 0x4642e8 (0xec2 bytes) 000267[0] 1785389239.283385 trunc from 0x4642e9 to 0x2ff914 (0x1649d4 bytes) 000268[1] 1785389239.400280 write 0x8b6ddd thru 0x8c2d53 (0xbf77 bytes) 000269[0] 1785389239.481467 mapread 0x442a49 thru 0x446018 (0x35d0 bytes) 000272[0] 1785389239.814487 trunc from 0x8c2d54 to 0x246615 (0x67c73e bytes) 000274[1] 1785389240.018213 read 0x10863a thru 0x1092c8 (0xc8f bytes) 000275[0] 1785389240.178778 mapread 0x18fccc thru 0x19fbfc (0xff31 bytes) 000276[1] 1785389240.269163 trunc from 0x246616 to 0x3271d5 (0xe0bc0 bytes) 000277[0] 1785389240.384303 write 0x029f37 thru 0x037f81 (0xe04b bytes) 000278[1] 1785389240.462222 mapread 0x10af74 thru 0x112cc3 (0x7d50 bytes) 000279[0] 1785389240.716280 read 0x11f7a5 thru 0x128567 (0x8dc3 bytes) 000280[0] 1785389240.860805 read 0x31381b thru 0x31ede5 (0xb5cb bytes) 000282[1] 1785389240.887743 mapwrite 0x4d2d29 thru 0x4d38f4 (0xbcc bytes) 000285[0] 1785389241.239849 mapwrite 0x9f7110 thru 0x9ff268 (0x8159 bytes) 000289[1] 1785389241.777611 mapwrite 0x0ec15f thru 0x0f9776 (0xd618 bytes) 000290[0] 1785389242.199070 mapread 0x7eea8b thru 0x7f1143 (0x26b9 bytes) 000291[0] 1785389242.241073 write 0x0c394b thru 0x0d2d27 (0xf3dd bytes) 000292[1] 1785389242.278107 read 0x2a9e55 thru 0x2b97d7 (0xf983 bytes) 000293[1] 1785389242.363448 mapwrite 0x65597c thru 0x65d683 (0x7d08 bytes) 000294[0] 1785389242.474918 read 0x739a19 thru 0x747fb5 (0xe59d bytes) 000298[0] 1785389242.506620 mapwrite 0x7a6928 thru 0x7a8ad9 (0x21b2 bytes) 000299[0] 1785389242.609540 trunc from 0x9ff269 to 0x731bf9 (0x2cd66f bytes) 000301[1] 1785389242.692084 trunc from 0x731bfa to 0x754728 (0x22b2f bytes) 000302[1] 1785389242.722491 read 0x352eb7 thru 0x3568ed (0x3a37 bytes) 000306[0] 1785389242.769949 write 0x543722 thru 0x54775e (0x403d bytes) 000308[1] 1785389242.833925 trunc from 0x754729 to 0x72e743 (0x25fe5 bytes) 000309[0] 1785389242.873121 mapread 0x110a73 thru 0x1163ed (0x597b bytes) 000312[1] 1785389243.026487 trunc from 0x72e744 to 0x17187d (0x5bcec6 bytes) 000313[1] 1785389243.102760 read 0x11cc67 thru 0x11e32f (0x16c9 bytes) 000315[0] 1785389243.165468 mapread 0x0f198d thru 0x0fab30 (0x91a4 bytes) 000317[1] 1785389243.232447 write 0x2393cc thru 0x24500b (0xbc40 bytes) 000318[0] 1785389243.248311 trunc from 0x24500c to 0x5eff9a (0x3aaf8f bytes) 000320[0] 1785389243.309653 read 0x15707d thru 0x161455 (0xa3d9 bytes) 000321[1] 1785389243.344090 write 0x62af92 thru 0x63936b (0xe3da bytes) 000322[0] 1785389243.368692 write 0x9851af thru 0x98b5fd (0x644f bytes) 000323[1] 1785389243.386054 mapread 0x236079 thru 0x23dd63 (0x7ceb bytes) 000325[1] 1785389243.573189 mapread 0x49915c thru 0x4a1175 (0x801a bytes) 000326[0] 1785389243.625058 mapread 0x868746 thru 0x869984 (0x123f bytes) 000328[1] 1785389243.684050 trunc from 0x98b5fe to 0x4afd21 (0x4db8dc bytes) 000329[1] 1785389243.807192 mapread 0x3f2822 thru 0x3fa3de (0x7bbd bytes) 000330[0] 1785389243.972698 mapread 0x420de2 thru 0x425f91 (0x51b0 bytes) truncating to largest ever: 0x9c6848 000332[0] 1785389244.133735 trunc from 0x4afd22 to 0x9c6847 (0x516b26 bytes) 000336[1] 1785389244.308631 mapwrite 0x098d5a thru 0x0a601f (0xd2c6 bytes) 000338[1] 1785389244.886168 mapread 0x45f9ac thru 0x46613b (0x6790 bytes) 000340[0] 1785389245.082471 trunc from 0x9c6848 to 0x9793ce (0x4d479 bytes) 000341[0] 1785389245.202781 mapwrite 0x93fa55 thru 0x9469a4 (0x6f50 bytes) 000342[0] 1785389245.408122 mapwrite 0x5b795e thru 0x5bd347 (0x59ea bytes) 000344[0] 1785389245.633487 read 0x165982 thru 0x16d6b2 (0x7d31 bytes) 000346[0] 1785389245.704027 read 0x37e996 thru 0x384f30 (0x659b bytes) 000349[0] 1785389245.901204 write 0x85db7b thru 0x85ed03 (0x1189 bytes) 000350[1] 1785389245.923785 mapwrite 0x3adbc4 thru 0x3b4324 (0x6761 bytes) 000352[1] 1785389246.297694 read 0x1a8634 thru 0x1b3262 (0xac2f bytes) 000355[0] 1785389246.499344 mapwrite 0x62a4e4 thru 0x62f3f4 (0x4f11 bytes) 000356[1] 1785389246.849724 trunc from 0x9793cf to 0x0e13b8 (0x898016 bytes) 000357[1] 1785389246.973846 read 0x034b5e thru 0x040436 (0xb8d9 bytes) 000359[1] 1785389247.118975 trunc from 0x0e13b9 to 0x54257f (0x4611c7 bytes) 000360[1] 1785389247.216868 mapread 0x454423 thru 0x45cad3 (0x86b1 bytes) 000361[1] 1785389247.343136 read 0x0aadc6 thru 0x0b3563 (0x879e bytes) 000362[0] 1785389247.415430 trunc from 0x542580 to 0x2a6d3b (0x29b844 bytes) 000363[1] 1785389247.559117 read 0x10c174 thru 0x117fae (0xbe3b bytes) 000367[1] 1785389247.737250 mapread 0x169a04 thru 0x16c9f6 (0x2ff3 bytes) 000368[0] 1785389247.753281 read 0x25511f thru 0x258c3b (0x3b1d bytes) 000369[0] 1785389247.860047 mapwrite 0x8e5ae4 thru 0x8e9c84 (0x41a1 bytes) 000370[0] 1785389248.219151 read 0x357c29 thru 0x35dd55 (0x612d bytes) 000372[0] 1785389248.240251 write 0x3f6f5e thru 0x3fd203 (0x62a6 bytes) 000373[1] 1785389248.301631 mapwrite 0x0c65b4 thru 0x0d5cc7 (0xf714 bytes) 000378[1] 1785389249.107089 read 0x4058b5 thru 0x412c4e (0xd39a bytes) 000379[1] 1785389249.153009 trunc from 0x8e9c85 to 0x69227e (0x257a06 bytes) 000380[1] 1785389249.258325 mapread 0x46013d thru 0x4680a4 (0x7f68 bytes) 000381[1] 1785389249.414650 write 0x6facf5 thru 0x6ff852 (0x4b5e bytes) 000383[1] 1785389249.509087 trunc from 0x6ff853 to 0x97308d (0x27383b bytes) 000384[0] 1785389249.646261 mapwrite 0x3c1147 thru 0x3c7a9f (0x6959 bytes) 000389[0] 1785389250.107183 mapread 0x42d670 thru 0x43461b (0x6fac bytes) 000390[1] 1785389250.190884 mapread 0x62b67d thru 0x632008 (0x698c bytes) 000391[1] 1785389250.352603 mapwrite 0x440ba9 thru 0x442e0e (0x2266 bytes) 000392[1] 1785389250.676469 trunc from 0x97308e to 0x58a486 (0x3e8c07 bytes) truncating to largest ever: 0x9e1774 000400[0] 1785389251.155281 trunc from 0x58a487 to 0x9e1773 (0x4572ed bytes) 000401[0] 1785389251.279951 write 0x6f0f34 thru 0x700659 (0xf726 bytes) 000403[1] 1785389251.415787 write 0x38eccc thru 0x39af98 (0xc2cd bytes) 000405[1] 1785389251.529811 write 0x5188cd thru 0x521dce (0x9502 bytes) 000408[1] 1785389251.728962 mapread 0x2ee0bc thru 0x2f381d (0x5762 bytes) 000411[1] 1785389251.934955 mapwrite 0x13fdf1 thru 0x1444e4 (0x46f4 bytes) 000415[1] 1785389252.190526 mapwrite 0x6bd40b thru 0x6c3aa5 (0x669b bytes) 000416[1] 1785389252.483667 write 0x07e0d4 thru 0x0801aa (0x20d7 bytes) 000417[0] 1785389252.562647 mapwrite 0x50ea28 thru 0x51aef2 (0xc4cb bytes) 000419[1] 1785389252.877034 mapread 0x429472 thru 0x42fdc8 (0x6957 bytes) 000421[0] 1785389253.162654 mapread 0x0474ed thru 0x04eae4 (0x75f8 bytes) 000422[1] 1785389253.311516 read 0x99bc4e thru 0x9a5698 (0x9a4b bytes) 000423[1] 1785389253.328577 mapwrite 0x7ebeec thru 0x7f8325 (0xc43a bytes) 000424[1] 1785389253.660879 write 0x66db6c thru 0x66f6b2 (0x1b47 bytes) 000425[0] 1785389253.724570 mapwrite 0x3b253c thru 0x3be1d8 (0xbc9d bytes) 000426[0] 1785389254.140455 trunc from 0x9e1774 to 0x8fe2e2 (0xe3491 bytes) 000427[0] 1785389254.257570 write 0x2a2b91 thru 0x2ab32c (0x879c bytes) 000429[0] 1785389254.350625 read 0x75420b thru 0x761d59 (0xdb4f bytes) 000431[1] 1785389254.401264 read 0x118476 thru 0x126782 (0xe30d bytes) 000433[1] 1785389254.544792 mapread 0x39b874 thru 0x39c0a7 (0x834 bytes) 000440[0] 1785389254.645118 read 0x44796f thru 0x448e35 (0x14c7 bytes) 000441[0] 1785389254.672726 mapread 0x352ed8 thru 0x3538f3 (0xa1c bytes) 000442[0] 1785389254.741116 mapread 0x55acb0 thru 0x561832 (0x6b83 bytes) 000443[1] 1785389254.881769 read 0x378f53 thru 0x38471b (0xb7c9 bytes) 000444[0] 1785389254.982183 mapread 0x71be00 thru 0x71ccfe (0xeff bytes) 000445[1] 1785389255.009909 mapread 0x3a7692 thru 0x3ad61a (0x5f89 bytes) 000447[0] 1785389255.117799 read 0x3a9d61 thru 0x3b0b57 (0x6df7 bytes) 000448[1] 1785389255.174186 mapwrite 0x0fab13 thru 0x105ec2 (0xb3b0 bytes) 000449[0] 1785389255.452069 mapwrite 0x5e7084 thru 0x5f4817 (0xd794 bytes) 000450[1] 1785389255.692093 trunc from 0x8fe2e3 to 0x4e3c62 (0x41a680 bytes) 000452[0] 1785389255.738026 mapwrite 0x09cb59 thru 0x0a310e (0x65b6 bytes) 000458[0] 1785389256.235359 write 0x6b33d8 thru 0x6b6ccc (0x38f5 bytes) 000459[1] 1785389256.278966 read 0x4a6c7e thru 0x4b49c9 (0xdd4c bytes) 000460[1] 1785389256.310235 write 0x5d601b thru 0x5dc771 (0x6757 bytes) 000461[0] 1785389256.360162 trunc from 0x6b6ccd to 0x07332c (0x6439a0 bytes) 000463[1] 1785389256.412637 mapread 0x05d954 thru 0x063892 (0x5f3f bytes) 000464[1] 1785389256.451807 write 0x04e3ae thru 0x04e7dc (0x42f bytes) 000465[0] 1785389256.481531 mapread 0x038d4b thru 0x041f30 (0x91e6 bytes) 000466[1] 1785389256.535923 write 0x748c4c thru 0x74d1ef (0x45a4 bytes) 000467[0] 1785389256.564377 mapread 0x5b8596 thru 0x5ba31b (0x1d86 bytes) 000470[0] 1785389256.635918 mapwrite 0x1dcf9a thru 0x1ebc90 (0xecf7 bytes) 000473[0] 1785389257.149930 read 0x6a5f24 thru 0x6acd04 (0x6de1 bytes) 000474[1] 1785389257.185042 write 0x17ae6a thru 0x1805d3 (0x576a bytes) 000478[1] 1785389257.300517 mapwrite 0x40a398 thru 0x41029e (0x5f07 bytes) 000479[0] 1785389257.515625 write 0x5b8cf3 thru 0x5bb54f (0x285d bytes) 000481[0] 1785389257.551357 mapwrite 0x705369 thru 0x70e933 (0x95cb bytes) 000485[0] 1785389257.759403 mapwrite 0x391d83 thru 0x397877 (0x5af5 bytes) 000488[0] 1785389258.066485 trunc from 0x74d1f0 to 0x726b38 (0x266b7 bytes) 000489[0] 1785389258.104121 write 0x0fd342 thru 0x1053ff (0x80be bytes) 000491[1] 1785389258.151174 trunc from 0x726b39 to 0x9b2fb6 (0x28c47e bytes) 000492[0] 1785389258.195796 write 0x8e0d30 thru 0x8e52ea (0x45bb bytes) 000493[1] 1785389258.231702 write 0x1b525f thru 0x1c1023 (0xbdc5 bytes) 000495[0] 1785389258.282806 trunc from 0x9b2fb7 to 0x7c3153 (0x1efe63 bytes) 000497[0] 1785389258.371147 mapwrite 0x4b2b7a thru 0x4b2fb4 (0x43b bytes) 000498[1] 1785389258.442207 mapwrite 0x42911a thru 0x432b06 (0x99ed bytes) 000499[1] 1785389258.548147 mapwrite 0x7e5b06 thru 0x7f2903 (0xcdfe bytes) 000500[1] 1785389258.799310 trunc from 0x7f2904 to 0x94a394 (0x157a91 bytes) All operations completed A-OK! Chance of close/open is 1 in 50 Seed set to 9259 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported truncating to largest ever: 0x445570 000001[0] 1785389260.504675 trunc from 00000000 to 0x44556f (0x445570 bytes) 000003[1] 1785389260.528697 trunc from 0x445570 to 0x3ef900 (0x55c6f bytes) 000008[1] 1785389260.697664 read 0x3c7000 thru 0x3c8317 (0x1318 bytes) 000009[0] 1785389260.755596 trunc from 0x3ef901 to 0x3cc301 (0x235ff bytes) 000011[0] 1785389260.946449 mapread 0x0ab000 thru 0x0ae96d (0x396e bytes) 000012[1] 1785389260.994588 mapwrite 0x301000 thru 0x30785e (0x685f bytes) truncating to largest ever: 0x4beddf 000013[1] 1785389261.305594 trunc from 0x3cc302 to 0x4bedde (0xf2add bytes) 000014[1] 1785389261.402016 mapread 0x13e000 thru 0x1470bd (0x90be bytes) 000015[1] 1785389261.467641 write 0x268000 thru 0x26a66d (0x266e bytes) 000016[0] 1785389261.482889 write 0x047000 thru 0x053600 (0xc601 bytes) 000017[1] 1785389261.552306 mapread 0x08c000 thru 0x099754 (0xd755 bytes) 000019[0] 1785389261.728749 mapwrite 0x640000 thru 0x64e0cc (0xe0cd bytes) 000021[0] 1785389262.064054 trunc from 0x64e0cd to 0x0bea51 (0x58f67b bytes) 000025[1] 1785389262.219497 mapread 0x07a000 thru 0x07a039 (0x3a bytes) 000026[1] 1785389262.287388 mapread 0x043000 thru 0x049f94 (0x6f95 bytes) 000030[1] 1785389262.359059 trunc from 0x0bea52 to 0x0cc69f (0xdc4e bytes) 000032[1] 1785389262.433951 trunc from 0x0cc6a0 to 0x14b43c (0x7ed9d bytes) 000037[1] 1785389262.701231 mapread 0x104000 thru 0x109416 (0x5417 bytes) 000038[1] 1785389262.744661 mapwrite 0x123000 thru 0x1276fe (0x46ff bytes) 000040[0] 1785389262.873975 trunc from 0x14b43d to 0x2b5035 (0x169bf9 bytes) 000041[0] 1785389262.935815 mapwrite 0x03d000 thru 0x043c3c (0x6c3d bytes) 000044[1] 1785389263.127720 write 0x8bd000 thru 0x8c9394 (0xc395 bytes) 000045[0] 1785389263.199982 mapread 0x302000 thru 0x30ff5d (0xdf5e bytes) 000047[1] 1785389263.381356 mapread 0x60b000 thru 0x614a21 (0x9a22 bytes) 000050[0] 1785389263.520874 write_OD 0x894000 thru 0x896fff (0x3000 bytes) 000051[0] 1785389263.638349 read_OD 0x3ed000 thru 0x3edfff (0x1000 bytes) 000055[1] 1785389263.658849 write 0x111000 thru 0x11b0f5 (0xa0f6 bytes) 000056[0] 1785389263.706920 trunc from 0x8c9395 to 0x46fbd4 (0x4597c0 bytes) 000057[0] 1785389263.783301 write_OD 0x054000 thru 0x055fff (0x2000 bytes) truncating to largest ever: 0x835881 000063[0] 1785389263.953148 trunc from 0x46fbd5 to 0x835880 (0x3c5cac bytes) 000065[1] 1785389264.099560 trunc from 0x835881 to 0x0ba740 (0x77b140 bytes) 000066[1] 1785389264.146291 read 0x05c000 thru 0x06040a (0x440b bytes) 000067[1] 1785389264.187642 mapread 0x0b2000 thru 0x0b9dce (0x7dcf bytes) 000068[0] 1785389264.224189 mapwrite 0x03b000 thru 0x043ca7 (0x8ca8 bytes) 000070[1] 1785389264.376998 mapwrite 0x1e5000 thru 0x1f2251 (0xd252 bytes) 000071[0] 1785389264.624576 read_OD 0x072000 thru 0x074fff (0x3000 bytes) 000072[1] 1785389264.689324 read 0x0c9000 thru 0x0d82c1 (0xf2c2 bytes) 000073[0] 1785389264.726403 trunc from 0x1f2252 to 0x4e546b (0x2f321a bytes) 000075[1] 1785389264.796339 mapread 0x120000 thru 0x12d89f (0xd8a0 bytes) 000076[1] 1785389265.006781 mapwrite 0x66e000 thru 0x677d5c (0x9d5d bytes) 000078[1] 1785389265.211742 mapread 0x152000 thru 0x157adf (0x5ae0 bytes) 000079[0] 1785389265.245158 mapread 0x501000 thru 0x50ccfd (0xbcfe bytes) 000080[1] 1785389265.381762 write 0x53a000 thru 0x53fd17 (0x5d18 bytes) 000083[1] 1785389265.435702 read 0x069000 thru 0x070112 (0x7113 bytes) 000084[0] 1785389265.457839 mapwrite 0x993000 thru 0x9a18f4 (0xe8f5 bytes) 000086[0] 1785389265.700054 read_OD 0x8fe000 thru 0x904fff (0x7000 bytes) 000087[1] 1785389265.770950 write 0x4a7000 thru 0x4adc64 (0x6c65 bytes) 000091[0] 1785389265.898929 mapread 0x0b7000 thru 0x0bf30b (0x830c bytes) 000095[1] 1785389266.023667 mapwrite 0x775000 thru 0x77f249 (0xa24a bytes) 000099[0] 1785389266.309968 trunc from 0x9a18f5 to 0x09b893 (0x906061 bytes) truncating to largest ever: 0x9d4cde 000100[0] 1785389266.458983 trunc from 0x09b894 to 0x9d4cdd (0x93944a bytes) 000103[0] 1785389266.612732 mapread 0x0b5000 thru 0x0c1e78 (0xce79 bytes) 000109[0] 1785389266.848289 write_OD 0x562000 thru 0x562fff (0x1000 bytes) 000111[0] 1785389266.981785 trunc from 0x9d4cde to 0x3210a4 (0x6b3c39 bytes) 000113[1] 1785389267.096351 trunc from 0x3210a5 to 0x4dcfbf (0x1bbf1b bytes) 000115[0] 1785389267.190594 mapwrite 0x311000 thru 0x318a0f (0x7a10 bytes) 000118[1] 1785389267.499646 write 0x566000 thru 0x56c8ff (0x6900 bytes) 000119[0] 1785389267.577946 write_OD 0x603000 thru 0x611fff (0xf000 bytes) 000120[1] 1785389267.727393 write 0x2d1000 thru 0x2d8dce (0x7dcf bytes) 000122[0] 1785389267.830593 mapwrite 0x087000 thru 0x089e55 (0x2e56 bytes) 000123[1] 1785389268.137687 mapread 0x310000 thru 0x3129bf (0x29c0 bytes) 000125[1] 1785389268.262654 trunc from 0x612000 to 0x53733d (0xdacc2 bytes) 000126[1] 1785389268.372140 mapwrite 0x8b4000 thru 0x8b61da (0x21db bytes) 000127[0] 1785389268.541449 trunc from 0x8b61db to 0x8005ec (0xb5bee bytes) 000128[0] 1785389268.610471 read_OD 0x5fe000 thru 0x5fffff (0x2000 bytes) 000129[0] 1785389268.648852 mapread 0x113000 thru 0x118b00 (0x5b01 bytes) 000130[0] 1785389268.710170 write_OD 0x119000 thru 0x11afff (0x2000 bytes) 000131[0] 1785389268.804742 mapread 0x673000 thru 0x682320 (0xf321 bytes) 000134[0] 1785389268.942725 mapwrite 0x1b3000 thru 0x1b9296 (0x6297 bytes) 000135[0] 1785389269.220953 trunc from 0x8005ed to 0x7a6441 (0x5a1ab bytes) 000138[1] 1785389269.387129 mapwrite 0x037000 thru 0x046156 (0xf157 bytes) 000140[1] 1785389269.547504 read 0x3ea000 thru 0x3f3608 (0x9609 bytes) 000141[1] 1785389269.567719 read 0x5de000 thru 0x5e2780 (0x4781 bytes) 000144[1] 1785389269.635238 mapwrite 0x578000 thru 0x57d872 (0x5873 bytes) 000148[1] 1785389269.858215 read 0x299000 thru 0x29df9d (0x4f9e bytes) 000149[1] 1785389269.900289 write 0x242000 thru 0x247ac8 (0x5ac9 bytes) 000150[1] 1785389269.908470 mapwrite 0x03b000 thru 0x0428ce (0x78cf bytes) 000151[1] 1785389270.182156 write 0x24f000 thru 0x258b4a (0x9b4b bytes) 000159[1] 1785389270.260619 write 0x56a000 thru 0x579668 (0xf669 bytes) 000160[1] 1785389270.325877 trunc from 0x7a6442 to 0x6fdf1f (0xa8522 bytes) 000161[1] 1785389270.393597 write 0x828000 thru 0x835760 (0xd761 bytes) 000164[1] 1785389270.440117 trunc from 0x835761 to 0x36e929 (0x4c6e37 bytes) 000165[0] 1785389270.458690 trunc from 0x36e92a to 0x09f401 (0x2cf528 bytes) 000168[1] 1785389270.562817 trunc from 0x09f402 to 0x8b4dfd (0x8159fc bytes) 000171[0] 1785389270.686812 write_OD 0x860000 thru 0x869fff (0xa000 bytes) 000172[0] 1785389270.809849 mapwrite 0x016000 thru 0x01c100 (0x6101 bytes) 000178[1] 1785389271.102802 trunc from 0x8b4dfe to 0x0b65ac (0x7fe851 bytes) 000181[0] 1785389271.157347 mapread 0x01f000 thru 0x02331b (0x431c bytes) 000182[0] 1785389271.245933 mapwrite 0x352000 thru 0x353cce (0x1ccf bytes) 000185[0] 1785389271.564224 read_OD 0x2db000 thru 0x2dffff (0x5000 bytes) 000186[0] 1785389271.640224 trunc from 0x353ccf to 0x1afacd (0x1a4201 bytes) 000187[0] 1785389271.700177 trunc from 0x1aface to 0x318257 (0x16878a bytes) 000189[1] 1785389271.729726 write 0x137000 thru 0x13e658 (0x7659 bytes) 000191[0] 1785389271.764950 mapread 0x14d000 thru 0x159f70 (0xcf71 bytes) 000192[0] 1785389271.849161 mapwrite 0x0fb000 thru 0x108567 (0xd568 bytes) 000193[1] 1785389272.251447 trunc from 0x318258 to 0x8c869b (0x5b0444 bytes) 000196[0] 1785389272.401350 trunc from 0x8c869c to 0x0a587a (0x822e21 bytes) 000197[1] 1785389272.455835 read 0x060000 thru 0x06fd9c (0xfd9d bytes) 000202[0] 1785389272.519364 mapread 0x070000 thru 0x0792ea (0x92eb bytes) 000203[1] 1785389272.674733 read 0x09e000 thru 0x0a57fe (0x77ff bytes) 000206[1] 1785389272.721573 mapread 0x062000 thru 0x06a3b7 (0x83b8 bytes) 000208[1] 1785389272.725159 mapwrite 0x9ff000 thru 0x9ff63e (0x63f bytes) 000209[1] 1785389272.909606 write 0x1fa000 thru 0x202b66 (0x8b67 bytes) 000214[1] 1785389272.944181 mapread 0x095000 thru 0x09d619 (0x861a bytes) 000216[1] 1785389273.022642 mapread 0x1df000 thru 0x1ea757 (0xb758 bytes) 000217[1] 1785389273.023958 read 0x954000 thru 0x95abb6 (0x6bb7 bytes) 000219[1] 1785389273.043802 write 0x556000 thru 0x55f8bd (0x98be bytes) 000220[1] 1785389273.052799 mapread 0x865000 thru 0x872496 (0xd497 bytes) 000222[0] 1785389273.087380 mapwrite 0x3a3000 thru 0x3b129d (0xe29e bytes) 000223[0] 1785389273.227914 mapwrite 0x41b000 thru 0x42640f (0xb410 bytes) 000224[1] 1785389273.277817 write 0x7f5000 thru 0x800b0b (0xbb0c bytes) 000225[0] 1785389273.301148 trunc from 0x9ff63f to 0x3d070e (0x62ef30 bytes) 000226[1] 1785389273.346767 mapread 0x2e0000 thru 0x2ea200 (0xa201 bytes) 000229[0] 1785389273.393009 write_OD 0x771000 thru 0x774fff (0x4000 bytes) 000233[1] 1785389273.484588 mapread 0x6c3000 thru 0x6cb2a8 (0x82a9 bytes) 000235[1] 1785389273.509292 mapwrite 0x348000 thru 0x356392 (0xe393 bytes) 000236[0] 1785389273.607670 mapread 0x0a3000 thru 0x0a6428 (0x3429 bytes) 000238[1] 1785389273.655651 mapread 0x4ac000 thru 0x4b53f6 (0x93f7 bytes) 000239[1] 1785389273.677360 mapread 0x1e4000 thru 0x1e429a (0x29b bytes) 000240[1] 1785389273.682700 write 0x8f9000 thru 0x8faebb (0x1ebc bytes) 000245[0] 1785389273.745467 mapread 0x6e0000 thru 0x6e5182 (0x5183 bytes) 000248[1] 1785389273.824403 mapread 0x146000 thru 0x150ba6 (0xaba7 bytes) 000249[0] 1785389273.865722 mapwrite 0x043000 thru 0x044f82 (0x1f83 bytes) 000254[0] 1785389274.006738 trunc from 0x8faebc to 0x8e42f8 (0x16bc3 bytes) 000255[1] 1785389274.042017 read 0x51f000 thru 0x52ef71 (0xff72 bytes) 000258[1] 1785389274.145808 read 0x4d9000 thru 0x4e7c04 (0xec05 bytes) 000261[0] 1785389274.178630 mapread 0x3ea000 thru 0x3ea774 (0x775 bytes) 000263[1] 1785389274.189695 mapread 0x164000 thru 0x168e3f (0x4e40 bytes) 000264[0] 1785389274.236683 mapwrite 0x822000 thru 0x82db90 (0xbb91 bytes) 000265[1] 1785389274.483760 mapread 0x64f000 thru 0x6533c3 (0x43c4 bytes) 000266[0] 1785389274.583102 read_OD 0x2d5000 thru 0x2e1fff (0xd000 bytes) 000267[1] 1785389274.623866 mapread 0x06c000 thru 0x06faca (0x3acb bytes) 000268[0] 1785389274.673542 read_OD 0x715000 thru 0x71dfff (0x9000 bytes) 000271[0] 1785389274.711354 mapread 0x21c000 thru 0x221b1d (0x5b1e bytes) 000276[1] 1785389274.801565 mapread 0x528000 thru 0x5348f8 (0xc8f9 bytes) 000279[1] 1785389274.910767 mapwrite 0x24f000 thru 0x2580aa (0x90ab bytes) 000280[0] 1785389275.120789 write_OD 0x170000 thru 0x172fff (0x3000 bytes) 000286[0] 1785389275.244477 write_OD 0x0ec000 thru 0x0f5fff (0xa000 bytes) 000287[0] 1785389275.334166 write_OD 0x475000 thru 0x476fff (0x2000 bytes) 000288[0] 1785389275.408354 write_OD 0x278000 thru 0x27efff (0x7000 bytes) 000291[1] 1785389275.457956 trunc from 0x8e42f9 to 0x46257a (0x481d7e bytes) 000292[0] 1785389275.512469 mapread 0x3f3000 thru 0x3f630e (0x330f bytes) 000293[1] 1785389275.601944 trunc from 0x46257b to 0x6c1b2f (0x25f5b5 bytes) 000303[0] 1785389275.867792 mapwrite 0x540000 thru 0x542e12 (0x2e13 bytes) 000304[1] 1785389276.165519 trunc from 0x6c1b30 to 0x649c9a (0x77e95 bytes) 000305[0] 1785389276.243636 write_OD 0x9d1000 thru 0x9ddfff (0xd000 bytes) 000307[0] 1785389276.449859 write_OD 0x903000 thru 0x903fff (0x1000 bytes) 000308[0] 1785389276.518775 mapwrite 0x7ce000 thru 0x7da69b (0xc69c bytes) 000309[1] 1785389276.652843 read 0x933000 thru 0x940c34 (0xdc35 bytes) 000310[0] 1785389276.693288 write_OD 0x210000 thru 0x21cfff (0xd000 bytes) 000313[1] 1785389276.751954 read 0x8b3000 thru 0x8b6e93 (0x3e94 bytes) 000318[1] 1785389276.771875 read 0x59e000 thru 0x5aabaa (0xcbab bytes) 000319[1] 1785389276.784620 read 0x68c000 thru 0x68f79c (0x379d bytes) 000321[0] 1785389276.793368 write_OD 0x3a2000 thru 0x3a8fff (0x7000 bytes) 000322[0] 1785389276.899023 trunc from 0x9de000 to 0x0c2a19 (0x91b5e6 bytes) 000323[1] 1785389276.956394 trunc from 0x0c2a1a to 0x867fc7 (0x7a55ae bytes) 000325[1] 1785389277.017916 mapwrite 0x873000 thru 0x87380a (0x80b bytes) 000327[1] 1785389277.143219 write 0x7bd000 thru 0x7cb4be (0xe4bf bytes) 000330[0] 1785389277.236504 trunc from 0x87380b to 0x624a7b (0x24ed8f bytes) 000332[1] 1785389277.337890 mapread 0x52e000 thru 0x52e6e5 (0x6e6 bytes) 000334[1] 1785389277.405050 write 0x6be000 thru 0x6c1745 (0x3746 bytes) 000340[1] 1785389277.523320 read_OD 0x62b000 thru 0x62cfff (0x2000 bytes) 000341[1] 1785389277.577299 read_OD 0x546000 thru 0x54afff (0x5000 bytes) 000343[0] 1785389277.603035 read_OD 0x1ab000 thru 0x1b0fff (0x6000 bytes) 000345[1] 1785389277.617117 mapwrite 0x1e7000 thru 0x1ea38d (0x338e bytes) 000350[1] 1785389277.794057 mapwrite 0x8d4000 thru 0x8df6da (0xb6db bytes) 000353[1] 1785389278.038051 mapread 0x0c6000 thru 0x0d1e06 (0xbe07 bytes) 000357[1] 1785389278.101757 mapwrite 0x0af000 thru 0x0be756 (0xf757 bytes) 000359[1] 1785389278.187735 write_OD 0x4b4000 thru 0x4b8fff (0x5000 bytes) 000360[0] 1785389278.240099 mapread 0x29e000 thru 0x2adabb (0xfabc bytes) 000361[1] 1785389278.269706 mapread 0x83d000 thru 0x847e6d (0xae6e bytes) 000362[1] 1785389278.299827 mapread 0x721000 thru 0x72c8ea (0xb8eb bytes) 000364[1] 1785389278.356641 read_OD 0x149000 thru 0x151fff (0x9000 bytes) 000365[0] 1785389278.364463 write_OD 0x875000 thru 0x882fff (0xe000 bytes) 000367[1] 1785389278.442382 trunc from 0x8df6db to 0x35d658 (0x582082 bytes) 000368[1] 1785389278.478310 mapread 0x12f000 thru 0x13af88 (0xbf89 bytes) 000369[0] 1785389278.524714 write_OD 0x3da000 thru 0x3e6fff (0xd000 bytes) 000372[0] 1785389278.582979 read_OD 0x093000 thru 0x09cfff (0xa000 bytes) 000373[0] 1785389278.597373 write_OD 0x084000 thru 0x086fff (0x3000 bytes) 000374[1] 1785389278.649186 mapwrite 0x11f000 thru 0x127c82 (0x8c83 bytes) 000377[0] 1785389278.775079 write_OD 0x1a4000 thru 0x1aafff (0x7000 bytes) 000380[0] 1785389278.816794 read_OD 0x34c000 thru 0x34cfff (0x1000 bytes) 000381[0] 1785389278.821563 mapread 0x35a000 thru 0x365726 (0xb727 bytes) 000382[0] 1785389278.884546 mapread 0x32e000 thru 0x3396c6 (0xb6c7 bytes) 000384[1] 1785389278.893926 read_OD 0x2c4000 thru 0x2cefff (0xb000 bytes) 000385[1] 1785389278.904594 read_OD 0x234000 thru 0x239fff (0x6000 bytes) 000386[0] 1785389278.911104 mapread 0x1ae000 thru 0x1b4de0 (0x6de1 bytes) 000388[0] 1785389278.912329 mapwrite 0x6d9000 thru 0x6e0e03 (0x7e04 bytes) 000390[1] 1785389279.044889 read_OD 0x4ba000 thru 0x4c5fff (0xc000 bytes) 000395[1] 1785389279.075246 trunc from 0x6e0e04 to 0x8489f0 (0x167bed bytes) 000396[1] 1785389279.119074 mapread 0x6fa000 thru 0x708cb6 (0xecb7 bytes) 000397[1] 1785389279.141323 mapread 0x461000 thru 0x46d3d6 (0xc3d7 bytes) 000398[1] 1785389279.175572 read_OD 0x454000 thru 0x459fff (0x6000 bytes) 000399[1] 1785389279.192361 trunc from 0x8489f1 to 0x17083d (0x6d81b3 bytes) 000400[1] 1785389279.228052 mapread 0x0fd000 thru 0x10ccf9 (0xfcfa bytes) 000405[1] 1785389279.315137 trunc from 0x17083e to 0x59f1e4 (0x42e9a7 bytes) 000406[1] 1785389279.354016 mapwrite 0x26f000 thru 0x272026 (0x3027 bytes) 000407[0] 1785389279.448666 mapread 0x342000 thru 0x351b9c (0xfb9d bytes) 000409[0] 1785389279.548771 mapwrite 0x0c1000 thru 0x0c7abb (0x6abc bytes) 000413[1] 1785389279.782060 trunc from 0x59f1e5 to 0x814452 (0x27526e bytes) 000416[0] 1785389279.846457 mapwrite 0x429000 thru 0x435aaf (0xcab0 bytes) 000420[1] 1785389280.138830 mapread 0x4dc000 thru 0x4dcaa0 (0xaa1 bytes) 000423[1] 1785389280.172075 write_OD 0x6bf000 thru 0x6c9fff (0xb000 bytes) 000424[0] 1785389280.246342 mapwrite 0x240000 thru 0x24f2a4 (0xf2a5 bytes) 000428[0] 1785389280.425860 trunc from 0x814453 to 0x143d10 (0x6d0742 bytes) 000435[1] 1785389280.540330 write_OD 0x8a5000 thru 0x8affff (0xb000 bytes) 000437[1] 1785389280.593758 mapread 0x105000 thru 0x11441e (0xf41f bytes) 000441[0] 1785389280.630429 read_OD 0x0ac000 thru 0x0bafff (0xf000 bytes) 000444[0] 1785389280.643336 mapread 0x83f000 thru 0x84d743 (0xe744 bytes) 000446[0] 1785389280.675243 trunc from 0x8b0000 to 0x33294b (0x57d6b4 bytes) 000447[1] 1785389280.721904 read_OD 0x205000 thru 0x212fff (0xe000 bytes) 000448[1] 1785389280.766386 read_OD 0x0f4000 thru 0x0f8fff (0x5000 bytes) 000449[0] 1785389280.776486 read_OD 0x20a000 thru 0x217fff (0xe000 bytes) 000450[0] 1785389280.785390 mapread 0x014000 thru 0x0215e5 (0xd5e6 bytes) 000452[1] 1785389280.836709 read_OD 0x1cf000 thru 0x1d0fff (0x2000 bytes) 000453[0] 1785389280.855354 trunc from 0x33294c to 0x307820 (0x2b12b bytes) 000454[0] 1785389280.940035 mapread 0x27e000 thru 0x28da03 (0xfa04 bytes) 000455[1] 1785389281.185555 mapread 0x2f5000 thru 0x3026df (0xd6e0 bytes) 000456[0] 1785389281.357123 trunc from 0x307821 to 0x02ec61 (0x2d8bbf bytes) 000457[1] 1785389281.459039 write_OD 0x73d000 thru 0x745fff (0x9000 bytes) 000459[0] 1785389281.636579 mapwrite 0x8b5000 thru 0x8bf81b (0xa81c bytes) 000460[0] 1785389281.867293 mapread 0x6d3000 thru 0x6dd8a1 (0xa8a2 bytes) 000461[0] 1785389281.872651 mapread 0x5f9000 thru 0x5fa97b (0x197c bytes) 000464[0] 1785389281.910068 read_OD 0x74c000 thru 0x74dfff (0x2000 bytes) 000467[0] 1785389282.003282 read_OD 0x778000 thru 0x780fff (0x9000 bytes) 000468[1] 1785389282.025139 mapwrite 0x699000 thru 0x69a96e (0x196f bytes) 000469[1] 1785389282.492322 write_OD 0x66f000 thru 0x678fff (0xa000 bytes) 000476[1] 1785389282.686546 mapwrite 0x3e9000 thru 0x3e9eb1 (0xeb2 bytes) 000479[0] 1785389282.917962 read_OD 0x1d2000 thru 0x1d7fff (0x6000 bytes) 000483[1] 1785389282.942708 mapread 0x43c000 thru 0x441e64 (0x5e65 bytes) 000485[1] 1785389282.965629 mapread 0x4c8000 thru 0x4cca36 (0x4a37 bytes) 000486[0] 1785389282.991550 mapwrite 00000000 thru 0x00b23f (0xb240 bytes) 000491[0] 1785389283.121688 write_OD 0x86a000 thru 0x875fff (0xc000 bytes) 000492[0] 1785389283.177364 read_OD 0x112000 thru 0x11bfff (0xa000 bytes) 000494[0] 1785389283.188716 write_OD 0x939000 thru 0x947fff (0xf000 bytes) All operations completed A-OK! PASS 16a (72s) == sanityn test 16b: 500 iterations of dual-mount fsx at small size ========================================================== 01:28:13 (1785389293) fallocate on zfs doesn't consume space fallocate not supported 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.942334 s, 8.9 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.83195 s, 10.1 MB/s Chance of close/open is 1 in 50 Seed set to 9299 fd 0: /mnt/lustre/f16b.sanityn fd 1: /mnt/lustre2/f16b.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000001[0] 1785389301.552253 mapwrite 0x3d3b73 thru 0x3d4f10 (0x139e bytes) 000002[0] 1785389301.916737 write 0x1dd662 thru 0x1debd0 (0x156f bytes) 000003[0] 1785389302.026620 mapread 0x2ca808 thru 0x2cadfe (0x5f7 bytes) truncating to largest ever: 0x44362c 000004[0] 1785389302.108574 trunc from 0x3d4f11 to 0x44362b (0x6e71b bytes) 000008[1] 1785389302.319610 write 0x95b6db thru 0x95b77c (0xa2 bytes) truncating to largest ever: 0x612563 000009[0] 1785389302.381617 trunc from 0x95b77d to 0x612562 (0x34921a bytes) 000011[0] 1785389302.565020 trunc from 0x612563 to 0x2ab186 (0x3673dc bytes) truncating to largest ever: 0x8d4a90 000013[0] 1785389302.703546 trunc from 0x2ab187 to 0x8d4a8f (0x629909 bytes) 000014[1] 1785389302.819079 read 0x1d318d thru 0x1d47a6 (0x161a bytes) 000016[1] 1785389302.903936 read 0x493288 thru 0x49426e (0xfe7 bytes) 000017[0] 1785389302.954129 read 0x4b3e2e thru 0x4b4105 (0x2d8 bytes) 000018[0] 1785389302.978113 trunc from 0x8d4a90 to 0x51b2b5 (0x3b97da bytes) 000019[1] 1785389303.081046 read 0x01feb5 thru 0x021264 (0x13b0 bytes) 000022[1] 1785389303.138921 trunc from 0x51b2b6 to 0x45921f (0xc2096 bytes) 000023[1] 1785389303.222277 mapwrite 0x0d7f8a thru 0x0d8130 (0x1a7 bytes) 000024[1] 1785389303.403009 mapwrite 0x1fcf91 thru 0x1febd0 (0x1c40 bytes) 000026[1] 1785389303.584231 mapread 0x1b4338 thru 0x1b43bb (0x84 bytes) 000028[1] 1785389303.652053 mapwrite 0x12eb7c thru 0x13004e (0x14d3 bytes) 000029[0] 1785389303.788778 write 0x6864fb thru 0x686c9e (0x7a4 bytes) 000031[1] 1785389303.947547 mapread 0x52e292 thru 0x530180 (0x1eef bytes) 000033[0] 1785389304.160853 read 0x6225ab thru 0x622e20 (0x876 bytes) 000035[0] 1785389304.190592 write 0x87f849 thru 0x88046b (0xc23 bytes) 000038[1] 1785389304.284864 read 0x6e7d87 thru 0x6e9805 (0x1a7f bytes) 000040[0] 1785389304.357904 write 0x2aa722 thru 0x2ab471 (0xd50 bytes) 000041[1] 1785389304.382976 write 0x1d9f7a thru 0x1dbf15 (0x1f9c bytes) 000042[0] 1785389304.426842 write 0x9a3f8c thru 0x9a582e (0x18a3 bytes) 000043[1] 1785389304.473068 read 0x05b0fb thru 0x05c5a6 (0x14ac bytes) 000044[1] 1785389304.518928 mapread 0x141384 thru 0x142e5d (0x1ada bytes) 000046[0] 1785389304.539255 read 0x982ea8 thru 0x984693 (0x17ec bytes) truncating to largest ever: 0x8f0296 000047[1] 1785389304.563906 trunc from 0x9a582f to 0x8f0295 (0xb5599 bytes) 000050[1] 1785389304.826176 read 0x794c4d thru 0x79542c (0x7e0 bytes) 000053[0] 1785389304.959440 read 0x1de899 thru 0x1df9ec (0x1154 bytes) 000054[1] 1785389305.009856 write 0x4eb37c thru 0x4eb81f (0x4a4 bytes) 000055[1] 1785389305.044941 trunc from 0x8f0296 to 0x2c4e69 (0x62b42c bytes) truncating to largest ever: 0x97d302 000058[0] 1785389305.204008 trunc from 0x2c4e6a to 0x97d301 (0x6b8498 bytes) 000059[0] 1785389305.370281 mapwrite 0x92b141 thru 0x92be99 (0xd59 bytes) 000061[1] 1785389305.553479 mapread 0x8cd21c thru 0x8ce1e4 (0xfc9 bytes) 000062[0] 1785389305.672555 write 0x654714 thru 0x655f4e (0x183b bytes) 000063[1] 1785389305.769294 write 0x4c570f thru 0x4c630e (0xc00 bytes) 000064[1] 1785389305.881482 trunc from 0x97d302 to 0x3374c3 (0x645e3e bytes) 000067[0] 1785389306.093751 trunc from 0x3374c4 to 0x468d20 (0x13185d bytes) 000068[1] 1785389306.266764 read 0x0100ba thru 0x01048a (0x3d1 bytes) 000071[1] 1785389306.412368 write 0x4e849b thru 0x4e991f (0x1485 bytes) 000072[0] 1785389306.521235 mapread 0x21212d thru 0x213ee9 (0x1dbd bytes) 000074[0] 1785389306.688216 mapwrite 0x05ce16 thru 0x05cfdd (0x1c8 bytes) 000080[1] 1785389307.281114 mapread 0x13ae68 thru 0x13b1ca (0x363 bytes) 000081[0] 1785389307.367133 read 0x1fe05a thru 0x1fe0ed (0x94 bytes) 000082[0] 1785389307.382250 read 0x31f696 thru 0x32129d (0x1c08 bytes) 000083[1] 1785389307.678342 mapwrite 0x15d64e thru 0x15f116 (0x1ac9 bytes) 000084[0] 1785389307.957900 mapread 0x1558f4 thru 0x156a8f (0x119c bytes) 000085[0] 1785389308.117148 write 0x7a5eaa thru 0x7a7752 (0x18a9 bytes) 000086[0] 1785389308.167805 write 0x969b85 thru 0x96a449 (0x8c5 bytes) 000089[0] 1785389308.293138 write 0x02b494 thru 0x02bb1f (0x68c bytes) 000091[0] 1785389308.329554 write 0x6c2713 thru 0x6c43bf (0x1cad bytes) 000092[0] 1785389308.643991 trunc from 0x96a44a to 0x0432d7 (0x927172 bytes) 000096[1] 1785389309.000132 mapread 0x03e704 thru 0x03f4a6 (0xda3 bytes) 000097[0] 1785389309.184681 mapread 0x0358fd thru 0x0372b6 (0x19ba bytes) 000098[1] 1785389309.247057 read 0x0260b8 thru 0x02683c (0x785 bytes) 000100[0] 1785389309.351798 read 0x02534c thru 0x025cc7 (0x97c bytes) 000101[1] 1785389309.372468 mapwrite 0x31cadd thru 0x31e29a (0x17be bytes) 000102[0] 1785389309.874793 write 0x8905c1 thru 0x890ba3 (0x5e3 bytes) 000103[0] 1785389309.969262 trunc from 0x890ba4 to 0x815c9f (0x7af04 bytes) 000105[0] 1785389310.223755 read 0x04809c thru 0x04828b (0x1f0 bytes) 000106[1] 1785389310.245538 write 0x3d550e thru 0x3d5bb3 (0x6a6 bytes) 000108[1] 1785389310.481083 read 0x53dbae thru 0x53f317 (0x176a bytes) 000109[1] 1785389310.527013 mapread 0x0afbb6 thru 0x0b06be (0xb09 bytes) 000112[1] 1785389310.677784 mapwrite 0x40e5df thru 0x4100f7 (0x1b19 bytes) 000113[0] 1785389310.865633 trunc from 0x815ca0 to 0x7c7e01 (0x4de9e bytes) 000116[1] 1785389310.984450 read 0x25f231 thru 0x2600da (0xeaa bytes) 000117[0] 1785389311.026184 mapwrite 0x5de503 thru 0x5df099 (0xb97 bytes) 000118[1] 1785389311.201565 mapread 0x3289cf thru 0x32a44b (0x1a7d bytes) 000119[0] 1785389311.257943 mapread 0x527898 thru 0x52860f (0xd78 bytes) 000120[0] 1785389311.282614 mapwrite 0x8254b9 thru 0x8270d9 (0x1c21 bytes) 000121[0] 1785389311.552272 trunc from 0x8270da to 0x39a04e (0x48d08b bytes) 000122[0] 1785389311.681369 write 0x84bcae thru 0x84c44f (0x7a2 bytes) 000123[0] 1785389311.717156 mapwrite 0x8e9448 thru 0x8ea58b (0x1144 bytes) 000124[0] 1785389312.019948 write 0x43d2ea thru 0x43ee2f (0x1b46 bytes) 000125[0] 1785389312.125487 read 0x695391 thru 0x696d05 (0x1975 bytes) 000126[1] 1785389312.202511 mapread 0x3f5140 thru 0x3f638a (0x124b bytes) 000127[1] 1785389312.319813 mapwrite 0x7f8e82 thru 0x7f9296 (0x415 bytes) 000132[1] 1785389312.756778 write 0x018e55 thru 0x019644 (0x7f0 bytes) 000134[1] 1785389312.880281 read 0x3c784b thru 0x3c91a1 (0x1957 bytes) truncating to largest ever: 0x9a75b9 000136[0] 1785389312.987089 trunc from 0x8ea58c to 0x9a75b8 (0xbd02d bytes) 000137[0] 1785389313.089896 trunc from 0x9a75b9 to 0x26d491 (0x73a127 bytes) 000138[0] 1785389313.149868 trunc from 0x26d492 to 0x4f2480 (0x284fef bytes) 000141[1] 1785389313.205142 write 0x34c8ac thru 0x34e778 (0x1ecd bytes) 000142[1] 1785389313.246840 read 0x1a535a thru 0x1a6fcc (0x1c73 bytes) 000144[0] 1785389313.290089 read 0x3295af thru 0x32a18b (0xbdd bytes) 000152[0] 1785389313.426857 write 0x6f5a10 thru 0x6f709f (0x1690 bytes) 000154[1] 1785389313.457876 mapwrite 0x3d66ea thru 0x3d704e (0x965 bytes) 000155[1] 1785389313.765395 trunc from 0x6f70a0 to 0x106d3b (0x5f0364 bytes) 000157[1] 1785389313.904625 write 0x75be5d thru 0x75c6ba (0x85e bytes) 000159[1] 1785389314.025843 mapread 0x1fa85a thru 0x1faf2a (0x6d1 bytes) 000160[0] 1785389314.043064 mapwrite 0x07511b thru 0x0755ca (0x4b0 bytes) 000161[0] 1785389314.360771 read 0x248fda thru 0x24a11f (0x1146 bytes) 000165[0] 1785389314.582250 mapwrite 0x5142fc thru 0x515282 (0xf87 bytes) 000168[0] 1785389314.800158 mapwrite 0x1a5373 thru 0x1a59df (0x66d bytes) 000169[1] 1785389314.909779 mapwrite 0x1e114d thru 0x1e1576 (0x42a bytes) 000171[0] 1785389315.259030 mapwrite 0x595b27 thru 0x5978db (0x1db5 bytes) 000172[1] 1785389315.464501 read 0x13fd32 thru 0x140ad5 (0xda4 bytes) 000174[0] 1785389315.554085 mapwrite 0x534332 thru 0x534379 (0x48 bytes) 000175[1] 1785389315.785368 mapread 0x2d8930 thru 0x2da273 (0x1944 bytes) 000176[1] 1785389315.905150 mapread 0x3ba0d5 thru 0x3baf2d (0xe59 bytes) 000177[0] 1785389315.964662 mapread 0x164440 thru 0x165297 (0xe58 bytes) 000178[0] 1785389316.038594 read 0x326272 thru 0x328259 (0x1fe8 bytes) 000180[1] 1785389316.176042 read 0x4b2066 thru 0x4b2c1a (0xbb5 bytes) 000181[0] 1785389316.207400 write 0x507568 thru 0x508701 (0x119a bytes) 000183[0] 1785389316.237032 mapwrite 0x29bfab thru 0x29bfbf (0x15 bytes) 000187[1] 1785389316.479025 read 0x3ee694 thru 0x3ee892 (0x1ff bytes) 000189[0] 1785389316.584208 write 0x62560a thru 0x626f71 (0x1968 bytes) 000191[1] 1785389316.679363 read 0x1ee7a3 thru 0x1ef946 (0x11a4 bytes) 000198[1] 1785389316.804874 trunc from 0x75c6bb to 0x4a2ee4 (0x2b97d6 bytes) 000200[0] 1785389316.953689 mapwrite 0x737fe5 thru 0x7388ec (0x908 bytes) 000203[0] 1785389317.247836 mapread 0x5388f8 thru 0x539d7d (0x1486 bytes) 000204[1] 1785389317.287323 trunc from 0x7388ed to 0x0ecf86 (0x64b966 bytes) 000205[0] 1785389317.402870 trunc from 0x0ecf87 to 0x1bdb8d (0xd0c07 bytes) 000209[0] 1785389317.648958 write 0x0f65ed thru 0x0f7065 (0xa79 bytes) 000211[0] 1785389317.658615 read 0x0dd173 thru 0x0de36d (0x11fb bytes) 000212[0] 1785389317.716814 read 0x0e7596 thru 0x0e92f0 (0x1d5b bytes) 000214[0] 1785389317.832129 trunc from 0x1bdb8e to 0x568d28 (0x3ab19b bytes) 000215[0] 1785389317.853393 write 0x3329bb thru 0x332c7b (0x2c1 bytes) 000217[0] 1785389317.969382 read 0x037159 thru 0x037d2f (0xbd7 bytes) 000218[1] 1785389318.011356 trunc from 0x568d29 to 0x5efec7 (0x8719f bytes) 000220[0] 1785389318.226963 trunc from 0x5efec8 to 0x51f766 (0xd0761 bytes) 000221[1] 1785389318.332173 write 0x545729 thru 0x547212 (0x1aea bytes) 000222[1] 1785389318.386529 write 0x12c249 thru 0x12d735 (0x14ed bytes) 000223[0] 1785389318.439015 mapwrite 0x18e9e7 thru 0x190940 (0x1f5a bytes) 000224[1] 1785389318.689023 read 0x3259e9 thru 0x3271aa (0x17c2 bytes) 000225[1] 1785389318.733510 mapwrite 0x05a50b thru 0x05bfe2 (0x1ad8 bytes) 000226[1] 1785389318.874261 write 0x0535a1 thru 0x0550a9 (0x1b09 bytes) 000227[0] 1785389318.903881 trunc from 0x547213 to 0x228186 (0x31f08c bytes) 000231[1] 1785389319.051140 trunc from 0x228187 to 0x9a080d (0x778687 bytes) 000232[0] 1785389319.198165 mapwrite 0x969716 thru 0x9699bf (0x2aa bytes) 000233[1] 1785389319.448684 mapread 0x576383 thru 0x577d98 (0x1a16 bytes) truncating to largest ever: 0x9e944f 000235[0] 1785389319.552917 trunc from 0x9a080e to 0x9e944e (0x48c41 bytes) 000236[1] 1785389319.710101 mapwrite 0x708860 thru 0x70a693 (0x1e34 bytes) 000237[1] 1785389319.918889 read 0x0824d1 thru 0x083527 (0x1057 bytes) 000239[1] 1785389319.991142 trunc from 0x9e944f to 0x9c5d37 (0x23717 bytes) 000242[0] 1785389320.094045 mapread 0x3ed277 thru 0x3ede60 (0xbea bytes) 000245[1] 1785389320.236779 write 0x863b89 thru 0x863d94 (0x20c bytes) 000246[1] 1785389320.324697 mapread 0x3aad2f thru 0x3ab7cd (0xa9f bytes) 000247[0] 1785389320.353224 read 0x20b39c thru 0x20c8d1 (0x1536 bytes) 000249[1] 1785389320.420856 mapread 0x6d1510 thru 0x6d1f1d (0xa0e bytes) 000251[1] 1785389320.470544 write 0x85a539 thru 0x85b556 (0x101e bytes) 000252[0] 1785389320.576953 write 0x0845ee thru 0x086067 (0x1a7a bytes) 000254[0] 1785389320.647366 trunc from 0x9c5d38 to 0x25896a (0x76d3cd bytes) 000256[0] 1785389320.813129 mapread 0x0b24b2 thru 0x0b2600 (0x14f bytes) 000258[0] 1785389320.894476 trunc from 0x25896b to 0x990c74 (0x73830a bytes) 000259[1] 1785389321.003819 mapwrite 0x7c05b8 thru 0x7c0fe2 (0xa2b bytes) 000264[1] 1785389321.403854 write 0x663360 thru 0x66451d (0x11be bytes) 000265[0] 1785389321.472068 mapwrite 0x9064c0 thru 0x906662 (0x1a3 bytes) 000267[0] 1785389321.744093 mapread 0x573ab9 thru 0x5747aa (0xcf2 bytes) 000268[1] 1785389321.795939 trunc from 0x990c75 to 0x45444c (0x53c828 bytes) 000270[1] 1785389321.877059 trunc from 0x45444d to 0x4f864f (0xa4203 bytes) 000271[0] 1785389321.908737 mapwrite 0x0bafea thru 0x0bcbc3 (0x1bda bytes) 000279[0] 1785389322.473460 trunc from 0x4f8650 to 0x1efc6f (0x3089e0 bytes) 000281[0] 1785389322.557101 mapread 0x103743 thru 0x1053c1 (0x1c7f bytes) 000282[0] 1785389322.582310 trunc from 0x1efc70 to 0x686672 (0x496a03 bytes) 000285[1] 1785389322.617712 mapwrite 0x2231d2 thru 0x223232 (0x61 bytes) 000287[1] 1785389322.825288 trunc from 0x686673 to 0x36c248 (0x31a42a bytes) 000288[0] 1785389322.923099 mapread 0x33e629 thru 0x33f0cf (0xaa7 bytes) 000290[0] 1785389323.095254 mapread 0x0a1c96 thru 0x0a1caf (0x1a bytes) 000294[1] 1785389323.168353 trunc from 0x36c249 to 0x8cc4d5 (0x56028d bytes) 000295[1] 1785389323.234534 write 0x64b713 thru 0x64d30f (0x1bfd bytes) 000297[0] 1785389323.362133 trunc from 0x8cc4d6 to 0x588fba (0x34351b bytes) 000300[0] 1785389323.581543 write 0x2f8c8b thru 0x2fa258 (0x15ce bytes) 000301[0] 1785389323.623212 write 0x75e15c thru 0x75f815 (0x16ba bytes) 000302[0] 1785389323.679274 mapwrite 0x5fb171 thru 0x5fca25 (0x18b5 bytes) 000304[0] 1785389323.971793 trunc from 0x75f816 to 0x5ae085 (0x1b1790 bytes) 000306[1] 1785389324.034131 mapwrite 0x611805 thru 0x613146 (0x1942 bytes) 000308[1] 1785389324.380506 read 0x17ae35 thru 0x17c937 (0x1b03 bytes) 000311[1] 1785389324.459659 write 0x32623a thru 0x326982 (0x749 bytes) 000312[1] 1785389324.503767 read 0x41abb3 thru 0x41c47d (0x18cb bytes) 000317[0] 1785389324.635512 read 0x1728fd thru 0x17315b (0x85f bytes) 000320[0] 1785389324.700048 write 0x838d09 thru 0x83ac1b (0x1f13 bytes) 000321[1] 1785389324.760195 read 0x3a4fd6 thru 0x3a6bbf (0x1bea bytes) 000322[1] 1785389324.813572 read 0x0a4b2e thru 0x0a4ef0 (0x3c3 bytes) 000323[0] 1785389324.830999 trunc from 0x83ac1c to 0x840d85 (0x616a bytes) 000325[1] 1785389324.897811 trunc from 0x840d86 to 0x076f72 (0x7c9e13 bytes) 000328[1] 1785389325.051490 trunc from 0x076f73 to 0x9c5355 (0x94e3e3 bytes) 000335[0] 1785389325.248932 mapwrite 0x599ea6 thru 0x59a607 (0x762 bytes) 000336[1] 1785389325.474369 write 0x69f23c thru 0x6a06e8 (0x14ad bytes) 000338[0] 1785389325.555219 trunc from 0x9c5356 to 0x3f89db (0x5cc97a bytes) 000341[1] 1785389325.738072 trunc from 0x3f89dc to 0x430afa (0x3811f bytes) 000343[0] 1785389325.813207 write 0x763448 thru 0x763d28 (0x8e1 bytes) 000347[1] 1785389325.991135 read 0x233fb9 thru 0x23466f (0x6b7 bytes) 000348[0] 1785389326.053067 write 0x46da04 thru 0x46e45a (0xa57 bytes) 000350[1] 1785389326.140287 write 0x98e867 thru 0x98e92b (0xc5 bytes) 000351[1] 1785389326.168555 mapwrite 0x2aa163 thru 0x2aa1f4 (0x92 bytes) 000352[0] 1785389326.346169 read 0x5018bd thru 0x5018f8 (0x3c bytes) 000355[1] 1785389326.428389 mapwrite 0x3a92df thru 0x3ab0ec (0x1e0e bytes) 000357[0] 1785389326.649585 read 0x6ef4f8 thru 0x6f0242 (0xd4b bytes) 000359[1] 1785389326.732803 mapwrite 0x71f908 thru 0x721491 (0x1b8a bytes) 000361[0] 1785389327.115049 mapread 0x40046c thru 0x402111 (0x1ca6 bytes) 000370[0] 1785389327.287714 mapwrite 0x1634fc thru 0x1650a7 (0x1bac bytes) 000372[1] 1785389327.521620 mapwrite 0x653857 thru 0x654ce0 (0x148a bytes) 000373[0] 1785389327.700265 trunc from 0x98e92c to 0x387316 (0x607615 bytes) 000374[1] 1785389327.807838 trunc from 0x387317 to 0x19c0f0 (0x1eb226 bytes) 000376[0] 1785389327.952960 mapwrite 0x65df25 thru 0x65f3ad (0x1489 bytes) 000378[1] 1785389328.281575 trunc from 0x65f3ae to 0x519fda (0x1453d3 bytes) 000379[1] 1785389328.412509 mapread 0x3cf9c6 thru 0x3d1979 (0x1fb4 bytes) 000380[0] 1785389328.503088 read 0x1f6859 thru 0x1f713b (0x8e3 bytes) 000382[0] 1785389328.593791 write 0x766095 thru 0x766171 (0xdd bytes) 000385[1] 1785389328.683778 write 0x9321a2 thru 0x93301a (0xe79 bytes) 000386[1] 1785389328.761300 trunc from 0x93301b to 0x79e840 (0x1947da bytes) 000390[1] 1785389328.924378 mapread 0x774ea8 thru 0x776a51 (0x1baa bytes) 000392[1] 1785389329.086240 mapread 0x67f2c2 thru 0x67f579 (0x2b8 bytes) 000393[1] 1785389329.100630 read 0x5b30d5 thru 0x5b3e83 (0xdaf bytes) 000394[0] 1785389329.108781 read 0x666413 thru 0x66795b (0x1549 bytes) 000395[0] 1785389329.174067 mapread 0x1827d5 thru 0x182bba (0x3e6 bytes) 000397[0] 1785389329.262162 write 0x31f126 thru 0x320804 (0x16df bytes) 000398[1] 1785389329.333647 write 0x87e299 thru 0x87e9e5 (0x74d bytes) 000399[0] 1785389329.402658 mapwrite 0x568d31 thru 0x568da3 (0x73 bytes) 000401[0] 1785389329.685290 mapwrite 0x79216c thru 0x7930cf (0xf64 bytes) 000403[1] 1785389329.928017 mapread 0x2778db thru 0x27986a (0x1f90 bytes) 000405[1] 1785389330.058616 mapread 0x2feb9b thru 0x300587 (0x19ed bytes) 000408[0] 1785389330.268654 trunc from 0x87e9e6 to 0x27fda9 (0x5fec3c bytes) 000409[0] 1785389330.382082 read 0x023294 thru 0x024c15 (0x1982 bytes) 000410[1] 1785389330.486275 trunc from 0x27fdaa to 0x1c6fc2 (0xb8de7 bytes) 000412[0] 1785389330.634915 mapwrite 0x28d791 thru 0x28e766 (0xfd6 bytes) 000413[1] 1785389330.849362 trunc from 0x28e767 to 0x88f990 (0x60122a bytes) 000414[1] 1785389330.979725 mapwrite 0x33dea7 thru 0x33df10 (0x6a bytes) 000417[1] 1785389331.250529 mapread 0x5a5e69 thru 0x5a6673 (0x80b bytes) 000418[0] 1785389331.274540 read 0x1f7591 thru 0x1f7f2a (0x99a bytes) 000419[1] 1785389331.340380 read 0x414b45 thru 0x4161b6 (0x1672 bytes) 000421[0] 1785389331.370149 read 0x601254 thru 0x601421 (0x1ce bytes) 000422[0] 1785389331.395940 trunc from 0x88f991 to 0x13c52e (0x753462 bytes) 000423[0] 1785389331.511148 mapread 0x0792a4 thru 0x07a8e3 (0x1640 bytes) 000424[1] 1785389331.583303 write 0x1171cb thru 0x118c31 (0x1a67 bytes) 000425[1] 1785389331.678013 mapwrite 0x23278c thru 0x23365b (0xed0 bytes) 000429[1] 1785389331.865622 read 0x0faa55 thru 0x0fc1f6 (0x17a2 bytes) 000430[0] 1785389331.938348 trunc from 0x23365c to 0x40f376 (0x1dbd1b bytes) 000431[1] 1785389332.027639 mapread 0x175fa1 thru 0x176e35 (0xe95 bytes) 000434[1] 1785389332.246230 trunc from 0x40f377 to 0x0969ad (0x3789c9 bytes) 000436[1] 1785389332.357055 read 0x047d0f thru 0x048271 (0x563 bytes) 000437[1] 1785389332.384517 mapwrite 0x85711d thru 0x85818e (0x1072 bytes) 000438[1] 1785389332.832826 mapwrite 0x797ee5 thru 0x7993af (0x14cb bytes) 000439[0] 1785389332.993238 write 0x63e699 thru 0x63f219 (0xb81 bytes) 000442[1] 1785389333.129314 trunc from 0x85818f to 0x7b3ba3 (0xa45eb bytes) 000443[1] 1785389333.239587 mapread 0x683535 thru 0x684af7 (0x15c3 bytes) 000445[0] 1785389333.350994 read 0x55b650 thru 0x55cfca (0x197b bytes) 000446[0] 1785389333.411873 read 0x6bd4ac thru 0x6be1fa (0xd4f bytes) 000447[1] 1785389333.434630 write 0x63eb86 thru 0x63fea8 (0x1323 bytes) 000449[1] 1785389333.541999 mapread 0x485490 thru 0x486bb4 (0x1725 bytes) 000451[1] 1785389333.635199 mapwrite 0x2d773a thru 0x2d8fd5 (0x189c bytes) 000456[0] 1785389333.795941 write 0x51f5f4 thru 0x521031 (0x1a3e bytes) 000457[0] 1785389333.890216 trunc from 0x7b3ba4 to 0x8ea0c1 (0x13651e bytes) 000458[1] 1785389333.947437 mapread 0x4ee483 thru 0x4efbc9 (0x1747 bytes) 000459[0] 1785389334.029024 write 0x085e56 thru 0x0865d8 (0x783 bytes) 000460[0] 1785389334.068945 mapwrite 0x794ef5 thru 0x7957d5 (0x8e1 bytes) 000462[0] 1785389334.208720 mapwrite 0x11e72c thru 0x120674 (0x1f49 bytes) 000463[0] 1785389334.345138 trunc from 0x8ea0c2 to 0x01f09c (0x8cb025 bytes) 000464[1] 1785389334.373851 mapread 0x0084cc thru 0x009880 (0x13b5 bytes) 000465[1] 1785389334.412992 trunc from 0x01f09d to 0x5366dc (0x517640 bytes) 000467[1] 1785389334.520560 mapread 0x49633a thru 0x496e35 (0xafc bytes) 000468[0] 1785389334.561658 read 0x0a218f thru 0x0a371c (0x158e bytes) 000469[1] 1785389334.607204 mapread 0x4c6230 thru 0x4c7e57 (0x1c28 bytes) 000470[1] 1785389334.626451 write 0x80708f thru 0x808556 (0x14c8 bytes) 000471[0] 1785389334.676354 read 0x79cf41 thru 0x79e014 (0x10d4 bytes) 000474[0] 1785389334.744535 mapwrite 0x1a210b thru 0x1a2715 (0x60b bytes) 000476[1] 1785389334.874317 write 0x608a60 thru 0x60a665 (0x1c06 bytes) 000479[1] 1785389334.975475 read 0x70e7e3 thru 0x70f205 (0xa23 bytes) 000480[1] 1785389335.038464 write 0x2b7f23 thru 0x2b96ea (0x17c8 bytes) 000482[0] 1785389335.140258 read 0x78520f thru 0x785740 (0x532 bytes) 000483[0] 1785389335.235151 write 0x89c562 thru 0x89cb16 (0x5b5 bytes) 000486[0] 1785389335.351530 read 0x61782f thru 0x6193d7 (0x1ba9 bytes) 000489[0] 1785389335.437323 trunc from 0x89cb17 to 0x19dd72 (0x6feda4 bytes) 000490[1] 1785389335.538329 write 0x9da03f thru 0x9dbc26 (0x1be8 bytes) 000493[1] 1785389335.645882 mapread 0x3aaf36 thru 0x3ab144 (0x20f bytes) 000494[0] 1785389335.716637 mapwrite 0x72551a thru 0x726ad9 (0x15c0 bytes) 000495[1] 1785389335.958783 read 0x810c8b thru 0x811e68 (0x11de bytes) 000496[1] 1785389336.010596 trunc from 0x9dbc27 to 0x46f2aa (0x56c97c bytes) 000497[1] 1785389336.141999 write 0x1262cf thru 0x128232 (0x1f64 bytes) 000498[0] 1785389336.212919 write 0x18d460 thru 0x18f222 (0x1dc3 bytes) All operations completed A-OK! PASS 16b (51s) == sanityn test 16c: verify data consistency on ldiskfs with cache disabled (b=17397) ========================================================== 01:29:04 (1785389344) SKIP: sanityn test_16c dio on ldiskfs only SKIP 16c (6s) == sanityn test 16d: Verify DIO and buffer IO with two clients ========================================================== 01:29:10 (1785389350) 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.847591 s, 9.9 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.846177 s, 9.9 MB/s ldlm.namespaces.MGC192.168.204.141@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8be707e69000.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8be710a2d000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8be707e69000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8be710a2d000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8be707e69000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8be710a2d000.lru_size=clear 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 18.0551 s, 5.8 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.59406 s, 15.9 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.01123 s, 26.1 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.07576 s, 13.0 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 15.0537 s, 7.0 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.30295 s, 19.8 MB/s PASS 16d (76s) == sanityn test 16e: Verify size consistency for O_DIRECT write ========================================================== 01:30:26 (1785389426) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.49796 s, 21.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.224883 s, 4.7 MB/s PASS 16e (12s) == sanityn test 16f: rw sequential consistency vs drop_caches ========================================================== 01:30:38 (1785389438) TIMEOUT PASS 16f (32s) == sanityn test 16g: mmap rw sequential consistency vs drop_caches ========================================================== 01:31:11 (1785389471) TIMEOUT PASS 16g (33s) == sanityn test 16h: mmap read after truncate file ======= 01:31:43 (1785389503) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16h (11s) == sanityn test 16i: read after truncate file ============ 01:31:54 (1785389514) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16i (11s) == sanityn test 16j: race dio with buffered i/o ========== 01:32:05 (1785389525) 1+0 records in 1+0 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.9912 s, 8.8 MB/s bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.1927 s, 12.0 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.36183 s, 7.8 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.6855 s, 9.8 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.96563 s, 6.6 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.93805 s, 8.9 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.36243 s, 7.8 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.36985 s, 7.8 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 4.00002 s, 6.6 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 5.00461 s, 5.2 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 5.3094 s, 4.9 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 4.65014 s, 5.6 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 4.99977 s, 5.2 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 6.17453 s, 4.2 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 6.28999 s, 4.2 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 4.27448 s, 6.1 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 4.49075 s, 5.8 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 4.48636 s, 5.8 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 4.80349 s, 5.5 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK PASS 16j (59s) == sanityn test 16k: Parallel FSX and drop caches should not panic ========================================================== 01:33:04 (1785389584) + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.1 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.2 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.3 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.4 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.5 -l 67108864 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported truncating to largest ever: 0x1a7062c fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported truncating to largest ever: 0x1a7062c skipping zero size read truncating to largest ever: 0x232e1d3 skipping zero size read truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x1a7062c truncating to largest ever: 0x32cce75 truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! + fsx 1 done: rc=0 All operations completed A-OK! Thu Jul 30 01:34:06 EDT 2026 + fsx 2 done: rc=0 Thu Jul 30 01:34:06 EDT 2026 + fsx 3 done: rc=0 Thu Jul 30 01:34:06 EDT 2026 + fsx 4 done: rc=0 Thu Jul 30 01:34:06 EDT 2026 + fsx 5 done: rc=0 Thu Jul 30 01:34:06 EDT 2026 PASS 16k (76s) == sanityn test 17: resource creation/LVB creation race ========================================================================= 01:34:20 (1785389660) fail_loc=0x8000030a PASS 17 (13s) == sanityn test 18: mmap sanity check =========================================================================================== 01:34:34 (1785389674) mmap test1: basic mmap operation (PASS, 0.13204s) mmap test2: MAP_PRIVATE not write back (PASS, 0.046114s) mmap test3: concurrent mmap ops on two nodes (PASS, 8.5408s) mmap test4: c1 write to f1 from mmapped f2, c2 write to f1 from mmapped f1 (PASS, 4.8052s) mmap test5: read/write file to/from the buffer which mmapped to just this file (PASS, 0.18087s) mmap test6: check mmap write/read content on two nodes (PASS, 0.36547s) mmap test7: file i/o with an unmapped buffer (SKIPPED, 0s) mmap test8: SIGBUS for beyond file size (SKIPPED, 0s) mmap test9: SIGBUS for negative file offset (SKIPPED, 0s) mmap test10: multi-thread mmap access (PASS, 2.7021s) mmap test11: multi-thread shared mmap (PASS, 3.318s) mmap test10: mtime not change for readonly mmap access (PASS, 1.2004s) PASS 18 (32s) == sanityn test 19: test concurrent uncached read races ========================================================================= 01:35:06 (1785389706) oleg441-server: error: get_param: param_path 'osd-*/lustre-OST*/read_cache_enable': No such file or directory pdsh@oleg441-client: oleg441-server: ssh exited with exit code 2 SKIP: sanityn test_19 not cache-capable obdfilter SKIP 19 (5s) == sanityn test 20: test extra readahead page left in cache ============================================================== 01:35:12 (1785389712) PASS 20 (9s) == sanityn test 21: Try to remove mountpoint on another dir ============================================================== 01:35:22 (1785389722) rmdir: removing directory, '/mnt/lustre/d21.sanityn' rmdir: failed to remove '/mnt/lustre/d21.sanityn': Device or resource busy rmdir: removing directory, '/mnt/lustre2/d21.sanityn' Removed mounted directory from another mountpoint, needs to be fixed PASS 21 (9s) == sanityn test 23: others should see updated atime while another read============================================================== 01:35:30 (1785389730) now is 1785389734 starting reads multiop /mnt/lustre/f23.sanityn vor20_c TMPPIPE=/tmp/multiop_open_wait_pipe.7625 new atime is 1785389795 PASS 23 (73s) == sanityn test 24a: lfs df [-ih] [path] test =================================================================================== 01:36:43 (1785389803) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 5120 3764224 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 5120 3764224 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre2 UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 115.3K 403 114.9K 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 114.5K 399 114.1K 1% /mnt/lustre[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.3K 403 114.9K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 399 114.1K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre2 UUID bytes Used Available Use% Mounted on lustre-MDT0000_UUID 2.1G 2.1M 2.1G 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3.6G 5.0M 3.6G 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3.6G 32.0M 3.6G 1% /mnt/lustre[OST:1] filesystem_summary: 7.2G 37.0M 7.2G 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118099 403 117696 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117231 399 116832 1% /mnt/lustre2[OST:1] filesystem_summary: 234914 386 234528 1% /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 5120 3764224 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.3K 403 114.9K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 399 114.1K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 118099 403 117696 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 117231 399 116832 1% /mnt/lustre[OST:1] filesystem_summary: 234914 386 234528 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118099 403 117696 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117231 399 116832 1% /mnt/lustre2[OST:1] filesystem_summary: 234914 386 234528 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 5120 3764224 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 5120 3764224 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre2 PASS 24a (12s) == sanityn test 24b: lfs df should show both filesystems ========================================================================= 01:36:55 (1785389815) PASS 24b (11s) == sanityn test 25a: change ACL on one mountpoint be seen on another ============================================================= 01:37:06 (1785389826) running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] PASS 25a (8s) == sanityn test 25b: change ACL under remote dir on one mountpoint be seen on another ========================================================== 01:37:15 (1785389835) SKIP: sanityn test_25b needs >= 2 MDTs SKIP 25b (4s) == sanityn test 26a: allow mtime to get older ============ 01:37:19 (1785389839) utime: good mknod times 1785389840 <= 1785389841 <= 1785389841 for /mnt/lustre/f26a utime: good mknod times 1785389840 <= 1785389841 <= 1785389841 for /mnt/lustre2/f26a utime: good utime mtimes 100000, atime 200000 utime: updated times for /mnt/lustre2/f26a PASS 26a (9s) == sanityn test 26b: sync mtime between ost and mds ====== 01:37:28 (1785389848) PASS 26b (11s) == sanityn test 26c: set-in-past on open file is not lost on close ========================================================== 01:37:39 (1785389859) multiop /mnt/lustre/f26c.sanityn vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7625 File: /mnt/lustre/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502398 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-07-30 01:37:41.000000000 -0400 Birth: 2026-07-30 01:37:42.000000000 -0400 File: /mnt/lustre2/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502398 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-07-30 01:37:41.000000000 -0400 Birth: 2026-07-30 01:37:42.000000000 -0400 PASS 26c (9s) SKIP: sanityn test_27 skipping excluded test 27 == sanityn test 30: recreate file race =================== 01:37:50 (1785389870) 0 PASS 30 (11s) == sanityn test 31a: voluntary cancel / blocking ast race======================================================================== 01:38:02 (1785389882) fail_loc=0x314 PASS 31a (9s) == sanityn test 31b: voluntary OST cancel / blocking ast race======================================================================== 01:38:11 (1785389891) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg441-server mds-ost sync done. sleep 5 for ZFS MDS Waiting for MDT destroys to complete fail_loc=0x314 fail_loc=0x316 fail_loc=0 fail_loc=0 Filesystem 1K-blocks Used Available Use% Mounted on 192.168.204.141@tcp:/lustre 7542784 40960 7497728 1% /mnt/lustre2 PASS 31b (31s) == sanityn test 31r: open-rename(replace) race =========== 01:38:42 (1785389922) -rw-r--r-- 1 root root 0 Jul 30 01:38 /mnt/lustre/f31r.sanityn.target fail_loc=0x1419 fail_val=3 PASS 31r (13s) == sanityn test 31s: open should not revalidate invalid dentry ========================================================== 01:38:55 (1785389935) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 1205: 43169 Terminated tail -f $tfile PASS 31s (11s) == sanityn test 31t: getattr should not revalidate invalid dentry ========================================================== 01:39:07 (1785389947) PASS 31t (9s) == sanityn test 32b: lockless i/o ======================== 01:39:16 (1785389956) SKIP: sanityn test_32b max_nolock_bytes is removed >= 2.17.53 SKIP 32b (5s) == sanityn test 32c: contention detection ================ 01:39:22 (1785389962) ldlm.namespaces.filter-lustre-OST0000_UUID.contended_locks=2 ldlm.namespaces.filter-lustre-OST0001_UUID.contended_locks=2 ldlm.namespaces.filter-lustre-OST0000_UUID.contention_events=0 ldlm.namespaces.filter-lustre-OST0001_UUID.contention_events=0 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.54632 s, 33.9 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.22402 s, 42.8 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 3.06265 s, 17.1 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 3.22646 s, 16.2 MB/s contention_seconds 4, contention_events 24 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.66573 s, 19.7 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 3.03079 s, 17.3 MB/s ldlm.namespaces.filter-lustre-OST0000_UUID.contention_events=0 ldlm.namespaces.filter-lustre-OST0001_UUID.contention_events=0 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.9439 s, 27.0 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.01972 s, 26.0 MB/s PASS 32c (40s) SKIP: sanityn test_33a skipping SLOW test 33a == sanityn test 33b: COS: cross create/delete, 2 clients, benchmark under remote dir ========================================================== 01:40:05 (1785390005) SKIP: sanityn test_33b Need two or more clients, have 1 SKIP 33b (5s) == sanityn test 33c: Cancel cross-MDT lock should trigger Sync-on-Lock-Cancel ========================================================== 01:40:11 (1785390011) SKIP: sanityn test_33c needs >= 2 MDTs SKIP 33c (3s) == sanityn test 33d: dependent transactions should trigger COS ========================================================== 01:40:15 (1785390015) SKIP: sanityn test_33d needs >= 2 MDTs SKIP 33d (5s) == sanityn test 33e: independent transactions shouldn't trigger COS ========================================================== 01:40:20 (1785390020) SKIP: sanityn test_33e needs >= 2 MDTs SKIP 33e (6s) == sanityn test 34: no lock timeout under IO ============= 01:40:25 (1785390025) fail_loc=0x512 fail_loc=0x512 lock should not expire writing on client1 reading on client2 success fail_loc=0x511 fail_loc=0x511 lock should expire writing on client1 reading on client2 success oleg441-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8be707e69000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8be707e69000.ost_server_uuid in IDLE state after 0 sec oleg441-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8be707e69000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8be707e69000.ost_server_uuid in IDLE state after 0 sec oleg441-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8be707e69000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8be707e69000.ost_server_uuid in IDLE state after 0 sec oleg441-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8be707e69000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8be707e69000.ost_server_uuid in IDLE state after 0 sec oleg441-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8be707e69000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8be707e69000.ost_server_uuid in IDLE state after 0 sec oleg441-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8be707e69000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8be707e69000.ost_server_uuid in IDLE state after 0 sec PASS 34 (170s) == sanityn test 35: -EINTR cp_ast vs. bl_ast race does not evict client ========================================================== 01:43:15 (1785390195) fail_loc=0x80000317 Race attempt 0 Wait for 50488 50501 for 60 sec... /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 1855: 50488 Killed createmany -o $MOUNT2/$tdir/a 4000 fail_loc=0x0 Filesystem Size Used Avail Use% Mounted on 192.168.204.141@tcp:/lustre 7.2G 40M 7.2G 1% /mnt/lustre 192.168.204.141@tcp:/lustre 7.2G 40M 7.2G 1% /mnt/lustre2 - open/close 470 (time 1785390217.60 total 10.02 last 46.93) - open/close 862 (time 1785390227.60 total 20.02 last 39.18) - open/close 1312 (time 1785390237.64 total 30.05 last 44.86) - open/close 1705 (time 1785390247.64 total 40.06 last 39.28) - open/close 2271 (time 1785390257.64 total 50.06 last 56.59) - open/close 3127 (time 1785390267.66 total 60.08 last 85.40) - open/close 3750 (time 1785390277.68 total 70.10 last 62.18) total: 4000 open/close in 73.50 seconds: 54.42 ops/second PASS 35 (86s) == sanityn test 36: handle ESTALE/open-unlink correctly == 01:44:41 (1785390281) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg441-server mds-ost sync done. Waiting for MDT destroys to complete 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.883727 s, 59.3 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7625 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(0) *** before(7497728) after_dd(7439360) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.02548 s, 51.1 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7625 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(1) *** before(7497728) after_dd(7425024) after(7476224) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.663806 s, 79.0 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7625 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(2) *** before(7476224) after_dd(7432192) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.06133 s, 49.4 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7625 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(3) *** before(7497728) after_dd(7440384) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.10532 s, 47.4 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7625 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(4) *** before(7497728) after_dd(7440384) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.92353 s, 56.8 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7625 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(5) *** before(7497728) after_dd(7434240) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.01367 s, 51.7 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7625 sleep 5 for ZFS MDS Waiting for MDT destroys to complete *** cycle(6) *** before(7497728) after_dd(7436288) after(7487488) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.677882 s, 77.3 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7625 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(7) *** before(7487488) after_dd(7422976) after(7474176) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.921233 s, 56.9 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7625 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(8) *** before(7474176) after_dd(7440384) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.00954 s, 51.9 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7625 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(9) *** before(7497728) after_dd(7439360) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.985464 s, 53.2 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7625 sleep 5 for ZFS MDS Waiting for MDT destroys to complete *** cycle(10) *** before(7497728) after_dd(7439360) after(7490560) PASS 36 (358s) == sanityn test 37: check i_size is not updated for directory on close (bug 18695) ======================================================================== 01:50:39 (1785390639) multiop /mnt/lustre/d37.sanityn vD_c TMPPIPE=/tmp/multiop_open_wait_pipe.7625 - create 1138 (time 1785390654.58 total 10.01 last 113.74) - create 2323 (time 1785390664.58 total 20.01 last 118.47) - create 4229 (time 1785390674.58 total 30.01 last 190.58) - create 5255 (time 1785390684.59 total 40.02 last 102.50) - create 6474 (time 1785390694.60 total 50.03 last 121.76) - create 7753 (time 1785390704.60 total 60.03 last 127.89) - create 9286 (time 1785390714.63 total 70.06 last 152.83) - create 9956 (time 1785390724.67 total 80.10 last 66.74) total: 10000 create in 80.67 seconds: 123.97 ops/second PASS 37 (100s) == sanityn test 39a: file mtime does not change after rename ========================================================== 01:52:19 (1785390739) repeat after cancel_lru_locks PASS 39a (9s) == sanityn test 39b: file mtime the same on clients with/out lock ========================================================== 01:52:28 (1785390748) repeat after cancel_lru_locks PASS 39b (10s) == sanityn test 39c: check truncate mtime update ================================================================================ 01:52:38 (1785390758) repeat after cancel_lru_locks PASS 39c (11s) == sanityn test 39d: sync write should update mtime ====== 01:52:49 (1785390769) fail_loc=0x411 fail_loc=0 PASS 39d (10s) SKIP: sanityn test_40a skipping ALWAYS excluded test 40a == sanityn test 40b: pdirops: open|create and others ======================================================================== 01:53:01 (1785390781) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40b (22s) == sanityn test 40c: pdirops: link and others ======================================================================== 01:53:23 (1785390803) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40c (22s) == sanityn test 40d: pdirops: unlink and others ======================================================================== 01:53:45 (1785390825) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40d (27s) == sanityn test 40e: pdirops: rename and others ======================================================================== 01:54:12 (1785390852) No conflict No conflict No conflict No conflict No conflict No conflict PASS 40e (19s) == sanityn test 41a: pdirops: create vs mkdir ======================================================================== 01:54:31 (1785390871) mkdir: cannot create directory '/mnt/lustre2/f41a.sanityn': File exists Conflict PASS 41a (15s) == sanityn test 41b: pdirops: create vs create ======================================================================== 01:54:46 (1785390886) open: File exists Conflict PASS 41b (16s) == sanityn test 41c: pdirops: create vs link ======================================================================== 01:55:02 (1785390902) link: cannot create link '/mnt/lustre2/f41c.sanityn' to '/mnt/lustre2/f41c.sanityn-2': File exists Conflict PASS 41c (17s) == sanityn test 41d: pdirops: create vs unlink ======================================================================== 01:55:19 (1785390919) Conflict PASS 41d (17s) == sanityn test 41e: pdirops: create and rename (tgt) ======================================================================== 01:55:36 (1785390936) Conflict PASS 41e (17s) == sanityn test 41f: pdirops: create and rename (src) ======================================================================== 01:55:53 (1785390953) Conflict PASS 41f (15s) == sanityn test 41g: pdirops: create vs getattr ======================================================================== 01:56:08 (1785390968) Conflict PASS 41g (15s) == sanityn test 41h: pdirops: create vs readdir ======================================================================== 01:56:24 (1785390984) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 2427: lctl set_param -n fail_loc=0 2>/dev/null || true: No such file or directory Conflict PASS 41h (16s) == sanityn test 41i: reint_open: create vs create ======== 01:56:39 (1785390999) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 41i (1280s) == sanityn test 42a: pdirops: mkdir vs mkdir ======================================================================== 02:17:59 (1785392279) mkdir: cannot create directory '/mnt/lustre2/f42a.sanityn': File exists Conflict PASS 42a (17s) == sanityn test 42b: pdirops: mkdir vs create ======================================================================== 02:18:16 (1785392296) open: File exists Conflict PASS 42b (17s) == sanityn test 42c: pdirops: mkdir vs link ======================================================================== 02:18:33 (1785392313) link: cannot create link '/mnt/lustre2/f42c.sanityn' to '/mnt/lustre2/f42c.sanityn-2': File exists Conflict PASS 42c (20s) == sanityn test 42d: pdirops: mkdir vs unlink ======================================================================== 02:18:53 (1785392333) Conflict PASS 42d (17s) == sanityn test 42e: pdirops: mkdir and rename (tgt) ======================================================================== 02:19:10 (1785392350) mv: cannot overwrite directory '/mnt/lustre2/f42e.sanityn' with non-directory Conflict PASS 42e (16s) == sanityn test 42f: pdirops: mkdir and rename (src) ======================================================================== 02:19:26 (1785392366) Conflict PASS 42f (16s) == sanityn test 42g: pdirops: mkdir vs getattr ======================================================================== 02:19:43 (1785392383) Conflict PASS 42g (17s) == sanityn test 42h: pdirops: mkdir vs readdir ======================================================================== 02:19:59 (1785392399) Conflict PASS 42h (21s) == sanityn test 43a: rmdir,mkdir doesn't return -EEXIST ======================================================================== 02:20:21 (1785392421) PASS 43a (79s) == sanityn test 43b: pdirops: unlink vs create ======================================================================== 02:21:40 (1785392500) Conflict PASS 43b (18s) == sanityn test 43c: pdirops: unlink vs link ======================================================================== 02:21:59 (1785392519) Conflict PASS 43c (26s) == sanityn test 43d: pdirops: unlink vs unlink ======================================================================== 02:22:25 (1785392545) rm: cannot remove '/mnt/lustre2/f43d.sanityn': No such file or directory Conflict PASS 43d (17s) == sanityn test 43e: pdirops: unlink and rename (tgt) ======================================================================== 02:22:42 (1785392562) Conflict PASS 43e (17s) == sanityn test 43f: pdirops: unlink and rename (src) ======================================================================== 02:22:59 (1785392579) mv: cannot stat '/mnt/lustre2/f43f.sanityn': No such file or directory Conflict PASS 43f (18s) == sanityn test 43g: pdirops: unlink vs getattr ======================================================================== 02:23:18 (1785392598) stat: cannot statx '/mnt/lustre2/f43g.sanityn': No such file or directory Conflict PASS 43g (19s) == sanityn test 43h: pdirops: unlink vs readdir ======================================================================== 02:23:37 (1785392617) Conflict PASS 43h (22s) == sanityn test 43i: pdirops: unlink vs remote mkdir ===== 02:23:59 (1785392639) SKIP: sanityn test_43i needs >= 2 MDTs SKIP 43i (5s) == sanityn test 43j: racy mkdir return EEXIST ======================================================================== 02:24:05 (1785392645) mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists PASS 43j (174s) == sanityn test 43k: unlink vs create ==================== 02:26:59 (1785392819) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 43k (1485s) == sanityn test 44a: pdirops: rename tgt vs mkdir ======================================================================== 02:51:44 (1785394304) mkdir: cannot create directory '/mnt/lustre2/f44a.sanityn': File exists Conflict Thu Jul 30 02:51:55 EDT 2026 PASS 44a (18s) == sanityn test 44b: pdirops: rename tgt vs create ======================================================================== 02:52:02 (1785394322) open: File exists Conflict PASS 44b (18s) == sanityn test 44c: pdirops: rename tgt vs link ======================================================================== 02:52:21 (1785394341) link: cannot create link '/mnt/lustre2/f44c.sanityn' to '/mnt/lustre2/f44c.sanityn-3': File exists Conflict PASS 44c (16s) == sanityn test 44d: pdirops: rename tgt vs unlink ======================================================================== 02:52:37 (1785394357) Conflict PASS 44d (14s) == sanityn test 44e: pdirops: rename tgt and rename (tgt) ======================================================================== 02:52:51 (1785394371) Conflict PASS 44e (16s) == sanityn test 44f: pdirops: rename tgt and rename (src) ======================================================================== 02:53:07 (1785394387) Conflict PASS 44f (17s) == sanityn test 44g: pdirops: rename tgt vs getattr ======================================================================== 02:53:25 (1785394405) Conflict PASS 44g (20s) == sanityn test 44h: pdirops: rename tgt vs readdir ======================================================================== 02:53:45 (1785394425) Conflict PASS 44h (18s) == sanityn test 44i: pdirops: rename tgt vs remote mkdir ========================================================== 02:54:03 (1785394443) SKIP: sanityn test_44i needs >= 2 MDTs SKIP 44i (4s) == sanityn test 45a: rename,mkdir doesn't return -EEXIST ======================================================================== 02:54:07 (1785394447) PASS 45a (146s) == sanityn test 45b: pdirops: rename src vs create ======================================================================== 02:56:33 (1785394593) Conflict PASS 45b (15s) == sanityn test 45c: pdirops: rename src vs link ======================================================================== 02:56:48 (1785394608) Conflict PASS 45c (14s) == sanityn test 45d: pdirops: rename src vs unlink ======================================================================== 02:57:02 (1785394622) rm: cannot remove '/mnt/lustre2/f45d.sanityn': No such file or directory Conflict PASS 45d (17s) == sanityn test 45e: pdirops: rename src and rename (tgt) ======================================================================== 02:57:20 (1785394640) Conflict PASS 45e (21s) == sanityn test 45f: pdirops: rename src and rename (src) ======================================================================== 02:57:41 (1785394661) mv: cannot stat '/mnt/lustre2/f45f.sanityn': No such file or directory Conflict PASS 45f (16s) == sanityn test 45g: pdirops: rename src vs getattr ======================================================================== 02:57:57 (1785394677) stat: cannot statx '/mnt/lustre2/f45g.sanityn': No such file or directory Conflict PASS 45g (16s) == sanityn test 45h: pdirops: unlink vs readdir ======================================================================== 02:58:14 (1785394694) Conflict PASS 45h (14s) == sanityn test 45i: pdirops: rename src vs remote mkdir ========================================================== 02:58:28 (1785394708) SKIP: sanityn test_45i needs >= 2 MDTs SKIP 45i (4s) == sanityn test 45j: read vs rename ====================== 02:58:32 (1785394712) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 45j (1577s) == sanityn test 46a: pdirops: link vs mkdir ======================================================================== 03:24:49 (1785396289) mkdir: cannot create directory '/mnt/lustre2/f46a.sanityn': File exists Conflict PASS 46a (20s) == sanityn test 46b: pdirops: link vs create ======================================================================== 03:25:09 (1785396309) open: File exists Conflict PASS 46b (22s) == sanityn test 46c: pdirops: link vs link ======================================================================== 03:25:31 (1785396331) link: cannot create link '/mnt/lustre2/f46c.sanityn' to '/mnt/lustre2/f46c.sanityn': File exists Conflict PASS 46c (22s) == sanityn test 46d: pdirops: link vs unlink ======================================================================== 03:25:53 (1785396353) Conflict PASS 46d (19s) == sanityn test 46e: pdirops: link and rename (tgt) ======================================================================== 03:26:12 (1785396372) Conflict PASS 46e (17s) == sanityn test 46f: pdirops: link and rename (src) ======================================================================== 03:26:29 (1785396389) Conflict PASS 46f (16s) == sanityn test 46g: pdirops: link vs getattr ======================================================================== 03:26:45 (1785396405) Conflict PASS 46g (19s) == sanityn test 46h: pdirops: link vs readdir ======================================================================== 03:27:04 (1785396424) Conflict PASS 46h (28s) == sanityn test 46i: pdirops: link vs remote mkdir ======= 03:27:32 (1785396452) SKIP: sanityn test_46i needs >= 2 MDTs SKIP 46i (6s) == sanityn test 47a: pdirops: remote mkdir vs mkdir ====== 03:27:39 (1785396459) SKIP: sanityn test_47a needs >= 2 MDTs SKIP 47a (6s) == sanityn test 47b: pdirops: remote mkdir vs create ===== 03:27:45 (1785396465) SKIP: sanityn test_47b needs >= 2 MDTs SKIP 47b (5s) == sanityn test 47c: pdirops: remote mkdir vs link ======= 03:27:50 (1785396470) SKIP: sanityn test_47c needs >= 2 MDTs SKIP 47c (6s) == sanityn test 47d: pdirops: remote mkdir vs unlink ===== 03:27:56 (1785396476) SKIP: sanityn test_47d needs >= 2 MDTs SKIP 47d (5s) == sanityn test 47e: pdirops: remote mkdir and rename (tgt) ========================================================== 03:28:01 (1785396481) SKIP: sanityn test_47e needs >= 2 MDTs SKIP 47e (6s) == sanityn test 47f: pdirops: remote mkdir and rename (src) ========================================================== 03:28:07 (1785396487) SKIP: sanityn test_47f needs >= 2 MDTs SKIP 47f (6s) == sanityn test 47g: pdirops: remote mkdir vs getattr ==== 03:28:13 (1785396493) SKIP: sanityn test_47g needs >= 2 MDTs SKIP 47g (5s) == sanityn test 50: osc lvb attrs: enqueue vs. CP AST ======================================================================== 03:28:18 (1785396498) 10+0 records in 10+0 records out 10240 bytes (10 kB, 10 KiB) copied, 0.207957 s, 49.2 kB/s fail_loc=0x410 fail_loc=0x0 PASS 50 (15s) == sanityn test 51a: layout lock: refresh layout should work ========================================================== 03:28:33 (1785396513) 0+1 records in 0+1 records out 158 bytes copied, 0.068171 s, 2.3 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 3690: kill: (218806) - No such process PASS 51a (17s) == sanityn test 51b: layout lock: glimpse should be able to restart if layout changed ========================================================== 03:28:50 (1785396530) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0420259 s, 24.4 kB/s fail_loc=0x1404 fail_val=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183579 s, 5.7 MB/s 3145728 PASS 51b (24s) == sanityn test 51c: layout lock: IT_LAYOUT blocked and correct layout can be returned ========================================================== 03:29:14 (1785396554) fail_loc=0x172 Setting layout to have 2 stripes ... 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0263843 s, 38.8 kB/s PASS 51c (15s) == sanityn test 51d: layout lock: losing layout lock should clean up memory map region ========================================================== 03:29:30 (1785396570) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0321823 s, 32.6 MB/s Before revoking layout lock: 1024 KB mapped PASS 51d (11s) == sanityn test 51e: lfs getstripe does not break leases, part 2 ========================================================== 03:29:41 (1785396581) /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 2454 0x996 0x240000400 write lease(2) has applied. /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 2454 0x996 0x240000400 read lease(1) has applied. PASS 51e (12s) == sanityn test 54: rename locking ======================= 03:29:53 (1785396593) llite.lustre-ffff8be707e69000.xattr_cache=1 llite.lustre-ffff8be710a2d000.xattr_cache=1 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 7680 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505015 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-07-30 03:29:56.000000000 -0400 Modify: 2026-07-30 03:29:56.000000000 -0400 Change: 2026-07-30 03:29:56.000000000 -0400 Birth: 2026-07-30 03:29:57.000000000 -0400 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000154 File: /mnt/lustre/d1/d2 Size: 7680 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505018 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-07-30 03:30:03.000000000 -0400 Modify: 2026-07-30 03:30:04.000000000 -0400 Change: 2026-07-30 03:30:04.000000000 -0400 Birth: 2026-07-30 03:30:03.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 512 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505021 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-07-30 03:30:10.000000000 -0400 Modify: 2026-07-30 03:30:10.000000000 -0400 Change: 2026-07-30 03:30:10.000000000 -0400 Birth: 2026-07-30 03:30:11.000000000 -0400 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1 Size: 7680 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505020 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-07-30 03:30:10.000000000 -0400 Modify: 2026-07-30 03:30:11.000000000 -0400 Change: 2026-07-30 03:30:11.000000000 -0400 Birth: 2026-07-30 03:30:10.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000154 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1/d2 Size: 7680 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505024 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-07-30 03:30:16.000000000 -0400 Modify: 2026-07-30 03:30:17.000000000 -0400 Change: 2026-07-30 03:30:17.000000000 -0400 Birth: 2026-07-30 03:30:17.000000000 -0400 File: /mnt/lustre/d1 Size: 7680 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505023 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-07-30 03:30:16.000000000 -0400 Modify: 2026-07-30 03:30:17.000000000 -0400 Change: 2026-07-30 03:30:17.000000000 -0400 Birth: 2026-07-30 03:30:17.000000000 -0400 PASS 54 (36s) == sanityn test 55a: rename vs unlink target dir ========= 03:30:29 (1785396629) fail_loc=0x80000156 mv: cannot move '/mnt/lustre/d55a.sanityn/d1/d2' to '/mnt/lustre/d55a.sanityn/d3/d2': No such file or directory PASS 55a (17s) == sanityn test 55b: rename vs unlink source dir ========= 03:30:46 (1785396646) fail_loc=0x80000156 mv: cannot stat '/mnt/lustre/d55b.sanityn/d1/d2': No such file or directory PASS 55b (15s) == sanityn test 55c: rename vs unlink orphan target dir == 03:31:01 (1785396661) fail_loc=0x156 mv: cannot move '/mnt/lustre/d55c.sanityn/d1/d2' to '/mnt/lustre/d55c.sanityn/d3/d2': No such file or directory PASS 55c (20s) == sanityn test 55d: rename file vs link ================= 03:31:21 (1785396681) fail_loc=0x155 mv: '/mnt/lustre/d55d.sanityn/f1' and '/mnt/lustre/d55d.sanityn/d55d.sanityn/f1' are the same file PASS 55d (17s) == sanityn test 55e: rename race AB/BA under the same parent dir ========================================================== 03:31:38 (1785396698) SKIP: sanityn test_55e needs >= 2 MDTs SKIP 55e (6s) == sanityn test 55f: rename: (P1/A -> P2/B) race with (P2/B -> P1/A) ========================================================== 03:31:45 (1785396705) fail_loc=0x155 We survived after P1A_P2B/P2B_P1A race test fail_loc=0x0 PASS 55f (21s) == sanityn test 55g: rename: race with trylock =========== 03:32:07 (1785396727) mdt.lustre-MDT0000.enable_rename_trylock=1 fail_loc=0x155 mv: cannot move '/mnt/lustre/d55g.sanityn/a1/a2' to a subdirectory of itself, '/mnt/lustre/d55g.sanityn/b1/b2/b3/b4/a2' We survived race with trylock test fail_loc=0x0 PASS 55g (29s) == sanityn test 56a: test llverdev with single large stripe ========================================================== 03:32:36 (1785396756) run_llverdev /mnt/lustre/f56a.sanityn -p -s 16000000 -c 4k llverdev: /mnt/lustre/f56a.sanityn is 16777216000000 bytes (15625 GB) in size timestamp: 1785396759 chunksize: 4096 size: 16777216000000 write offset: 16384000000kB 0.418076 MB/s read offset: 16384000000kB inf MB/s llverdev: data verified successfully PASS 56a (156s) == sanityn test 56b: test llverdev and partial verify of wide stripe file ========================================================== 03:35:12 (1785396912) run_llverdev /mnt/lustre/f56b.sanityn -p -s 32000000 -c 4k llverdev: /mnt/lustre/f56b.sanityn is 33554432000000 bytes (31250 GB) in size timestamp: 1785396924 chunksize: 4096 size: 33554432000000 write offset: 32768000000kB 0.891053 MB/s read offset: 32768000000kB inf MB/s llverdev: data verified successfully PASS 56b (159s) == sanityn test 60: Verify data_version behaviour ======== 03:37:52 (1785397072) PASS 60 (10s) == sanityn test 70a: cd directory && rm directory ======== 03:38:02 (1785397082) PASS 70a (9s) == sanityn test 70b: remove files after calling rm_entry ========================================================== 03:38:12 (1785397092) PASS 70b (10s) == sanityn test 71a: correct file map just after write operation is finished ========================================================== 03:38:22 (1785397102) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.022112 s, 3.0 MB/s SKIP: sanityn test_71a ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71a (5s) == sanityn test 71b: check fiemap support for stripecount > 1 ========================================================== 03:38:27 (1785397107) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0221122 s, 3.0 MB/s SKIP: sanityn test_71b ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71b (5s) == sanityn test 71c: check FIEMAP_EXTENT_LAST flag with different extents number ========================================================== 03:38:32 (1785397112) SKIP: sanityn test_71c support only ldiskfs ost SKIP 71c (3s) == sanityn test 71d: fiemap corruption test with fm_extent_count=0 ========================================================== 03:38:36 (1785397116) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0228459 s, 2.9 MB/s SKIP: sanityn test_71d ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71d (5s) == sanityn test 72: getxattr/setxattr cache should be consistent between nodes ========================================================== 03:38:40 (1785397120) llite.lustre-ffff8be707e69000.xattr_cache=1 llite.lustre-ffff8be710a2d000.xattr_cache=1 getfattr: Removing leading '/' from absolute path names user.attr1="value1" getfattr: Removing leading '/' from absolute path names user.attr1="value2" getfattr: Removing leading '/' from absolute path names getfattr: Removing leading '/' from absolute path names b362740ad371eae580e7f85452e8830b - 870e072da1ebfc20ea8c0cd3d6bdf3b6 - PASS 72 (11s) == sanityn test 73: getxattr should not cause xattr lock cancellation ========================================================== 03:38:51 (1785397131) llite.lustre-ffff8be707e69000.xattr_cache=1 llite.lustre-ffff8be710a2d000.xattr_cache=1 getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" PASS 73 (9s) == sanityn test 74: flock deadlock: different mounts ======================================================================== 03:39:01 (1785397141) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0365115 s, 28.0 kB/s 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0254182 s, 40.3 kB/s 231642: get lock1 231641: get lock2 231641: done 231641: get lock1 231642: done 231642: get lock2 fcntl cmd 7 failed: Resource deadlock avoided 231642: cannot get lock2: Resource deadlock avoided 231642: exit rc=0 231641: done 231641: exit rc=0 PASS 74 (11s) == sanityn test 75: osc: upcall after unuse lock============================================================================= 03:39:12 (1785397152) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0598675 s, 35.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0346355 s, 30.3 MB/s fail_loc=0x31d /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 4373: 232223 Killed stat -c %s $DIR1/$tfile fail_loc=0 PASS 75 (13s) == sanityn test 76: Verify MDT open_files listing ======== 03:39:25 (1785397165) opening files: ........ closing files: ........ PASS 76 (82s) == sanityn test 77a: check FIFO NRS policy =============== 03:40:48 (1785397248) ost.OSS.ost.nrs_policies=fifo ost.OSS.ost_create.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_out.nrs_policies=fifo ost.OSS.ost_seq.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.37129 s, 45.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0256888 s, 40.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0311703 s, 33.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0365179 s, 28.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0263444 s, 39.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0405381 s, 25.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0683382 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102587 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13973 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0366943 s, 28.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0505116 s, 20.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0517297 s, 20.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164122 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0748624 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0906128 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215914 s, 4.9 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8be707e69000.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181341 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177698 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205677 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.250404 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211237 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200453 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131604 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14097 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169766 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23966 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.241606 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168231 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.243125 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184381 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197056 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.268212 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.326537 s, 3.2 MB/s removed '/mnt/lustre/d77a.sanityn/nrs_w_oleg441-client.virtnet' removed '/mnt/lustre/d77a.sanityn/nrs_r_oleg441-client.virtnet' removed directory '/mnt/lustre/d77a.sanityn' PASS 77a (13s) == sanityn test 77b: check CRR-N NRS policy ============== 03:41:00 (1785397260) ost.OSS.ost.nrs_policies=crrn ost.OSS.ost_create.nrs_policies=crrn ost.OSS.ost_io.nrs_policies=crrn ost.OSS.ost_out.nrs_policies=crrn ost.OSS.ost_seq.nrs_policies=crrn ost.OSS.ost.nrs_crrn_quantum=1 ost.OSS.ost_create.nrs_crrn_quantum=1 ost.OSS.ost_io.nrs_crrn_quantum=1 ost.OSS.ost_out.nrs_crrn_quantum=1 ost.OSS.ost_seq.nrs_crrn_quantum=1 policy: crr-n, crrn_quantum 1 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.235574 s, 71.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0281358 s, 37.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142169 s, 7.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0967992 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101573 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0818932 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0743653 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0383667 s, 27.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.217002 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.24944 s, 4.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151641 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199031 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221498 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.2101 s, 5.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8be707e69000.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffff8be707e69000.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171013 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.280772 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.304179 s, 3.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8be707e69000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0021842 s, 480 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00217323 s, 482 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00219003 s, 479 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132589 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00619797 s, 169 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113942 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.238843 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174795 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10235 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0939317 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0600271 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0547543 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0385875 s, 27.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0502444 s, 20.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0369877 s, 28.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0265541 s, 39.5 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg441-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg441-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost.nrs_crrn_quantum=64 ost.OSS.ost_create.nrs_crrn_quantum=64 ost.OSS.ost_io.nrs_crrn_quantum=64 ost.OSS.ost_out.nrs_crrn_quantum=64 ost.OSS.ost_seq.nrs_crrn_quantum=64 policy: crr-n, crrn_quantum 64 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.394455 s, 42.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0347671 s, 30.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0228752 s, 45.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0504968 s, 20.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0424653 s, 24.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120174 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0194129 s, 54.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148974 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0650476 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0355918 s, 29.5 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8be707e69000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0420187 s, 25.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0274728 s, 38.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.268832 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16187 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144322 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155037 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208379 s, 5.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8be707e69000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184935 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220763 s, 4.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159559 s, 6.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0433604 s, 24.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163757 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180654 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0626648 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185689 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194822 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195973 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.233632 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205306 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153175 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.254548 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.257289 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.228733 s, 4.6 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg441-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg441-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77b (15s) == sanityn test 77c: check ORR NRS policy ================ 03:41:16 (1785397276) ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_orr_quantum=1 ost.OSS.ost_io.nrs_orr_offset_type=physical ost.OSS.ost_io.nrs_orr_supported=reads policy: orr, orr_quantum 1, orr_offset_type physical, orr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.306777 s, 54.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0341138 s, 30.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0361947 s, 29.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135012 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148963 s, 7.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.235325 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200407 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175826 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.282408 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.267022 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222984 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207853 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191568 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.24128 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.320466 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.284187 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.461181 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186144 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0747976 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0597009 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0729653 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0668033 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0725183 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.283027 s, 3.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0978643 s, 10.7 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0979844 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103975 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122261 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122846 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127209 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138249 s, 7.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141464 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.342821 s, 3.1 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg441-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg441-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=writes ost.OSS.ost_io.nrs_orr_quantum=64 policy: orr, orr_quantum 64, orr_offset_type physical, orr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.389451 s, 43.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128175 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151365 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17841 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181926 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.226965 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108773 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202432 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134198 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.267017 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.297397 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21305 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201207 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.225599 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.338557 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.330103 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.328402 s, 3.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8be707e69000.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.063824 s, 16.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116639 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0712029 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0802637 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0271546 s, 38.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0892412 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0920217 s, 11.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0832352 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0866859 s, 12.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.062092 s, 16.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0859939 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.099356 s, 10.6 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0972902 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0439736 s, 23.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.010622 s, 98.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136867 s, 7.7 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg441-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg441-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=reads_and_writes ost.OSS.ost_io.nrs_orr_offset_type=logical policy: orr, orr_quantum 64, orr_offset_type logical, orr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.298852 s, 56.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0564059 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0769481 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108162 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0940493 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.096863 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144729 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0967498 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153566 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157664 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130848 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183353 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0927569 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174748 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166994 s, 6.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8be707e69000.lock_unused_count=3 ldlm.namespaces.lustre-OST0001-osc-ffff8be707e69000.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188848 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.355038 s, 3.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8be707e69000.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0312751 s, 33.5 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0879822 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0241969 s, 43.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166994 s, 62.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0735511 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14973 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0824979 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0102559 s, 102 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00222432 s, 471 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177559 s, 5.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21471 s, 4.9 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192518 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170933 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211673 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0490108 s, 21.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0189976 s, 55.2 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg441-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg441-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77c (21s) == sanityn test 77d: check TRR nrs policy ================ 03:41:36 (1785397296) ost.OSS.ost_io.nrs_policies=trr ost.OSS.ost_io.nrs_trr_quantum=1 ost.OSS.ost_io.nrs_trr_offset_type=physical ost.OSS.ost_io.nrs_trr_supported=reads policy: trr, trr_quantum 1, trr_offset_type physical, trr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.303185 s, 55.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0467275 s, 22.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0531637 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13958 s, 7.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101439 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169401 s, 6.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112517 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102983 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179203 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162388 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205714 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138492 s, 7.6 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8be707e69000.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153418 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1495 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175171 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.241668 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.315277 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0574097 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0773337 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0503501 s, 20.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0618924 s, 16.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0596977 s, 17.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0749178 s, 14.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8be707e69000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121418 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0722721 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0936741 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0750533 s, 14.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104612 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0947904 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0866933 s, 12.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0857843 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117946 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0966789 s, 10.8 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg441-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg441-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=writes ost.OSS.ost_io.nrs_trr_quantum=64 policy: trr, trr_quantum 64, trr_offset_type physical, trr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.36691 s, 45.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0506165 s, 20.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0838541 s, 12.5 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0800581 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.418978 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.332162 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188199 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.251632 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155826 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.438991 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.270865 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.2081 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.367707 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.289791 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.257003 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.433741 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.64527 s, 1.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8be707e69000.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff8be707e69000.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0714889 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0822937 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164741 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183948 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101382 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0990093 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216048 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103959 s, 10.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209723 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107707 s, 9.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0968601 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11512 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206513 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123156 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107464 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.2664 s, 3.9 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg441-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg441-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=reads_and_writes ost.OSS.ost_io.nrs_trr_offset_type=logical policy: trr, trr_quantum 64, trr_offset_type logical, trr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.482437 s, 34.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0213988 s, 49.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0650976 s, 16.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.048531 s, 21.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106212 s, 9.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0450474 s, 23.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0982389 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.26681 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180167 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.265787 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.302751 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.303811 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212749 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0309195 s, 33.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.350543 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.313926 s, 3.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8be707e69000.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffff8be707e69000.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.342483 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107295 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0679251 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0571302 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0692969 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0811509 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0909054 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0659602 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0905982 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0838901 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0921836 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12259 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112148 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100162 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161642 s, 64.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116919 s, 9.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8be707e69000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19078 s, 5.5 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg441-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg441-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77d (23s) == sanityn test 77e: check TBF NID nrs policy ============ 03:42:00 (1785397320) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_tbf_rule=start localhost nid={0@lo} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start clients nid={192.168.204.41@tcp} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start others nid={*.*.*.*@tcp} rate=50 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.418122 s, 40.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0266024 s, 39.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0384017 s, 27.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0663041 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138776 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160245 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0946906 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146404 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186558 s, 5.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177548 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175756 s, 6.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178481 s, 5.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184105 s, 5.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22165 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207537 s, 5.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8be707e69000.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff8be707e69000.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.255143 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.334767 s, 3.1 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8be707e69000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0779755 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0653835 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166478 s, 63.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0628234 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0375411 s, 27.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.043297 s, 24.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0477805 s, 21.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0448559 s, 23.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.09475 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0329912 s, 31.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121737 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0397764 s, 26.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0486036 s, 21.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0714167 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113796 s, 9.2 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0609363 s, 17.2 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg441-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg441-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change localhost rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change clients rate=101 ost.OSS.ost_io.nrs_tbf_rule=change others rate=51 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.308012 s, 54.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0398959 s, 26.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0637516 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0422946 s, 24.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0743992 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0517253 s, 20.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0331949 s, 31.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0509136 s, 20.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179889 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210855 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.233663 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161296 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140841 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168506 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0931282 s, 11.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8be707e69000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff8be707e69000.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1944 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.225204 s, 4.7 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8be707e69000.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00435492 s, 241 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176173 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00219257 s, 478 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131573 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185776 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136677 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0938269 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159621 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12337 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0870805 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0812173 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0804711 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0734084 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0623266 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0542413 s, 19.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0370342 s, 28.3 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg441-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg441-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop localhost ost.OSS.ost_io.nrs_tbf_rule=stop clients ost.OSS.ost_io.nrs_tbf_rule=stop others 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.349116 s, 48.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.066046 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0850839 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0895564 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1881 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0902647 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120534 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176856 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223055 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208265 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208159 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.235872 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196587 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.217296 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189874 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.27115 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.307721 s, 3.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8be707e69000.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff8be707e69000.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0502532 s, 20.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169959 s, 61.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0194648 s, 53.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00617103 s, 170 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0528104 s, 19.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0594783 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101132 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0280879 s, 37.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0696216 s, 15.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0378269 s, 27.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0430522 s, 24.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0428993 s, 24.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.043867 s, 23.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0834738 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0912616 s, 11.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8be707e69000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103163 s, 10.2 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg441-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg441-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.305684 s, 54.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168809 s, 62.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0305052 s, 34.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0363352 s, 28.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0760958 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0288019 s, 36.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.053515 s, 19.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0950693 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0488665 s, 21.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0529819 s, 19.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0279704 s, 37.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18067 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.214967 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17879 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172452 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145779 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.264611 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0712659 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0759359 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0383446 s, 27.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0936885 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0799783 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0986641 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0476249 s, 22.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0652517 s, 16.1 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0761185 s, 13.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0856673 s, 12.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126974 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0840882 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0823417 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0835613 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118679 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120224 s, 8.7 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg441-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg441-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' PASS 77e (28s) == sanityn test 77f: check TBF JobID nrs policy ========== 03:42:27 (1785397347) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start runas jobid={iozone.500 dd.500 tiotest.500} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start iozone_runas jobid={iozone.500} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg441-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.214677 s, 78.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0499421 s, 21.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.029766 s, 35.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0819629 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121529 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151481 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131232 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104753 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183982 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145265 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173893 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.225637 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.234009 s, 4.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23649 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.258626 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.29838 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.268133 s, 3.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8be707e69000.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff8be707e69000.lock_unused_count=6 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00403884 s, 260 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0334317 s, 31.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016585 s, 63.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0384182 s, 27.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0817345 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00449876 s, 233 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0106603 s, 98.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00538941 s, 195 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00314379 s, 334 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00348271 s, 301 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0652487 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0527691 s, 19.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0536733 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0289949 s, 36.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0204075 s, 51.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00813935 s, 129 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg441-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg441-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change runas rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change iozone_runas rate=101 ost.OSS.ost_io.nrs_tbf_rule=change dd_runas rate=51 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg441-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.373931 s, 44.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0773559 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124159 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107692 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.218838 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150795 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177893 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103202 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149406 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.34952 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.410582 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.368452 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.344746 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.357562 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.346721 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.405703 s, 2.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.35748 s, 2.9 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8be707e69000.lock_unused_count=7 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0629745 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0687556 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.077358 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0893065 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0899434 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110735 s, 9.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16854 s, 6.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166812 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108444 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108736 s, 9.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115302 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117045 s, 9.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108853 s, 9.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11849 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132052 s, 7.9 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132842 s, 7.9 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg441-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg441-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop runas ost.OSS.ost_io.nrs_tbf_rule=stop iozone_runas ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg441-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.355576 s, 47.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0424276 s, 24.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0974777 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0926827 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113314 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0721535 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151768 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175492 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116959 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185453 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199508 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222603 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197769 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.246328 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.240355 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189217 s, 5.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8be707e69000.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff8be707e69000.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.244365 s, 4.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0875968 s, 12.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0828575 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0454181 s, 23.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0509414 s, 20.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0697093 s, 15.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0847122 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0690151 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0701648 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0691863 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.061984 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.064746 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0803947 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0906286 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0660329 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0929742 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.085408 s, 12.3 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg441-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg441-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_policies=fifo running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg441-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.350326 s, 47.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0347844 s, 30.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0464902 s, 22.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0886848 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0692697 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158509 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190607 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0349307 s, 30.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173745 s, 6.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.234925 s, 4.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165168 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107372 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208826 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.235657 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202831 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195804 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.288476 s, 3.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8be707e69000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff8be707e69000.lock_unused_count=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00510411 s, 205 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8be707e69000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.055771 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0111537 s, 94.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0257954 s, 40.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0104073 s, 101 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00665171 s, 158 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136633 s, 76.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0929504 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0971995 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131667 s, 79.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103077 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171887 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108378 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0648279 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0710966 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0690766 s, 15.2 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg441-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg441-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' PASS 77f (28s) == sanityn test 77g: Change TBF type directly ============ 03:42:56 (1785397376) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 ost.OSS.ost_io.nrs_policies=fifo PASS 77g (15s) == sanityn test 77h: Wrong policy name should report error, not LBUG ========================================================== 03:43:11 (1785397391) oleg441-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=abc: No such file or directory oleg441-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='abc': No such file or directory pdsh@oleg441-client: oleg441-server: ssh exited with exit code 2 oleg441-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc: Invalid argument oleg441-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc': Invalid argument pdsh@oleg441-client: oleg441-server: ssh exited with exit code 22 oleg441-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf reg abc: Invalid argument oleg441-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf reg abc': Invalid argument pdsh@oleg441-client: oleg441-server: ssh exited with exit code 22 oleg441-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc efg: Invalid argument oleg441-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc efg': Invalid argument pdsh@oleg441-client: oleg441-server: ssh exited with exit code 22 PASS 77h (14s) == sanityn test 77i: Change rank of TBF rule ============= 03:43:25 (1785397405) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start before jobid={jobid} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start after jobid={jobid} rate=1000 rank=default ost.OSS.ost_io.nrs_tbf_rule=start target jobid={jobid} rate=1000 rank=after Move before itself ost.OSS.ost_io.nrs_tbf_rule=change target rank=target Move to higher rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=before Move to lower rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=after Move before default ost.OSS.ost_io.nrs_tbf_rule=change target rank=default ost.OSS.ost_io.nrs_policies=fifo PASS 77i (30s) == sanityn test 77j: check TBF-OPCode NRS policy ========= 03:43:55 (1785397435) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf opcode ost.OSS.ost_io.nrs_tbf_rule=start ost_r opcode={ost_read} rate=5 ost.OSS.ost_io.nrs_tbf_rule=start ost_w opcode={ost_write} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.640392 s, 26.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0621466 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0357979 s, 29.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0277693 s, 37.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120961 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.245003 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.274887 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.229305 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191977 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.251207 s, 4.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.345547 s, 3.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.314387 s, 3.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.321969 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193001 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205961 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.248353 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.349849 s, 3.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8be707e69000.lock_unused_count=5 ldlm.namespaces.lustre-OST0001-osc-ffff8be707e69000.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.291454 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.311878 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.283571 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221825 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.295587 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.312184 s, 3.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.261851 s, 4.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.327098 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.315489 s, 3.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.334119 s, 3.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.324944 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.234688 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.360058 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.348313 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.385284 s, 2.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8be707e69000.lock_unused_count=12 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.372502 s, 2.8 MB/s removed '/mnt/lustre/d77j.sanityn/nrs_w_oleg441-client.virtnet' removed '/mnt/lustre/d77j.sanityn/nrs_r_oleg441-client.virtnet' removed directory '/mnt/lustre/d77j.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 5 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 14.3326 s, 7.3 MB/s Write runtime is 16 s, speed is 6.250000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.7599 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77j.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_r ost.OSS.ost_io.nrs_tbf_rule=stop ost_w ost.OSS.ost_io.nrs_policies=fifo PASS 77j (78s) == sanityn test 77ja: check TBF-UID/GID NRS policy ======= 03:45:13 (1785397513) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg441-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg441-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg441-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg441-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg441-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.279 s, 13.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.085397 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0973945 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.054563 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144774 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0810886 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186546 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187716 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118414 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149343 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15006 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151932 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127795 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108315 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140903 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.288858 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.442798 s, 2.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121234 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0479093 s, 21.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150604 s, 69.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00216807 s, 484 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144428 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175044 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146455 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179221 s, 5.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152316 s, 6.9 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0887027 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0776061 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0392415 s, 26.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188438 s, 5.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.09868 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113516 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205284 s, 5.1 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg441-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg441-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.5787 s, 5.1 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.7779 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg441-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg441-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg441-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg441-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg441-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.26169 s, 13.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107012 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12456 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0856339 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149248 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.247951 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222347 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.301275 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.236063 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.273385 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.248661 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.304879 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.32297 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.340375 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.355883 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.228875 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.331238 s, 3.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0842523 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107743 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0538024 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0458063 s, 22.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0961427 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0923835 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012441 s, 84.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161527 s, 64.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0995215 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0113164 s, 92.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8be707e69000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00672505 s, 156 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0338857 s, 30.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131229 s, 8.0 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.028541 s, 36.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00828307 s, 127 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170754 s, 6.1 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg441-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg441-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 21.5566 s, 4.9 MB/s Write runtime is 22 s, speed is 4.545454 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8801 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77ja (167s) == sanityn test 77jb: check TBF-UID/GID NRS policy on files that don't belong to us ========================================================== 03:48:01 (1785397681) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg441-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg441-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg441-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg441-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg441-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.33205 s, 12.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162769 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180293 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182377 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.262597 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.266474 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.229349 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.372088 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.374054 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.36862 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.359971 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.342907 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.376764 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.38203 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.352324 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.435814 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.544345 s, 1.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0782783 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110462 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124445 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126588 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1355 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0291588 s, 36.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131073 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110052 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0917394 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158517 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0652798 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117582 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131053 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173566 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179005 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180708 s, 5.8 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg441-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg441-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.6293 s, 5.1 MB/s Write runtime is 22 s, speed is 4.545454 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.0708 s, 5.2 MB/s Read runtime is 22 s, speed is 4.545454 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg441-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg441-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg441-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg441-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg441-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.26557 s, 13.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.060421 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0639462 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147409 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154739 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133085 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166198 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132033 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.253326 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.267949 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.317483 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129415 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108887 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.370185 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.236209 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21574 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.495083 s, 2.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0361657 s, 29.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0547531 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0348999 s, 30.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0406965 s, 25.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0597498 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.047991 s, 21.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0670851 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0542167 s, 19.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0558635 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0676023 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0668824 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0790553 s, 13.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0937579 s, 11.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0721883 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0604036 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.088466 s, 11.9 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg441-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg441-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.4377 s, 5.1 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.7938 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77jb (164s) == sanityn test 77k: check TBF policy with UID/GID/JobID/OPCode expression ========================================================== 03:50:45 (1785397845) ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg441-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.479909 s, 35.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0361331 s, 29.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0386595 s, 27.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0317144 s, 33.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155574 s, 67.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0340003 s, 30.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0555171 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115937 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0893764 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108359 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0975716 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0751897 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0868558 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189847 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185541 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170605 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169017 s, 6.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107184 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118566 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0252247 s, 41.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123564 s, 8.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144391 s, 7.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0599458 s, 17.5 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131319 s, 8.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139873 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108374 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14742 s, 7.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139896 s, 7.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00264052 s, 397 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147969 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17143 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0892866 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123965 s, 8.5 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg441-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg441-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.6591 s, 10.9 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.2802 s, 9.3 MB/s Read runtime is 12 s, speed is 8.333333 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.204.41@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.204.41@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.646556 s, 25.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0805152 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0951808 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0769463 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0696911 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146173 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0457877 s, 22.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0711452 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0541698 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0506318 s, 20.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.249017 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.238293 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.245961 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188185 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.27901 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200743 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.271906 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.077908 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0750916 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0410816 s, 25.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0574397 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0584462 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.074672 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0542054 s, 19.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0655605 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111477 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0667026 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0865024 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10039 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0961057 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116315 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0944959 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127605 s, 8.2 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg441-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg441-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 12.7059 s, 8.3 MB/s Write runtime is 14 s, speed is 7.142857 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.3643 s, 10.1 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.204.41@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg441-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.643316 s, 26.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0337401 s, 31.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0732739 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0939125 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11923 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.218183 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207169 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201091 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.234518 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.231419 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.233611 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180459 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.226324 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176219 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19229 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195717 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.291836 s, 3.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.071586 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00713229 s, 147 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0850809 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00993156 s, 106 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00346439 s, 303 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0992566 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149481 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00542258 s, 193 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0400377 s, 26.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0295267 s, 35.5 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0358757 s, 29.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0211974 s, 49.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212969 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0402179 s, 26.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0557448 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146323 s, 7.2 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg441-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg441-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.2733 s, 9.3 MB/s Write runtime is 13 s, speed is 7.692307 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.03793 s, 17.4 MB/s Read runtime is 7 s, speed is 14.285714 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg441-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.749742 s, 22.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0355001 s, 29.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0367237 s, 28.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107131 s, 9.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180081 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105888 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138818 s, 7.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0347922 s, 30.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162859 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151051 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208517 s, 5.0 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147436 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123202 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.246658 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.282598 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.301905 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.274194 s, 3.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0351343 s, 29.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00424179 s, 247 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00220563 s, 475 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00242362 s, 433 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0821402 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146993 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149992 s, 69.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0297799 s, 35.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161478 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0334442 s, 31.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117157 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158165 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108524 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127195 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0594079 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110407 s, 9.5 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg441-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg441-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 12.9178 s, 8.1 MB/s Write runtime is 14 s, speed is 7.142857 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0979 s, 10.4 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 12.8414 s, 8.2 MB/s Write runtime is 14 s, speed is 7.142857 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.049 s, 10.4 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg441-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.10509 s, 15.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0783411 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.056988 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180183 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129483 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128805 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189815 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185605 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.337525 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.322155 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.288152 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.267485 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.344073 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.380506 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.325269 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.391589 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.467388 s, 2.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0391715 s, 26.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0519705 s, 20.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0528612 s, 19.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0573001 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0781491 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0866581 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0850211 s, 12.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0886392 s, 11.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10999 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156498 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121471 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122954 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124251 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125165 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109721 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148663 s, 7.1 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg441-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg441-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 21.0467 s, 5.0 MB/s Write runtime is 23 s, speed is 4.347826 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.7026 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_ug ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg441-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.371277 s, 45.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0368138 s, 28.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.050302 s, 20.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0308687 s, 34.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139155 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166232 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100443 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143037 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115814 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166245 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106968 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207659 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147221 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180189 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177026 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.245165 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205872 s, 5.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.064242 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0364901 s, 28.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.070304 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0252546 s, 41.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0497381 s, 21.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0580998 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.059477 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0483763 s, 21.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0495401 s, 21.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.065814 s, 15.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.053439 s, 19.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.060575 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0969884 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0841369 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0996001 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00223509 s, 469 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg441-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg441-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 12.0944 s, 8.7 MB/s Write runtime is 13 s, speed is 7.692307 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.2532 s, 10.2 MB/s Read runtime is 12 s, speed is 8.333333 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg441-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.725035 s, 23.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0819087 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100867 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16764 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222705 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155504 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180652 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200363 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143133 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.25856 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18925 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.218938 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.256868 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.241525 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.314335 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.357507 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.276406 s, 3.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0878281 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110043 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0427587 s, 24.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123931 s, 8.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0591461 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0610477 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0566158 s, 18.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0651814 s, 16.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141928 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0790946 s, 13.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151794 s, 69.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0681351 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0896053 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117509 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100392 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132925 s, 7.9 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg441-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg441-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 14.4884 s, 7.2 MB/s Write runtime is 15 s, speed is 6.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.2379 s, 10.2 MB/s Read runtime is 12 s, speed is 8.333333 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 13.9004 s, 7.5 MB/s Write runtime is 15 s, speed is 6.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0748 s, 10.4 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_policies=fifo PASS 77k (502s) == sanityn test 77kb: Check different granular TBF type combination: jobid+opcode ========================================================== 03:59:08 (1785398348) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/nrs_r_oleg441-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.713156 s, 23.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0644302 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0462572 s, 22.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115103 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0511553 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10376 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.279789 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.267619 s, 3.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.267263 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.290558 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.310947 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.316771 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199022 s, 5.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215494 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.375675 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.347361 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.392497 s, 2.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8be707e69000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff8be707e69000.lock_unused_count=2 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0791208 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0677755 s, 15.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0500064 s, 21.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0578813 s, 18.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0625129 s, 16.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0620426 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.065704 s, 16.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0950306 s, 11.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10084 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0803615 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0936642 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0967908 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0790766 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101152 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14039 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111882 s, 9.4 MB/s removed '/mnt/lustre/d77kb.sanityn/nrs_w_oleg441-client.virtnet' removed '/mnt/lustre/d77kb.sanityn/nrs_r_oleg441-client.virtnet' removed directory '/mnt/lustre/d77kb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 14.9905 s, 7.0 MB/s Write runtime is 16 s, speed is 6.250000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0567 s, 10.4 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kb (65s) == sanityn test 77kc: Check different granular TBF type combination: nid+opcode ========================================================== 04:00:13 (1785398413) ost.OSS.ost_io.nrs_policies=tbf nid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.204.41@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.204.41@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.64857 s, 25.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0307872 s, 34.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0652545 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0642841 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119525 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.052193 s, 20.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0476437 s, 22.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0695858 s, 15.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0181899 s, 57.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0667827 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183523 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127679 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118054 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177161 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116134 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0904937 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101529 s, 10.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119529 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0591752 s, 17.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119089 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0772916 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142894 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121267 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0674706 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0827665 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111123 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112649 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.091169 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0921935 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0776891 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.085984 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102778 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165447 s, 6.3 MB/s removed '/mnt/lustre/d77kc.sanityn/nrs_r_oleg441-client.virtnet' removed '/mnt/lustre/d77kc.sanityn/nrs_w_oleg441-client.virtnet' removed directory '/mnt/lustre/d77kc.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 14.0767 s, 7.4 MB/s Write runtime is 15 s, speed is 6.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.2238 s, 10.3 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kc.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kc (61s) == sanityn test 77kd: Check different granular TBF type combination: nid+jobid ========================================================== 04:01:14 (1785398474) ost.OSS.ost_io.nrs_policies=tbf nid+jobid ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.204.41@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/nrs_r_oleg441-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.50541 s, 33.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0623754 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0742683 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0662387 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0928413 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14164 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0855252 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177666 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220591 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174317 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220455 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127712 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208959 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.27167 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.273667 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221865 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.259234 s, 4.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0935555 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0912191 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0536816 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0307087 s, 34.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107739 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147912 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0405318 s, 25.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0586337 s, 17.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11382 s, 9.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.04957 s, 21.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0518704 s, 20.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0596292 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0675768 s, 15.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0246231 s, 42.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00626718 s, 167 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0394705 s, 26.6 MB/s removed '/mnt/lustre/d77kd.sanityn/nrs_w_oleg441-client.virtnet' removed '/mnt/lustre/d77kd.sanityn/nrs_r_oleg441-client.virtnet' removed directory '/mnt/lustre/d77kd.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 13.4571 s, 7.8 MB/s Write runtime is 15 s, speed is 6.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kd.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.53709 s, 16.0 MB/s Read runtime is 8 s, speed is 12.500000 IOPS cleanup_tbf /mnt/lustre/d77kd.sanityn sleep 5 for ZFS MDS Waiting for MDT destroys to complete PASS 77kd (50s) == sanityn test 77ke: Check different granular TBF type combination: jobid+opcode ========================================================== 04:02:04 (1785398524) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/nrs_r_oleg441-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.699895 s, 24.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.070131 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0683487 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0280879 s, 37.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0814346 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0457308 s, 22.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.041204 s, 25.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0473066 s, 22.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0962298 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161511 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143264 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0198503 s, 52.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.261202 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205353 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150595 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.276805 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.406541 s, 2.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0751928 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0832396 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0949144 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116775 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0900751 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0721175 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106917 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0883826 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.097332 s, 10.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119341 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0819268 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0936202 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107225 s, 9.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117288 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0630334 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178208 s, 5.9 MB/s removed '/mnt/lustre/d77ke.sanityn/nrs_w_oleg441-client.virtnet' removed '/mnt/lustre/d77ke.sanityn/nrs_r_oleg441-client.virtnet' removed directory '/mnt/lustre/d77ke.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 13.9431 s, 7.5 MB/s Write runtime is 15 s, speed is 6.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ke.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.1851 s, 10.3 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.02959 s, 13.1 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.3381 s, 10.1 MB/s Read runtime is 12 s, speed is 8.333333 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77ke (104s) == sanityn test 77kf: Check different granular TBF type combination: uid+gid ========================================================== 04:03:48 (1785398628) ost.OSS.ost_io.nrs_policies=tbf uid+gid ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/nrs_r_oleg441-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.09555 s, 15.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0709007 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0855476 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0609992 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102962 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140591 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215683 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227428 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136625 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147387 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186107 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.261025 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175155 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.299135 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.328145 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.340351 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.36786 s, 2.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10896 s, 9.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0310755 s, 33.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0255058 s, 41.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0282212 s, 37.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0361323 s, 29.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0517607 s, 20.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0525599 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136444 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0641419 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0574122 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0684426 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115844 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127773 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152158 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.087151 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102238 s, 10.3 MB/s removed '/mnt/lustre/d77kf.sanityn/nrs_r_oleg441-client.virtnet' removed '/mnt/lustre/d77kf.sanityn/nrs_w_oleg441-client.virtnet' removed directory '/mnt/lustre/d77kf.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.3885 s, 5.1 MB/s Write runtime is 22 s, speed is 4.545454 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kf.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.819 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77kf.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kf (80s) == sanityn test 77kg: check different granular TBF type combination: uid+opcode ========================================================== 04:05:08 (1785398708) ost.OSS.ost_io.nrs_policies=tbf uid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg441-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.495694 s, 33.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0520584 s, 20.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0618924 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121044 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145738 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181092 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16963 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193805 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220002 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199483 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190732 s, 5.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.226846 s, 4.6 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211793 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.286285 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.25123 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.27346 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.258484 s, 4.1 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8be707e69000.lock_unused_count=2 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0579916 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0276784 s, 37.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0732472 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0621313 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0749494 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0608827 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0439557 s, 23.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0801694 s, 13.1 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0647146 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123891 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0657583 s, 15.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0652234 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0780206 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0909514 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0841911 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113283 s, 9.3 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg441-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg441-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 14.6097 s, 7.2 MB/s Write runtime is 16 s, speed is 6.250000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.228 s, 10.3 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg441-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.57892 s, 29.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0293665 s, 35.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0633303 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171988 s, 6.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220094 s, 4.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171982 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175813 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194812 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129094 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148797 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175921 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.275341 s, 3.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200624 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.241862 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.241052 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222638 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.337987 s, 3.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0488753 s, 21.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.05097 s, 20.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0758803 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0563062 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0291289 s, 36.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0600378 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0730865 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0569704 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0481397 s, 21.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0576569 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0843777 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0676859 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0978861 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0722394 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0867035 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0968389 s, 10.8 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg441-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg441-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 15.5939 s, 6.7 MB/s Write runtime is 17 s, speed is 5.882352 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.027 s, 10.5 MB/s Read runtime is 12 s, speed is 8.333333 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 13.6061 s, 7.7 MB/s Write runtime is 15 s, speed is 6.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.9208 s, 9.6 MB/s Read runtime is 12 s, speed is 8.333333 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kg (171s) == sanityn test 77kh: Verify that Project ID support for NRS TBF rule ========================================================== 04:08:00 (1785398880) using SAVE_PROJECT_SUPPORTED=0 Stopping clients: oleg441-client.virtnet /mnt/lustre (opts:) Stopping client oleg441-client.virtnet /mnt/lustre opts: Stopping clients: oleg441-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg441-client.virtnet /mnt/lustre2 opts: Stopping /mnt/lustre-mds1 (opts:-f) on oleg441-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg441-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg441-server sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime) proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) devtmpfs on /dev type devtmpfs (rw,nosuid,mode=755) securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime) tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev) devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000) tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755) tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755) cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd) pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime) bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700) cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids) cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory) cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma) cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio) cgroup on /sys/fs/cgroup/misc type cgroup (rw,nosuid,nodev,noexec,relatime,misc) cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer) cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct) cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices) cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event) cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio) cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset) cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb) none on /sys/kernel/tracing type tracefs (rw,relatime) configfs on /sys/kernel/config type configfs (rw,relatime) /dev/nbd0 on / type ext4 (ro,relatime,stripe=32) rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M) mqueue on /dev/mqueue type mqueue (rw,relatime) debugfs on /sys/kernel/debug type debugfs (rw,relatime) none on /mnt type ramfs (rw,relatime) /dev/vda on /home/green/git/lustre-release type squashfs (ro,relatime) none on /var/lib/stateless/writable type tmpfs (rw,relatime) none on /var/cache/man type tmpfs (rw,relatime) none on /var/log type tmpfs (rw,relatime) none on /var/lib/dbus type tmpfs (rw,relatime) none on /tmp type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) none on /var/tmp type tmpfs (rw,relatime) none on /var/lib/NetworkManager type tmpfs (rw,relatime) none on /var/lib/systemd/random-seed type tmpfs (rw,relatime) none on /var/spool type tmpfs (rw,relatime) none on /var/lib/nfs type tmpfs (rw,relatime) none on /var/lib/gssproxy type tmpfs (rw,relatime) none on /var/lib/logrotate type tmpfs (rw,relatime) none on /etc type tmpfs (rw,relatime) none on /var/lib/rsyslog type tmpfs (rw,relatime) none on /root/.ssh type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) 192.168.200.253:/exports/state/oleg441-client.virtnet on /var/lib/stateless/state type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.204.41,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg441-client.virtnet/boot on /boot type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.204.41,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg441-client.virtnet/etc/kdump.conf on /etc/kdump.conf type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.204.41,local_lock=none,addr=192.168.200.253) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=378180k,mode=700) 192.168.200.253:/exports/testreports/66906/testresults/sanityn-zfs-rocky8.10_x86_64-rocky8.10_x86_64 on /tmp/testlogs type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.204.41,local_lock=none,addr=192.168.200.253) /dev/vda on /usr/sbin/mount.lustre type squashfs (ro,relatime) Checking servers environments Checking clients oleg441-client.virtnet environments /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg441-server' oleg441-server: oleg441-server.virtnet: executing load_modules_local oleg441-server: Loading modules from /home/green/git/lustre-release/lustre oleg441-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg441-server: detected 4 online CPUs by sysfs oleg441-server: MODOPTS_LIBCFS= oleg441-server: Force libcfs to create 2 CPU partitions Setup mgs, mdt, osts Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg441-server: oleg441-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg441-client: oleg441-server: ssh exited with exit code 1 Started lustre-MDT0000 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg441-server: oleg441-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg441-client: oleg441-server: ssh exited with exit code 1 Started lustre-OST0000 Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg441-server: oleg441-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg441-client: oleg441-server: ssh exited with exit code 1 Started lustre-OST0001 Starting client: oleg441-client.virtnet: -o user_xattr,flock 192.168.204.141@tcp:/lustre /mnt/lustre Starting client oleg441-client.virtnet: -o user_xattr,flock 192.168.204.141@tcp:/lustre /mnt/lustre Started clients oleg441-client.virtnet: 192.168.204.141@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Starting client: oleg441-client.virtnet: -o user_xattr,flock 192.168.204.141@tcp:/lustre /mnt/lustre2 Starting client oleg441-client.virtnet: -o user_xattr,flock 192.168.204.141@tcp:/lustre /mnt/lustre2 Started clients oleg441-client.virtnet: 192.168.204.141@tcp:/lustre on /mnt/lustre2 type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8be7081c9000.idle_timeout=debug osc.lustre-OST0000-osc-ffff8be711c65000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8be7081c9000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8be711c65000.idle_timeout=debug disable quota as required ost.OSS.ost_io.nrs_policies=tbf projid ost.OSS.ost_io.nrs_tbf_rule=start ext_proj_rw projid={100} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.584379 s, 28.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0642059 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0868618 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119415 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22196 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.241 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216849 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.257538 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.352607 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.396288 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.347522 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.430727 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.340821 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.373169 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.328605 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.460847 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.376539 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165257 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192443 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161353 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112775 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132792 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18672 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195381 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140809 s, 7.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192014 s, 5.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170836 s, 6.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158876 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165221 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166186 s, 6.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173803 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187619 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189265 s, 5.5 MB/s removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg441-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg441-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 12.7521 s, 8.2 MB/s Write runtime is 14 s, speed is 7.142857 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.78902 s, 18.1 MB/s Read runtime is 7 s, speed is 14.285714 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_proj_rw ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf projid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w projid={100}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r projid={100}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.865181 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155858 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160533 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.308539 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.3247 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.400835 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.44118 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.442075 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.501471 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.567719 s, 1.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.593523 s, 1.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.558425 s, 1.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.66522 s, 1.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.668064 s, 1.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.757183 s, 1.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.770704 s, 1.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.755702 s, 1.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.431741 s, 2.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146469 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.274004 s, 3.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180349 s, 5.8 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154957 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169425 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.225778 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.249385 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.538824 s, 1.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.228443 s, 4.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.226331 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18588 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.234123 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.233374 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.238221 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.225123 s, 4.7 MB/s removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg441-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg441-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 16.1647 s, 6.5 MB/s Write runtime is 17 s, speed is 5.882352 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0984 s, 10.4 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77kh (284s) == sanityn test 77ki: Add rule with unsupported TBF type should fail for generic TBF ========================================================== 04:12:43 (1785399163) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 oleg441-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_fail projid={100}&opcode={ost_write} rate=20: Invalid argument oleg441-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_fail projid={100}&opcode={ost_write} rate=20': Invalid argument pdsh@oleg441-client: oleg441-server: ssh exited with exit code 22 ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77ki (25s) == sanityn test 77kj: Verify nodemap support for NRS TBF rule ========================================================== 04:13:08 (1785399188) On MGS 192.168.204.141, active = nodemap.active=1 On MGS 192.168.204.141, TBF = ost.OSS.ost_io.nrs_policies=tbf nodemap ost.OSS.ost_io.nrs_tbf_rule=start ext_nm_rw nodemap={TBF} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.434899 s, 38.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107315 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164275 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145872 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170457 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.263482 s, 4.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.262993 s, 4.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.330544 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.353923 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.226097 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.315627 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.481609 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.377513 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.520825 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.40576 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.488619 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.574382 s, 1.8 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8be7081c9000.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0478817 s, 21.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0491499 s, 21.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0650315 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116163 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0566146 s, 18.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0596411 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0641023 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0644342 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0649065 s, 16.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0782835 s, 13.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0933718 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0948439 s, 11.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0950652 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10086 s, 10.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10484 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126184 s, 8.3 MB/s removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg441-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg441-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 14.244 s, 7.4 MB/s Write runtime is 16 s, speed is 6.250000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.6794 s, 18.5 MB/s Read runtime is 7 s, speed is 14.285714 IOPS cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete On MGS 192.168.204.141, TBF.id = On MGS 192.168.204.141, active = nodemap.active=0 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.897883 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128077 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190239 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.308952 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.232923 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.267216 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.39463 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.489133 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.437576 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.433254 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.34094 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.553587 s, 1.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.479793 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.421617 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.527942 s, 2.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8be7081c9000.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff8be7081c9000.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.641103 s, 1.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.597726 s, 1.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191725 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141014 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155487 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150329 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200686 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174832 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121576 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148023 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172652 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169554 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205651 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163624 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149648 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174029 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159807 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184184 s, 5.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8be7081c9000.lock_unused_count=2 removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg441-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg441-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 1 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 16.178 s, 6.5 MB/s Write runtime is 18 s, speed is 5.555555 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.33245 s, 24.2 MB/s Read runtime is 5 s, speed is 20.000000 IOPS cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_nm_rw ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf nodemap+opcode oleg441-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_w nodemap={TBF}&opcode={ost_write} rate=20: Invalid argument oleg441-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_w nodemap={TBF}&opcode={ost_write} rate=20': Invalid argument oleg441-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_r nodemap={TBF}&opcode={ost_read} rate=10: Invalid argument oleg441-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_r nodemap={TBF}&opcode={ost_read} rate=10': Invalid argument pdsh@oleg441-client: oleg441-server: ssh exited with exit code 22 On MGS 192.168.204.141, active = nodemap.active=1 On MGS 192.168.204.141, TBF = ost.OSS.ost_io.nrs_tbf_rule=start ext_w nodemap={TBF}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nodemap={TBF}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.852546 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0724342 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0579903 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137006 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0940771 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14156 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10355 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109005 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136584 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.214731 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195729 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0985114 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193032 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.267144 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165416 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.240675 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.265947 s, 3.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8be7081c9000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff8be7081c9000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.270273 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.246211 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.217147 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220356 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.260924 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.231929 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.28373 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23901 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.241084 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.240852 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.254728 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.259092 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.280545 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.290747 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.289185 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.301824 s, 3.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8be7081c9000.lock_unused_count=4 removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg441-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg441-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 13.6527 s, 7.7 MB/s Write runtime is 15 s, speed is 6.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.2389 s, 10.2 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo On MGS 192.168.204.141, TBF.id = On MGS 192.168.204.141, active = nodemap.active=0 PASS 77kj (231s) == sanityn test 77l: check the output of NRS policies for generic TBF ========================================================== 04:16:59 (1785399419) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_policies=fifo PASS 77l (12s) == sanityn test 77m: check NRS Delay slows write RPC processing ========================================================== 04:17:11 (1785399431) ost.OSS.ost_io.nrs_policies=delay ost.OSS.ost_io.nrs_delay_min=4 ost.OSS.ost_io.nrs_delay_max=4 ost.OSS.ost_io.nrs_delay_pct=100 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg441-client.virtnet] [bs=1M] [count=1] [oflag=direct] [conv=fdatasync] 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 3.47217 s, 302 kB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg441-client.virtnet] [bs=1M] [count=10] [oflag=direct] [conv=fdatasync] 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 40.0821 s, 262 kB/s ost.OSS.ost_io.nrs_policies=fifo PASS 77m (63s) == sanityn test 77n: check wildcard support for TBF JobID NRS policy ========================================================== 04:18:14 (1785399494) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={*.500} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.632146 s, 26.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0667758 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0672393 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0600504 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0420257 s, 25.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118067 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142282 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.343334 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.335644 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.344699 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.323357 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.282693 s, 3.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8be7081c9000.lock_unused_count=5 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.37697 s, 2.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.356961 s, 2.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.499602 s, 2.1 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0821288 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100703 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11676 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0926317 s, 11.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0993162 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0685487 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121952 s, 8.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.522842 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.472379 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0120465 s, 87.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016687 s, 62.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0103538 s, 101 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0118463 s, 88.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0977368 s, 10.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0687826 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0176355 s, 59.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.248464 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0192584 s, 54.4 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg441-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg441-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77n.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 14.633 s, 7.2 MB/s Write runtime is 16 s, speed is 6.250000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77n.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.64092 s, 15.8 MB/s Read runtime is 8 s, speed is 12.500000 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.*} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.859619 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160351 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163056 s, 6.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.39991 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.368989 s, 2.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.331399 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.401496 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.427122 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.421374 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.435899 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.546874 s, 1.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.369232 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.591209 s, 1.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.550351 s, 1.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.544225 s, 1.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.440317 s, 2.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8be7081c9000.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff8be7081c9000.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.664976 s, 1.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.491191 s, 2.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.450784 s, 2.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.491276 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.51262 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.479085 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.487412 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.48872 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.543705 s, 1.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.583943 s, 1.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.516825 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.529995 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.540034 s, 1.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.543269 s, 1.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.542461 s, 1.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.609816 s, 1.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.652524 s, 1.6 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8be7081c9000.lock_unused_count=2 removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg441-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg441-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 13.6266 s, 7.7 MB/s Write runtime is 15 s, speed is 6.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.9416 s, 15.1 MB/s Read runtime is 8 s, speed is 12.500000 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_policies=fifo PASS 77n (115s) == sanityn test 77o: Changing rank should not panic ====== 04:20:09 (1785399609) mds.MDS.mdt.nrs_policies=tbf nid mds.MDS.mdt.nrs_tbf_rule=start name nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=start name1 nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=change name1 rank=name mds.MDS.mdt.nrs_tbf_rule=stop name mds.MDS.mdt.nrs_policies=fifo PASS 77o (18s) == sanityn test 77q: Parallel TBF rule definitions should not panic ========================================================== 04:20:28 (1785399628) mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_policies=fifo PASS 77q (154s) == sanityn test 77p: Check validity of rule names for TBF policies ========================================================== 04:23:02 (1785399782) mds.MDS.mdt.nrs_policies=tbf oleg441-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_overflo uid={500} rate=500: File name too long oleg441-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_overflo uid={500} rate=500': File name too long pdsh@oleg441-client: oleg441-server: ssh exited with exit code 36 oleg441-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start uid={500} rate=500: Invalid argument oleg441-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start uid={500} rate=500': Invalid argument pdsh@oleg441-client: oleg441-server: ssh exited with exit code 22 oleg441-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_empty: Invalid argument oleg441-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_empty': Invalid argument pdsh@oleg441-client: oleg441-server: ssh exited with exit code 22 oleg441-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p@spec uid={500} rate=500: Invalid argument oleg441-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p@spec uid={500} rate=500': Invalid argument pdsh@oleg441-client: oleg441-server: ssh exited with exit code 22 oleg441-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p.spec uid={500} rate=500: Invalid argument oleg441-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p.spec uid={500} rate=500': Invalid argument pdsh@oleg441-client: oleg441-server: ssh exited with exit code 22 oleg441-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p~spec uid={500} rate=500: Invalid argument oleg441-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p~spec uid={500} rate=500': Invalid argument pdsh@oleg441-client: oleg441-server: ssh exited with exit code 22 oleg441-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p#spec uid={500} rate=500: Invalid argument oleg441-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p#spec uid={500} rate=500': Invalid argument pdsh@oleg441-client: oleg441-server: ssh exited with exit code 22 oleg441-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p/spec uid={500} rate=500: Invalid argument oleg441-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p/spec uid={500} rate=500': Invalid argument pdsh@oleg441-client: oleg441-server: ssh exited with exit code 22 oleg441-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p^spec uid={500} rate=500: Invalid argument oleg441-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p^spec uid={500} rate=500': Invalid argument pdsh@oleg441-client: oleg441-server: ssh exited with exit code 22 oleg441-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p%spec uid={500} rate=500: Invalid argument oleg441-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p%spec uid={500} rate=500': Invalid argument pdsh@oleg441-client: oleg441-server: ssh exited with exit code 22 oleg441-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p*spec uid={500} rate=500: Invalid argument oleg441-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p*spec uid={500} rate=500': Invalid argument pdsh@oleg441-client: oleg441-server: ssh exited with exit code 22 oleg441-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p;spec uid={500} rate=500: Invalid argument oleg441-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p;spec uid={500} rate=500': Invalid argument pdsh@oleg441-client: oleg441-server: ssh exited with exit code 22 oleg441-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p,spec uid={500} rate=500: Invalid argument oleg441-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p,spec uid={500} rate=500': Invalid argument pdsh@oleg441-client: oleg441-server: ssh exited with exit code 22 oleg441-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p?spec uid={500} rate=500: Invalid argument oleg441-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p?spec uid={500} rate=500': Invalid argument pdsh@oleg441-client: oleg441-server: ssh exited with exit code 22 oleg441-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77pspec uid={500} rate=500: Invalid argument oleg441-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p>spec uid={500} rate=500': Invalid argument pdsh@oleg441-client: oleg441-server: ssh exited with exit code 22 oleg441-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p:spec uid={500} rate=500: Invalid argument oleg441-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p:spec uid={500} rate=500': Invalid argument pdsh@oleg441-client: oleg441-server: ssh exited with exit code 22 oleg441-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p+spec uid={500} rate=500: Invalid argument oleg441-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p+spec uid={500} rate=500': Invalid argument pdsh@oleg441-client: oleg441-server: ssh exited with exit code 22 oleg441-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p=spec uid={500} rate=500: Invalid argument oleg441-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p=spec uid={500} rate=500': Invalid argument pdsh@oleg441-client: oleg441-server: ssh exited with exit code 22 oleg441-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p)spec uid={500} rate=500: Invalid argument oleg441-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p)spec uid={500} rate=500': Invalid argument pdsh@oleg441-client: oleg441-server: ssh exited with exit code 22 oleg441-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p(spec uid={500} rate=500: Invalid argument oleg441-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p(spec uid={500} rate=500': Invalid argument pdsh@oleg441-client: oleg441-server: ssh exited with exit code 22 oleg441-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p{spec uid={500} rate=500: Invalid argument oleg441-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p{spec uid={500} rate=500': Invalid argument pdsh@oleg441-client: oleg441-server: ssh exited with exit code 22 oleg441-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p}spec uid={500} rate=500: Invalid argument oleg441-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p}spec uid={500} rate=500': Invalid argument pdsh@oleg441-client: oleg441-server: ssh exited with exit code 22 oleg441-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p|spec uid={500} rate=500: Invalid argument oleg441-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p|spec uid={500} rate=500': Invalid argument pdsh@oleg441-client: oleg441-server: ssh exited with exit code 22 oleg441-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p[spec uid={500} rate=500: Invalid argument oleg441-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p[spec uid={500} rate=500': Invalid argument pdsh@oleg441-client: oleg441-server: ssh exited with exit code 22 oleg441-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p]spec uid={500} rate=500: Invalid argument oleg441-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p]spec uid={500} rate=500': Invalid argument pdsh@oleg441-client: oleg441-server: ssh exited with exit code 22 oleg441-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p!spec uid={500} rate=500: Invalid argument oleg441-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p!spec uid={500} rate=500': Invalid argument pdsh@oleg441-client: oleg441-server: ssh exited with exit code 22 oleg441-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p&spec uid={500} rate=500: Invalid argument oleg441-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p&spec uid={500} rate=500': Invalid argument pdsh@oleg441-client: oleg441-server: ssh exited with exit code 22 oleg441-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p$spec uid={500} rate=500: Invalid argument oleg441-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p$spec uid={500} rate=500': Invalid argument pdsh@oleg441-client: oleg441-server: ssh exited with exit code 22 oleg441-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p`spec uid={500} rate=500: Invalid argument oleg441-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p`spec uid={500} rate=500': Invalid argument pdsh@oleg441-client: oleg441-server: ssh exited with exit code 22 oleg441-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p\spec uid={500} rate=500: Invalid argument oleg441-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p\spec uid={500} rate=500': Invalid argument pdsh@oleg441-client: oleg441-server: ssh exited with exit code 22 mds.MDS.mdt.nrs_policies=fifo PASS 77p (49s) == sanityn test 77r: Change type of tbf policy at run time ========================================================== 04:23:51 (1785399831) mds.MDS.mdt.nrs_policies=tbf uid mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf gid mds.MDS.mdt.nrs_tbf_rule=start md_rule gid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start md_rule jobid={*.500} rate=1 mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=fifo check the number of requests in queue: fifo: 0 crrn: 0 tbf: 0 delay: 0 fifo: 0 crrn: 0 tbf: 0 delay: 0 cleanup 77r 272897 kill md_thread xargs PASS 77r (64s) == sanityn test 77s: Check TBF LRU shrinker ============== 04:24:55 (1785399895) jobid_name=%e.%p jobid_var=TEST_77s_JOBVAR mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start touch jobid={touch.*} rate=1000 mds.MDS.mdt.nrs_tbf_rule=start unlink jobid={unlink.*} rate=1000 touch jobid={touch.*} 1000, ref 10 touch jobid={touch.*} 1000, ref 0 unlink jobid={unlink.*} 1000, ref 10 unlink jobid={unlink.*} 1000, ref 0 mds.MDS.mdt.nrs_policies=fifo jobid_var=procname_uid jobid_name=%e.%u PASS 77s (80s) == sanityn test 78: Enable policy and specify tunings right away ========================================================== 04:26:15 (1785399975) oleg441-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_orr_quantum=1: No such device oleg441-server: error: set_param: setting 'ost/OSS/*/nrs_orr_quantum'='1': No such device pdsh@oleg441-client: oleg441-server: ssh exited with exit code 19 ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_policies=fifo PASS 78 (13s) == sanityn test 79: xattr: intent error ================== 04:26:29 (1785399989) File: /mnt/lustre/d79.sanityn Size: 512 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115238826935613 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-07-30 04:26:31.000000000 -0400 Modify: 2026-07-30 04:26:31.000000000 -0400 Change: 2026-07-30 04:26:31.000000000 -0400 Birth: 2026-07-30 04:26:31.000000000 -0400 fail_val=0 fail_loc=0x80000160 fail_val=0 fail_loc=0x80000131 # file: mnt/lustre/d79.sanityn trusted.name1="value1" PASS 79 (21s) == sanityn test 80a: migrate directory when some children is being opened ========================================================== 04:26:50 (1785400010) SKIP: sanityn test_80a needs >= 2 MDTs SKIP 80a (4s) == sanityn test 80b: Accessing directory during migration ========================================================== 04:26:54 (1785400014) SKIP: sanityn test_80b needs >= 2 MDTs SKIP 80b (5s) == sanityn test 81a: rename and stat under striped directory ========================================================== 04:26:59 (1785400019) SKIP: sanityn test_81a needs >= 2 MDTs SKIP 81a (5s) == sanityn test 81b: rename under striped directory doesn't deadlock ========================================================== 04:27:04 (1785400024) SKIP: sanityn test_81b We need at least 2 MDTs for this test SKIP 81b (5s) == sanityn test 81c: rename revoke LOOKUP lock for remote object ========================================================== 04:27:09 (1785400029) SKIP: sanityn test_81c needs >= 4 MDTs SKIP 81c (5s) == sanityn test 81d: parallel rename file cross-dir on same MDT ========================================================== 04:27:14 (1785400034) - open/close 846 (time 1785400053.49 total 10.01 last 84.53) - open/close 1558 (time 1785400063.49 total 20.02 last 71.15) - open/close 2116 (time 1785400073.50 total 30.02 last 55.79) total: 2500 open/close in 34.73 seconds: 71.98 ops/second snapshot_time 1785400202.688617290 secs.nsecs start_time 1785400040.358770632 secs.nsecs elapsed_time 162.329846658 secs.nsecs open 2505 samples [usecs] 152 19652 3234875 8344786931 close 2505 samples [usecs] 155 16014 784021 1016412299 mknod 2501 samples [usecs] 519 19556 2863717 7086584825 unlink 2500 samples [usecs] 706 108942 5487258 69949393754 mkdir 2 samples [usecs] 3474 6660 10134 56424276 rmdir 2 samples [usecs] 7089 22244 29333 545049457 rename 2500 samples [usecs] 447 80853 25352187 619512600567 getattr 3760 samples [usecs] 9 26606 276314 1444991880 parallel_rename_file 2500 samples [usecs] 447 80853 25352187 619512600567 crossdir_rename 2500 samples [usecs] 447 80853 25352187 619512600567 PASS 81d (175s) == sanityn test 82: fsetxattr and fgetxattr on orphan files ========================================================== 04:30:10 (1785400210) multiop /mnt/lustre/f82.sanityn vO_ac TMPPIPE=/tmp/multiop_open_wait_pipe.7625 multiop /mnt/lustre2/f82.sanityn vo_Ac TMPPIPE=/tmp/multiop_open_wait_pipe.7625 PASS 82 (10s) == sanityn test 83: access striped directory while it is being created/unlinked ========================================================== 04:30:21 (1785400221) SKIP: sanityn test_83 needs >= 2 MDTs SKIP 83 (5s) == sanityn test 84: 0-nlink race in lu_object_find() ===== 04:30:26 (1785400226) fail_loc=0x8000060b PASS 84 (16s) == sanityn test 85: Lustre API root cache race =========== 04:30:42 (1785400242) Starting: llapi_root_test /mnt/lustre/f85.sanityn /mnt/lustre2/f85.sanityn Starting test test1 at 1785400245 Finishing test test1 at 1785400253 PASS 85 (19s) == sanityn test 90: open/create and unlink striped directory ========================================================== 04:31:01 (1785400261) SKIP: sanityn test_90 needs >= 2 MDTs SKIP 90 (7s) == sanityn test 91: chmod and unlink striped directory === 04:31:09 (1785400269) SKIP: sanityn test_91 needs >= 2 MDTs SKIP 91 (9s) == sanityn test 92: create remote directory under orphan directory ========================================================== 04:31:18 (1785400278) SKIP: sanityn test_92 needs >= 2 MDTs SKIP 92 (7s) == sanityn test 93: alloc_rr should not allocate on same ost ========================================================== 04:31:25 (1785400285) fail_loc=0x00000163 fail_loc=0x0 /mnt/lustre/f93.sanityn-1/file1 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5748 0x1674 0x240000400 1 5707 0x164b 0x280000400 /mnt/lustre2/f93.sanityn-2/file2 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 5708 0x164c 0x280000400 0 5749 0x1675 0x240000400 rc1=0 and rc2=0 PASS 93 (25s) == sanityn test 94: signal vs CP callback race =========== 04:31:50 (1785400310) osc.lustre-OST0000-osc-ffff8be7081c9000.idle_timeout=0 osc.lustre-OST0000-osc-ffff8be711c65000.idle_timeout=0 osc.lustre-OST0001-osc-ffff8be7081c9000.idle_timeout=0 osc.lustre-OST0001-osc-ffff8be711c65000.idle_timeout=0 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.238664 s, 34.3 kB/s fail_val=5 fail_loc=0x80000312 fail_val=6 fail_loc=0x80000329 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 2.99938 s, 1.4 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 6389: 289483 Alarm clock dd if=/dev/zero of=$DIR/$tfile conv=notrunc oflag=append bs=4k count=1 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.194041 s, 21.1 kB/s osc.lustre-OST0000-osc-ffff8be7081c9000.idle_timeout=debug osc.lustre-OST0000-osc-ffff8be711c65000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8be7081c9000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8be711c65000.idle_timeout=debug PASS 94 (20s) == sanityn test 95a: Check readpage() on a page that was removed from page cache ========================================================== 04:32:10 (1785400330) llite.lustre-ffff8be7081c9000.fast_read=0 llite.lustre-ffff8be711c65000.fast_read=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0786232 s, 26.7 MB/s fail_loc=0x80001421 fail_val=7 fail_loc=0x80001422 fail_val=10 llite.lustre-ffff8be7081c9000.fast_read=1 llite.lustre-ffff8be711c65000.fast_read=1 PASS 95a (24s) == sanityn test 95b: Check readpage() on a page that is no longer uptodate ========================================================== 04:32:34 (1785400354) llite.lustre-ffff8be7081c9000.fast_read=0 llite.lustre-ffff8be711c65000.fast_read=0 1+0 records in 1+0 records out 12288 bytes (12 kB, 12 KiB) copied, 0.0145723 s, 843 kB/s fail_loc=0x80001424 fail_val=5 fail_loc=0 llite.lustre-ffff8be7081c9000.fast_read=1 llite.lustre-ffff8be711c65000.fast_read=1 PASS 95b (15s) == sanityn test 100a: DoM: glimpse RPCs for stat without IO lock (DoM only file) ========================================================== 04:32:49 (1785400369) SKIP: sanityn test_100a Reserved for glimpse-ahead SKIP 100a (5s) == sanityn test 100b: DoM: no glimpse RPC for stat with IO lock (DoM only file) ========================================================== 04:32:54 (1785400374) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00492729 s, 831 kB/s /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK PASS 100b (10s) == sanityn test 100c: DoM: write vs stat without IO lock (combined file) ========================================================== 04:33:05 (1785400385) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.251497 s, 8.3 MB/s /mnt/lustre/d100c.sanityn/dom has type file OK /mnt/lustre/d100c.sanityn/dom has size 2097152 OK PASS 100c (11s) == sanityn test 100d: DoM: write+truncate vs stat without IO lock (combined file) ========================================================== 04:33:16 (1785400396) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.189418 s, 11.1 MB/s /mnt/lustre/d100d.sanityn/dom has type file OK /mnt/lustre/d100d.sanityn/dom has size 4096 OK PASS 100d (10s) == sanityn test 100e: DoM: read on open and file size ==== 04:33:26 (1785400406) 1+0 records in 1+0 records out 12000 bytes (12 kB, 12 KiB) copied, 0.0256961 s, 467 kB/s PASS 100e (8s) == sanityn test 101a: Discard DoM data on unlink ========= 04:33:35 (1785400415) /mnt/lustre/f101a.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0202971 s, 202 kB/s PASS 101a (11s) == sanityn test 101b: Discard DoM data on rename ========= 04:33:46 (1785400426) /mnt/lustre/f101b.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0164565 s, 249 kB/s PASS 101b (11s) == sanityn test 101c: Discard DoM data on close-unlink === 04:33:58 (1785400438) /mnt/lustre/f101c.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0203635 s, 201 kB/s PASS 101c (12s) == sanityn test 102: Test open by handle of unlinked file ========================================================== 04:34:10 (1785400450) file: /mnt/lustre/f102.sanityn fh_bytes: 32 fh_type: 151 fh_data: d1 0b 00 00 02 00 00 00 1e 0f 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 file's parent FID is [0:0x0:0x0] file FID is [0x200000bd1:0xf1e:0x0] access via mount point '/mnt/lustre2' - OK access after unlink - OK check_fhandle_syscalls test Passed! PASS 102 (9s) == sanityn test 103: Test size correctness with lockahead ========================================================== 04:34:20 (1785400460) fail_loc=0x415 fail_loc=0x415 fail_val=2 Incorrect size expected (no glimpse fix): Starting test test23 at 1785400464 lockahead_test: lockahead_test.c:1171: test23: assertion 'sb.st_size == sb2.st_size' failed: size on /mnt/lustre/d103.sanityn/f103.sanityn and /mnt/lustre2/d103.sanityn/f103.sanityn differs: 0 vs 1048576 fail_loc=0 fail_loc=0x214 fail_val=2 Starting test test23 at 1785400468 Finishing test test23 at 1785400468 PASS 103 (16s) == sanityn test 104: Verify that MDS stores atime/mtime/ctime during close ========================================================== 04:34:36 (1785400476) SKIP: sanityn test_104 ldiskfs only test SKIP 104 (4s) == sanityn test 105: Glimpse and lock cancel race ======== 04:34:40 (1785400480) fail_loc=0x416 PASS 105 (30s) == sanityn test 106a: Verify the btime via statx() ======= 04:35:11 (1785400511) SKIP: sanityn test_106a Test only for ldiskfs and statx() supported SKIP 106a (5s) == sanityn test 106b: Glimpse RPCs test for statx ======== 04:35:15 (1785400515) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0174273 s, 58.8 kB/s File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238826938149 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-07-30 04:35:17.000000000 -0400 Modify: 2026-07-30 04:35:17.000000000 -0400 Change: 2026-07-30 04:35:17.000000000 -0400 Birth: 2026-07-30 04:35:18.000000000 -0400 /mnt/lustre/f106b.sanityn 144115238826938149 -rw-r--r-- 1 0 0 1785400518 1785400517 1785400517 File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238826938149 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-07-30 04:35:17.000000000 -0400 Modify: 2026-07-30 04:35:17.000000000 -0400 Change: 2026-07-30 04:35:17.000000000 -0400 Birth: 2026-07-30 04:35:18.000000000 -0400 1785400517 1024 1 PASS 106b (10s) == sanityn test 106c: Verify statx attributes mask ======= 04:35:25 (1785400525) PASS 106c (9s) == sanityn test 107a: Basic grouplock conflict =========== 04:35:34 (1785400534) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.495146 s, 21.2 MB/s /mnt/lustre/f107a.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 5715 0x1653 0x280000400 multiop /mnt/lustre/f107a.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7625 multiop /mnt/lustre2/f107a.sanityn vO_G16022000r10g16022000c TMPPIPE=/tmp/multiop_open_wait_pipe.7625 First grouplock blocks second one PASS 107a (14s) == sanityn test 107b: Grouplock is added to the head of waiting list ========================================================== 04:35:48 (1785400548) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.686434 s, 15.3 MB/s /mnt/lustre/f107b.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5758 0x167e 0x240000400 multiop /mnt/lustre/f107b.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7625 Grouplock blocks IO First grouplock blocks second one Second grouplock blocks IO PASS 107b (18s) == sanityn test 108a: lseek: parallel updates ============ 04:36:06 (1785400566) 1+0 records in 1+0 records out 10000 bytes (10 kB, 9.8 KiB) copied, 0.0149333 s, 670 kB/s llite.lustre-ffff8be7081c9000.hybrid_io=0 llite.lustre-ffff8be711c65000.hybrid_io=0 fail_val=4 fail_loc=0x80000414 1+0 records in 1+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 4.69962 s, 1.8 MB/s PASS 108a (14s) == sanityn test 109: Race with several mount instances on 1 node ========================================================== 04:36:21 (1785400581) 192.168.204.141@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg441-client.virtnet /mnt/lustre (opts:) 192.168.204.141@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg441-client.virtnet /mnt/lustre2 (opts:) Starting race between client mounts (duration 30) Iteration 0 Starting client: oleg441-client.virtnet: -o user_xattr,flock 192.168.204.141@tcp:/lustre /mnt/lustre2 Starting client: oleg441-client.virtnet: -o user_xattr,flock 192.168.204.141@tcp:/lustre /mnt/lustre 192.168.204.141@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.141@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg441-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg441-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg441-server' oleg441-server: oleg441-server.virtnet: executing load_modules_local oleg441-server: Loading modules from /home/green/git/lustre-release/lustre oleg441-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg441-server: detected 4 online CPUs by sysfs oleg441-server: MODOPTS_LIBCFS= oleg441-server: Force libcfs to create 2 CPU partitions Starting client: oleg441-client.virtnet: -o user_xattr,flock 192.168.204.141@tcp:/lustre /mnt/lustre Starting client: oleg441-client.virtnet: -o user_xattr,flock 192.168.204.141@tcp:/lustre /mnt/lustre2 PASS 109 (62s) == sanityn test 110: do not grant another lock on resend ========================================================== 04:37:23 (1785400643) fail_loc=0x525 fail_val=3 fail_loc=0x534 File: /mnt/lustre/f110.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238860488707 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-07-30 04:37:26.000000000 -0400 Modify: 2026-07-30 04:37:26.000000000 -0400 Change: 2026-07-30 04:37:26.000000000 -0400 Birth: 2026-07-30 04:37:27.000000000 -0400 fail_loc=0 fail_val=0 Connected clients: oleg441-client.virtnet oleg441-client.virtnet PASS 110 (84s) == sanityn test 111: A racy rename/link an open file should not cause fs corruption ========================================================== 04:38:48 (1785400728) SKIP: sanityn test_111 needs >= 2 MDTs SKIP 111 (6s) == sanityn test 112: update max-inherit in default LMV === 04:38:53 (1785400733) SKIP: sanityn test_112 We need at least 2 MDTs for this test SKIP 112 (5s) == sanityn test 113: check servers of specified fs ======= 04:38:58 (1785400738) lustre-OST0000-osc-ffff8be7116e2800 active. lustre-OST0001-osc-ffff8be7116e2800 active. lustre-OST0000-osc-ffff8be7116e2800 active. lustre-OST0001-osc-ffff8be7116e2800 active. MGC192.168.204.141@tcp active. PASS 113 (10s) == sanityn test 114: implicit default LMV inherit ======== 04:39:08 (1785400748) SKIP: sanityn test_114 We need at least 2 MDTs for this test SKIP 114 (6s) == sanityn test 115: ldiskfs doesn't check direntry for uniqueness ========================================================== 04:39:14 (1785400754) SKIP: sanityn test_115 ldiskfs only test SKIP 115 (4s) == sanityn test 116: DNE: Set default LMV layout from a remote client ========================================================== 04:39:18 (1785400758) SKIP: sanityn test_116 needs >= 2 MDTs SKIP 116 (5s) == sanityn test 121: trunc append race =================== 04:39:23 (1785400763) 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0192662 s, 1.1 MB/s fail_loc=0x80001436 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.163813 s, 125 kB/s 0000000 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0005000 0000 0000 0000 0000 0000 0000 0000 0000 * 000c800 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0011800 PASS 121 (13s) == sanityn test 200: service remains healthy while able to process request ========================================================== 04:39:37 (1785400777) at_max=10 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=9 fail_loc=0x1e1 fail_val=20 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.311 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.2996 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.3478 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.4011 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.342 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.3962 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.4249 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1965 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.2291 s, 0.1 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.3374 s, 0.1 kB/s 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.3658 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.2423 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.2807 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.3912 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.4912 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.6176 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.5715 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.6592 s, 0.1 kB/s at_max=600 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=116 oleg441-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8be70b93b000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8be70b93b000.ost_server_uuid in IDLE state after 0 sec PASS 200 (95s) cleanup: ====================================================== == sanityn test complete, duration 12223 sec ============= 04:41:14 (1785400874) === sanityn: start cleanup 04:41:15 (1785400875) === Stopping clients: oleg441-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg441-client.virtnet /mnt/lustre2 opts: === sanityn: finish cleanup 04:46:10 (1785401170) ===