-----============= acceptance-small: sanityn ============----- Tue Aug 11 16:31:34 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg109-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanityn.*ex': No such file or directory excepting tests: 27 40a skipping tests SLOW=no: 33a === sanityn: start setup 16:31:57 (1786480317) === Starting client oleg109-client.virtnet: -o user_xattr,flock 192.168.201.109@tcp:/lustre /mnt/lustre2 Mount client oleg109-client.virtnet: mount -t lustre -o user_xattr,flock 192.168.201.109@tcp:/lustre /mnt/lustre2 Started clients oleg109-client.virtnet: 192.168.201.109@tcp:/lustre on /mnt/lustre2 type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) oleg109-client.virtnet: executing check_config_client /mnt/lustre oleg109-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg109-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9882c4faf800.idle_timeout=debug osc.lustre-OST0000-osc-ffff9882c5c8a000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9882c4faf800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9882c5c8a000.idle_timeout=debug disable quota as required oleg109-server: oleg109-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanityn: finish setup 16:32:33 (1786480353) === 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.040838 s, 25.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7632] == sanityn test 0: do_node_vp() and do_facet_vp() do the right thing ========================================================== 16:32:35 (1786480355) PASS 0 (8s) == sanityn test 1: Check attribute updates on 2 mount points ========================================================== 16:32:43 (1786480363) /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0777 OK /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0666 OK /mnt/lustre/f1.sanityn: absent OK PASS 1 (8s) == sanityn test 2a: check cached attribute updates on 2 mtpt's ================================================================== 16:32:51 (1786480371) -rw-r--r-- 1 root root 0 Aug 11 16:32 /mnt/lustre2/f2a /mnt/lustre/f2a has type file OK /mnt/lustre/f2a has perms 0777 OK PASS 2a (6s) == sanityn test 2b: check cached attribute updates on 2 mtpt's ================================================================== 16:32:57 (1786480377) -rw-r--r-- 1 root root 0 Aug 11 16:32 /mnt/lustre2/f2b /mnt/lustre2/f2b has type file OK /mnt/lustre2/f2b has perms 0777 OK PASS 2b (6s) == sanityn test 2c: check cached attribute updates on 2 mtpt's root ============================================================= 16:33:03 (1786480383) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0777 OK PASS 2c (7s) == sanityn test 2d: check cached attribute updates on 2 mtpt's root ============================================================= 16:33:10 (1786480390) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0755 OK PASS 2d (7s) == sanityn test 2e: check chmod on root is propagated to others ========================================================== 16:33:17 (1786480397) total 1 -rwxrwxrwx 1 root root 0 Aug 11 16:32 f2a -rwxrwxrwx 1 root root 0 Aug 11 16:32 f2b total 1 -rwxrwxrwx 1 root root 0 Aug 11 16:32 f2a -rwxrwxrwx 1 root root 0 Aug 11 16:32 f2b running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre2/f2e.sanityn] [count=1] 1+0 records in 1+0 records out 512 bytes copied, 0.0248394 s, 20.6 kB/s PASS 2e (7s) == sanityn test 2f: check attr/owner updates on DNE with 2 mtpt's ========================================================== 16:33:24 (1786480404) SKIP: sanityn test_2f needs >= 2 MDTs SKIP 2f (4s) == sanityn test 2g: check blocks update on sync write ==== 16:33:28 (1786480408) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.401631 s, 5.2 MB/s /mnt/lustre/f2g.sanityn has 4103 blocks /mnt/lustre2/f2g.sanityn has 4103 blocks PASS 2g (7s) == sanityn test 3: symlink on one mtpt, readlink on another ===================================================================== 16:33:35 (1786480415) PASS 3 (6s) == sanityn test 4: fstat validation on multiple mount points ==================================================================== 16:33:41 (1786480421) PASS 4 (7s) == sanityn test 5: create a file on one mount, truncate it on the other ========================================================== 16:33:48 (1786480428) /mnt/lustre/f5 has type file OK /mnt/lustre/f5 has size 100 OK PASS 5 (6s) == sanityn test 6: remove of open file on other node ============================================================================ 16:33:54 (1786480434) opening writing unlinking /mnt/lustre2/f6.sanityn accessing (1) seeking (1) accessing (2) fstat... reading comparing data truncating seeking (2) writing again seeking (3) reading again comparing data again closing SUCCESS - goto beer PASS 6 (6s) == sanityn test 7: remove of open directory on other node ======================================================================= 16:34:00 (1786480440) creating directory /mnt/lustre/d7 opening directory unlinking /mnt/lustre/d7 Ok, everything goes well. PASS 7 (6s) == sanityn test 8: remove of open special file on other node ==================================================================== 16:34:06 (1786480446) creating special file /mnt/lustre/f8.sanityn opening file unlinking /mnt/lustre/f8.sanityn Ok, everything goes well. PASS 8 (6s) == sanityn test 9a: append of file with sub-page size on multiple mounts ========================================================== 16:34:12 (1786480452) PASS 9a (7s) == sanityn test 9b: append to striped sparse file ======== 16:34:19 (1786480459) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0722057 s, 14.5 MB/s 3+0 records in 3+0 records out 3 bytes copied, 0.0681668 s, 0.0 kB/s Data read (expecting 'foo'): foo PASS 9b (7s) == sanityn test 10a: write of file with sub-page size on multiple mounts ========================================================== 16:34:26 (1786480466) 1+0 records in 1+0 records out 1 byte copied, 0.0229956 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0171191 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0180341 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00953418 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0114269 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00911072 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00854316 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00810118 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0067719 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00901175 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00548618 s, 0.2 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00533594 s, 0.2 kB/s PASS 10a (7s) == sanityn test 10b: write of file with sub-page size on multiple mounts ========================================================== 16:34:33 (1786480473) 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.0234788 s, 131 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0514745 s, 79.6 kB/s 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.00205372 s, 1.5 MB/s PASS 10b (5s) == sanityn test 11: execution of file opened for write should return error ============================================================== 16:34:38 (1786480478) multiop /mnt/lustre/d11/f vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7632 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 298: /mnt/lustre2/d11/f: Text file busy PASS 11 (5s) == sanityn test 12: test lock ordering (link, stat, unlink) ========================================================== 16:34:44 (1786480484) warning: '-runtime' deprecated, use '-t runtime' instead - link 430 (time 1786480495.87 total 10.01 last 42.97) - link 795 (time 1786480505.88 total 20.02 last 36.47) - link 1206 (time 1786480515.89 total 30.03 last 41.05) - link 1570 (time 1786480525.90 total 40.04 last 36.35) - link 1930 (time 1786480535.93 total 50.06 last 35.91) - link 2290 (time 1786480545.95 total 60.09 last 35.92) - link 2601 (time 1786480555.98 total 70.12 last 31.00) - link 2926 (time 1786480566.00 total 80.14 last 32.45) - link 3261 (time 1786480576.00 total 90.14 last 33.48) total: 3670 link in 100.00 seconds: 36.70 ops/second using seed 1100858040 running for 100 seconds - stat 5970 (time 1786480497 ; total 11 ; last 11) - stat 10000 (time 1786480504 ; total 18 ; last 7) - stat 16368 (time 1786480515 ; total 29 ; last 11) - stat 20000 (time 1786480521 ; total 35 ; last 6) - stat 26166 (time 1786480532 ; total 46 ; last 11) - stat 30000 (time 1786480539 ; total 53 ; last 7) - stat 32571 (time 1786480550 ; total 64 ; last 11) - stat 35482 (time 1786480561 ; total 75 ; last 11) - stat 38391 (time 1786480572 ; total 86 ; last 11) - stat 40000 (time 1786480577 ; total 91 ; last 5) total: 42900 stats in 100 seconds: 429.000000 stats/second - unlinked 0 (time 1786480536 ; total 0 ; last 0) unlink(/mnt/lustre2/lockdir/lockfile3670) error: No such file or directory total: 3669 unlinks in 85 seconds: 43.164707 unlinks/second /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 77: kill: (21434) - No such process /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 78: kill: (21436) - No such process PASS 12 (142s) == sanityn test 13: test directory page revocation ======= 16:37:05 (1786480625) f13 f13 PASS 13 (7s) == sanityn test 14aa: execution of file open for write returns -ETXTBSY ========================================================== 16:37:12 (1786480632) multiop /mnt/lustre/d14aa.sanityn/f14aa.sanityn vOw_c TMPPIPE=/tmp/multiop_open_wait_pipe.7632 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 333: /mnt/lustre2/d14aa.sanityn/f14aa.sanityn: Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 22620 Terminated $MULTIOP_PROG $FILE v$ARGS > $TMPPIPE (wd: ~) PASS 14aa (5s) == sanityn test 14ab: open(RDWR) of executing file returns -ETXTBSY ========================================================== 16:37:17 (1786480637) open(O_RDWR|O_CREAT): Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 23198 Terminated $DIR1/$tdir/sleep 60 (wd: ~) PASS 14ab (6s) == sanityn test 14b: truncate of executing file returns -ETXTBSY ================================================================ 16:37:23 (1786480643) truncate: cannot truncate '/mnt/lustre2/d14b.sanityn/sleep' to length 60: Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 348: 23777 Terminated $DIR1/$tdir/sleep 60 PASS 14b (7s) == sanityn test 14c: open(O_TRUNC) of executing file return -ETXTBSY ============================================================ 16:37:30 (1786480650) cp: cannot create regular file '/mnt/lustre2/d14c.sanityn/sleep': Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 360: 24359 Terminated $DIR1/$tdir/sleep 60 PASS 14c (6s) == sanityn test 14d: chmod of executing file is still possible ================================================================== 16:37:36 (1786480656) chmod /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 371: 24942 Terminated $DIR1/$tdir/sleep 60 PASS 14d (8s) == sanityn test 15: test out-of-space with multiple writers ===================================================================== 16:37:44 (1786480664) PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin: MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky sleep 5 for ZFS MDS Waiting for MDT destroys to complete fallocate on zfs doesn't consume space STRIPECOUNT=2 ORIGFREE=7531520 MAXFREE=800000 SKIP: oos2 test_15 oos2.sh: 7531520KB free > 800000KB, increase MAXFREE (or reduce fs size) sleep 5 for ZFS MDS Waiting for MDT destroys to complete SKIP 15 (50s) == sanityn test 16a: 500 iterations of dual-mount fsx ==== 16:38:34 (1786480714) fallocate on zfs doesn't consume space fallocate not supported 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.401058 s, 20.9 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.288356 s, 29.1 MB/s Chance of close/open is 1 in 50 Seed set to 717 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000005[0] 1786480718.555815 mapwrite 0x77dfd5 thru 0x784c00 (0x6c2c bytes) 000007[0] 1786480718.689489 read 0x6989ff thru 0x69a634 (0x1c36 bytes) 000008[1] 1786480718.721661 read 0x6c315e thru 0x6cb74f (0x85f2 bytes) truncating to largest ever: 0x14570b 000010[1] 1786480718.775650 trunc from 0x784c01 to 0x14570a (0x63f4f6 bytes) 000011[1] 1786480718.815312 mapread 0x0487d3 thru 0x053861 (0xb08f bytes) 000013[1] 1786480718.866830 write 0x04f001 thru 0x05e35c (0xf35c bytes) 000014[1] 1786480718.879619 mapwrite 0x783752 thru 0x78e90a (0xb1b9 bytes) 000015[0] 1786480719.001437 write 0x3a799e thru 0x3b6f23 (0xf586 bytes) 000017[0] 1786480719.073213 read 0x502065 thru 0x50272c (0x6c8 bytes) 000021[0] 1786480719.122887 mapwrite 0x4b755d thru 0x4bd93a (0x63de bytes) 000023[1] 1786480719.218250 read 0x7415fb thru 0x749f5b (0x8961 bytes) 000026[1] 1786480719.253149 read 0x262602 thru 0x26932e (0x6d2d bytes) 000028[1] 1786480719.283006 mapread 0x21c381 thru 0x228d02 (0xc982 bytes) truncating to largest ever: 0x835351 000029[1] 1786480719.309429 trunc from 0x78e90b to 0x835350 (0xa6a46 bytes) 000030[0] 1786480719.344456 mapread 0x2818d3 thru 0x28b286 (0x99b4 bytes) 000031[0] 1786480719.398538 write 0x8c3f13 thru 0x8cd8e0 (0x99ce bytes) 000033[1] 1786480719.490695 read 0x82d09d thru 0x82f751 (0x26b5 bytes) 000034[0] 1786480719.549017 mapread 0x787778 thru 0x793b4c (0xc3d5 bytes) 000035[0] 1786480719.610644 mapwrite 0x5b184f thru 0x5b5668 (0x3e1a bytes) 000038[0] 1786480719.723073 read 0x53ef18 thru 0x5494d1 (0xa5ba bytes) 000039[0] 1786480719.738642 mapwrite 0x60f185 thru 0x61743e (0x82ba bytes) 000041[1] 1786480719.803280 trunc from 0x8cd8e1 to 0x6867fd (0x2470e3 bytes) truncating to largest ever: 0x91ed41 000042[1] 1786480719.939519 trunc from 0x6867fe to 0x91ed40 (0x298543 bytes) 000045[1] 1786480720.107369 read 0x79ab7c thru 0x79b4f2 (0x977 bytes) 000046[1] 1786480720.146342 write 0x300201 thru 0x300d5e (0xb5e bytes) 000049[1] 1786480720.218802 mapread 0x3a0353 thru 0x3aa1ff (0x9ead bytes) 000050[1] 1786480720.304036 trunc from 0x91ed41 to 0x20ac23 (0x71411d bytes) 000054[1] 1786480720.420551 read 0x173745 thru 0x180d28 (0xd5e4 bytes) 000055[0] 1786480720.488469 write 0x1505df thru 0x1515be (0xfe0 bytes) 000056[1] 1786480720.536618 mapread 0x18cf0f thru 0x190143 (0x3235 bytes) 000057[0] 1786480720.614839 trunc from 0x20ac24 to 0x8c070b (0x6b5ae8 bytes) 000059[0] 1786480720.672441 read 0x0f306a thru 0x100392 (0xd329 bytes) 000065[0] 1786480720.805597 read 0x3a95c8 thru 0x3ab64c (0x2085 bytes) 000068[1] 1786480720.924181 mapwrite 0x075b3e thru 0x0764f7 (0x9ba bytes) 000069[0] 1786480721.160965 read 0x33436d thru 0x339f80 (0x5c14 bytes) 000070[1] 1786480721.205147 mapread 0x8690e7 thru 0x877e7c (0xed96 bytes) 000071[1] 1786480721.395173 mapwrite 0x5f79fb thru 0x6070ae (0xf6b4 bytes) 000072[1] 1786480721.619980 read 0x56d67f thru 0x56f52b (0x1ead bytes) 000073[1] 1786480721.639166 read 0x607a6e thru 0x6081a1 (0x734 bytes) 000074[1] 1786480721.660482 mapwrite 0x4175fd thru 0x41d2b9 (0x5cbd bytes) 000075[1] 1786480721.751495 write 0x3dba7b thru 0x3dc96b (0xef1 bytes) 000078[0] 1786480721.837607 read 0x693ea4 thru 0x6978ea (0x3a47 bytes) 000081[1] 1786480721.915385 mapwrite 0x8e2193 thru 0x8e70cd (0x4f3b bytes) 000084[1] 1786480722.141036 write 0x6abaf3 thru 0x6b622b (0xa739 bytes) 000085[0] 1786480722.185385 mapwrite 0x087d1b thru 0x093660 (0xb946 bytes) 000086[1] 1786480722.287902 trunc from 0x8e70ce to 0x696d3e (0x25038f bytes) 000087[1] 1786480722.315388 read 0x40f391 thru 0x41e96b (0xf5db bytes) 000089[1] 1786480722.323409 mapwrite 0x510a9b thru 0x51f5b2 (0xeb18 bytes) 000090[1] 1786480722.430543 trunc from 0x696d3f to 0x2d17e1 (0x3c555d bytes) 000092[1] 1786480722.492759 write 0x71a872 thru 0x71fa41 (0x51d0 bytes) 000095[1] 1786480722.541651 mapwrite 0x0673cb thru 0x067b46 (0x77c bytes) 000097[1] 1786480722.592941 mapwrite 0x598ae1 thru 0x5a7371 (0xe891 bytes) 000099[1] 1786480722.754935 mapread 0x4caf7d thru 0x4d6e14 (0xbe98 bytes) 000103[1] 1786480722.878437 write 0x772649 thru 0x782512 (0xfeca bytes) 000104[1] 1786480722.882399 write 0x7020d4 thru 0x707472 (0x539f bytes) 000105[0] 1786480722.917752 mapread 0x50ba8d thru 0x519556 (0xdaca bytes) 000106[1] 1786480722.986613 mapwrite 0x12d287 thru 0x13032a (0x30a4 bytes) 000109[0] 1786480723.119691 read 0x21861f thru 0x21f6f5 (0x70d7 bytes) 000111[1] 1786480723.150320 mapwrite 0x1e5a32 thru 0x1ea395 (0x4964 bytes) 000112[1] 1786480723.255933 mapwrite 0x0bc827 thru 0x0c3bb0 (0x738a bytes) 000114[1] 1786480723.343281 write 0x970b63 thru 0x97be15 (0xb2b3 bytes) 000118[1] 1786480723.378640 trunc from 0x97be16 to 0x38b58d (0x5f0888 bytes) 000119[1] 1786480723.408846 write 0x8c7539 thru 0x8d4edd (0xd9a5 bytes) 000120[0] 1786480723.442347 mapwrite 0x3e85b9 thru 0x3f4722 (0xc16a bytes) 000121[0] 1786480723.548705 trunc from 0x8d4ede to 0x082a37 (0x8524a6 bytes) 000124[0] 1786480723.612945 mapwrite 0x2ee1e5 thru 0x2f6ec7 (0x8ce3 bytes) 000125[0] 1786480723.708047 mapread 0x2c0cf0 thru 0x2c96bb (0x89cc bytes) 000133[0] 1786480723.973672 trunc from 0x2f6ec8 to 0x6ef6d4 (0x3f880d bytes) 000136[1] 1786480724.043841 mapwrite 0x84a405 thru 0x84edb6 (0x49b2 bytes) 000137[1] 1786480724.264194 mapwrite 0x2fd9a2 thru 0x30153f (0x3b9e bytes) 000142[0] 1786480724.461420 trunc from 0x84edb7 to 0x3ca25b (0x484b5b bytes) 000144[0] 1786480724.503726 read 0x1f5c61 thru 0x2058fe (0xfc9e bytes) 000146[1] 1786480724.581013 read 0x26ef77 thru 0x270fd9 (0x2063 bytes) 000149[1] 1786480724.743576 read 0x0a97b5 thru 0x0b824f (0xea9b bytes) 000150[1] 1786480724.749961 write 0x296728 thru 0x29cd79 (0x6652 bytes) 000151[1] 1786480724.854983 write 0x1688fb thru 0x1786dd (0xfde3 bytes) 000153[1] 1786480724.887057 write 0x4ddfaa thru 0x4e26fd (0x4754 bytes) 000156[1] 1786480724.944016 read 0x0154e4 thru 0x0225a7 (0xd0c4 bytes) 000157[0] 1786480724.944842 write 0x0b211a thru 0x0b706c (0x4f53 bytes) 000158[0] 1786480725.002491 mapwrite 0x646acb thru 0x656231 (0xf767 bytes) 000160[1] 1786480725.155746 trunc from 0x656232 to 0x356237 (0x2ffffa bytes) 000161[1] 1786480725.204705 read 0x0bf634 thru 0x0c0c7e (0x164b bytes) 000166[0] 1786480725.314978 write 0x7312a3 thru 0x73dc64 (0xc9c2 bytes) 000167[1] 1786480725.357623 mapwrite 0x6c97fc thru 0x6d8742 (0xef47 bytes) 000168[1] 1786480725.591408 read 0x4ea676 thru 0x4ed42a (0x2db5 bytes) 000170[1] 1786480725.619781 mapread 0x412632 thru 0x41d96b (0xb33a bytes) 000172[0] 1786480725.654664 mapread 0x33814c thru 0x344015 (0xbeca bytes) 000175[0] 1786480725.780531 write 0x0c62e7 thru 0x0c804d (0x1d67 bytes) 000176[1] 1786480725.850712 read 0x1cc700 thru 0x1d1e94 (0x5795 bytes) 000177[0] 1786480725.928598 read 0x01e054 thru 0x020b53 (0x2b00 bytes) 000178[1] 1786480725.948205 write 0x2830c4 thru 0x28961c (0x6559 bytes) 000180[1] 1786480726.017258 mapread 0x6ca647 thru 0x6d26b2 (0x806c bytes) 000185[1] 1786480726.042378 trunc from 0x73dc65 to 0x08b4bb (0x6b27a9 bytes) 000187[1] 1786480726.061389 trunc from 0x08b4bc to 0x5442fe (0x4b8e43 bytes) 000189[0] 1786480726.077624 read 0x427464 thru 0x42d98a (0x6527 bytes) 000191[0] 1786480726.113642 mapread 0x24707e thru 0x24d57f (0x6502 bytes) 000192[1] 1786480726.148893 trunc from 0x5442ff to 0x1645af (0x3dfd4f bytes) 000193[0] 1786480726.221606 write 0x7c212d thru 0x7c7ab5 (0x5989 bytes) 000196[1] 1786480726.278451 write 0x138e2e thru 0x13cc1f (0x3df2 bytes) 000198[0] 1786480726.330195 mapread 0x57fcf4 thru 0x585aad (0x5dba bytes) 000199[0] 1786480726.371511 mapwrite 0x5a7747 thru 0x5b4bca (0xd484 bytes) 000200[0] 1786480726.477578 mapread 0x783cd4 thru 0x793047 (0xf374 bytes) 000202[0] 1786480726.615356 trunc from 0x7c7ab6 to 0x0fc4eb (0x6cb5ca bytes) truncating to largest ever: 0x9dae33 000204[1] 1786480726.693179 trunc from 0x0fc4ec to 0x9dae32 (0x8de947 bytes) 000205[1] 1786480726.774653 read 0x861c8e thru 0x86593d (0x3cb0 bytes) 000208[0] 1786480726.868276 trunc from 0x9dae33 to 0x6f509d (0x2e5d95 bytes) 000210[0] 1786480727.028256 write 0x7d6b57 thru 0x7d89cc (0x1e76 bytes) 000212[1] 1786480727.041524 mapread 0x40d4f5 thru 0x40ece4 (0x17f0 bytes) 000215[0] 1786480727.183672 mapwrite 0x91fbfa thru 0x922221 (0x2628 bytes) 000217[1] 1786480727.331955 mapwrite 0x263228 thru 0x26fdc1 (0xcb9a bytes) 000218[1] 1786480727.533308 write 0x5c6fbe thru 0x5d2cd2 (0xbd15 bytes) 000219[0] 1786480727.591477 mapwrite 0x9bba20 thru 0x9be62d (0x2c0e bytes) 000224[0] 1786480727.896513 read 0x558273 thru 0x558c5b (0x9e9 bytes) 000227[0] 1786480727.974392 mapread 0x21e69c thru 0x220699 (0x1ffe bytes) 000228[1] 1786480728.032526 mapread 0x93120a thru 0x931f17 (0xd0e bytes) 000230[0] 1786480728.117302 mapread 0x6d20d1 thru 0x6d297d (0x8ad bytes) 000231[0] 1786480728.146810 mapread 0x2ca652 thru 0x2d4522 (0x9ed1 bytes) 000233[0] 1786480728.189250 write 0x91a13c thru 0x9284ab (0xe370 bytes) 000234[1] 1786480728.210672 mapwrite 0x9184e4 thru 0x921b4e (0x966b bytes) 000236[1] 1786480728.339558 write 0x1ee2af thru 0x1f3ff4 (0x5d46 bytes) 000237[0] 1786480728.377914 mapwrite 0x003b16 thru 0x009043 (0x552e bytes) 000239[1] 1786480728.539837 trunc from 0x9be62e to 0x11566c (0x8a8fc1 bytes) 000240[0] 1786480728.586514 read 0x0b74e9 thru 0x0bdb94 (0x66ac bytes) 000242[1] 1786480728.628305 trunc from 0x11566d to 0x88d81d (0x7781b1 bytes) 000243[1] 1786480728.697595 trunc from 0x88d81e to 0x1a0827 (0x6ecff6 bytes) 000244[0] 1786480728.720390 mapread 0x18d13d thru 0x1968e5 (0x97a9 bytes) 000245[1] 1786480728.770016 mapread 0x0394d4 thru 0x04705a (0xdb87 bytes) 000246[1] 1786480728.797492 trunc from 0x1a0828 to 0x93e960 (0x79e139 bytes) 000249[0] 1786480728.829347 write 0x60c9b6 thru 0x61c4bf (0xfb0a bytes) 000252[1] 1786480728.901444 read 0x8e9590 thru 0x8f3796 (0xa207 bytes) 000253[1] 1786480728.928239 mapwrite 0x42b4be thru 0x430b24 (0x5667 bytes) 000256[0] 1786480729.056663 mapread 0x440bae thru 0x444b2e (0x3f81 bytes) 000258[0] 1786480729.127518 mapwrite 0x088207 thru 0x088349 (0x143 bytes) 000259[0] 1786480729.183484 trunc from 0x93e961 to 0x3cb09d (0x5738c3 bytes) 000266[0] 1786480729.262563 write 0x44f144 thru 0x44f3cc (0x289 bytes) 000267[0] 1786480729.263832 mapwrite 0x8e3646 thru 0x8edeba (0xa875 bytes) 000270[1] 1786480729.330200 mapwrite 0x0a2409 thru 0x0a44ef (0x20e7 bytes) 000273[0] 1786480729.502524 trunc from 0x8edebb to 0x31e148 (0x5cfd72 bytes) 000280[0] 1786480729.635404 mapwrite 0x76e1d6 thru 0x7733a7 (0x51d2 bytes) 000281[0] 1786480729.728534 mapwrite 0x7cbebd thru 0x7cca64 (0xba8 bytes) 000285[1] 1786480729.893841 trunc from 0x7cca65 to 0x619a6f (0x1b2ff5 bytes) 000286[1] 1786480729.929155 mapread 0x3345a4 thru 0x335688 (0x10e5 bytes) 000287[0] 1786480729.939429 mapread 0x5fd50d thru 0x601b22 (0x4616 bytes) 000289[1] 1786480730.006625 write 0x83ccfc thru 0x840f49 (0x424e bytes) 000296[0] 1786480730.081952 trunc from 0x840f4a to 0x7d979b (0x677ae bytes) 000297[1] 1786480730.167173 write 0x24bd06 thru 0x259507 (0xd802 bytes) 000301[1] 1786480730.252458 write 0x14a1a2 thru 0x157a8a (0xd8e9 bytes) 000303[1] 1786480730.293436 trunc from 0x7d979c to 0x064104 (0x775697 bytes) 000304[1] 1786480730.356482 mapread 0x02f403 thru 0x036cd5 (0x78d3 bytes) 000305[1] 1786480730.398501 write 0x747b98 thru 0x75067e (0x8ae7 bytes) 000307[1] 1786480730.431500 mapwrite 0x5a3ed8 thru 0x5b0ee4 (0xd00d bytes) 000309[1] 1786480730.549891 trunc from 0x75067f to 0x891c9a (0x14161c bytes) 000310[1] 1786480730.569054 write 0x0c3bf8 thru 0x0d233f (0xe748 bytes) 000311[0] 1786480730.609359 write 0x1140d3 thru 0x118090 (0x3fbe bytes) 000312[0] 1786480730.663053 mapwrite 0x596c51 thru 0x59c7a4 (0x5b54 bytes) 000313[1] 1786480730.776381 write 0x65e48b thru 0x669eed (0xba63 bytes) 000314[1] 1786480730.816670 write 0x93e9fd thru 0x9402eb (0x18ef bytes) 000315[0] 1786480730.836761 write 0x17a511 thru 0x181287 (0x6d77 bytes) 000317[1] 1786480730.892763 write 0x6280a3 thru 0x628d95 (0xcf3 bytes) 000318[0] 1786480730.914226 mapwrite 0x49b5f5 thru 0x4a4707 (0x9113 bytes) 000319[1] 1786480731.042932 trunc from 0x9402ec to 0x88daf3 (0xb27f8 bytes) 000321[1] 1786480731.091913 read 0x285ade thru 0x2918d2 (0xbdf5 bytes) 000325[1] 1786480731.109737 write 0x1a8ed8 thru 0x1ae530 (0x5659 bytes) 000326[1] 1786480731.119486 write 0x4d8892 thru 0x4deb98 (0x6307 bytes) 000327[1] 1786480731.128423 read 0x886edf thru 0x88daf3 (0x6c15 bytes) 000331[0] 1786480731.163935 write 0x28d2f7 thru 0x291ff3 (0x4cfd bytes) 000336[0] 1786480731.257893 trunc from 0x88daf4 to 0x8d0184 (0x42691 bytes) 000337[1] 1786480731.313358 read 0x47760f thru 0x48172d (0xa11f bytes) 000338[0] 1786480731.335817 trunc from 0x8d0185 to 0x5e712c (0x2e9058 bytes) 000339[0] 1786480731.367146 trunc from 0x5e712d to 0x65444b (0x6d31f bytes) 000340[1] 1786480731.389008 mapwrite 0x691f96 thru 0x699593 (0x75fe bytes) 000342[1] 1786480731.565912 mapwrite 0x1284b3 thru 0x12bdbc (0x390a bytes) 000345[0] 1786480731.704240 write 0x8cc045 thru 0x8da6fa (0xe6b6 bytes) 000347[0] 1786480731.746459 write 0x2c9cda thru 0x2cc458 (0x277f bytes) 000349[1] 1786480731.785885 write 0x224c7e thru 0x23356e (0xe8f1 bytes) 000350[0] 1786480731.840663 mapwrite 0x44e80d thru 0x450d2b (0x251f bytes) 000351[1] 1786480731.967525 trunc from 0x8da6fb to 0x844033 (0x966c7 bytes) 000352[1] 1786480732.009502 mapread 0x6f3b80 thru 0x703831 (0xfcb2 bytes) 000354[1] 1786480732.043262 write 0x2354e7 thru 0x23cc11 (0x772b bytes) 000358[1] 1786480732.106351 mapwrite 0x52bd57 thru 0x538d3a (0xcfe4 bytes) 000359[0] 1786480732.217706 trunc from 0x844034 to 0x82a086 (0x19fad bytes) 000361[1] 1786480732.268441 trunc from 0x82a087 to 0x4a4bfd (0x385489 bytes) 000364[1] 1786480732.373828 read 0x23c48c thru 0x24b298 (0xee0d bytes) 000366[1] 1786480732.424200 mapwrite 0x0ac402 thru 0x0b80cd (0xbccc bytes) 000367[1] 1786480732.556817 mapwrite 0x04511e thru 0x0484bb (0x339e bytes) 000368[0] 1786480732.640856 trunc from 0x4a4bfe to 0x275a98 (0x22f165 bytes) 000370[1] 1786480732.743751 write 0x9c9ff1 thru 0x9d493d (0xa94d bytes) 000371[1] 1786480732.772571 read 0x05ea09 thru 0x068cc2 (0xa2ba bytes) 000372[0] 1786480732.802967 read 0x08cebc thru 0x08e45b (0x15a0 bytes) 000373[0] 1786480732.931282 write 0x89ab84 thru 0x89ba5a (0xed7 bytes) 000374[1] 1786480733.052695 mapread 0x6d2273 thru 0x6d2a8a (0x818 bytes) 000378[1] 1786480733.133385 write 0x6c3e1d thru 0x6d2f27 (0xf10b bytes) 000385[0] 1786480733.281221 read 0x7a4ba8 thru 0x7a90a3 (0x44fc bytes) 000386[0] 1786480733.338291 trunc from 0x9d493e to 0x003f4f (0x9d09ee bytes) 000387[0] 1786480733.385383 write 0x2275ee thru 0x233d54 (0xc767 bytes) 000388[1] 1786480733.430181 mapwrite 0x0495da thru 0x05021e (0x6c45 bytes) 000390[0] 1786480733.582895 write 0x487faf thru 0x4888b3 (0x905 bytes) 000393[0] 1786480733.641971 trunc from 0x4888b4 to 0x1f36a4 (0x29520f bytes) 000394[1] 1786480733.669265 mapwrite 0x1e1ea0 thru 0x1f0a5d (0xebbe bytes) 000396[1] 1786480733.864768 read 0x0a7ccb thru 0x0b6b2d (0xee63 bytes) 000398[1] 1786480733.931829 mapread 0x0a0827 thru 0x0aad28 (0xa502 bytes) 000401[0] 1786480733.978877 mapread 0x1a07d0 thru 0x1ac882 (0xc0b3 bytes) 000402[0] 1786480734.036431 mapwrite 0x961b09 thru 0x965899 (0x3d91 bytes) 000403[0] 1786480734.141884 mapwrite 0x3d4393 thru 0x3d43ac (0x1a bytes) 000404[0] 1786480734.207795 mapread 0x32de7a thru 0x33dc78 (0xfdff bytes) 000406[0] 1786480734.255484 write 0x25415f thru 0x256010 (0x1eb2 bytes) 000409[1] 1786480734.300465 mapread 0x70e8f3 thru 0x719bcb (0xb2d9 bytes) 000410[1] 1786480734.395607 read 0x4a1b13 thru 0x4b1487 (0xf975 bytes) 000411[0] 1786480734.412886 write 0x3a250a thru 0x3ab300 (0x8df7 bytes) 000415[0] 1786480734.444262 read 0x5d69b7 thru 0x5da235 (0x387f bytes) 000416[1] 1786480734.451641 write 0x0af093 thru 0x0b70b4 (0x8022 bytes) 000418[0] 1786480734.503812 write 0x9bdb63 thru 0x9cb9a8 (0xde46 bytes) 000420[0] 1786480734.544044 write 0x66a5c4 thru 0x66b732 (0x116f bytes) 000421[1] 1786480734.589964 read 0x7f6f53 thru 0x801573 (0xa621 bytes) 000424[0] 1786480734.648194 write 0x99dac0 thru 0x9ac05a (0xe59b bytes) 000425[1] 1786480734.675710 mapread 0x18c001 thru 0x19bcb4 (0xfcb4 bytes) 000427[0] 1786480734.735420 trunc from 0x9cb9a9 to 0x524373 (0x4a7635 bytes) 000431[0] 1786480734.820626 read 0x1f60b7 thru 0x2047f7 (0xe741 bytes) 000432[0] 1786480734.845454 mapread 0x2a083d thru 0x2ade1a (0xd5de bytes) 000433[1] 1786480734.893966 mapread 0x2b07cc thru 0x2b71a6 (0x69db bytes) 000434[0] 1786480734.949706 trunc from 0x524374 to 0x9263eb (0x402078 bytes) 000435[1] 1786480734.980623 mapwrite 0x13d437 thru 0x13f9b1 (0x257b bytes) 000438[0] 1786480735.101857 trunc from 0x9263ec to 0x6e9bc0 (0x23c82b bytes) 000439[0] 1786480735.159934 write 0x74cc1e thru 0x759659 (0xca3c bytes) 000440[0] 1786480735.190613 mapread 0x09ea02 thru 0x09f5a8 (0xba7 bytes) 000441[1] 1786480735.202449 read 0x0a2899 thru 0x0a35b9 (0xd21 bytes) 000442[0] 1786480735.224020 read 0x2d1d20 thru 0x2d9fe7 (0x82c8 bytes) 000443[0] 1786480735.244926 mapread 0x5681bf thru 0x56efb3 (0x6df5 bytes) 000444[0] 1786480735.261730 mapwrite 0x2095e9 thru 0x214b6c (0xb584 bytes) 000446[1] 1786480735.409738 write 0x5f88e3 thru 0x5fdea4 (0x55c2 bytes) 000447[1] 1786480735.470188 mapread 0x5b447c thru 0x5bda78 (0x95fd bytes) 000449[0] 1786480735.535404 trunc from 0x75965a to 0x82e2f5 (0xd4c9c bytes) 000450[0] 1786480735.598954 write 0x857cb4 thru 0x85e78a (0x6ad7 bytes) 000451[0] 1786480735.604333 write 0x52c82d thru 0x533e6d (0x7641 bytes) 000453[0] 1786480735.668463 write 0x865ec8 thru 0x86d3d0 (0x7509 bytes) 000454[1] 1786480735.670889 read 0x14140b thru 0x14c37f (0xaf75 bytes) 000455[1] 1786480735.717386 write 0x915eef thru 0x918054 (0x2166 bytes) 000457[0] 1786480735.761778 read 0x08be9c thru 0x093720 (0x7885 bytes) 000458[1] 1786480735.801999 read 0x8ee65c thru 0x8f077c (0x2121 bytes) 000462[0] 1786480735.871319 trunc from 0x918055 to 0x3514cd (0x5c6b87 bytes) 000463[0] 1786480735.934272 mapwrite 0x017b63 thru 0x022285 (0xa723 bytes) 000464[1] 1786480736.090200 write 0x3d3dfd thru 0x3d45a3 (0x7a7 bytes) 000466[0] 1786480736.153016 mapwrite 0x9ec552 thru 0x9f1e16 (0x58c5 bytes) 000467[1] 1786480736.326179 trunc from 0x9f1e17 to 0x3b5f7c (0x63be9a bytes) 000468[0] 1786480736.441908 read 0x1b992c thru 0x1bd6f7 (0x3dcc bytes) 000470[1] 1786480736.487143 mapwrite 0x90a35b thru 0x913ba2 (0x9848 bytes) 000471[1] 1786480736.717373 write 0x24f23f thru 0x254c7e (0x5a40 bytes) 000472[0] 1786480736.769429 read 0x251250 thru 0x25ad5f (0x9b10 bytes) 000473[1] 1786480736.852490 trunc from 0x913ba3 to 0x87d9f2 (0x961b0 bytes) 000474[0] 1786480736.989266 write 0x6ba978 thru 0x6c5ff6 (0xb67f bytes) 000476[0] 1786480737.115502 mapwrite 0x7c8133 thru 0x7d247b (0xa349 bytes) 000477[1] 1786480737.233736 trunc from 0x87d9f3 to 0x210988 (0x66d06a bytes) 000479[0] 1786480737.310667 mapread 0x082483 thru 0x084dad (0x292b bytes) 000480[0] 1786480737.366552 mapread 0x11667c thru 0x11fd4d (0x96d2 bytes) 000482[1] 1786480737.455521 mapwrite 0x8b01d4 thru 0x8bdcd6 (0xdb03 bytes) 000485[1] 1786480737.676327 mapwrite 0x215bde thru 0x2181d5 (0x25f8 bytes) 000486[1] 1786480737.786692 mapwrite 0x45e73b thru 0x467c2a (0x94f0 bytes) 000487[1] 1786480737.905621 mapwrite 0x448916 thru 0x44bd50 (0x343b bytes) 000490[0] 1786480737.959242 read 0x2692fe thru 0x26aaf2 (0x17f5 bytes) 000493[1] 1786480737.984296 write 0x0cd74e thru 0x0d83fa (0xacad bytes) 000495[0] 1786480738.042939 mapread 0x56cac6 thru 0x57928d (0xc7c8 bytes) All operations completed A-OK! Chance of close/open is 1 in 50 Seed set to 738 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000004[0] 1786480739.009805 write_OD 0x0ca000 thru 0x0d8fff (0xf000 bytes) 000005[1] 1786480739.118384 read_OD 0x05b000 thru 0x067fff (0xd000 bytes) 000006[1] 1786480739.142435 read_OD 0x0ce000 thru 0x0d7fff (0xa000 bytes) 000009[0] 1786480739.151739 read_OD 0x0bd000 thru 0x0c0fff (0x4000 bytes) 000014[1] 1786480739.161179 mapwrite 0x51e000 thru 0x52d201 (0xf202 bytes) truncating to largest ever: 0x21a69 000017[1] 1786480739.338151 trunc from 0x52d202 to 0x021a68 (0x50b799 bytes) truncating to largest ever: 0x2cc449 000021[0] 1786480739.389986 trunc from 0x021a69 to 0x2cc448 (0x2aa9e0 bytes) 000024[1] 1786480739.472055 mapread 0x0f3000 thru 0x0fc67c (0x967d bytes) 000026[1] 1786480739.533223 write_OD 0x7ab000 thru 0x7acfff (0x2000 bytes) 000029[0] 1786480739.581717 mapwrite 0x6d7000 thru 0x6e4390 (0xd391 bytes) 000030[0] 1786480739.660966 mapread 0x5e3000 thru 0x5f2bde (0xfbdf bytes) 000031[0] 1786480739.703154 read_OD 0x2f7000 thru 0x303fff (0xd000 bytes) 000033[0] 1786480739.719783 trunc from 0x7ad000 to 0x2c5e72 (0x4e718d bytes) 000034[0] 1786480739.747217 mapwrite 0x980000 thru 0x9835cf (0x35d0 bytes) 000035[0] 1786480739.820606 write_OD 0x092000 thru 0x09afff (0x9000 bytes) 000037[1] 1786480739.872753 mapwrite 0x23d000 thru 0x249881 (0xc882 bytes) truncating to largest ever: 0x9f4a1a 000038[1] 1786480739.975477 trunc from 0x9835d0 to 0x9f4a19 (0x7144a bytes) 000039[0] 1786480740.018455 write_OD 0x577000 thru 0x57bfff (0x5000 bytes) 000040[0] 1786480740.074675 write_OD 0x247000 thru 0x254fff (0xe000 bytes) 000041[0] 1786480740.118395 mapwrite 0x4c2000 thru 0x4c4190 (0x2191 bytes) 000042[1] 1786480740.177744 trunc from 0x9f4a1a to 0x518fb7 (0x4dba62 bytes) 000043[1] 1786480740.202471 read_OD 0x2f6000 thru 0x2f7fff (0x2000 bytes) 000046[1] 1786480740.217526 trunc from 0x518fb8 to 0x05ed29 (0x4ba28e bytes) 000050[0] 1786480740.271816 read_OD 0x04e000 thru 0x050fff (0x3000 bytes) 000053[0] 1786480740.297176 mapwrite 0x78f000 thru 0x79cb1c (0xdb1d bytes) 000055[0] 1786480740.512050 mapwrite 0x50b000 thru 0x50f5eb (0x45ec bytes) 000060[1] 1786480740.600967 mapread 0x677000 thru 0x67a96d (0x396e bytes) 000061[0] 1786480740.630720 mapwrite 0x298000 thru 0x2a67bb (0xe7bc bytes) 000064[0] 1786480740.791260 mapwrite 0x456000 thru 0x461e49 (0xbe4a bytes) 000065[1] 1786480740.920225 write_OD 0x0b0000 thru 0x0b5fff (0x6000 bytes) 000066[1] 1786480740.971557 trunc from 0x79cb1d to 0x40b5eb (0x391531 bytes) 000067[1] 1786480741.032763 mapread 0x0d8000 thru 0x0de43d (0x643e bytes) 000070[0] 1786480741.085771 trunc from 0x40b5ec to 0x046cc8 (0x3c4923 bytes) 000071[1] 1786480741.134472 write_OD 0x12a000 thru 0x136fff (0xd000 bytes) 000073[1] 1786480741.184780 write_OD 0x457000 thru 0x463fff (0xd000 bytes) 000080[1] 1786480741.234110 trunc from 0x464000 to 0x508ee7 (0xa4ee8 bytes) 000082[1] 1786480741.297679 read_OD 0x399000 thru 0x3a0fff (0x8000 bytes) 000085[0] 1786480741.321139 mapread 0x35d000 thru 0x3696be (0xc6bf bytes) 000086[1] 1786480741.381829 trunc from 0x508ee8 to 0x83a4dd (0x3315f6 bytes) 000087[1] 1786480741.417336 read_OD 0x531000 thru 0x539fff (0x9000 bytes) 000088[1] 1786480741.443980 mapread 0x7cb000 thru 0x7cf487 (0x4488 bytes) 000090[0] 1786480741.471948 mapwrite 0x3bc000 thru 0x3c750e (0xb50f bytes) 000091[1] 1786480741.608932 read_OD 0x1dd000 thru 0x1e6fff (0xa000 bytes) 000092[1] 1786480741.640743 mapread 0x3a2000 thru 0x3b03a1 (0xe3a2 bytes) 000093[1] 1786480741.674378 mapread 0x5b2000 thru 0x5bd06f (0xb070 bytes) 000094[1] 1786480741.709280 read_OD 0x7eb000 thru 0x7effff (0x5000 bytes) 000096[1] 1786480741.718226 read_OD 0x4fe000 thru 0x4fefff (0x1000 bytes) 000102[1] 1786480741.723506 mapread 0x274000 thru 0x2795e5 (0x55e6 bytes) 000104[1] 1786480741.740875 write_OD 0x0fd000 thru 0x100fff (0x4000 bytes) 000106[0] 1786480741.801764 read_OD 0x34e000 thru 0x354fff (0x7000 bytes) 000110[0] 1786480741.811808 write_OD 0x2f2000 thru 0x2f9fff (0x8000 bytes) 000113[1] 1786480741.852920 read_OD 0x6d5000 thru 0x6defff (0xa000 bytes) 000114[1] 1786480741.881754 mapread 0x1bb000 thru 0x1c0ee0 (0x5ee1 bytes) 000116[1] 1786480741.901453 trunc from 0x83a4de to 0x02eb39 (0x80b9a4 bytes) 000118[1] 1786480741.942333 trunc from 0x02eb3a to 0x48a471 (0x45b938 bytes) 000119[1] 1786480741.960891 read_OD 0x41a000 thru 0x420fff (0x7000 bytes) 000123[0] 1786480741.988266 read_OD 0x315000 thru 0x31afff (0x6000 bytes) 000127[1] 1786480741.994796 read_OD 0x276000 thru 0x279fff (0x4000 bytes) 000128[1] 1786480741.999007 trunc from 0x48a472 to 0x9394cb (0x4af05a bytes) 000129[1] 1786480742.038787 mapread 0x474000 thru 0x475315 (0x1316 bytes) 000131[1] 1786480742.077574 write_OD 0x837000 thru 0x83bfff (0x5000 bytes) 000132[1] 1786480742.141544 write_OD 0x61b000 thru 0x61cfff (0x2000 bytes) 000137[0] 1786480742.209364 write_OD 0x27e000 thru 0x27efff (0x1000 bytes) 000138[1] 1786480742.266905 mapwrite 0x79e000 thru 0x7a9fb2 (0xbfb3 bytes) 000139[1] 1786480742.380372 read_OD 0x8c1000 thru 0x8c4fff (0x4000 bytes) 000141[0] 1786480742.398721 mapwrite 0x454000 thru 0x4624e1 (0xe4e2 bytes) 000142[0] 1786480742.487448 mapread 0x5ed000 thru 0x5fcb9c (0xfb9d bytes) 000144[1] 1786480742.550754 trunc from 0x9394cc to 0x49c482 (0x49d049 bytes) 000149[0] 1786480742.606367 trunc from 0x49c483 to 0x2dc473 (0x1c000f bytes) 000151[0] 1786480742.648467 write_OD 0x78b000 thru 0x795fff (0xb000 bytes) 000153[0] 1786480742.708316 mapread 0x40d000 thru 0x419438 (0xc439 bytes) 000154[1] 1786480742.759342 read_OD 0x72a000 thru 0x734fff (0xb000 bytes) 000158[1] 1786480742.774224 mapwrite 0x241000 thru 0x243d33 (0x2d34 bytes) 000160[1] 1786480742.900625 read_OD 0x220000 thru 0x22efff (0xf000 bytes) 000165[0] 1786480742.954003 read_OD 0x71a000 thru 0x71ffff (0x6000 bytes) 000166[1] 1786480742.972882 mapwrite 0x0db000 thru 0x0e1ff0 (0x6ff1 bytes) 000167[0] 1786480743.106192 trunc from 0x796000 to 0x1808a5 (0x61575a bytes) 000168[0] 1786480743.135369 mapread 0x042000 thru 0x04e95a (0xc95b bytes) 000169[0] 1786480743.186549 mapread 0x129000 thru 0x132a0c (0x9a0d bytes) 000171[0] 1786480743.218713 mapwrite 0x145000 thru 0x149adc (0x4add bytes) 000172[0] 1786480743.328799 mapread 0x0e4000 thru 0x0ee359 (0xa35a bytes) 000173[1] 1786480743.345900 write_OD 0x0ed000 thru 0x0f1fff (0x5000 bytes) 000174[1] 1786480743.389834 mapwrite 0x174000 thru 0x18153e (0xd53f bytes) 000175[1] 1786480743.505217 trunc from 0x18153f to 0x30ee20 (0x18d8e2 bytes) 000176[0] 1786480743.519857 mapread 0x0c4000 thru 0x0ccf76 (0x8f77 bytes) 000177[0] 1786480743.596589 mapread 0x1c5000 thru 0x1d28c8 (0xd8c9 bytes) 000178[1] 1786480743.649534 write_OD 0x364000 thru 0x364fff (0x1000 bytes) 000179[0] 1786480743.754813 mapwrite 0x82d000 thru 0x83431b (0x731c bytes) 000181[1] 1786480743.893820 write_OD 0x012000 thru 0x01afff (0x9000 bytes) 000182[0] 1786480743.959167 mapwrite 0x229000 thru 0x238bac (0xfbad bytes) 000183[1] 1786480744.065243 write_OD 0x891000 thru 0x891fff (0x1000 bytes) 000184[1] 1786480744.120223 mapread 0x6c3000 thru 0x6cd507 (0xa508 bytes) 000187[0] 1786480744.161225 read_OD 0x7eb000 thru 0x7f4fff (0xa000 bytes) 000189[1] 1786480744.181908 mapread 0x018000 thru 0x021529 (0x952a bytes) 000191[1] 1786480744.198321 mapwrite 0x606000 thru 0x6090a5 (0x30a6 bytes) 000192[0] 1786480744.254852 trunc from 0x892000 to 0x5c8afc (0x2c9503 bytes) 000193[1] 1786480744.280728 mapread 0x218000 thru 0x2183f3 (0x3f4 bytes) 000197[1] 1786480744.308456 write_OD 0x75f000 thru 0x76bfff (0xd000 bytes) 000198[0] 1786480744.357786 read_OD 0x2a2000 thru 0x2acfff (0xb000 bytes) 000201[0] 1786480744.369936 write_OD 0x6a9000 thru 0x6b6fff (0xe000 bytes) 000203[0] 1786480744.428025 write_OD 0x1dc000 thru 0x1eafff (0xf000 bytes) 000204[0] 1786480744.497863 read_OD 0x0bd000 thru 0x0cafff (0xe000 bytes) 000206[1] 1786480744.511046 write_OD 0x4fd000 thru 0x502fff (0x6000 bytes) 000207[0] 1786480744.570248 trunc from 0x76c000 to 0x7d6160 (0x6a161 bytes) 000214[0] 1786480744.678931 read_OD 0x5c0000 thru 0x5cafff (0xb000 bytes) 000216[1] 1786480744.724610 mapread 0x794000 thru 0x7a05d4 (0xc5d5 bytes) 000217[1] 1786480744.762062 read_OD 0x0d0000 thru 0x0d3fff (0x4000 bytes) 000220[1] 1786480744.770254 read_OD 0x2a9000 thru 0x2acfff (0x4000 bytes) 000221[1] 1786480744.780027 mapwrite 0x672000 thru 0x67c4b1 (0xa4b2 bytes) truncating to largest ever: 0x9fb24c 000224[0] 1786480744.907844 trunc from 0x7d6161 to 0x9fb24b (0x2250eb bytes) 000225[0] 1786480744.957889 read_OD 0x816000 thru 0x81cfff (0x7000 bytes) 000226[1] 1786480744.975360 write_OD 0x791000 thru 0x79ffff (0xf000 bytes) 000227[1] 1786480745.038118 write_OD 0x2fc000 thru 0x300fff (0x5000 bytes) 000228[0] 1786480745.106057 mapread 0x85c000 thru 0x869611 (0xd612 bytes) 000231[0] 1786480745.164163 read_OD 0x762000 thru 0x765fff (0x4000 bytes) 000232[1] 1786480745.170198 read_OD 0x90b000 thru 0x90bfff (0x1000 bytes) 000234[0] 1786480745.183058 mapwrite 0x34d000 thru 0x3502ad (0x32ae bytes) 000235[1] 1786480745.344584 mapread 0x14d000 thru 0x14f271 (0x2272 bytes) 000237[1] 1786480745.454964 read_OD 0x693000 thru 0x694fff (0x2000 bytes) 000241[0] 1786480745.473925 trunc from 0x9fb24c to 0x4f456b (0x506ce0 bytes) 000242[1] 1786480745.513569 write_OD 0x486000 thru 0x48bfff (0x6000 bytes) 000255[1] 1786480745.596266 mapread 0x3c9000 thru 0x3d3d08 (0xad09 bytes) 000257[0] 1786480745.637942 write_OD 0x977000 thru 0x980fff (0xa000 bytes) 000260[1] 1786480745.840476 mapread 0x1b9000 thru 0x1c89cd (0xf9ce bytes) 000261[0] 1786480745.910115 mapread 0x6c8000 thru 0x6d2eae (0xaeaf bytes) 000262[0] 1786480745.928724 read_OD 0x5c2000 thru 0x5c9fff (0x8000 bytes) 000263[1] 1786480745.936150 trunc from 0x981000 to 0x72d04a (0x253fb5 bytes) 000265[1] 1786480745.987542 read_OD 0x19c000 thru 0x19cfff (0x1000 bytes) 000266[1] 1786480746.008459 read_OD 0x4af000 thru 0x4b2fff (0x4000 bytes) 000269[1] 1786480746.015775 write_OD 0x381000 thru 0x389fff (0x9000 bytes) 000271[0] 1786480746.079200 mapread 0x585000 thru 0x58e4a9 (0x94aa bytes) 000272[1] 1786480746.099736 read_OD 0x4dd000 thru 0x4dffff (0x3000 bytes) 000273[0] 1786480746.105921 mapwrite 0x038000 thru 0x0463bd (0xe3be bytes) 000277[1] 1786480746.229033 write_OD 0x156000 thru 0x161fff (0xc000 bytes) 000278[0] 1786480746.292924 read_OD 0x01c000 thru 0x01dfff (0x2000 bytes) 000283[1] 1786480746.315323 mapwrite 0x7f0000 thru 0x7fdd00 (0xdd01 bytes) 000284[1] 1786480746.520131 mapread 0x536000 thru 0x541b15 (0xbb16 bytes) 000285[1] 1786480746.583728 mapread 0x538000 thru 0x545e94 (0xde95 bytes) 000286[1] 1786480746.592467 write_OD 0x5e2000 thru 0x5e4fff (0x3000 bytes) 000287[1] 1786480746.692768 mapread 0x792000 thru 0x79ef75 (0xcf76 bytes) 000288[1] 1786480746.750204 write_OD 0x335000 thru 0x337fff (0x3000 bytes) 000291[0] 1786480746.815551 write_OD 0x6ea000 thru 0x6f0fff (0x7000 bytes) 000292[0] 1786480746.867245 write_OD 0x47d000 thru 0x489fff (0xd000 bytes) 000296[0] 1786480746.933425 mapwrite 0x854000 thru 0x859a19 (0x5a1a bytes) 000298[0] 1786480747.073013 write_OD 0x37f000 thru 0x388fff (0xa000 bytes) 000299[1] 1786480747.171060 read_OD 0x76b000 thru 0x76efff (0x4000 bytes) 000303[0] 1786480747.188772 read_OD 0x1f3000 thru 0x1f3fff (0x1000 bytes) 000305[0] 1786480747.204585 trunc from 0x859a1a to 0x4de543 (0x37b4d6 bytes) 000306[1] 1786480747.285572 trunc from 0x4de544 to 0x8b287a (0x3d4337 bytes) 000307[0] 1786480747.343277 trunc from 0x8b287b to 0x3ea1ff (0x4c867b bytes) 000308[1] 1786480747.367944 mapwrite 0x4ac000 thru 0x4b049a (0x449b bytes) 000310[0] 1786480747.484042 write_OD 0x9e5000 thru 0x9f1fff (0xd000 bytes) 000311[1] 1786480747.534720 trunc from 0x9f2000 to 0x6bffd1 (0x33202e bytes) 000314[0] 1786480747.587276 read_OD 0x309000 thru 0x312fff (0xa000 bytes) 000317[0] 1786480747.626847 mapread 0x575000 thru 0x577d74 (0x2d75 bytes) 000318[1] 1786480747.657201 trunc from 0x6bffd2 to 0x911b95 (0x251bc4 bytes) 000321[0] 1786480747.741787 trunc from 0x911b96 to 0x98c129 (0x7a594 bytes) 000325[1] 1786480747.823397 write_OD 0x495000 thru 0x499fff (0x5000 bytes) 000331[0] 1786480747.890522 mapwrite 0x8ef000 thru 0x8f3344 (0x4345 bytes) 000332[0] 1786480747.992951 mapwrite 0x483000 thru 0x48feac (0xcead bytes) 000335[1] 1786480748.172326 read_OD 0x4ab000 thru 0x4acfff (0x2000 bytes) 000340[0] 1786480748.205230 mapwrite 0x8f9000 thru 0x908d53 (0xfd54 bytes) 000342[0] 1786480748.327814 mapwrite 0x803000 thru 0x811a87 (0xea88 bytes) 000344[1] 1786480748.450057 write_OD 0x895000 thru 0x8a2fff (0xe000 bytes) 000347[1] 1786480748.570760 trunc from 0x98c12a to 0x5d732e (0x3b4dfb bytes) 000348[0] 1786480748.622887 read_OD 0x0fc000 thru 0x108fff (0xd000 bytes) 000350[1] 1786480748.668139 mapwrite 0x2f0000 thru 0x2f349b (0x349c bytes) 000352[1] 1786480748.821882 mapread 0x01b000 thru 0x029fb7 (0xefb8 bytes) 000353[0] 1786480748.897882 trunc from 0x5d732f to 0x2be9c4 (0x31896a bytes) 000355[0] 1786480748.971969 mapread 0x124000 thru 0x132a7a (0xea7b bytes) 000356[1] 1786480749.008888 mapwrite 0x072000 thru 0x08197e (0xf97f bytes) 000357[0] 1786480749.115571 trunc from 0x2be9c5 to 0x836754 (0x577d90 bytes) 000359[0] 1786480749.167052 write_OD 0x950000 thru 0x958fff (0x9000 bytes) 000362[0] 1786480749.224667 mapwrite 0x713000 thru 0x71fb70 (0xcb71 bytes) 000363[0] 1786480749.351125 mapwrite 0x51e000 thru 0x529c33 (0xbc34 bytes) 000364[0] 1786480749.443573 mapwrite 0x8aa000 thru 0x8b2992 (0x8993 bytes) 000366[1] 1786480749.527781 mapread 0x410000 thru 0x414a74 (0x4a75 bytes) 000368[0] 1786480749.573536 mapread 0x46c000 thru 0x477c74 (0xbc75 bytes) 000369[1] 1786480749.610114 mapwrite 0x55c000 thru 0x563e9e (0x7e9f bytes) 000370[1] 1786480749.686232 mapread 0x41b000 thru 0x41f0ba (0x40bb bytes) 000373[1] 1786480749.726072 mapwrite 0x0db000 thru 0x0ea6b2 (0xf6b3 bytes) 000374[0] 1786480749.802949 write_OD 0x627000 thru 0x62cfff (0x6000 bytes) 000375[1] 1786480749.855814 read_OD 0x1f4000 thru 0x1fcfff (0x9000 bytes) 000377[0] 1786480749.872063 mapwrite 0x832000 thru 0x83f1e2 (0xd1e3 bytes) 000378[0] 1786480749.958772 read_OD 0x8cf000 thru 0x8dcfff (0xe000 bytes) 000380[1] 1786480749.992785 write_OD 0x2c7000 thru 0x2cefff (0x8000 bytes) 000385[0] 1786480750.047889 mapread 0x79b000 thru 0x7a76b7 (0xc6b8 bytes) 000388[1] 1786480750.074738 mapwrite 0x430000 thru 0x437541 (0x7542 bytes) 000390[0] 1786480750.150770 read_OD 0x374000 thru 0x37afff (0x7000 bytes) 000392[1] 1786480750.206716 mapwrite 0x95e000 thru 0x966487 (0x8488 bytes) 000394[1] 1786480750.305556 read_OD 0x5f3000 thru 0x5f7fff (0x5000 bytes) 000395[1] 1786480750.319650 read_OD 0x94d000 thru 0x950fff (0x4000 bytes) 000397[0] 1786480750.325927 trunc from 0x966488 to 0x2d88f6 (0x68db91 bytes) 000398[0] 1786480750.343455 mapread 0x295000 thru 0x2971cd (0x21ce bytes) 000401[0] 1786480750.403244 read_OD 0x0c0000 thru 0x0c6fff (0x7000 bytes) 000403[0] 1786480750.433218 mapread 0x0b7000 thru 0x0c5c23 (0xec24 bytes) 000406[0] 1786480750.484506 mapread 0x05d000 thru 0x067b27 (0xab28 bytes) 000407[1] 1786480750.485173 mapwrite 0x624000 thru 0x633b78 (0xfb79 bytes) 000408[1] 1786480750.606919 read_OD 0x3b0000 thru 0x3bdfff (0xe000 bytes) 000409[1] 1786480750.633968 write_OD 0x145000 thru 0x14afff (0x6000 bytes) 000413[0] 1786480750.691315 read_OD 0x508000 thru 0x515fff (0xe000 bytes) 000414[1] 1786480750.702039 mapwrite 0x904000 thru 0x909a69 (0x5a6a bytes) 000415[0] 1786480750.798767 mapread 0x2a4000 thru 0x2a7e4e (0x3e4f bytes) 000416[1] 1786480750.850740 read_OD 0x13b000 thru 0x142fff (0x8000 bytes) 000417[0] 1786480750.862471 trunc from 0x909a6a to 0x906cca (0x2d9f bytes) 000425[0] 1786480750.961629 write_OD 0x6ba000 thru 0x6c4fff (0xb000 bytes) 000427[0] 1786480751.017987 trunc from 0x906ccb to 0x6b4472 (0x252858 bytes) 000428[1] 1786480751.082463 mapwrite 0x528000 thru 0x536f1d (0xef1e bytes) 000430[0] 1786480751.212741 mapread 0x54f000 thru 0x555630 (0x6631 bytes) 000431[1] 1786480751.250522 mapwrite 0x97d000 thru 0x97e8e6 (0x18e7 bytes) 000432[1] 1786480751.326602 trunc from 0x97e8e7 to 0x1c01fb (0x7be6eb bytes) 000433[1] 1786480751.343415 read_OD 0x19d000 thru 0x1abfff (0xf000 bytes) 000434[1] 1786480751.370045 write_OD 0x32c000 thru 0x32efff (0x3000 bytes) 000438[0] 1786480751.423186 trunc from 0x32f000 to 0x9837d3 (0x6547d4 bytes) 000439[0] 1786480751.451581 mapwrite 0x8e4000 thru 0x8e7feb (0x3fec bytes) 000441[1] 1786480751.497262 mapwrite 0x9f5000 thru 0x9ffa01 (0xaa02 bytes) 000442[0] 1786480751.741695 read_OD 0x724000 thru 0x725fff (0x2000 bytes) 000445[1] 1786480751.814962 write_OD 0x97a000 thru 0x984fff (0xb000 bytes) 000447[1] 1786480751.875305 mapread 0x74e000 thru 0x75c4e6 (0xe4e7 bytes) 000452[0] 1786480751.948017 read_OD 0x024000 thru 0x024fff (0x1000 bytes) 000454[0] 1786480751.953276 write_OD 0x9fe000 thru 0x9fefff (0x1000 bytes) 000459[0] 1786480752.002082 read_OD 0x244000 thru 0x248fff (0x5000 bytes) 000470[0] 1786480752.019668 mapwrite 0x23a000 thru 0x248479 (0xe47a bytes) 000474[0] 1786480752.231219 trunc from 0x9ffa02 to 0x7a4abd (0x25af44 bytes) 000476[1] 1786480752.260195 mapwrite 0x6ec000 thru 0x6f668c (0xa68d bytes) 000478[1] 1786480752.419984 mapwrite 0x55e000 thru 0x565653 (0x7654 bytes) 000482[0] 1786480752.596810 trunc from 0x7a4abe to 0x299007 (0x50bab6 bytes) 000485[1] 1786480752.656501 trunc from 0x299008 to 0x478895 (0x1df88e bytes) 000488[0] 1786480752.725461 mapwrite 0x80a000 thru 0x812691 (0x8692 bytes) 000491[0] 1786480752.840081 write_OD 0x3c1000 thru 0x3cbfff (0xb000 bytes) 000492[0] 1786480752.901489 read_OD 0x523000 thru 0x524fff (0x2000 bytes) 000493[1] 1786480752.913834 read_OD 0x1a0000 thru 0x1adfff (0xe000 bytes) 000500[0] 1786480752.928409 write_OD 0x7ba000 thru 0x7c2fff (0x9000 bytes) All operations completed A-OK! PASS 16a (43s) == sanityn test 16b: 500 iterations of dual-mount fsx at small size ========================================================== 16:39:17 (1786480757) fallocate on zfs doesn't consume space fallocate not supported 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.399086 s, 21.0 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.311299 s, 26.9 MB/s Chance of close/open is 1 in 50 Seed set to 759 fd 0: /mnt/lustre/f16b.sanityn fd 1: /mnt/lustre2/f16b.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read truncating to largest ever: 0x32c360 000003[1] 1786480760.559954 trunc from 00000000 to 0x32c35f (0x32c360 bytes) 000007[0] 1786480760.660787 mapwrite 0x3c3f7d thru 0x3c54ac (0x1530 bytes) truncating to largest ever: 0x3a11cd 000008[0] 1786480760.775893 trunc from 0x3c54ad to 0x3a11cc (0x242e0 bytes) 000011[0] 1786480760.819336 mapwrite 0x5e3c14 thru 0x5e400c (0x3f9 bytes) 000012[0] 1786480760.971721 mapwrite 0x863617 thru 0x8648d1 (0x12bb bytes) 000013[0] 1786480761.094922 read 0x18097b thru 0x181b86 (0x120c bytes) 000015[0] 1786480761.105677 read 0x2f8666 thru 0x2fa486 (0x1e21 bytes) 000017[1] 1786480761.133552 mapwrite 0x1c8e01 thru 0x1ca2e4 (0x14e4 bytes) 000018[1] 1786480761.215374 mapwrite 0x7a143a thru 0x7a3156 (0x1d1d bytes) 000019[0] 1786480761.314806 read 0x4b1d9d thru 0x4b2f3e (0x11a2 bytes) 000020[0] 1786480761.348557 mapwrite 0x42cae1 thru 0x42e2c3 (0x17e3 bytes) truncating to largest ever: 0x445cfc 000021[1] 1786480761.464658 trunc from 0x8648d2 to 0x445cfb (0x41ebd6 bytes) 000022[1] 1786480761.492389 mapwrite 0x716754 thru 0x718455 (0x1d02 bytes) 000024[0] 1786480761.569694 trunc from 0x718456 to 0x082922 (0x695b33 bytes) 000025[0] 1786480761.594469 mapwrite 0x5d2bae thru 0x5d44f0 (0x1943 bytes) 000026[1] 1786480761.669237 mapwrite 0x2ed83d thru 0x2eedbb (0x157f bytes) 000027[0] 1786480761.767701 mapwrite 0x47c7d9 thru 0x47c8da (0x102 bytes) 000029[0] 1786480761.909665 read 0x3f6f3d thru 0x3f71e7 (0x2ab bytes) 000030[0] 1786480761.949446 read 0x28377d thru 0x284df7 (0x167b bytes) 000031[1] 1786480761.972387 write 0x6b4d48 thru 0x6b6a84 (0x1d3d bytes) truncating to largest ever: 0x4b4ec3 000033[1] 1786480762.013859 trunc from 0x6b6a85 to 0x4b4ec2 (0x201bc2 bytes) 000034[1] 1786480762.064628 read 0x47784b thru 0x477f4b (0x701 bytes) truncating to largest ever: 0x503721 000036[1] 1786480762.116443 trunc from 0x4b4ec3 to 0x503720 (0x4e85e bytes) 000038[1] 1786480762.166368 write 0x97fcae thru 0x9804fa (0x84d bytes) 000039[0] 1786480762.195728 write 0x855abb thru 0x8579bf (0x1f05 bytes) 000044[1] 1786480762.297777 read 0x58b547 thru 0x58baa9 (0x563 bytes) 000045[1] 1786480762.315847 write 0x26116a thru 0x26264d (0x14e4 bytes) 000046[1] 1786480762.331426 read 0x4b7e9a thru 0x4b9559 (0x16c0 bytes) 000047[0] 1786480762.337006 mapread 0x2eb5d0 thru 0x2ebb1f (0x550 bytes) 000048[1] 1786480762.390823 write 0x5ba21a thru 0x5baf68 (0xd4f bytes) 000049[0] 1786480762.409480 read 0x8e5495 thru 0x8e73bd (0x1f29 bytes) 000054[1] 1786480762.464063 write 0x8748b2 thru 0x876122 (0x1871 bytes) 000057[0] 1786480762.514160 mapwrite 0x2c9bef thru 0x2ca9cf (0xde1 bytes) 000058[1] 1786480762.609391 mapwrite 0x20a5bf thru 0x20c3e7 (0x1e29 bytes) 000061[0] 1786480762.723757 mapread 0x59da34 thru 0x59e56d (0xb3a bytes) 000062[1] 1786480762.766755 mapread 0x8ee254 thru 0x8ef857 (0x1604 bytes) 000063[1] 1786480762.793584 read 0x4d77a9 thru 0x4d8298 (0xaf0 bytes) 000064[0] 1786480762.800746 read 0x13d921 thru 0x13e716 (0xdf6 bytes) 000065[1] 1786480762.825761 trunc from 0x9804fb to 0x1fff71 (0x780589 bytes) 000069[1] 1786480762.925817 read 0x0b8fff thru 0x0ba0aa (0x10ac bytes) 000070[1] 1786480762.996342 mapwrite 0x92ef4d thru 0x9304f7 (0x15ab bytes) 000071[1] 1786480763.186550 trunc from 0x9304f8 to 0x04b9a5 (0x8e4b52 bytes) 000072[0] 1786480763.225662 mapread 0x01774e thru 0x01892c (0x11df bytes) 000073[1] 1786480763.288400 mapwrite 0x6b7b8d thru 0x6b8c20 (0x1094 bytes) truncating to largest ever: 0x93c95b 000074[0] 1786480763.392082 trunc from 0x6b8c21 to 0x93c95a (0x283d3a bytes) 000075[1] 1786480763.452428 read 0x1a692d thru 0x1a7e97 (0x156b bytes) 000079[0] 1786480763.495670 mapwrite 0x744622 thru 0x745e04 (0x17e3 bytes) 000080[0] 1786480763.580821 write 0x748666 thru 0x748b14 (0x4af bytes) 000081[0] 1786480763.589254 trunc from 0x93c95b to 0x41fbc8 (0x51cd92 bytes) 000082[1] 1786480763.654711 read 0x0543d9 thru 0x054a95 (0x6bd bytes) 000084[0] 1786480763.697531 mapwrite 0x5d0c09 thru 0x5d22c2 (0x16ba bytes) 000085[1] 1786480763.803672 mapread 0x4f8ce2 thru 0x4f9c80 (0xf9f bytes) 000088[1] 1786480763.880552 mapread 0x3966b3 thru 0x397ae2 (0x1430 bytes) 000089[1] 1786480763.914796 read 0x018167 thru 0x01879e (0x638 bytes) 000090[0] 1786480763.919318 mapwrite 0x672805 thru 0x672f80 (0x77c bytes) 000091[1] 1786480764.011821 mapread 0x5b30cd thru 0x5b4879 (0x17ad bytes) 000093[0] 1786480764.068015 trunc from 0x672f81 to 0x6e525f (0x722df bytes) 000096[0] 1786480764.186849 read 0x14859b thru 0x1497e3 (0x1249 bytes) 000097[1] 1786480764.215397 mapread 0x504d0c thru 0x5053d5 (0x6ca bytes) 000100[1] 1786480764.259889 write 0x2db3be thru 0x2dcda8 (0x19eb bytes) 000101[1] 1786480764.284149 write 0x8ffb6e thru 0x900767 (0xbfa bytes) 000103[1] 1786480764.305721 trunc from 0x900768 to 0x74546b (0x1bb2fc bytes) 000107[0] 1786480764.364857 mapread 0x4171c6 thru 0x418273 (0x10ae bytes) 000108[0] 1786480764.401629 mapwrite 0x65e940 thru 0x65fcfb (0x13bc bytes) 000110[0] 1786480764.516871 trunc from 0x74546c to 0x6be2d2 (0x87199 bytes) 000112[0] 1786480764.548926 mapread 0x2d3af5 thru 0x2d541a (0x1926 bytes) 000115[1] 1786480764.586807 mapread 0x25e998 thru 0x25fc6a (0x12d3 bytes) 000116[0] 1786480764.628420 read 0x17fc9a thru 0x1813ba (0x1721 bytes) 000118[0] 1786480764.661806 read 0x6430ca thru 0x643101 (0x38 bytes) 000119[1] 1786480764.667866 mapread 0x1a2981 thru 0x1a320b (0x88b bytes) 000121[0] 1786480764.720955 write 0x47fcc2 thru 0x48086d (0xbac bytes) 000124[1] 1786480764.754440 mapwrite 0x4400f1 thru 0x441256 (0x1166 bytes) 000125[0] 1786480764.832278 write 0x70e57a thru 0x70f8bc (0x1343 bytes) 000127[1] 1786480764.844797 write 0x9f6d2c thru 0x9f7817 (0xaec bytes) 000130[1] 1786480764.893272 write 0x30b857 thru 0x30d650 (0x1dfa bytes) 000132[0] 1786480764.927895 read 0x75c39b thru 0x75ceb5 (0xb1b bytes) 000134[0] 1786480764.974422 trunc from 0x9f7818 to 0x78d5cd (0x26a24a bytes) 000135[0] 1786480765.057868 write 0x32707d thru 0x328819 (0x179d bytes) 000136[1] 1786480765.077537 mapwrite 0x740cd6 thru 0x741e2e (0x1159 bytes) 000140[1] 1786480765.212319 write 0x220378 thru 0x22157f (0x1208 bytes) 000141[1] 1786480765.254163 mapread 0x71c87f thru 0x71e113 (0x1895 bytes) 000142[1] 1786480765.263370 write 0x66d2f2 thru 0x66e9d4 (0x16e3 bytes) 000143[1] 1786480765.293156 mapwrite 0x36c170 thru 0x36d90d (0x179e bytes) 000147[1] 1786480765.416429 mapread 0x59717c thru 0x598e66 (0x1ceb bytes) 000152[0] 1786480765.503787 read 0x666809 thru 0x66874e (0x1f46 bytes) 000153[1] 1786480765.543461 read 0x0d3013 thru 0x0d4ffe (0x1fec bytes) 000155[1] 1786480765.579794 mapwrite 0x6674b3 thru 0x667847 (0x395 bytes) 000157[0] 1786480765.642647 read 0x133b45 thru 0x13442a (0x8e6 bytes) 000159[0] 1786480765.683495 mapwrite 0x3a7f20 thru 0x3a8da7 (0xe88 bytes) 000163[1] 1786480765.807720 read 0x61f821 thru 0x620b7f (0x135f bytes) 000166[0] 1786480765.849947 mapread 0x378c74 thru 0x37a1ec (0x1579 bytes) 000167[1] 1786480765.880702 mapwrite 0x48b3c8 thru 0x48d1c5 (0x1dfe bytes) 000168[0] 1786480766.082046 mapread 0x4b1dbe thru 0x4b3238 (0x147b bytes) 000170[1] 1786480766.156419 write 0x88d362 thru 0x88e488 (0x1127 bytes) 000172[0] 1786480766.229533 mapread 0x23c524 thru 0x23cb85 (0x662 bytes) 000174[0] 1786480766.268685 trunc from 0x88e489 to 0x8a6b08 (0x18680 bytes) 000175[0] 1786480766.365388 mapwrite 0x462ed8 thru 0x464132 (0x125b bytes) 000177[0] 1786480766.451404 mapwrite 0x1b4d74 thru 0x1b6782 (0x1a0f bytes) 000179[1] 1786480766.562725 mapread 0x38954e thru 0x389cda (0x78d bytes) 000185[0] 1786480766.684675 mapwrite 0x1ad3f2 thru 0x1ae497 (0x10a6 bytes) 000186[0] 1786480766.767045 mapwrite 0x888154 thru 0x888dc1 (0xc6e bytes) 000189[1] 1786480766.841836 mapwrite 0x457e1a thru 0x458a4c (0xc33 bytes) 000190[1] 1786480766.924312 write 0x065897 thru 0x066e96 (0x1600 bytes) truncating to largest ever: 0x9847bc 000191[1] 1786480766.976264 trunc from 0x8a6b09 to 0x9847bb (0xddcb3 bytes) 000194[1] 1786480767.093713 write 0x56e594 thru 0x56f667 (0x10d4 bytes) 000196[0] 1786480767.114449 mapread 0x3f0610 thru 0x3f21bb (0x1bac bytes) truncating to largest ever: 0x99b17c 000198[0] 1786480767.263721 trunc from 0x9847bc to 0x99b17b (0x169c0 bytes) 000199[1] 1786480767.355490 mapwrite 0x9b2280 thru 0x9b22ed (0x6e bytes) 000201[1] 1786480767.498183 write 0x53b762 thru 0x53cde9 (0x1688 bytes) 000203[0] 1786480767.567705 mapwrite 0x2a1406 thru 0x2a234a (0xf45 bytes) 000206[0] 1786480767.741440 mapread 0x115435 thru 0x11739c (0x1f68 bytes) 000207[0] 1786480767.789348 mapread 0x7bde77 thru 0x7be0cb (0x255 bytes) 000209[0] 1786480767.857054 write 0x6d56f6 thru 0x6d708c (0x1997 bytes) 000210[1] 1786480767.879031 mapwrite 0x69a3c7 thru 0x69c323 (0x1f5d bytes) 000213[0] 1786480768.017803 read 0x1d8aba thru 0x1da484 (0x19cb bytes) 000215[0] 1786480768.053766 mapread 0x5cb828 thru 0x5ccda2 (0x157b bytes) 000216[1] 1786480768.092917 write 0x0442d2 thru 0x04622e (0x1f5d bytes) 000217[1] 1786480768.122438 read 0x0be0b3 thru 0x0bfbfd (0x1b4b bytes) 000218[0] 1786480768.155751 write 0x60c51f thru 0x60db90 (0x1672 bytes) 000220[0] 1786480768.222227 mapread 0x8a1270 thru 0x8a22be (0x104f bytes) 000221[0] 1786480768.291698 read 0x5789e8 thru 0x578f10 (0x529 bytes) 000224[1] 1786480768.317295 trunc from 0x9b22ee to 0x2e736b (0x6caf82 bytes) 000225[0] 1786480768.417439 read 0x27b7ae thru 0x27d1bd (0x1a10 bytes) 000227[0] 1786480768.504829 mapwrite 0x12cebd thru 0x12d2d4 (0x418 bytes) 000228[1] 1786480768.616598 read 0x1a699c thru 0x1a6ba3 (0x208 bytes) 000231[0] 1786480768.692881 read 0x1d31a3 thru 0x1d395c (0x7ba bytes) 000233[1] 1786480768.739851 read 0x137119 thru 0x1389d2 (0x18ba bytes) 000234[0] 1786480768.740775 mapwrite 0x0569b5 thru 0x056fa9 (0x5f5 bytes) 000237[1] 1786480768.870643 mapwrite 0x094ada thru 0x096379 (0x18a0 bytes) 000239[1] 1786480768.990605 write 0x5e1af1 thru 0x5e22e6 (0x7f6 bytes) 000242[1] 1786480769.037897 mapread 0x4bb7e3 thru 0x4bcbf4 (0x1412 bytes) 000244[1] 1786480769.065384 read 0x1a6279 thru 0x1a6c6b (0x9f3 bytes) 000246[0] 1786480769.092646 read 0x01d0ee thru 0x01d973 (0x886 bytes) 000249[0] 1786480769.148599 trunc from 0x5e22e7 to 0x8654fe (0x283218 bytes) 000250[1] 1786480769.266909 mapread 0x7a69e4 thru 0x7a7542 (0xb5f bytes) 000258[1] 1786480769.347445 read 0x086a5f thru 0x0876df (0xc81 bytes) 000261[0] 1786480769.373808 read 0x3d1da8 thru 0x3d293e (0xb97 bytes) 000262[0] 1786480769.378798 mapwrite 0x14e9d4 thru 0x1506f8 (0x1d25 bytes) 000264[1] 1786480769.510519 write 0x6be6e5 thru 0x6c0312 (0x1c2e bytes) 000265[1] 1786480769.538614 trunc from 0x8654ff to 0x95e3ac (0xf8eae bytes) 000266[0] 1786480769.593180 read 0x0a9c23 thru 0x0ab867 (0x1c45 bytes) 000268[1] 1786480769.616615 mapread 0x6a8826 thru 0x6a96dd (0xeb8 bytes) 000269[0] 1786480769.624553 read 0x4da4b9 thru 0x4db46d (0xfb5 bytes) 000271[1] 1786480769.644872 mapread 0x216cb5 thru 0x21839a (0x16e6 bytes) 000272[0] 1786480769.661400 mapwrite 0x61a892 thru 0x61a8b4 (0x23 bytes) 000275[0] 1786480769.823847 read 0x151c47 thru 0x152de8 (0x11a2 bytes) 000277[0] 1786480769.850843 read 0x2d7b6d thru 0x2d95a7 (0x1a3b bytes) 000278[1] 1786480769.877718 read 0x76aa0a thru 0x76bc76 (0x126d bytes) 000279[1] 1786480769.906593 read 0x1a4e5d thru 0x1a5916 (0xaba bytes) 000284[0] 1786480769.921761 trunc from 0x95e3ad to 0x6d37ad (0x28abff bytes) 000286[1] 1786480769.997852 write 0x4d3a1b thru 0x4d59ed (0x1fd3 bytes) 000288[1] 1786480770.051571 mapread 0x4a6f91 thru 0x4a799f (0xa0f bytes) 000290[0] 1786480770.102796 mapwrite 0x2064e8 thru 0x2081fc (0x1d15 bytes) 000291[0] 1786480770.225366 trunc from 0x6d37ae to 0x738ee0 (0x65733 bytes) 000293[1] 1786480770.264743 write 0x275d2c thru 0x275e7d (0x152 bytes) 000294[0] 1786480770.282229 read 0x4a996d thru 0x4aa1ef (0x883 bytes) 000296[1] 1786480770.316233 mapread 0x4a55e1 thru 0x4a67b0 (0x11d0 bytes) 000298[1] 1786480770.336487 write 0x594c7b thru 0x5961c5 (0x154b bytes) 000299[0] 1786480770.349478 trunc from 0x738ee1 to 0x2e4838 (0x4546a8 bytes) 000301[0] 1786480770.387563 mapwrite 0x02ae14 thru 0x02bf01 (0x10ee bytes) 000302[0] 1786480770.450704 mapwrite 0x5efc9a thru 0x5f04e6 (0x84d bytes) 000303[0] 1786480770.521597 mapread 0x2f4eb2 thru 0x2f6cfe (0x1e4d bytes) 000305[1] 1786480770.542676 read 0x3b26ba thru 0x3b3c77 (0x15be bytes) 000306[1] 1786480770.645842 read 0x1601e1 thru 0x16092f (0x74f bytes) 000308[0] 1786480770.675143 write 0x5375b6 thru 0x538427 (0xe72 bytes) 000309[1] 1786480770.706503 write 0x1531eb thru 0x1545ba (0x13d0 bytes) 000312[0] 1786480770.811235 read 0x1de33d thru 0x1de9b2 (0x676 bytes) 000316[0] 1786480770.858791 write 0x7a3386 thru 0x7a3780 (0x3fb bytes) 000317[0] 1786480770.862257 read 0x5efe34 thru 0x5f170c (0x18d9 bytes) 000319[0] 1786480770.892804 mapwrite 0x02c534 thru 0x02d81a (0x12e7 bytes) 000320[1] 1786480770.999449 read 0x29ce4e thru 0x29e194 (0x1347 bytes) 000321[0] 1786480771.100616 mapread 0x20cff3 thru 0x20de24 (0xe32 bytes) 000322[1] 1786480771.119867 trunc from 0x7a3781 to 0x824e53 (0x816d3 bytes) 000324[0] 1786480771.172506 mapwrite 0x0f1bed thru 0x0f2684 (0xa98 bytes) 000325[0] 1786480771.247924 read 0x156a87 thru 0x156e22 (0x39c bytes) 000327[1] 1786480771.289507 write 0x6e6f62 thru 0x6e8e62 (0x1f01 bytes) 000329[1] 1786480771.324784 mapread 0x51e04c thru 0x51ea8c (0xa41 bytes) 000331[0] 1786480771.365184 trunc from 0x824e54 to 0x705a2c (0x11f427 bytes) 000332[0] 1786480771.436390 trunc from 0x705a2d to 0x1eca1b (0x519011 bytes) 000334[1] 1786480771.453069 trunc from 0x1eca1c to 0x78b8ca (0x59eeaf bytes) 000335[0] 1786480771.503206 read 0x6f2b99 thru 0x6f4ab8 (0x1f20 bytes) 000337[1] 1786480771.525710 mapread 0x24f353 thru 0x24f4a1 (0x14f bytes) 000338[1] 1786480771.533252 write 0x7efc27 thru 0x7f00f1 (0x4cb bytes) 000341[1] 1786480771.590796 trunc from 0x7f00f2 to 0x06d7fc (0x7828f5 bytes) 000342[1] 1786480771.639885 mapread 0x016ab5 thru 0x0178ee (0xe3a bytes) 000343[0] 1786480771.653336 write 0x76db44 thru 0x76fa85 (0x1f42 bytes) 000345[1] 1786480771.714455 trunc from 0x76fa86 to 0x6dc0f2 (0x93993 bytes) 000346[0] 1786480771.792390 read 0x47af2d thru 0x47b054 (0x128 bytes) 000347[0] 1786480771.825341 trunc from 0x6dc0f3 to 0x04f775 (0x68c97d bytes) 000348[0] 1786480771.860938 write 0x7dca0b thru 0x7dd4b0 (0xaa6 bytes) 000349[1] 1786480771.894283 mapread 0x1364c0 thru 0x137dc3 (0x1904 bytes) 000350[0] 1786480771.964619 read 0x79be73 thru 0x79cf80 (0x110e bytes) 000351[0] 1786480771.980239 mapwrite 0x642694 thru 0x643d19 (0x1686 bytes) 000352[1] 1786480772.128265 mapread 0x632975 thru 0x632d9d (0x429 bytes) 000353[1] 1786480772.158845 mapread 0x1d4c71 thru 0x1d5613 (0x9a3 bytes) 000354[1] 1786480772.180622 trunc from 0x7dd4b1 to 0x397aa3 (0x445a0d bytes) 000355[1] 1786480772.280412 mapwrite 0x3cbae7 thru 0x3cc988 (0xea2 bytes) 000356[0] 1786480772.452206 mapread 0x04ef76 thru 0x04f45c (0x4e7 bytes) 000358[0] 1786480772.547476 mapwrite 0x33f913 thru 0x33fe4c (0x53a bytes) 000361[0] 1786480772.756420 read 0x0ae88b thru 0x0afa49 (0x11bf bytes) 000362[0] 1786480772.851415 write 0x65d474 thru 0x65ebc1 (0x174e bytes) 000364[1] 1786480772.879464 mapwrite 0x8ee1e3 thru 0x8ef6cb (0x14e9 bytes) 000366[0] 1786480773.135773 mapwrite 0x8a0c05 thru 0x8a1d28 (0x1124 bytes) 000367[0] 1786480773.238474 write 0x73cb52 thru 0x73e9f1 (0x1ea0 bytes) 000368[1] 1786480773.341906 mapread 0x2e9e57 thru 0x2ebc7c (0x1e26 bytes) 000371[1] 1786480773.422812 write 0x3c5103 thru 0x3c6d7a (0x1c78 bytes) 000372[1] 1786480773.451797 mapwrite 0x971b5e thru 0x97232d (0x7d0 bytes) 000373[1] 1786480773.527399 mapread 0x970c02 thru 0x971073 (0x472 bytes) 000375[0] 1786480773.564858 mapread 0x774f78 thru 0x7751d2 (0x25b bytes) 000377[1] 1786480773.646552 mapread 0x8a1c81 thru 0x8a2db4 (0x1134 bytes) 000380[0] 1786480773.700781 read 0x520a6c thru 0x5211cb (0x760 bytes) 000386[0] 1786480773.804175 mapwrite 0x3c5057 thru 0x3c5059 (0x3 bytes) 000387[0] 1786480773.905558 mapread 0x0b2917 thru 0x0b4677 (0x1d61 bytes) 000389[0] 1786480773.956270 trunc from 0x97232e to 0x05dc38 (0x9146f5 bytes) 000391[1] 1786480774.050294 read 0x004ef7 thru 0x0064a2 (0x15ac bytes) 000392[0] 1786480774.136834 read 0x0098c1 thru 0x00aac2 (0x1202 bytes) 000394[1] 1786480774.234840 mapwrite 0x4f77a5 thru 0x4f872f (0xf8b bytes) 000395[1] 1786480774.559580 mapwrite 0x7aeae3 thru 0x7aff36 (0x1454 bytes) 000396[1] 1786480774.716828 mapwrite 0x217ed4 thru 0x217f7a (0xa7 bytes) 000398[0] 1786480774.864760 read 0x7394fa thru 0x73a8be (0x13c5 bytes) 000399[0] 1786480774.897282 trunc from 0x7aff37 to 0x995a30 (0x1e5afa bytes) 000403[1] 1786480775.042246 trunc from 0x995a31 to 0x8e6694 (0xaf39c bytes) 000408[0] 1786480775.213535 read 0x74234e thru 0x742d25 (0x9d8 bytes) 000411[0] 1786480775.287225 read 0x20f85a thru 0x2106c0 (0xe67 bytes) 000415[0] 1786480775.326815 mapwrite 0x2d7550 thru 0x2d874b (0x11fc bytes) 000417[1] 1786480775.483424 trunc from 0x8e6695 to 0x6eb3f0 (0x1fb2a4 bytes) 000418[0] 1786480775.578467 write 0x3f1ed2 thru 0x3f36ab (0x17da bytes) 000419[1] 1786480775.619529 read 0x40a26e thru 0x40b518 (0x12ab bytes) 000422[1] 1786480775.667964 trunc from 0x6eb3f1 to 0x48c8f4 (0x25eafc bytes) 000423[1] 1786480775.736614 write 0x3bcf1e thru 0x3bea65 (0x1b48 bytes) 000429[1] 1786480775.932397 write 0x9d74c2 thru 0x9d843a (0xf79 bytes) 000430[0] 1786480775.976460 mapwrite 0x8510be thru 0x85110e (0x51 bytes) 000431[0] 1786480776.109506 write 0x979a26 thru 0x97a297 (0x872 bytes) 000434[0] 1786480776.248496 read 0x705a1b thru 0x70781b (0x1e01 bytes) 000435[1] 1786480776.322704 trunc from 0x9d843b to 0x3a8d8e (0x62f6ac bytes) 000439[0] 1786480776.508480 trunc from 0x3a8d8f to 0x04f681 (0x35970d bytes) 000440[0] 1786480776.586564 read 0x0298a9 thru 0x02b6f4 (0x1e4c bytes) 000443[1] 1786480776.632331 trunc from 0x04f682 to 0x73eaec (0x6ef46b bytes) 000445[1] 1786480776.690199 mapread 0x357af3 thru 0x3586f1 (0xbff bytes) 000446[1] 1786480776.703424 read 0x660c79 thru 0x66258a (0x1912 bytes) 000447[1] 1786480776.712277 mapwrite 0x40b253 thru 0x40cf7a (0x1d28 bytes) 000448[0] 1786480776.772344 trunc from 0x73eaed to 0x1c01ec (0x57e900 bytes) 000449[0] 1786480776.823534 mapwrite 0x667628 thru 0x6693cf (0x1da8 bytes) 000450[1] 1786480777.035949 read 0x3b3bcf thru 0x3b44d5 (0x907 bytes) 000451[1] 1786480777.070571 write 0x1be043 thru 0x1bfce0 (0x1c9e bytes) 000452[0] 1786480777.130240 read 0x001a70 thru 0x003942 (0x1ed3 bytes) 000455[0] 1786480777.196905 read 0x207634 thru 0x20830e (0xcdb bytes) 000456[1] 1786480777.205863 read 0x186119 thru 0x186bf5 (0xadd bytes) 000458[1] 1786480777.214175 trunc from 0x6693d0 to 0x6f8565 (0x8f196 bytes) 000460[0] 1786480777.328209 write 0x1ce22d thru 0x1cf6a4 (0x1478 bytes) 000461[0] 1786480777.391848 trunc from 0x6f8566 to 0x8d6d9c (0x1de837 bytes) 000465[0] 1786480777.473728 write 0x8d928e thru 0x8dadd5 (0x1b48 bytes) 000466[0] 1786480777.507069 read 0x067788 thru 0x068ca7 (0x1520 bytes) 000467[1] 1786480777.518571 read 0x3eb78e thru 0x3ec866 (0x10d9 bytes) 000468[1] 1786480777.600630 mapread 0x729a2e thru 0x72b438 (0x1a0b bytes) 000469[0] 1786480777.717629 read 0x1fbace thru 0x1fd782 (0x1cb5 bytes) 000471[1] 1786480777.754699 mapwrite 0x641c13 thru 0x643448 (0x1836 bytes) 000475[0] 1786480777.966766 read 0x7bb656 thru 0x7bc3e7 (0xd92 bytes) 000477[0] 1786480777.996370 write 0x8da297 thru 0x8dc190 (0x1efa bytes) 000478[1] 1786480778.043470 write 0x0cbe6a thru 0x0cd0dc (0x1273 bytes) 000479[1] 1786480778.072649 read 0x631e97 thru 0x6333e7 (0x1551 bytes) 000481[1] 1786480778.092465 read 0x487266 thru 0x488b2d (0x18c8 bytes) 000482[0] 1786480778.100675 mapwrite 0x1f076c thru 0x1f17b8 (0x104d bytes) 000483[1] 1786480778.186594 mapread 0x74a3cf thru 0x74a93e (0x570 bytes) 000487[1] 1786480778.328849 mapwrite 0x339b89 thru 0x33b374 (0x17ec bytes) 000488[1] 1786480778.467133 write 0x9510bf thru 0x95302a (0x1f6c bytes) 000490[0] 1786480778.499813 mapwrite 0x3e19f2 thru 0x3e3715 (0x1d24 bytes) 000493[1] 1786480778.668840 trunc from 0x95302b to 0x74ff74 (0x2030b6 bytes) 000494[1] 1786480778.723344 write 0x91f68a thru 0x920951 (0x12c8 bytes) 000495[1] 1786480778.753867 mapwrite 0x04a5a6 thru 0x04b1d1 (0xc2c bytes) 000496[0] 1786480778.841767 read 0x2d6f43 thru 0x2d8631 (0x16ef bytes) 000499[1] 1786480778.910175 read 0x8ae971 thru 0x8b03f9 (0x1a89 bytes) 000500[1] 1786480778.930536 write 0x0254b4 thru 0x026e60 (0x19ad bytes) All operations completed A-OK! PASS 16b (26s) == sanityn test 16c: verify data consistency on ldiskfs with cache disabled (b=17397) ========================================================== 16:39:44 (1786480784) SKIP: sanityn test_16c dio on ldiskfs only SKIP 16c (3s) == sanityn test 16d: Verify DIO and buffer IO with two clients ========================================================== 16:39:47 (1786480787) 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.33056 s, 25.4 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.322017 s, 26.1 MB/s ldlm.namespaces.MGC192.168.201.109@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff9882c4faf800.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff9882c5c8a000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff9882c4faf800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff9882c5c8a000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff9882c4faf800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff9882c5c8a000.lru_size=clear 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.3442 s, 12.6 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 2.55605 s, 41.0 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 2.93891 s, 35.7 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.76723 s, 22.0 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.93306 s, 15.1 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 3.28535 s, 31.9 MB/s PASS 16d (40s) == sanityn test 16e: Verify size consistency for O_DIRECT write ========================================================== 16:40:27 (1786480827) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.206201 s, 50.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0784451 s, 13.4 MB/s PASS 16e (7s) == sanityn test 16f: rw sequential consistency vs drop_caches ========================================================== 16:40:34 (1786480834) TIMEOUT PASS 16f (26s) == sanityn test 16g: mmap rw sequential consistency vs drop_caches ========================================================== 16:41:00 (1786480860) TIMEOUT PASS 16g (26s) == sanityn test 16h: mmap read after truncate file ======= 16:41:26 (1786480886) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16h (5s) == sanityn test 16i: read after truncate file ============ 16:41:31 (1786480891) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16i (6s) == sanityn test 16j: race dio with buffered i/o ========== 16:41:37 (1786480897) 1+0 records in 1+0 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.64463 s, 15.9 MB/s bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.03527 s, 25.3 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.93436 s, 13.6 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.19407 s, 22.0 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.53081 s, 17.1 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.29682 s, 20.2 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.89522 s, 13.8 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.44163 s, 18.2 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.7779 s, 14.7 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.32205 s, 11.3 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.49603 s, 10.5 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.91894 s, 13.7 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.97691 s, 13.3 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.34449 s, 11.2 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.33202 s, 11.2 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.08175 s, 12.6 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.21969 s, 11.8 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.38608 s, 11.0 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.39949 s, 10.9 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK PASS 16j (27s) == sanityn test 16k: Parallel FSX and drop caches should not panic ========================================================== 16:42:04 (1786480924) + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.1 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.2 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.3 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.4 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.5 -l 67108864 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: fsx: Operation not supported test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported skipping zero size read skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x1a7062c truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! + fsx 1 done: rc=0 Tue Aug 11 16:42:38 EDT 2026 + fsx 2 done: rc=0 Tue Aug 11 16:42:38 EDT 2026 All operations completed A-OK! + fsx 3 done: rc=0 Tue Aug 11 16:42:38 EDT 2026 + fsx 4 done: rc=0 Tue Aug 11 16:42:38 EDT 2026 All operations completed A-OK! + fsx 5 done: rc=0 Tue Aug 11 16:42:38 EDT 2026 PASS 16k (43s) == sanityn test 17: resource creation/LVB creation race ========================================================================= 16:42:47 (1786480967) fail_loc=0x8000030a PASS 17 (10s) == sanityn test 18: mmap sanity check =========================================================================================== 16:42:57 (1786480977) mmap test1: basic mmap operation (PASS, 0.10103s) mmap test2: MAP_PRIVATE not write back (PASS, 0.089031s) mmap test3: concurrent mmap ops on two nodes (PASS, 7.8931s) mmap test4: c1 write to f1 from mmapped f2, c2 write to f1 from mmapped f1 (PASS, 4.5977s) mmap test5: read/write file to/from the buffer which mmapped to just this file (PASS, 0.15541s) mmap test6: check mmap write/read content on two nodes (PASS, 0.14585s) mmap test7: file i/o with an unmapped buffer (SKIPPED, 0s) mmap test8: SIGBUS for beyond file size (SKIPPED, 0s) mmap test9: SIGBUS for negative file offset (SKIPPED, 0s) mmap test10: multi-thread mmap access (PASS, 1.3449s) mmap test11: multi-thread shared mmap (PASS, 1.1401s) mmap test10: mtime not change for readonly mmap access (PASS, 1.1411s) PASS 18 (24s) == sanityn test 19: test concurrent uncached read races ========================================================================= 16:43:21 (1786481001) oleg109-server: error: get_param: param_path 'osd-*/lustre-OST*/read_cache_enable': No such file or directory pdsh@oleg109-client: oleg109-server: ssh exited with exit code 2 SKIP: sanityn test_19 not cache-capable obdfilter SKIP 19 (5s) == sanityn test 20: test extra readahead page left in cache ============================================================== 16:43:26 (1786481006) PASS 20 (6s) == sanityn test 21: Try to remove mountpoint on another dir ============================================================== 16:43:32 (1786481012) rmdir: removing directory, '/mnt/lustre/d21.sanityn' rmdir: failed to remove '/mnt/lustre/d21.sanityn': Device or resource busy rmdir: removing directory, '/mnt/lustre2/d21.sanityn' Removed mounted directory from another mountpoint, needs to be fixed PASS 21 (5s) == sanityn test 23: others should see updated atime while another read============================================================== 16:43:37 (1786481017) now is 1786481020 starting reads multiop /mnt/lustre/f23.sanityn vor20_c TMPPIPE=/tmp/multiop_open_wait_pipe.7632 new atime is 1786481081 PASS 23 (68s) == sanityn test 24a: lfs df [-ih] [path] test =================================================================================== 16:44:45 (1786481085) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 31744 3737600 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 31744 3737600 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre2 UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 115.2K 401 114.8K 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 114.5K 402 114.1K 1% /mnt/lustre[OST:1] filesystem_summary: 229.3K 386 229.0K 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.2K 401 114.8K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 402 114.1K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.3K 386 229.0K 1% /mnt/lustre2 UUID bytes Used Available Use% Mounted on lustre-MDT0000_UUID 2.1G 2.1M 2.1G 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3.6G 8.0M 3.6G 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3.6G 31.0M 3.6G 1% /mnt/lustre[OST:1] filesystem_summary: 7.2G 39.0M 7.2G 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118001 401 117600 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117266 402 116864 1% /mnt/lustre2[OST:1] filesystem_summary: 234850 386 234464 1% /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 31744 3737600 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.2K 401 114.8K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 402 114.1K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.3K 386 229.0K 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 118001 401 117600 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 117266 402 116864 1% /mnt/lustre[OST:1] filesystem_summary: 234850 386 234464 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118001 401 117600 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117266 402 116864 1% /mnt/lustre2[OST:1] filesystem_summary: 234850 386 234464 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 31744 3737600 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 31744 3737600 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre2 PASS 24a (6s) == sanityn test 24b: lfs df should show both filesystems ========================================================================= 16:44:51 (1786481091) PASS 24b (6s) == sanityn test 25a: change ACL on one mountpoint be seen on another ============================================================= 16:44:57 (1786481097) running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] PASS 25a (5s) == sanityn test 25b: change ACL under remote dir on one mountpoint be seen on another ========================================================== 16:45:02 (1786481102) SKIP: sanityn test_25b needs >= 2 MDTs SKIP 25b (3s) == sanityn test 26a: allow mtime to get older ============ 16:45:05 (1786481105) utime: good mknod times 1786481105 <= 1786481106 <= 1786481106 for /mnt/lustre/f26a utime: good mknod times 1786481105 <= 1786481106 <= 1786481106 for /mnt/lustre2/f26a utime: good utime mtimes 100000, atime 200000 utime: updated times for /mnt/lustre2/f26a PASS 26a (6s) == sanityn test 26b: sync mtime between ost and mds ====== 16:45:11 (1786481111) PASS 26b (7s) == sanityn test 26c: set-in-past on open file is not lost on close ========================================================== 16:45:18 (1786481118) multiop /mnt/lustre/f26c.sanityn vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7632 File: /mnt/lustre/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502398 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-08-11 16:45:19.000000000 -0400 Birth: 2026-08-11 16:45:20.000000000 -0400 File: /mnt/lustre2/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502398 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-08-11 16:45:19.000000000 -0400 Birth: 2026-08-11 16:45:20.000000000 -0400 PASS 26c (6s) SKIP: sanityn test_27 skipping excluded test 27 == sanityn test 30: recreate file race =================== 16:45:25 (1786481125) 0 PASS 30 (6s) == sanityn test 31a: voluntary cancel / blocking ast race======================================================================== 16:45:31 (1786481131) fail_loc=0x314 PASS 31a (6s) == sanityn test 31b: voluntary OST cancel / blocking ast race======================================================================== 16:45:37 (1786481137) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg109-server mds-ost sync done. sleep 5 for ZFS MDS Waiting for MDT destroys to complete fail_loc=0x314 fail_loc=0x316 fail_loc=0 fail_loc=0 Filesystem 1K-blocks Used Available Use% Mounted on 192.168.201.109@tcp:/lustre 7542784 41984 7494656 1% /mnt/lustre2 PASS 31b (19s) == sanityn test 31r: open-rename(replace) race =========== 16:45:56 (1786481156) -rw-r--r-- 1 root root 0 Aug 11 16:45 /mnt/lustre/f31r.sanityn.target fail_loc=0x1419 fail_val=3 PASS 31r (7s) == sanityn test 31s: open should not revalidate invalid dentry ========================================================== 16:46:03 (1786481163) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 1205: 43107 Terminated tail -f $tfile PASS 31s (5s) == sanityn test 31t: getattr should not revalidate invalid dentry ========================================================== 16:46:08 (1786481168) PASS 31t (6s) == sanityn test 32b: lockless i/o ======================== 16:46:14 (1786481174) SKIP: sanityn test_32b max_nolock_bytes is removed >= 2.17.53 SKIP 32b (3s) == sanityn test 32c: contention detection ================ 16:46:17 (1786481177) ldlm.namespaces.filter-lustre-OST0000_UUID.contended_locks=2 ldlm.namespaces.filter-lustre-OST0001_UUID.contended_locks=2 ldlm.namespaces.filter-lustre-OST0000_UUID.contention_events=0 ldlm.namespaces.filter-lustre-OST0001_UUID.contention_events=0 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.10024 s, 47.7 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.05737 s, 49.6 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.58623 s, 33.1 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.84374 s, 28.4 MB/s contention_seconds 4, contention_events 18 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.51139 s, 34.7 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.7151 s, 30.6 MB/s ldlm.namespaces.filter-lustre-OST0000_UUID.contention_events=0 ldlm.namespaces.filter-lustre-OST0001_UUID.contention_events=0 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.789132 s, 66.4 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.812566 s, 64.5 MB/s PASS 32c (24s) SKIP: sanityn test_33a skipping SLOW test 33a == sanityn test 33b: COS: cross create/delete, 2 clients, benchmark under remote dir ========================================================== 16:46:43 (1786481203) SKIP: sanityn test_33b Need two or more clients, have 1 SKIP 33b (7s) == sanityn test 33c: Cancel cross-MDT lock should trigger Sync-on-Lock-Cancel ========================================================== 16:46:51 (1786481211) SKIP: sanityn test_33c needs >= 2 MDTs SKIP 33c (3s) == sanityn test 33d: dependent transactions should trigger COS ========================================================== 16:46:54 (1786481214) SKIP: sanityn test_33d needs >= 2 MDTs SKIP 33d (4s) == sanityn test 33e: independent transactions shouldn't trigger COS ========================================================== 16:46:58 (1786481218) SKIP: sanityn test_33e needs >= 2 MDTs SKIP 33e (4s) == sanityn test 34: no lock timeout under IO ============= 16:47:02 (1786481222) fail_loc=0x512 fail_loc=0x512 lock should not expire writing on client1 reading on client2 success fail_loc=0x511 fail_loc=0x511 lock should expire writing on client1 reading on client2 success oleg109-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9882c4faf800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9882c4faf800.ost_server_uuid in IDLE state after 0 sec oleg109-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9882c4faf800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9882c4faf800.ost_server_uuid in IDLE state after 0 sec oleg109-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9882c4faf800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9882c4faf800.ost_server_uuid in IDLE state after 0 sec oleg109-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9882c4faf800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9882c4faf800.ost_server_uuid in IDLE state after 0 sec oleg109-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9882c4faf800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9882c4faf800.ost_server_uuid in IDLE state after 0 sec oleg109-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9882c4faf800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9882c4faf800.ost_server_uuid in IDLE state after 0 sec PASS 34 (135s) == sanityn test 35: -EINTR cp_ast vs. bl_ast race does not evict client ========================================================== 16:49:17 (1786481357) fail_loc=0x80000317 Race attempt 0 Wait for 50411 50460 for 60 sec... - open/close 753 (time 1786481374.10 total 10.01 last 75.26) - open/close 1527 (time 1786481384.10 total 20.01 last 77.39) - open/close 2406 (time 1786481394.12 total 30.03 last 87.69) - open/close 3305 (time 1786481404.13 total 40.03 last 89.88) total: 4000 open/close in 48.09 seconds: 83.18 ops/second fail_loc=0x0 Filesystem Size Used Avail Use% Mounted on 192.168.201.109@tcp:/lustre 7.2G 41M 7.2G 1% /mnt/lustre 192.168.201.109@tcp:/lustre 7.2G 41M 7.2G 1% /mnt/lustre2 PASS 35 (73s) == sanityn test 36: handle ESTALE/open-unlink correctly == 16:50:30 (1786481430) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg109-server mds-ost sync done. Waiting for MDT destroys to complete 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.56516 s, 33.5 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7632 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(0) *** before(7496704) after_dd(7441408) after(7492608) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.26355 s, 41.5 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7632 sleep 5 for ZFS MDS Waiting for MDT destroys to complete *** cycle(1) *** before(7492608) after_dd(7438336) after(7489536) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.36791 s, 38.3 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7632 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(2) *** before(7489536) after_dd(7439360) after(7490560) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.989443 s, 53.0 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7632 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(3) *** before(7490560) after_dd(7445504) after(7496704) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.17595 s, 44.6 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7632 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(4) *** before(7496704) after_dd(7441408) after(7496704) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.28435 s, 40.8 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7632 sleep 5 for ZFS MDS Waiting for MDT destroys to complete *** cycle(5) *** before(7496704) after_dd(7443456) after(7494656) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.35151 s, 38.8 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7632 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(6) *** before(7494656) after_dd(7443456) after(7494656) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.89788 s, 27.6 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7632 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(7) *** before(7494656) after_dd(7441408) after(7496704) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.61038 s, 32.6 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7632 sleep 5 for ZFS MDS Waiting for MDT destroys to complete *** cycle(8) *** before(7496704) after_dd(7441408) after(7492608) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.34226 s, 39.1 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7632 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(9) *** before(7492608) after_dd(7438336) after(7489536) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.09513 s, 47.9 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7632 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(10) *** before(7489536) after_dd(7443456) after(7494656) PASS 36 (292s) == sanityn test 37: check i_size is not updated for directory on close (bug 18695) ======================================================================== 16:55:22 (1786481722) multiop /mnt/lustre/d37.sanityn vD_c TMPPIPE=/tmp/multiop_open_wait_pipe.7632 - create 1725 (time 1786481736.67 total 10.01 last 172.38) - create 3161 (time 1786481746.67 total 20.01 last 143.58) - create 4659 (time 1786481756.68 total 30.01 last 149.70) - create 5962 (time 1786481766.68 total 40.02 last 130.27) - create 7392 (time 1786481776.68 total 50.02 last 142.94) - create 8936 (time 1786481786.69 total 60.03 last 154.24) total: 10000 create in 68.99 seconds: 144.95 ops/second PASS 37 (80s) == sanityn test 39a: file mtime does not change after rename ========================================================== 16:56:42 (1786481802) repeat after cancel_lru_locks PASS 39a (7s) == sanityn test 39b: file mtime the same on clients with/out lock ========================================================== 16:56:50 (1786481810) repeat after cancel_lru_locks PASS 39b (8s) == sanityn test 39c: check truncate mtime update ================================================================================ 16:56:57 (1786481817) repeat after cancel_lru_locks PASS 39c (8s) == sanityn test 39d: sync write should update mtime ====== 16:57:05 (1786481825) fail_loc=0x411 fail_loc=0 PASS 39d (6s) SKIP: sanityn test_40a skipping ALWAYS excluded test 40a == sanityn test 40b: pdirops: open|create and others ======================================================================== 16:57:13 (1786481833) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40b (16s) == sanityn test 40c: pdirops: link and others ======================================================================== 16:57:30 (1786481850) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40c (16s) == sanityn test 40d: pdirops: unlink and others ======================================================================== 16:57:46 (1786481866) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40d (14s) == sanityn test 40e: pdirops: rename and others ======================================================================== 16:58:00 (1786481880) No conflict No conflict No conflict No conflict No conflict No conflict PASS 40e (14s) == sanityn test 41a: pdirops: create vs mkdir ======================================================================== 16:58:14 (1786481894) mkdir: cannot create directory '/mnt/lustre2/f41a.sanityn': File exists Conflict PASS 41a (10s) == sanityn test 41b: pdirops: create vs create ======================================================================== 16:58:24 (1786481904) open: File exists Conflict PASS 41b (10s) == sanityn test 41c: pdirops: create vs link ======================================================================== 16:58:34 (1786481914) link: cannot create link '/mnt/lustre2/f41c.sanityn' to '/mnt/lustre2/f41c.sanityn-2': File exists Conflict PASS 41c (10s) == sanityn test 41d: pdirops: create vs unlink ======================================================================== 16:58:44 (1786481924) Conflict PASS 41d (9s) == sanityn test 41e: pdirops: create and rename (tgt) ======================================================================== 16:58:53 (1786481933) Conflict PASS 41e (9s) == sanityn test 41f: pdirops: create and rename (src) ======================================================================== 16:59:02 (1786481942) Conflict PASS 41f (10s) == sanityn test 41g: pdirops: create vs getattr ======================================================================== 16:59:13 (1786481953) Conflict PASS 41g (10s) == sanityn test 41h: pdirops: create vs readdir ======================================================================== 16:59:23 (1786481963) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 2427: lctl set_param -n fail_loc=0 2>/dev/null || true: No such file or directory Conflict PASS 41h (10s) == sanityn test 41i: reint_open: create vs create ======== 16:59:33 (1786481973) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 41i (880s) == sanityn test 42a: pdirops: mkdir vs mkdir ======================================================================== 17:14:13 (1786482853) mkdir: cannot create directory '/mnt/lustre2/f42a.sanityn': File exists Conflict PASS 42a (6s) == sanityn test 42b: pdirops: mkdir vs create ======================================================================== 17:14:19 (1786482859) open: File exists Conflict PASS 42b (6s) == sanityn test 42c: pdirops: mkdir vs link ======================================================================== 17:14:25 (1786482865) link: cannot create link '/mnt/lustre2/f42c.sanityn' to '/mnt/lustre2/f42c.sanityn-2': File exists Conflict PASS 42c (5s) == sanityn test 42d: pdirops: mkdir vs unlink ======================================================================== 17:14:30 (1786482870) Conflict PASS 42d (6s) == sanityn test 42e: pdirops: mkdir and rename (tgt) ======================================================================== 17:14:36 (1786482876) mv: cannot overwrite directory '/mnt/lustre2/f42e.sanityn' with non-directory Conflict PASS 42e (6s) == sanityn test 42f: pdirops: mkdir and rename (src) ======================================================================== 17:14:42 (1786482882) Conflict PASS 42f (6s) == sanityn test 42g: pdirops: mkdir vs getattr ======================================================================== 17:14:48 (1786482888) Conflict PASS 42g (5s) == sanityn test 42h: pdirops: mkdir vs readdir ======================================================================== 17:14:53 (1786482893) Conflict PASS 42h (6s) == sanityn test 43a: rmdir,mkdir doesn't return -EEXIST ======================================================================== 17:14:59 (1786482899) PASS 43a (25s) == sanityn test 43b: pdirops: unlink vs create ======================================================================== 17:15:24 (1786482924) Conflict PASS 43b (5s) == sanityn test 43c: pdirops: unlink vs link ======================================================================== 17:15:29 (1786482929) Conflict PASS 43c (5s) == sanityn test 43d: pdirops: unlink vs unlink ======================================================================== 17:15:34 (1786482934) rm: cannot remove '/mnt/lustre2/f43d.sanityn': No such file or directory Conflict PASS 43d (6s) == sanityn test 43e: pdirops: unlink and rename (tgt) ======================================================================== 17:15:40 (1786482940) Conflict PASS 43e (6s) == sanityn test 43f: pdirops: unlink and rename (src) ======================================================================== 17:15:46 (1786482946) mv: cannot stat '/mnt/lustre2/f43f.sanityn': No such file or directory Conflict PASS 43f (5s) == sanityn test 43g: pdirops: unlink vs getattr ======================================================================== 17:15:51 (1786482951) stat: cannot statx '/mnt/lustre2/f43g.sanityn': No such file or directory Conflict PASS 43g (6s) == sanityn test 43h: pdirops: unlink vs readdir ======================================================================== 17:15:57 (1786482957) Conflict PASS 43h (5s) == sanityn test 43i: pdirops: unlink vs remote mkdir ===== 17:16:02 (1786482962) SKIP: sanityn test_43i needs >= 2 MDTs SKIP 43i (2s) == sanityn test 43j: racy mkdir return EEXIST ======================================================================== 17:16:04 (1786482964) mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists PASS 43j (44s) == sanityn test 43k: unlink vs create ==================== 17:16:48 (1786483008) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 43k (422s) == sanityn test 44a: pdirops: rename tgt vs mkdir ======================================================================== 17:23:50 (1786483430) mkdir: cannot create directory '/mnt/lustre2/f44a.sanityn': File exists Conflict Tue Aug 11 17:23:54 EDT 2026 PASS 44a (5s) == sanityn test 44b: pdirops: rename tgt vs create ======================================================================== 17:23:55 (1786483435) open: File exists Conflict PASS 44b (5s) == sanityn test 44c: pdirops: rename tgt vs link ======================================================================== 17:24:00 (1786483440) link: cannot create link '/mnt/lustre2/f44c.sanityn' to '/mnt/lustre2/f44c.sanityn-3': File exists Conflict PASS 44c (5s) == sanityn test 44d: pdirops: rename tgt vs unlink ======================================================================== 17:24:05 (1786483445) Conflict PASS 44d (5s) == sanityn test 44e: pdirops: rename tgt and rename (tgt) ======================================================================== 17:24:10 (1786483450) Conflict PASS 44e (5s) == sanityn test 44f: pdirops: rename tgt and rename (src) ======================================================================== 17:24:15 (1786483455) Conflict PASS 44f (5s) == sanityn test 44g: pdirops: rename tgt vs getattr ======================================================================== 17:24:20 (1786483460) Conflict PASS 44g (5s) == sanityn test 44h: pdirops: rename tgt vs readdir ======================================================================== 17:24:25 (1786483465) Conflict PASS 44h (4s) == sanityn test 44i: pdirops: rename tgt vs remote mkdir ========================================================== 17:24:29 (1786483469) SKIP: sanityn test_44i needs >= 2 MDTs SKIP 44i (1s) == sanityn test 45a: rename,mkdir doesn't return -EEXIST ======================================================================== 17:24:30 (1786483470) PASS 45a (25s) == sanityn test 45b: pdirops: rename src vs create ======================================================================== 17:24:55 (1786483495) Conflict PASS 45b (5s) == sanityn test 45c: pdirops: rename src vs link ======================================================================== 17:25:00 (1786483500) Conflict PASS 45c (6s) == sanityn test 45d: pdirops: rename src vs unlink ======================================================================== 17:25:06 (1786483506) rm: cannot remove '/mnt/lustre2/f45d.sanityn': No such file or directory Conflict PASS 45d (5s) == sanityn test 45e: pdirops: rename src and rename (tgt) ======================================================================== 17:25:11 (1786483511) Conflict PASS 45e (4s) == sanityn test 45f: pdirops: rename src and rename (src) ======================================================================== 17:25:15 (1786483515) mv: cannot stat '/mnt/lustre2/f45f.sanityn': No such file or directory Conflict PASS 45f (5s) == sanityn test 45g: pdirops: rename src vs getattr ======================================================================== 17:25:20 (1786483520) stat: cannot statx '/mnt/lustre2/f45g.sanityn': No such file or directory Conflict PASS 45g (5s) == sanityn test 45h: pdirops: unlink vs readdir ======================================================================== 17:25:25 (1786483525) Conflict PASS 45h (4s) == sanityn test 45i: pdirops: rename src vs remote mkdir ========================================================== 17:25:30 (1786483530) SKIP: sanityn test_45i needs >= 2 MDTs SKIP 45i (1s) == sanityn test 45j: read vs rename ====================== 17:25:31 (1786483531) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 45j (424s) == sanityn test 46a: pdirops: link vs mkdir ======================================================================== 17:32:35 (1786483955) mkdir: cannot create directory '/mnt/lustre2/f46a.sanityn': File exists Conflict PASS 46a (6s) == sanityn test 46b: pdirops: link vs create ======================================================================== 17:32:41 (1786483961) open: File exists Conflict PASS 46b (7s) == sanityn test 46c: pdirops: link vs link ======================================================================== 17:32:48 (1786483968) link: cannot create link '/mnt/lustre2/f46c.sanityn' to '/mnt/lustre2/f46c.sanityn': File exists Conflict PASS 46c (8s) == sanityn test 46d: pdirops: link vs unlink ======================================================================== 17:32:56 (1786483976) Conflict PASS 46d (7s) == sanityn test 46e: pdirops: link and rename (tgt) ======================================================================== 17:33:04 (1786483984) Conflict PASS 46e (8s) == sanityn test 46f: pdirops: link and rename (src) ======================================================================== 17:33:11 (1786483991) Conflict PASS 46f (7s) == sanityn test 46g: pdirops: link vs getattr ======================================================================== 17:33:18 (1786483998) Conflict PASS 46g (6s) == sanityn test 46h: pdirops: link vs readdir ======================================================================== 17:33:24 (1786484004) Conflict PASS 46h (5s) == sanityn test 46i: pdirops: link vs remote mkdir ======= 17:33:29 (1786484009) SKIP: sanityn test_46i needs >= 2 MDTs SKIP 46i (1s) == sanityn test 47a: pdirops: remote mkdir vs mkdir ====== 17:33:30 (1786484010) SKIP: sanityn test_47a needs >= 2 MDTs SKIP 47a (1s) == sanityn test 47b: pdirops: remote mkdir vs create ===== 17:33:31 (1786484011) SKIP: sanityn test_47b needs >= 2 MDTs SKIP 47b (1s) == sanityn test 47c: pdirops: remote mkdir vs link ======= 17:33:33 (1786484013) SKIP: sanityn test_47c needs >= 2 MDTs SKIP 47c (2s) == sanityn test 47d: pdirops: remote mkdir vs unlink ===== 17:33:34 (1786484014) SKIP: sanityn test_47d needs >= 2 MDTs SKIP 47d (1s) == sanityn test 47e: pdirops: remote mkdir and rename (tgt) ========================================================== 17:33:35 (1786484015) SKIP: sanityn test_47e needs >= 2 MDTs SKIP 47e (1s) == sanityn test 47f: pdirops: remote mkdir and rename (src) ========================================================== 17:33:36 (1786484016) SKIP: sanityn test_47f needs >= 2 MDTs SKIP 47f (1s) == sanityn test 47g: pdirops: remote mkdir vs getattr ==== 17:33:38 (1786484018) SKIP: sanityn test_47g needs >= 2 MDTs SKIP 47g (1s) == sanityn test 50: osc lvb attrs: enqueue vs. CP AST ======================================================================== 17:33:39 (1786484019) 10+0 records in 10+0 records out 10240 bytes (10 kB, 10 KiB) copied, 0.0160177 s, 639 kB/s fail_loc=0x410 fail_loc=0x0 PASS 50 (13s) == sanityn test 51a: layout lock: refresh layout should work ========================================================== 17:33:52 (1786484032) 0+1 records in 0+1 records out 158 bytes copied, 0.0191185 s, 8.3 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 3690: kill: (218024) - No such process PASS 51a (7s) == sanityn test 51b: layout lock: glimpse should be able to restart if layout changed ========================================================== 17:33:59 (1786484039) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0104725 s, 97.8 kB/s fail_loc=0x1404 fail_val=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0404668 s, 25.9 MB/s 3145728 PASS 51b (24s) == sanityn test 51c: layout lock: IT_LAYOUT blocked and correct layout can be returned ========================================================== 17:34:23 (1786484063) fail_loc=0x172 Setting layout to have 2 stripes ... 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0549704 s, 18.6 kB/s PASS 51c (16s) == sanityn test 51d: layout lock: losing layout lock should clean up memory map region ========================================================== 17:34:40 (1786484080) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0896325 s, 11.7 MB/s Before revoking layout lock: 1024 KB mapped PASS 51d (10s) == sanityn test 51e: lfs getstripe does not break leases, part 2 ========================================================== 17:34:49 (1786484089) /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 2455 0x997 0x240000400 write lease(2) has applied. /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 2455 0x997 0x240000400 read lease(1) has applied. PASS 51e (9s) == sanityn test 54: rename locking ======================= 17:34:59 (1786484099) llite.lustre-ffff9882c4faf800.xattr_cache=1 llite.lustre-ffff9882c5c8a000.xattr_cache=1 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 512 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505015 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-11 17:35:00.000000000 -0400 Modify: 2026-08-11 17:35:00.000000000 -0400 Change: 2026-08-11 17:35:00.000000000 -0400 Birth: 2026-08-11 17:35:01.000000000 -0400 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000154 File: /mnt/lustre/d1/d2 Size: 7680 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505018 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-11 17:35:06.000000000 -0400 Modify: 2026-08-11 17:35:07.000000000 -0400 Change: 2026-08-11 17:35:07.000000000 -0400 Birth: 2026-08-11 17:35:07.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 512 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505021 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-11 17:35:13.000000000 -0400 Modify: 2026-08-11 17:35:13.000000000 -0400 Change: 2026-08-11 17:35:13.000000000 -0400 Birth: 2026-08-11 17:35:14.000000000 -0400 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1 Size: 7680 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505020 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-11 17:35:13.000000000 -0400 Modify: 2026-08-11 17:35:14.000000000 -0400 Change: 2026-08-11 17:35:14.000000000 -0400 Birth: 2026-08-11 17:35:14.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000154 File: /mnt/lustre/d1/d2 Size: 7680 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505024 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-11 17:35:20.000000000 -0400 Modify: 2026-08-11 17:35:21.000000000 -0400 Change: 2026-08-11 17:35:21.000000000 -0400 Birth: 2026-08-11 17:35:20.000000000 -0400 File: /mnt/lustre/d1 Size: 7680 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505023 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-11 17:35:19.000000000 -0400 Modify: 2026-08-11 17:35:21.000000000 -0400 Change: 2026-08-11 17:35:21.000000000 -0400 Birth: 2026-08-11 17:35:20.000000000 -0400 open(O_RDWR|O_CREAT): Is a directory PASS 54 (37s) == sanityn test 55a: rename vs unlink target dir ========= 17:35:36 (1786484136) fail_loc=0x80000156 mv: cannot move '/mnt/lustre/d55a.sanityn/d1/d2' to '/mnt/lustre/d55a.sanityn/d3/d2': No such file or directory PASS 55a (18s) == sanityn test 55b: rename vs unlink source dir ========= 17:35:54 (1786484154) fail_loc=0x80000156 mv: cannot stat '/mnt/lustre/d55b.sanityn/d1/d2': No such file or directory PASS 55b (16s) == sanityn test 55c: rename vs unlink orphan target dir == 17:36:10 (1786484170) fail_loc=0x156 mv: cannot move '/mnt/lustre/d55c.sanityn/d1/d2' to '/mnt/lustre/d55c.sanityn/d3/d2': No such file or directory PASS 55c (19s) == sanityn test 55d: rename file vs link ================= 17:36:29 (1786484189) fail_loc=0x155 mv: '/mnt/lustre/d55d.sanityn/f1' and '/mnt/lustre/d55d.sanityn/d55d.sanityn/f1' are the same file PASS 55d (20s) == sanityn test 55e: rename race AB/BA under the same parent dir ========================================================== 17:36:49 (1786484209) SKIP: sanityn test_55e needs >= 2 MDTs SKIP 55e (3s) == sanityn test 55f: rename: (P1/A -> P2/B) race with (P2/B -> P1/A) ========================================================== 17:36:52 (1786484212) fail_loc=0x155 We survived after P1A_P2B/P2B_P1A race test fail_loc=0x0 PASS 55f (20s) == sanityn test 55g: rename: race with trylock =========== 17:37:12 (1786484232) mdt.lustre-MDT0000.enable_rename_trylock=1 fail_loc=0x155 mv: cannot move '/mnt/lustre/d55g.sanityn/a1/a2' to a subdirectory of itself, '/mnt/lustre/d55g.sanityn/b1/b2/b3/b4/a2' We survived race with trylock test fail_loc=0x0 PASS 55g (23s) == sanityn test 56a: test llverdev with single large stripe ========================================================== 17:37:36 (1786484256) run_llverdev /mnt/lustre/f56a.sanityn -p -s 16000000 -c 4k llverdev: /mnt/lustre/f56a.sanityn is 16777216000000 bytes (15625 GB) in size timestamp: 1786484258 chunksize: 4096 size: 16777216000000 write offset: 16384000000kB 0.549901 MB/s read offset: 16384000000kB inf MB/s llverdev: data verified successfully PASS 56a (119s) == sanityn test 56b: test llverdev and partial verify of wide stripe file ========================================================== 17:39:34 (1786484374) run_llverdev /mnt/lustre/f56b.sanityn -p -s 32000000 -c 4k llverdev: /mnt/lustre/f56b.sanityn is 33554432000000 bytes (31250 GB) in size timestamp: 1786484379 chunksize: 4096 size: 33554432000000 write offset: 32768000000kB 1.40315 MB/s read offset: 32768000000kB inf MB/s llverdev: data verified successfully PASS 56b (100s) == sanityn test 60: Verify data_version behaviour ======== 17:41:14 (1786484474) PASS 60 (7s) == sanityn test 70a: cd directory && rm directory ======== 17:41:22 (1786484482) PASS 70a (7s) == sanityn test 70b: remove files after calling rm_entry ========================================================== 17:41:30 (1786484490) PASS 70b (8s) == sanityn test 71a: correct file map just after write operation is finished ========================================================== 17:41:38 (1786484498) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0045234 s, 14.5 MB/s SKIP: sanityn test_71a ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71a (3s) == sanityn test 71b: check fiemap support for stripecount > 1 ========================================================== 17:41:40 (1786484500) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.00829368 s, 7.9 MB/s SKIP: sanityn test_71b ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71b (4s) == sanityn test 71c: check FIEMAP_EXTENT_LAST flag with different extents number ========================================================== 17:41:44 (1786484504) SKIP: sanityn test_71c support only ldiskfs ost SKIP 71c (2s) == sanityn test 71d: fiemap corruption test with fm_extent_count=0 ========================================================== 17:41:47 (1786484507) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0194856 s, 3.4 MB/s SKIP: sanityn test_71d ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71d (4s) == sanityn test 72: getxattr/setxattr cache should be consistent between nodes ========================================================== 17:41:51 (1786484511) llite.lustre-ffff9882c4faf800.xattr_cache=1 llite.lustre-ffff9882c5c8a000.xattr_cache=1 getfattr: Removing leading '/' from absolute path names user.attr1="value1" getfattr: Removing leading '/' from absolute path names user.attr1="value2" getfattr: Removing leading '/' from absolute path names getfattr: Removing leading '/' from absolute path names b362740ad371eae580e7f85452e8830b - 870e072da1ebfc20ea8c0cd3d6bdf3b6 - PASS 72 (7s) == sanityn test 73: getxattr should not cause xattr lock cancellation ========================================================== 17:41:57 (1786484517) llite.lustre-ffff9882c4faf800.xattr_cache=1 llite.lustre-ffff9882c5c8a000.xattr_cache=1 getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" PASS 73 (6s) == sanityn test 74: flock deadlock: different mounts ======================================================================== 17:42:03 (1786484523) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0118935 s, 86.1 kB/s 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0164221 s, 62.4 kB/s 230848: get lock2 230850: get lock1 230850: done 230850: get lock2 230848: done 230848: get lock1 fcntl cmd 7 failed: Resource deadlock avoided 230848: cannot get lock1: Resource deadlock avoided 230850: done 230850: exit rc=0 230848: done 230848: exit rc=0 PASS 74 (11s) == sanityn test 75: osc: upcall after unuse lock============================================================================= 17:42:14 (1786484534) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0415484 s, 50.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162564 s, 64.5 MB/s fail_loc=0x31d /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 4373: 231431 Killed stat -c %s $DIR1/$tfile fail_loc=0 PASS 75 (12s) == sanityn test 76: Verify MDT open_files listing ======== 17:42:26 (1786484546) opening files: ........ closing files: ........ PASS 76 (53s) == sanityn test 77a: check FIFO NRS policy =============== 17:43:20 (1786484600) ost.OSS.ost.nrs_policies=fifo ost.OSS.ost_create.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_out.nrs_policies=fifo ost.OSS.ost_seq.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.279474 s, 60.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0337652 s, 31.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.042193 s, 24.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0963433 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0882008 s, 11.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9882c4faf800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0929978 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0932691 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0912345 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15699 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181416 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167026 s, 6.3 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144131 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16678 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161698 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153137 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194706 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220347 s, 4.8 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9882c4faf800.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.022761 s, 46.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0452151 s, 23.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0243338 s, 43.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0272332 s, 38.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0561629 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0392693 s, 26.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0289659 s, 36.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0354726 s, 29.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0375087 s, 28.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0383179 s, 27.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.047154 s, 22.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00965062 s, 109 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0640801 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0535963 s, 19.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0364777 s, 28.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00289803 s, 362 MB/s removed '/mnt/lustre/d77a.sanityn/nrs_r_oleg109-client.virtnet' removed '/mnt/lustre/d77a.sanityn/nrs_w_oleg109-client.virtnet' removed directory '/mnt/lustre/d77a.sanityn' PASS 77a (6s) == sanityn test 77b: check CRR-N NRS policy ============== 17:43:25 (1786484605) ost.OSS.ost.nrs_policies=crrn ost.OSS.ost_create.nrs_policies=crrn ost.OSS.ost_io.nrs_policies=crrn ost.OSS.ost_out.nrs_policies=crrn ost.OSS.ost_seq.nrs_policies=crrn ost.OSS.ost.nrs_crrn_quantum=1 ost.OSS.ost_create.nrs_crrn_quantum=1 ost.OSS.ost_io.nrs_crrn_quantum=1 ost.OSS.ost_out.nrs_crrn_quantum=1 ost.OSS.ost_seq.nrs_crrn_quantum=1 policy: crr-n, crrn_quantum 1 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.376326 s, 44.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0255386 s, 41.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0694124 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0794313 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0942366 s, 11.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0993401 s, 10.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118289 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127109 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0931649 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112535 s, 9.3 MB/s 1+0 records in 1+0 records out ldlm.namespaces.lustre-OST0000-osc-ffff9882c4faf800.lock_unused_count=2 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130346 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147837 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148368 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168826 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180421 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179016 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210871 s, 5.0 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9882c4faf800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0236512 s, 44.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.045157 s, 23.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128078 s, 81.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0077247 s, 136 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00308957 s, 339 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128441 s, 8.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11789 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0714533 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112735 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.089946 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00780983 s, 134 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.03899 s, 26.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0252759 s, 41.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0334966 s, 31.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0216436 s, 48.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0403666 s, 26.0 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg109-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg109-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost.nrs_crrn_quantum=64 ost.OSS.ost_create.nrs_crrn_quantum=64 ost.OSS.ost_io.nrs_crrn_quantum=64 ost.OSS.ost_out.nrs_crrn_quantum=64 ost.OSS.ost_seq.nrs_crrn_quantum=64 policy: crr-n, crrn_quantum 64 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.290487 s, 57.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0308158 s, 34.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0177692 s, 59.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0738559 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0286151 s, 36.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0359242 s, 29.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0775779 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101121 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112394 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0594618 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121401 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183632 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194036 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120346 s, 8.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9882c4faf800.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff9882c4faf800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213707 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205533 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195459 s, 5.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.010627 s, 98.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0317806 s, 33.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00982782 s, 107 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141328 s, 74.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0286981 s, 36.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9882c4faf800.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff9882c4faf800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0228751 s, 45.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0331236 s, 31.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0250863 s, 41.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0672597 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0478741 s, 21.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.061854 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0703589 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0471967 s, 22.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0642692 s, 16.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0683674 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0714179 s, 14.7 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg109-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg109-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77b (8s) == sanityn test 77c: check ORR NRS policy ================ 17:43:33 (1786484613) ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_orr_quantum=1 ost.OSS.ost_io.nrs_orr_offset_type=physical ost.OSS.ost_io.nrs_orr_supported=reads policy: orr, orr_quantum 1, orr_offset_type physical, orr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.209306 s, 80.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0322402 s, 32.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0722121 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0489548 s, 21.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0780001 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0585348 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0905731 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0897966 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102075 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116164 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0828753 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142412 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161283 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0943935 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191063 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185003 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158677 s, 6.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9882c4faf800.lock_unused_count=6 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00514835 s, 204 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0252756 s, 41.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0373099 s, 28.1 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0266286 s, 39.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0255774 s, 41.0 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0199494 s, 52.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0270858 s, 38.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0101648 s, 103 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0288873 s, 36.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0588133 s, 17.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00343595 s, 305 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0429064 s, 24.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0321056 s, 32.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0630612 s, 16.6 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0453869 s, 23.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0446241 s, 23.5 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg109-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg109-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=writes ost.OSS.ost_io.nrs_orr_quantum=64 policy: orr, orr_quantum 64, orr_offset_type physical, orr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.226664 s, 74.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0695798 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0656551 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.074471 s, 14.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0820289 s, 12.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0688201 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106418 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0791777 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0990991 s, 10.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0837605 s, 12.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102163 s, 10.3 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125849 s, 8.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112213 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0953962 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116824 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0993717 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120609 s, 8.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0291104 s, 36.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0338373 s, 31.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0246129 s, 42.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.020735 s, 50.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0298821 s, 35.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0234407 s, 44.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00798804 s, 131 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0319214 s, 32.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.025863 s, 40.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.011619 s, 90.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0355361 s, 29.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0353214 s, 29.7 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9882c4faf800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0199665 s, 52.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00752975 s, 139 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0191416 s, 54.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00453755 s, 231 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg109-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg109-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=reads_and_writes ost.OSS.ost_io.nrs_orr_offset_type=logical policy: orr, orr_quantum 64, orr_offset_type logical, orr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.169434 s, 99.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0386555 s, 27.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0287243 s, 36.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0438929 s, 23.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0383391 s, 27.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.045192 s, 23.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0786188 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0771666 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0966767 s, 10.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9882c4faf800.lock_unused_count=3 ldlm.namespaces.lustre-OST0001-osc-ffff9882c4faf800.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106425 s, 9.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103956 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0878795 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.097066 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0974385 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122852 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118173 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135964 s, 7.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9882c4faf800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9882c4faf800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0398473 s, 26.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0288951 s, 36.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0351434 s, 29.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0289562 s, 36.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0289438 s, 36.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0484915 s, 21.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0321972 s, 32.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0403078 s, 26.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0337448 s, 31.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.044321 s, 23.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.047477 s, 22.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0339677 s, 30.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0402497 s, 26.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0496644 s, 21.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0389919 s, 26.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0416029 s, 25.2 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg109-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg109-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77c (7s) == sanityn test 77d: check TRR nrs policy ================ 17:43:40 (1786484620) ost.OSS.ost_io.nrs_policies=trr ost.OSS.ost_io.nrs_trr_quantum=1 ost.OSS.ost_io.nrs_trr_offset_type=physical ost.OSS.ost_io.nrs_trr_supported=reads policy: trr, trr_quantum 1, trr_offset_type physical, trr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.229068 s, 73.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0322488 s, 32.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0317557 s, 33.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10531 s, 10.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9882c4faf800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9882c4faf800.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133009 s, 7.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147501 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130814 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127857 s, 8.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166394 s, 6.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16707 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162229 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14544 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155249 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171111 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152139 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150129 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178726 s, 5.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9882c4faf800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9882c4faf800.lock_unused_count=2 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0308581 s, 34.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0228299 s, 45.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0198028 s, 53.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0249439 s, 42.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00511573 s, 205 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0583621 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0428731 s, 24.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0368823 s, 28.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00298656 s, 351 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0493311 s, 21.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0412018 s, 25.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0413231 s, 25.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0425501 s, 24.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0116817 s, 89.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0189285 s, 55.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0108445 s, 96.7 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg109-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg109-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=writes ost.OSS.ost_io.nrs_trr_quantum=64 policy: trr, trr_quantum 64, trr_offset_type physical, trr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.197468 s, 85.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0360406 s, 29.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0504242 s, 20.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0857851 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0733471 s, 14.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128899 s, 8.1 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132241 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119475 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155794 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163433 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157001 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183332 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184993 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157143 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184836 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160971 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17563 s, 6.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9882c4faf800.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff9882c4faf800.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0180838 s, 58.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0206445 s, 50.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00400057 s, 262 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00366883 s, 286 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0330507 s, 31.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0284253 s, 36.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0635359 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0197171 s, 53.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0464457 s, 22.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0410658 s, 25.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.03321 s, 31.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.037337 s, 28.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00820629 s, 128 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0416174 s, 25.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0403246 s, 26.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0171237 s, 61.2 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg109-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg109-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=reads_and_writes ost.OSS.ost_io.nrs_trr_offset_type=logical policy: trr, trr_quantum 64, trr_offset_type logical, trr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.257569 s, 65.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0238448 s, 44.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0486357 s, 21.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0586851 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.06542 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0317043 s, 33.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0507058 s, 20.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0529936 s, 19.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0986499 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152769 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152064 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122245 s, 8.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9882c4faf800.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffff9882c4faf800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0857821 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163294 s, 6.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.047218 s, 22.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0480835 s, 21.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0216176 s, 48.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0451475 s, 23.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0340137 s, 30.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0232384 s, 45.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0698857 s, 15.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0238435 s, 44.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0501215 s, 20.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0301515 s, 34.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00278268 s, 377 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.039018 s, 26.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0427983 s, 24.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0581034 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195175 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170635 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00557284 s, 188 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0186023 s, 56.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200465 s, 5.2 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg109-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg109-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77d (8s) == sanityn test 77e: check TBF NID nrs policy ============ 17:43:48 (1786484628) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_tbf_rule=start localhost nid={0@lo} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start clients nid={192.168.201.9@tcp} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start others nid={*.*.*.*@tcp} rate=50 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.224403 s, 74.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0352465 s, 29.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.043944 s, 23.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0435508 s, 24.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0490157 s, 21.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15116 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151412 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155214 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170161 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160998 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173254 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178524 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165464 s, 6.3 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162602 s, 6.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9882c4faf800.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff9882c4faf800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164232 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163247 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182489 s, 5.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0701561 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0978123 s, 10.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0894772 s, 11.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0693894 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0741183 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0814684 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0787039 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1042 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0860112 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0720761 s, 14.5 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0734101 s, 14.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0977925 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0850261 s, 12.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0895583 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0866552 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119556 s, 8.8 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg109-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg109-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change localhost rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change clients rate=101 ost.OSS.ost_io.nrs_tbf_rule=change others rate=51 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.176064 s, 95.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0319093 s, 32.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0248842 s, 42.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0891365 s, 11.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0984131 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104473 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0938532 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110551 s, 9.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107081 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102861 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110217 s, 9.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120476 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109377 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115676 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119219 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1161 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122517 s, 8.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0239379 s, 43.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.026402 s, 39.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0266173 s, 39.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.031725 s, 33.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0317812 s, 33.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.027502 s, 38.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0280009 s, 37.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0358429 s, 29.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0279354 s, 37.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0344044 s, 30.5 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0344829 s, 30.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0301355 s, 34.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0402248 s, 26.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.039957 s, 26.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0354319 s, 29.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0359417 s, 29.2 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg109-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg109-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop localhost ost.OSS.ost_io.nrs_tbf_rule=stop clients ost.OSS.ost_io.nrs_tbf_rule=stop others 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.147568 s, 114 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0508705 s, 20.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0535918 s, 19.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0466651 s, 22.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0496518 s, 21.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0774132 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0730791 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0625162 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0800248 s, 13.1 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9882c4faf800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0507335 s, 20.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128592 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158924 s, 6.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14721 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157373 s, 6.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145555 s, 7.2 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154104 s, 6.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9882c4faf800.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15915 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0390944 s, 26.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0318846 s, 32.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0501015 s, 20.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0610844 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0492014 s, 21.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0434932 s, 24.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0315718 s, 33.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0489641 s, 21.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0367445 s, 28.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0462881 s, 22.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0365321 s, 28.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0338118 s, 31.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0490627 s, 21.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0408569 s, 25.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0357626 s, 29.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0435325 s, 24.1 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg109-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg109-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.181748 s, 92.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0428161 s, 24.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0493472 s, 21.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0461805 s, 22.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0846282 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0891583 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0868849 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0958628 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100041 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102138 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0986673 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138568 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115415 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116739 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131768 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129258 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146086 s, 7.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127719 s, 82.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0437696 s, 24.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0179946 s, 58.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0184489 s, 56.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0354832 s, 29.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164479 s, 63.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0485404 s, 21.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0255138 s, 41.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.052897 s, 19.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0535407 s, 19.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0272681 s, 38.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0281408 s, 37.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0291555 s, 36.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0291205 s, 36.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0565303 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0270503 s, 38.8 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg109-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg109-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' PASS 77e (10s) == sanityn test 77f: check TBF JobID nrs policy ========== 17:43:58 (1786484638) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start runas jobid={iozone.500 dd.500 tiotest.500} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start iozone_runas jobid={iozone.500} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg109-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.16691 s, 101 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0356392 s, 29.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0333364 s, 31.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0409805 s, 25.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0490967 s, 21.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0509642 s, 20.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0476957 s, 22.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0397639 s, 26.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0404477 s, 25.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0420156 s, 25.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0674624 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102525 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0892151 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0923831 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102209 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113015 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125365 s, 8.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157512 s, 66.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0193072 s, 54.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0191679 s, 54.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0200311 s, 52.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.024007 s, 43.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0255106 s, 41.1 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0311656 s, 33.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0192197 s, 54.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0262953 s, 39.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0268038 s, 39.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0243283 s, 43.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0241065 s, 43.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.026928 s, 38.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0301626 s, 34.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0247847 s, 42.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0297048 s, 35.3 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg109-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg109-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change runas rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change iozone_runas rate=101 ost.OSS.ost_io.nrs_tbf_rule=change dd_runas rate=51 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg109-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.151744 s, 111 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0297405 s, 35.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0270621 s, 38.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0784597 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0661108 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0661152 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0770541 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103602 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100257 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0981301 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109152 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100803 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106801 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108041 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104898 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104906 s, 10.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0203652 s, 51.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0200159 s, 52.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.019253 s, 54.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0238753 s, 43.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0217598 s, 48.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.025005 s, 41.9 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0201818 s, 52.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0281441 s, 37.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183978 s, 57.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0239584 s, 43.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00373858 s, 280 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0228492 s, 45.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0221973 s, 47.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0297599 s, 35.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.027669 s, 37.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0289054 s, 36.3 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg109-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg109-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop runas ost.OSS.ost_io.nrs_tbf_rule=stop iozone_runas ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg109-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.152061 s, 110 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0340907 s, 30.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0323999 s, 32.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0840078 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0772311 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111403 s, 9.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120202 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107484 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105455 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0939381 s, 11.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124477 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106727 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140471 s, 7.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138721 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130024 s, 8.1 MB/s execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148135 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14302 s, 7.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183378 s, 57.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0221432 s, 47.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0203625 s, 51.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0171815 s, 61.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158978 s, 66.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0250178 s, 41.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0224456 s, 46.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00423754 s, 247 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135006 s, 77.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0242798 s, 43.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00558463 s, 188 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0210923 s, 49.7 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00298417 s, 351 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0115355 s, 90.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.027466 s, 38.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9882c4faf800.lock_unused_count=3 ldlm.namespaces.lustre-OST0001-osc-ffff9882c4faf800.lock_unused_count=9 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153393 s, 68.4 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg109-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg109-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_policies=fifo running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg109-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.147777 s, 114 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0341004 s, 30.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0614769 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0621831 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.061383 s, 17.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0780535 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.049732 s, 21.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0743909 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0767146 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0598168 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0952102 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102163 s, 10.3 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9882c4faf800.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116843 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129804 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108317 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11658 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127789 s, 8.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160114 s, 65.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013956 s, 75.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0180082 s, 58.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0188186 s, 55.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.017235 s, 60.8 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0194616 s, 53.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.019369 s, 54.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148557 s, 70.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00669653 s, 157 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0208953 s, 50.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0205103 s, 51.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0217497 s, 48.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0188095 s, 55.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0232707 s, 45.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00639138 s, 164 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00163575 s, 641 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg109-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg109-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' PASS 77f (9s) == sanityn test 77g: Change TBF type directly ============ 17:44:07 (1786484647) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 ost.OSS.ost_io.nrs_policies=fifo PASS 77g (3s) == sanityn test 77h: Wrong policy name should report error, not LBUG ========================================================== 17:44:10 (1786484650) oleg109-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=abc: No such file or directory oleg109-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='abc': No such file or directory pdsh@oleg109-client: oleg109-server: ssh exited with exit code 2 oleg109-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc: Invalid argument oleg109-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc': Invalid argument pdsh@oleg109-client: oleg109-server: ssh exited with exit code 22 oleg109-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf reg abc: Invalid argument oleg109-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf reg abc': Invalid argument pdsh@oleg109-client: oleg109-server: ssh exited with exit code 22 oleg109-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc efg: Invalid argument oleg109-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc efg': Invalid argument pdsh@oleg109-client: oleg109-server: ssh exited with exit code 22 PASS 77h (4s) == sanityn test 77i: Change rank of TBF rule ============= 17:44:14 (1786484654) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start before jobid={jobid} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start after jobid={jobid} rate=1000 rank=default ost.OSS.ost_io.nrs_tbf_rule=start target jobid={jobid} rate=1000 rank=after Move before itself ost.OSS.ost_io.nrs_tbf_rule=change target rank=target Move to higher rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=before Move to lower rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=after Move before default ost.OSS.ost_io.nrs_tbf_rule=change target rank=default ost.OSS.ost_io.nrs_policies=fifo PASS 77i (8s) == sanityn test 77j: check TBF-OPCode NRS policy ========= 17:44:22 (1786484662) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf opcode ost.OSS.ost_io.nrs_tbf_rule=start ost_r opcode={ost_read} rate=5 ost.OSS.ost_io.nrs_tbf_rule=start ost_w opcode={ost_write} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.113629 s, 148 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0291541 s, 36.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0298791 s, 35.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0210988 s, 49.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0417614 s, 25.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1026 s, 10.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0970758 s, 10.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0878388 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0918288 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101434 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103929 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0880939 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0903977 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0948987 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.096807 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0927173 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108467 s, 9.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0172325 s, 60.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0186231 s, 56.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0176392 s, 59.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144195 s, 72.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153361 s, 68.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141534 s, 74.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0184067 s, 57.0 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134619 s, 77.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0196262 s, 53.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00508382 s, 206 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00951098 s, 110 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00888853 s, 118 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0104052 s, 101 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00508538 s, 206 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0097814 s, 107 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00253339 s, 414 MB/s removed '/mnt/lustre/d77j.sanityn/nrs_r_oleg109-client.virtnet' removed '/mnt/lustre/d77j.sanityn/nrs_w_oleg109-client.virtnet' removed directory '/mnt/lustre/d77j.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 5 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.94567 s, 21.2 MB/s Write runtime is 5 s, speed is 20.000000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.4309 s, 5.4 MB/s Read runtime is 20 s, speed is 5.000000 IOPS cleanup_tbf /mnt/lustre/d77j.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_r ost.OSS.ost_io.nrs_tbf_rule=stop ost_w ost.OSS.ost_io.nrs_policies=fifo PASS 77j (48s) == sanityn test 77ja: check TBF-UID/GID NRS policy ======= 17:45:10 (1786484710) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg109-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg109-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg109-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg109-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg109-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.465508 s, 36.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0709776 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0735626 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0782212 s, 13.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126409 s, 8.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125189 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115639 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141309 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11393 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138502 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125438 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13062 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124623 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131203 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143575 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155023 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140532 s, 7.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0111914 s, 93.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158644 s, 66.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157719 s, 66.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0203111 s, 51.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012485 s, 84.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0210285 s, 49.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143241 s, 73.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0222855 s, 47.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173912 s, 60.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.018716 s, 56.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00509768 s, 206 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0030962 s, 339 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00356657 s, 294 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00425821 s, 246 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00438653 s, 239 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00271281 s, 387 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg109-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg109-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8287 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9529 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg109-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg109-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg109-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg109-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg109-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.450241 s, 37.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0449572 s, 23.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0575413 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0719989 s, 14.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0908023 s, 11.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0980455 s, 10.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0865758 s, 12.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0918709 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0923967 s, 11.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128144 s, 8.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126768 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130389 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123718 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10846 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123548 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129655 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142834 s, 7.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.018104 s, 57.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183689 s, 57.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170276 s, 61.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0210202 s, 49.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.021519 s, 48.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162195 s, 64.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00851965 s, 123 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167935 s, 62.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0210898 s, 49.7 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0187194 s, 56.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0178269 s, 58.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00747729 s, 140 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00394467 s, 266 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0241981 s, 43.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.006347 s, 165 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00164595 s, 637 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg109-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg109-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8437 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.903 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77ja (133s) == sanityn test 77jb: check TBF-UID/GID NRS policy on files that don't belong to us ========================================================== 17:47:23 (1786484843) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg109-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg109-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg109-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg109-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg109-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.22426 s, 13.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0264152 s, 39.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0523658 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0541599 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0626814 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0625674 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0589314 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.07145 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0856847 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157344 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160525 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188204 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19345 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180504 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205926 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202947 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196842 s, 5.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157729 s, 66.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173505 s, 60.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0177947 s, 58.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0180722 s, 58.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0238 s, 44.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0171767 s, 61.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0228427 s, 45.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0121257 s, 86.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.022023 s, 47.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00672208 s, 156 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00384319 s, 273 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0244489 s, 42.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0097317 s, 108 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0235166 s, 44.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0216359 s, 48.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00426189 s, 246 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg109-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg109-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8291 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9654 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg109-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg109-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg109-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg109-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg109-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.25115 s, 13.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0404961 s, 25.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0741369 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0900397 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0826623 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112303 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11259 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0831719 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105876 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0876856 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104598 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0899572 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133007 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111583 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116061 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112884 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126543 s, 8.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0239187 s, 43.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0217305 s, 48.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0270095 s, 38.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0238743 s, 43.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0308404 s, 34.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00373726 s, 281 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0220725 s, 47.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0174987 s, 59.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0193323 s, 54.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0108999 s, 96.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183832 s, 57.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00370531 s, 283 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0061201 s, 171 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00404126 s, 259 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00510642 s, 205 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135749 s, 77.2 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg109-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg109-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8322 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9598 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77jb (135s) == sanityn test 77k: check TBF policy with UID/GID/JobID/OPCode expression ========================================================== 17:49:38 (1786484978) ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg109-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.272375 s, 61.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0239416 s, 43.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0249183 s, 42.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0757639 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0628518 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100675 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0821593 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0974963 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0921021 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0880622 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0983739 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1048 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0880974 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0980908 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0927456 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102009 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110073 s, 9.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0246646 s, 42.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0280402 s, 37.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016956 s, 61.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0236178 s, 44.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0226853 s, 46.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0245281 s, 42.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0337201 s, 31.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0277607 s, 37.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.021824 s, 48.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00903509 s, 116 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144957 s, 72.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0227311 s, 46.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0253445 s, 41.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0256001 s, 41.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.026097 s, 40.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0090241 s, 116 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg109-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg109-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.9393 s, 21.2 MB/s Write runtime is 5 s, speed is 20.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.7476 s, 10.8 MB/s Read runtime is 10 s, speed is 10.000000 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.201.9@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.201.9@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.137424 s, 122 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0243182 s, 43.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0340283 s, 30.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0716268 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0644435 s, 16.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0724073 s, 14.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0832326 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0870715 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0784731 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0853768 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103665 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0907559 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0874868 s, 12.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169272 s, 6.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163119 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207503 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220865 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0265527 s, 39.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0235742 s, 44.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0279901 s, 37.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0246321 s, 42.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00927993 s, 113 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00706192 s, 148 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155758 s, 67.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.022308 s, 47.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158241 s, 66.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0204965 s, 51.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0112502 s, 93.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0287962 s, 36.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0307615 s, 34.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0346413 s, 30.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0235048 s, 44.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0122593 s, 85.5 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg109-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg109-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.93036 s, 21.3 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73394 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.201.9@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg109-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.135418 s, 124 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0245426 s, 42.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.083205 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0786522 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0922933 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109866 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0966606 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105515 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0957059 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110113 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0922591 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0917724 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0968954 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106529 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106706 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0866071 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0822871 s, 12.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153209 s, 68.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0202278 s, 51.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0175316 s, 59.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0197537 s, 53.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0187925 s, 55.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157507 s, 66.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152128 s, 68.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0045096 s, 233 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00746281 s, 141 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0250982 s, 41.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0186049 s, 56.4 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0233066 s, 45.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0070486 s, 149 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134965 s, 77.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0094425 s, 111 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00563147 s, 186 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg109-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg109-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.94927 s, 21.2 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.88332 s, 21.5 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg109-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.130945 s, 128 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0276921 s, 37.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0294918 s, 35.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.055522 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0766906 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0664737 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0879565 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0862726 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101874 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0874978 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0853092 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101723 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0886165 s, 11.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15728 s, 6.7 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162154 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216661 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.225568 s, 4.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152715 s, 68.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164222 s, 63.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173623 s, 60.4 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0187036 s, 56.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0192518 s, 54.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0200828 s, 52.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0045725 s, 229 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00405711 s, 258 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0228506 s, 45.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.022713 s, 46.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00558629 s, 188 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170761 s, 61.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0219886 s, 47.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00542742 s, 193 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00314332 s, 334 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0307046 s, 34.2 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg109-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg109-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.8078 s, 10.7 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73886 s, 10.8 MB/s Read runtime is 10 s, speed is 10.000000 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.91399 s, 21.3 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73474 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg109-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.114228 s, 147 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0247937 s, 42.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0320226 s, 32.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0531967 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0415334 s, 25.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0607589 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0647692 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0619917 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.046829 s, 22.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0850577 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105675 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1089 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100562 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111827 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117508 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135681 s, 7.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.018346 s, 57.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0122873 s, 85.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0213615 s, 49.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0236429 s, 44.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0334889 s, 31.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183703 s, 57.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0303859 s, 34.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0245853 s, 42.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0239144 s, 43.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0041405 s, 253 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0328359 s, 31.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144737 s, 72.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0123412 s, 85.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0102762 s, 102 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0280811 s, 37.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0303185 s, 34.6 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg109-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg109-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.4879 s, 5.4 MB/s Write runtime is 20 s, speed is 5.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.4425 s, 5.4 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_ug ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg109-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.201664 s, 83.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0275505 s, 38.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.044155 s, 23.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.066215 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0831618 s, 12.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107466 s, 9.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0991103 s, 10.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102855 s, 10.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112056 s, 9.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118054 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111306 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125874 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106634 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112798 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122639 s, 8.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106123 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110075 s, 9.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0245317 s, 42.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0192464 s, 54.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.018968 s, 55.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0250681 s, 41.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0189744 s, 55.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.02665 s, 39.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00783746 s, 134 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00400829 s, 262 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.021715 s, 48.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00638671 s, 164 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0280249 s, 37.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0270093 s, 38.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0195382 s, 53.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0280631 s, 37.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0100179 s, 105 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014408 s, 72.8 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg109-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg109-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.94493 s, 21.2 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73403 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg109-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.180907 s, 92.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0445922 s, 23.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0307142 s, 34.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151827 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143443 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136316 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147888 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157525 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153853 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136545 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164569 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187021 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152146 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.263112 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.25425 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.310993 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.306339 s, 3.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0252489 s, 41.5 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0190532 s, 55.0 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164734 s, 63.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0206043 s, 50.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0290847 s, 36.1 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.021072 s, 49.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152975 s, 68.5 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0212435 s, 49.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0298809 s, 35.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0264059 s, 39.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00594024 s, 177 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013564 s, 77.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0113803 s, 92.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00927891 s, 113 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00704535 s, 149 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0328581 s, 31.9 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg109-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg109-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.84607 s, 10.6 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.74056 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.76543 s, 10.7 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73596 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_policies=fifo PASS 77k (357s) == sanityn test 77kb: Check different granular TBF type combination: jobid+opcode ========================================================== 17:55:35 (1786485335) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/nrs_r_oleg109-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.254385 s, 66.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0366634 s, 28.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0368326 s, 28.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133263 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133004 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141701 s, 7.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139272 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159331 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169805 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149507 s, 7.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165001 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17009 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165931 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172702 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157784 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185574 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169014 s, 6.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0184835 s, 56.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0185079 s, 56.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0206478 s, 50.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0268051 s, 39.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0272132 s, 38.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00622234 s, 169 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0234537 s, 44.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0193231 s, 54.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0256957 s, 40.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143065 s, 73.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00400127 s, 262 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0377242 s, 27.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128459 s, 81.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146926 s, 71.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.017499 s, 59.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00911715 s, 115 MB/s removed '/mnt/lustre/d77kb.sanityn/nrs_w_oleg109-client.virtnet' removed '/mnt/lustre/d77kb.sanityn/nrs_r_oleg109-client.virtnet' removed directory '/mnt/lustre/d77kb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.93774 s, 21.2 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73235 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kb (37s) == sanityn test 77kc: Check different granular TBF type combination: nid+opcode ========================================================== 17:56:12 (1786485372) ost.OSS.ost_io.nrs_policies=tbf nid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.201.9@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.201.9@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.31085 s, 54.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0448656 s, 23.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0762626 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0965481 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141126 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149928 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185207 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172495 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161513 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158933 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144211 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150484 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154875 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202504 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181894 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.217717 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.241303 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143577 s, 73.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0288752 s, 36.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.025404 s, 41.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0358863 s, 29.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161494 s, 64.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0334391 s, 31.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0306118 s, 34.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0245241 s, 42.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0314907 s, 33.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.033449 s, 31.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.033146 s, 31.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0105525 s, 99.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0542299 s, 19.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0338511 s, 31.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00605972 s, 173 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00214143 s, 490 MB/s removed '/mnt/lustre/d77kc.sanityn/nrs_r_oleg109-client.virtnet' removed '/mnt/lustre/d77kc.sanityn/nrs_w_oleg109-client.virtnet' removed directory '/mnt/lustre/d77kc.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.98966 s, 21.0 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.74011 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kc.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kc (41s) == sanityn test 77kd: Check different granular TBF type combination: nid+jobid ========================================================== 17:56:53 (1786485413) ost.OSS.ost_io.nrs_policies=tbf nid+jobid ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.201.9@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/nrs_r_oleg109-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.256287 s, 65.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0236056 s, 44.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0314118 s, 33.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0941036 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0697743 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11846 s, 8.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102637 s, 10.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107213 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122769 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0963488 s, 10.9 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112734 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111586 s, 9.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109706 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112215 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127259 s, 8.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135998 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123411 s, 8.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00694982 s, 151 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0112153 s, 93.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0123761 s, 84.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0232716 s, 45.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016678 s, 62.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0185295 s, 56.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0235602 s, 44.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136368 s, 76.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00528423 s, 198 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0227689 s, 46.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00540747 s, 194 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0123323 s, 85.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00526239 s, 199 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0273896 s, 38.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0198858 s, 52.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0172442 s, 60.8 MB/s removed '/mnt/lustre/d77kd.sanityn/nrs_r_oleg109-client.virtnet' removed '/mnt/lustre/d77kd.sanityn/nrs_w_oleg109-client.virtnet' removed directory '/mnt/lustre/d77kd.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.93595 s, 21.2 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kd.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.99655 s, 21.0 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kd.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kd (36s) == sanityn test 77ke: Check different granular TBF type combination: jobid+opcode ========================================================== 17:57:29 (1786485449) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/nrs_r_oleg109-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.562122 s, 29.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0770406 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0888862 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0731686 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13685 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130699 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146913 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164659 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200815 s, 5.2 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18664 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147072 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170369 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15586 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166776 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200197 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198644 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221628 s, 4.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0203992 s, 51.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0217341 s, 48.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0583825 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0349523 s, 30.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0313984 s, 33.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0633086 s, 16.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0758684 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00769014 s, 136 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0426277 s, 24.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0509217 s, 20.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0450974 s, 23.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0662166 s, 15.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0541836 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0539416 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0559025 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0577566 s, 18.2 MB/s removed '/mnt/lustre/d77ke.sanityn/nrs_r_oleg109-client.virtnet' removed '/mnt/lustre/d77ke.sanityn/nrs_w_oleg109-client.virtnet' removed directory '/mnt/lustre/d77ke.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.86937 s, 10.6 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ke.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.74267 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.9155 s, 21.3 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73141 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77ke (81s) == sanityn test 77kf: Check different granular TBF type combination: uid+gid ========================================================== 17:58:51 (1786485531) ost.OSS.ost_io.nrs_policies=tbf uid+gid ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/nrs_r_oleg109-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.85505 s, 19.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0216531 s, 48.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0493251 s, 21.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0538337 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0270441 s, 38.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0376818 s, 27.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0426088 s, 24.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0488035 s, 21.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0519758 s, 20.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161303 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.204796 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215123 s, 4.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223819 s, 4.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191236 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198873 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223753 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223097 s, 4.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0208864 s, 50.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0226661 s, 46.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.020379 s, 51.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0208956 s, 50.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153089 s, 68.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00432333 s, 243 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0188411 s, 55.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0266392 s, 39.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0280233 s, 37.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0207392 s, 50.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0254369 s, 41.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0224611 s, 46.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0209979 s, 49.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0272307 s, 38.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0269182 s, 39.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00411521 s, 255 MB/s removed '/mnt/lustre/d77kf.sanityn/nrs_w_oleg109-client.virtnet' removed '/mnt/lustre/d77kf.sanityn/nrs_r_oleg109-client.virtnet' removed directory '/mnt/lustre/d77kf.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.6712 s, 5.3 MB/s Write runtime is 20 s, speed is 5.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kf.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9572 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77kf.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kf (66s) == sanityn test 77kg: check different granular TBF type combination: uid+opcode ========================================================== 17:59:57 (1786485597) ost.OSS.ost_io.nrs_policies=tbf uid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg109-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.258507 s, 64.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.036238 s, 28.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0286899 s, 36.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0493884 s, 21.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0540221 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0454594 s, 23.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.048172 s, 21.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.038376 s, 27.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0536421 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0889396 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0798035 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104023 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0771314 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0976764 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101498 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0856095 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.097023 s, 10.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0179543 s, 58.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0194964 s, 53.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167732 s, 62.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0216122 s, 48.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0188834 s, 55.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0206329 s, 50.8 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.026746 s, 39.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0171277 s, 61.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168044 s, 62.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151722 s, 69.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0197349 s, 53.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0242593 s, 43.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00338027 s, 310 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0252477 s, 41.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00407568 s, 257 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00451761 s, 232 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg109-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg109-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.93521 s, 21.2 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73303 s, 10.8 MB/s Read runtime is 10 s, speed is 10.000000 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg109-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.459224 s, 36.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0227463 s, 46.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0358301 s, 29.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0450348 s, 23.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0660385 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0550368 s, 19.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0673392 s, 15.6 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0587856 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0999976 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0736133 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112642 s, 9.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.094566 s, 11.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0951345 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.255156 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.256921 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.309414 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.303631 s, 3.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143576 s, 73.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0182042 s, 57.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0177012 s, 59.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0179809 s, 58.3 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0201172 s, 52.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0207403 s, 50.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00384376 s, 273 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0079897 s, 131 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128664 s, 81.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0197285 s, 53.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0228996 s, 45.8 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00925076 s, 113 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0233794 s, 44.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173231 s, 60.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127141 s, 82.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0108509 s, 96.6 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg109-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg109-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.77796 s, 10.7 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73748 s, 10.8 MB/s Read runtime is 10 s, speed is 10.000000 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.76878 s, 10.7 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.74058 s, 10.8 MB/s Read runtime is 10 s, speed is 10.000000 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kg (118s) == sanityn test 77kh: Verify that Project ID support for NRS TBF rule ========================================================== 18:01:55 (1786485715) using SAVE_PROJECT_SUPPORTED=0 Stopping clients: oleg109-client.virtnet /mnt/lustre (opts:) Stopping client oleg109-client.virtnet /mnt/lustre opts: Stopping clients: oleg109-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg109-client.virtnet /mnt/lustre2 opts: Stopping /mnt/lustre-mds1 (opts:-f) on oleg109-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg109-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg109-server sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime) proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) devtmpfs on /dev type devtmpfs (rw,nosuid,mode=755) securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime) tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev) devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000) tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755) tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755) cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd) pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime) bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700) cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb) cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids) cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer) cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio) cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory) cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct) cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio) cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event) cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices) cgroup on /sys/fs/cgroup/misc type cgroup (rw,nosuid,nodev,noexec,relatime,misc) cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset) cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma) none on /sys/kernel/tracing type tracefs (rw,relatime) configfs on /sys/kernel/config type configfs (rw,relatime) /dev/nbd0 on / type ext4 (ro,relatime,stripe=32) rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) debugfs on /sys/kernel/debug type debugfs (rw,relatime) mqueue on /dev/mqueue type mqueue (rw,relatime) hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M) none on /mnt type ramfs (rw,relatime) none on /var/lib/stateless/writable type tmpfs (rw,relatime) /dev/vda on /home/green/git/lustre-release type squashfs (ro,relatime) none on /var/cache/man type tmpfs (rw,relatime) none on /var/log type tmpfs (rw,relatime) none on /var/lib/dbus type tmpfs (rw,relatime) none on /tmp type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) none on /var/tmp type tmpfs (rw,relatime) none on /var/lib/NetworkManager type tmpfs (rw,relatime) none on /var/lib/systemd/random-seed type tmpfs (rw,relatime) none on /var/spool type tmpfs (rw,relatime) none on /var/lib/nfs type tmpfs (rw,relatime) none on /var/lib/gssproxy type tmpfs (rw,relatime) none on /var/lib/logrotate type tmpfs (rw,relatime) none on /etc type tmpfs (rw,relatime) none on /var/lib/rsyslog type tmpfs (rw,relatime) none on /root/.ssh type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) 192.168.200.253:/exports/state/oleg109-client.virtnet on /var/lib/stateless/state type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.201.9,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg109-client.virtnet/boot on /boot type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.201.9,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg109-client.virtnet/etc/kdump.conf on /etc/kdump.conf type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.201.9,local_lock=none,addr=192.168.200.253) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=378180k,mode=700) 192.168.200.253:/exports/testreports/67687/testresults/sanityn-zfs-rocky8.10_x86_64-rocky8.10_x86_64 on /tmp/testlogs type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.201.9,local_lock=none,addr=192.168.200.253) /dev/vda on /usr/sbin/mount.lustre type squashfs (ro,relatime) Checking servers environments Checking clients oleg109-client.virtnet environments /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg109-server' oleg109-server: oleg109-server.virtnet: executing load_modules_local oleg109-server: Loading modules from /home/green/git/lustre-release/lustre oleg109-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg109-server: detected 4 online CPUs by sysfs oleg109-server: MODOPTS_LIBCFS= oleg109-server: Force libcfs to create 2 CPU partitions Setup mgs, mdt, osts Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg109-server: oleg109-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg109-client: oleg109-server: ssh exited with exit code 1 Started lustre-MDT0000 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg109-server: oleg109-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg109-client: oleg109-server: ssh exited with exit code 1 Started lustre-OST0000 Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg109-server: oleg109-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg109-client: oleg109-server: ssh exited with exit code 1 Started lustre-OST0001 Starting client: oleg109-client.virtnet: -o user_xattr,flock 192.168.201.109@tcp:/lustre /mnt/lustre Starting client oleg109-client.virtnet: -o user_xattr,flock 192.168.201.109@tcp:/lustre /mnt/lustre Started clients oleg109-client.virtnet: 192.168.201.109@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Starting client: oleg109-client.virtnet: -o user_xattr,flock 192.168.201.109@tcp:/lustre /mnt/lustre2 Starting client oleg109-client.virtnet: -o user_xattr,flock 192.168.201.109@tcp:/lustre /mnt/lustre2 Started clients oleg109-client.virtnet: 192.168.201.109@tcp:/lustre on /mnt/lustre2 type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9882c5c7e800.idle_timeout=debug osc.lustre-OST0000-osc-ffff9882ef6cc800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9882c5c7e800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9882ef6cc800.idle_timeout=debug disable quota as required ost.OSS.ost_io.nrs_policies=tbf projid ost.OSS.ost_io.nrs_tbf_rule=start ext_proj_rw projid={100} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.258304 s, 65.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0226264 s, 46.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0622331 s, 16.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0699556 s, 15.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0817695 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0741074 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0929243 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.066346 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0890449 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0869277 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101954 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108577 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14277 s, 7.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.250433 s, 4.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.263895 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.302089 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.305195 s, 3.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141082 s, 74.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144503 s, 72.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0109285 s, 95.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162927 s, 64.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170884 s, 61.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134374 s, 78.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00697151 s, 150 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0187896 s, 55.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0258945 s, 40.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0220306 s, 47.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0436999 s, 24.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00874339 s, 120 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0361132 s, 29.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0459686 s, 22.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00613187 s, 171 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00348436 s, 301 MB/s removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg109-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg109-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.99887 s, 21.0 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.98083 s, 21.1 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_proj_rw ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf projid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w projid={100}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r projid={100}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.323004 s, 51.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0264585 s, 39.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.028694 s, 36.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10843 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103149 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117759 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115004 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123133 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10306 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128036 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12731 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119338 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121719 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210978 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202366 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.279769 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.279756 s, 3.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183891 s, 57.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0191591 s, 54.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0226931 s, 46.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0210406 s, 49.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0194978 s, 53.8 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0176171 s, 59.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0188065 s, 55.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0194294 s, 54.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0184542 s, 56.8 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00446657 s, 235 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0224608 s, 46.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0176809 s, 59.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.020805 s, 50.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0249055 s, 42.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00279994 s, 374 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00247852 s, 423 MB/s removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg109-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg109-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.95394 s, 21.2 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73605 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77kh (133s) == sanityn test 77ki: Add rule with unsupported TBF type should fail for generic TBF ========================================================== 18:04:08 (1786485848) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 oleg109-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_fail projid={100}&opcode={ost_write} rate=20: Invalid argument oleg109-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_fail projid={100}&opcode={ost_write} rate=20': Invalid argument pdsh@oleg109-client: oleg109-server: ssh exited with exit code 22 ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77ki (8s) == sanityn test 77kj: Verify nodemap support for NRS TBF rule ========================================================== 18:04:16 (1786485856) On MGS 192.168.201.109, active = nodemap.active=1 On MGS 192.168.201.109, TBF = ost.OSS.ost_io.nrs_policies=tbf nodemap ost.OSS.ost_io.nrs_tbf_rule=start ext_nm_rw nodemap={TBF} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.244959 s, 68.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0324519 s, 32.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0789138 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0975766 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0974358 s, 10.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0883623 s, 11.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0923228 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.078419 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0745711 s, 14.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0913557 s, 11.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0926344 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14839 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145573 s, 7.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.249832 s, 4.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.240273 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.290498 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.310676 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0174793 s, 60.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0200031 s, 52.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0172884 s, 60.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137502 s, 76.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0228115 s, 46.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0251018 s, 41.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.020842 s, 50.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0285534 s, 36.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173328 s, 60.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0207093 s, 50.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0300549 s, 34.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0240467 s, 43.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0218373 s, 48.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0236759 s, 44.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0239237 s, 43.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0301295 s, 34.8 MB/s removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg109-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg109-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.94187 s, 21.2 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.97207 s, 21.1 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete On MGS 192.168.201.109, TBF.id = On MGS 192.168.201.109, active = nodemap.active=0 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.12812 s, 131 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0322975 s, 32.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0428514 s, 24.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0527717 s, 19.9 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0546712 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0607541 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0749808 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0769475 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0739345 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0879815 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0720356 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0961807 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0933422 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0888888 s, 11.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9882ef6cc800.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffff9882ef6cc800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0973383 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0959563 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12228 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00921728 s, 114 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166042 s, 63.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130564 s, 80.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00944058 s, 111 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133771 s, 78.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012349 s, 84.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0203318 s, 51.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00266246 s, 394 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0213827 s, 49.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00408871 s, 256 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00610859 s, 172 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00313196 s, 335 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9882ef6cc800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00349437 s, 300 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00285485 s, 367 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00347265 s, 302 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00367548 s, 285 MB/s removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg109-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg109-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 1 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 1.48647 s, 70.5 MB/s Write runtime is 3 s, speed is 33.333333 IOPS write rate (33.333333) should exceed limit (20 * 1) Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 0.428707 s, 245 MB/s Read runtime is 1 s, speed is 100.000000 IOPS read rate (100.000000) should exceed limit (20 * 1) cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_nm_rw ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf nodemap+opcode oleg109-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_w nodemap={TBF}&opcode={ost_write} rate=20: Invalid argument oleg109-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_w nodemap={TBF}&opcode={ost_write} rate=20': Invalid argument oleg109-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_r nodemap={TBF}&opcode={ost_read} rate=10: Invalid argument oleg109-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_r nodemap={TBF}&opcode={ost_read} rate=10': Invalid argument pdsh@oleg109-client: oleg109-server: ssh exited with exit code 22 On MGS 192.168.201.109, active = nodemap.active=1 On MGS 192.168.201.109, TBF = ost.OSS.ost_io.nrs_tbf_rule=start ext_w nodemap={TBF}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nodemap={TBF}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.247915 s, 67.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0461611 s, 22.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.059555 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.073799 s, 14.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0724984 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0725671 s, 14.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0638451 s, 16.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0889772 s, 11.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0909777 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141341 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147499 s, 7.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.264033 s, 4.0 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.252069 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.289111 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.30247 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.351055 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.359607 s, 2.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150953 s, 69.5 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161879 s, 64.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0193593 s, 54.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016597 s, 63.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.023964 s, 43.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00569214 s, 184 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167629 s, 62.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152534 s, 68.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0209335 s, 50.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0172504 s, 60.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0249835 s, 42.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00677921 s, 155 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0215 s, 48.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00806328 s, 130 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0263974 s, 39.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00516459 s, 203 MB/s removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg109-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg109-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.92901 s, 21.3 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73007 s, 10.8 MB/s Read runtime is 10 s, speed is 10.000000 IOPS cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo On MGS 192.168.201.109, TBF.id = On MGS 192.168.201.109, active = nodemap.active=0 PASS 77kj (116s) == sanityn test 77l: check the output of NRS policies for generic TBF ========================================================== 18:06:12 (1786485972) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_policies=fifo PASS 77l (4s) == sanityn test 77m: check NRS Delay slows write RPC processing ========================================================== 18:06:16 (1786485976) ost.OSS.ost_io.nrs_policies=delay ost.OSS.ost_io.nrs_delay_min=4 ost.OSS.ost_io.nrs_delay_max=4 ost.OSS.ost_io.nrs_delay_pct=100 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg109-client.virtnet] [bs=1M] [count=1] [oflag=direct] [conv=fdatasync] 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 3.27723 s, 320 kB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg109-client.virtnet] [bs=1M] [count=10] [oflag=direct] [conv=fdatasync] 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 40.0006 s, 262 kB/s ost.OSS.ost_io.nrs_policies=fifo PASS 77m (50s) == sanityn test 77n: check wildcard support for TBF JobID NRS policy ========================================================== 18:07:06 (1786486026) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={*.500} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.126492 s, 133 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0277517 s, 37.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0493726 s, 21.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0706583 s, 14.8 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9882ef6cc800.lock_unused_count=2 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0878052 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0833475 s, 12.6 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0941985 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0791974 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0848328 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0767941 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0867414 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0950897 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0888549 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11395 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104149 s, 10.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9882ef6cc800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0959984 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117893 s, 8.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0235375 s, 44.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0172092 s, 60.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.02014 s, 52.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0230567 s, 45.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0251478 s, 41.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145415 s, 72.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0191126 s, 54.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.024444 s, 42.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0209465 s, 50.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016157 s, 64.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0172366 s, 60.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0110548 s, 94.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0037879 s, 277 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0222485 s, 47.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0223493 s, 46.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00159148 s, 659 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg109-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg109-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77n.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.93506 s, 21.2 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77n.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.96514 s, 21.1 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.*} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.239784 s, 70.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0238356 s, 44.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0776547 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.083462 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0714861 s, 14.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0912009 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0849111 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0753983 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0798192 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0574282 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.077084 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0904937 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0647874 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.255865 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.259741 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.304608 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.300899 s, 3.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0215307 s, 48.7 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0196712 s, 53.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0253926 s, 41.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0195287 s, 53.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0210678 s, 49.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0257417 s, 40.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0175185 s, 59.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0255773 s, 41.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0214787 s, 48.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0274732 s, 38.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0213719 s, 49.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0225366 s, 46.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0248741 s, 42.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0238917 s, 43.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0268632 s, 39.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0277369 s, 37.8 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg109-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg109-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.93507 s, 21.2 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.96718 s, 21.1 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_policies=fifo PASS 77n (66s) == sanityn test 77o: Changing rank should not panic ====== 18:08:12 (1786486092) mds.MDS.mdt.nrs_policies=tbf nid mds.MDS.mdt.nrs_tbf_rule=start name nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=start name1 nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=change name1 rank=name mds.MDS.mdt.nrs_tbf_rule=stop name mds.MDS.mdt.nrs_policies=fifo PASS 77o (4s) == sanityn test 77q: Parallel TBF rule definitions should not panic ========================================================== 18:08:16 (1786486096) mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_policies=fifo PASS 77q (35s) == sanityn test 77p: Check validity of rule names for TBF policies ========================================================== 18:08:51 (1786486131) mds.MDS.mdt.nrs_policies=tbf oleg109-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_overflo uid={500} rate=500: File name too long oleg109-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_overflo uid={500} rate=500': File name too long pdsh@oleg109-client: oleg109-server: ssh exited with exit code 36 oleg109-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start uid={500} rate=500: Invalid argument oleg109-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start uid={500} rate=500': Invalid argument pdsh@oleg109-client: oleg109-server: ssh exited with exit code 22 oleg109-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_empty: Invalid argument oleg109-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_empty': Invalid argument pdsh@oleg109-client: oleg109-server: ssh exited with exit code 22 oleg109-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p@spec uid={500} rate=500: Invalid argument oleg109-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p@spec uid={500} rate=500': Invalid argument pdsh@oleg109-client: oleg109-server: ssh exited with exit code 22 oleg109-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p.spec uid={500} rate=500: Invalid argument oleg109-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p.spec uid={500} rate=500': Invalid argument pdsh@oleg109-client: oleg109-server: ssh exited with exit code 22 oleg109-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p~spec uid={500} rate=500: Invalid argument oleg109-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p~spec uid={500} rate=500': Invalid argument pdsh@oleg109-client: oleg109-server: ssh exited with exit code 22 oleg109-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p#spec uid={500} rate=500: Invalid argument oleg109-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p#spec uid={500} rate=500': Invalid argument pdsh@oleg109-client: oleg109-server: ssh exited with exit code 22 oleg109-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p/spec uid={500} rate=500: Invalid argument oleg109-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p/spec uid={500} rate=500': Invalid argument pdsh@oleg109-client: oleg109-server: ssh exited with exit code 22 oleg109-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p^spec uid={500} rate=500: Invalid argument oleg109-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p^spec uid={500} rate=500': Invalid argument pdsh@oleg109-client: oleg109-server: ssh exited with exit code 22 oleg109-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p%spec uid={500} rate=500: Invalid argument oleg109-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p%spec uid={500} rate=500': Invalid argument pdsh@oleg109-client: oleg109-server: ssh exited with exit code 22 oleg109-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p*spec uid={500} rate=500: Invalid argument oleg109-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p*spec uid={500} rate=500': Invalid argument pdsh@oleg109-client: oleg109-server: ssh exited with exit code 22 oleg109-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p;spec uid={500} rate=500: Invalid argument oleg109-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p;spec uid={500} rate=500': Invalid argument pdsh@oleg109-client: oleg109-server: ssh exited with exit code 22 oleg109-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p,spec uid={500} rate=500: Invalid argument oleg109-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p,spec uid={500} rate=500': Invalid argument pdsh@oleg109-client: oleg109-server: ssh exited with exit code 22 oleg109-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p?spec uid={500} rate=500: Invalid argument oleg109-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p?spec uid={500} rate=500': Invalid argument pdsh@oleg109-client: oleg109-server: ssh exited with exit code 22 oleg109-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77pspec uid={500} rate=500: Invalid argument oleg109-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p>spec uid={500} rate=500': Invalid argument pdsh@oleg109-client: oleg109-server: ssh exited with exit code 22 oleg109-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p:spec uid={500} rate=500: Invalid argument oleg109-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p:spec uid={500} rate=500': Invalid argument pdsh@oleg109-client: oleg109-server: ssh exited with exit code 22 oleg109-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p+spec uid={500} rate=500: Invalid argument oleg109-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p+spec uid={500} rate=500': Invalid argument pdsh@oleg109-client: oleg109-server: ssh exited with exit code 22 oleg109-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p=spec uid={500} rate=500: Invalid argument oleg109-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p=spec uid={500} rate=500': Invalid argument pdsh@oleg109-client: oleg109-server: ssh exited with exit code 22 oleg109-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p)spec uid={500} rate=500: Invalid argument oleg109-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p)spec uid={500} rate=500': Invalid argument pdsh@oleg109-client: oleg109-server: ssh exited with exit code 22 oleg109-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p(spec uid={500} rate=500: Invalid argument oleg109-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p(spec uid={500} rate=500': Invalid argument pdsh@oleg109-client: oleg109-server: ssh exited with exit code 22 oleg109-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p{spec uid={500} rate=500: Invalid argument oleg109-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p{spec uid={500} rate=500': Invalid argument pdsh@oleg109-client: oleg109-server: ssh exited with exit code 22 oleg109-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p}spec uid={500} rate=500: Invalid argument oleg109-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p}spec uid={500} rate=500': Invalid argument pdsh@oleg109-client: oleg109-server: ssh exited with exit code 22 oleg109-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p|spec uid={500} rate=500: Invalid argument oleg109-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p|spec uid={500} rate=500': Invalid argument pdsh@oleg109-client: oleg109-server: ssh exited with exit code 22 oleg109-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p[spec uid={500} rate=500: Invalid argument oleg109-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p[spec uid={500} rate=500': Invalid argument pdsh@oleg109-client: oleg109-server: ssh exited with exit code 22 oleg109-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p]spec uid={500} rate=500: Invalid argument oleg109-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p]spec uid={500} rate=500': Invalid argument pdsh@oleg109-client: oleg109-server: ssh exited with exit code 22 oleg109-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p!spec uid={500} rate=500: Invalid argument oleg109-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p!spec uid={500} rate=500': Invalid argument pdsh@oleg109-client: oleg109-server: ssh exited with exit code 22 oleg109-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p&spec uid={500} rate=500: Invalid argument oleg109-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p&spec uid={500} rate=500': Invalid argument pdsh@oleg109-client: oleg109-server: ssh exited with exit code 22 oleg109-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p$spec uid={500} rate=500: Invalid argument oleg109-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p$spec uid={500} rate=500': Invalid argument pdsh@oleg109-client: oleg109-server: ssh exited with exit code 22 oleg109-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p`spec uid={500} rate=500: Invalid argument oleg109-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p`spec uid={500} rate=500': Invalid argument pdsh@oleg109-client: oleg109-server: ssh exited with exit code 22 oleg109-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p\spec uid={500} rate=500: Invalid argument oleg109-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p\spec uid={500} rate=500': Invalid argument pdsh@oleg109-client: oleg109-server: ssh exited with exit code 22 mds.MDS.mdt.nrs_policies=fifo PASS 77p (12s) == sanityn test 77r: Change type of tbf policy at run time ========================================================== 18:09:03 (1786486143) mds.MDS.mdt.nrs_policies=tbf uid mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf gid mds.MDS.mdt.nrs_tbf_rule=start md_rule gid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start md_rule jobid={*.500} rate=1 mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=fifo check the number of requests in queue: fifo: 0 crrn: 0 tbf: 0 delay: 0 fifo: 0 crrn: 0 tbf: 0 delay: 0 cleanup 77r 276259 kill md_thread xargs PASS 77r (40s) == sanityn test 77s: Check TBF LRU shrinker ============== 18:09:44 (1786486184) jobid_name=%e.%p jobid_var=TEST_77s_JOBVAR mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start touch jobid={touch.*} rate=1000 mds.MDS.mdt.nrs_tbf_rule=start unlink jobid={unlink.*} rate=1000 touch jobid={touch.*} 1000, ref 10 touch jobid={touch.*} 1000, ref 0 unlink jobid={unlink.*} 1000, ref 10 unlink jobid={unlink.*} 1000, ref 0 mds.MDS.mdt.nrs_policies=fifo jobid_var=procname_uid jobid_name=%e.%u PASS 77s (10s) == sanityn test 78: Enable policy and specify tunings right away ========================================================== 18:09:55 (1786486195) oleg109-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_orr_quantum=1: No such device oleg109-server: error: set_param: setting 'ost/OSS/*/nrs_orr_quantum'='1': No such device pdsh@oleg109-client: oleg109-server: ssh exited with exit code 19 ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_policies=fifo PASS 78 (3s) == sanityn test 79: xattr: intent error ================== 18:09:57 (1786486197) File: /mnt/lustre/d79.sanityn Size: 512 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115238826936337 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-11 18:09:58.000000000 -0400 Modify: 2026-08-11 18:09:58.000000000 -0400 Change: 2026-08-11 18:09:58.000000000 -0400 Birth: 2026-08-11 18:09:58.000000000 -0400 fail_val=0 fail_loc=0x80000160 fail_val=0 fail_loc=0x80000131 # file: mnt/lustre/d79.sanityn trusted.name1="value1" PASS 79 (12s) == sanityn test 80a: migrate directory when some children is being opened ========================================================== 18:10:09 (1786486209) SKIP: sanityn test_80a needs >= 2 MDTs SKIP 80a (1s) == sanityn test 80b: Accessing directory during migration ========================================================== 18:10:10 (1786486210) SKIP: sanityn test_80b needs >= 2 MDTs SKIP 80b (1s) == sanityn test 81a: rename and stat under striped directory ========================================================== 18:10:11 (1786486211) SKIP: sanityn test_81a needs >= 2 MDTs SKIP 81a (1s) == sanityn test 81b: rename under striped directory doesn't deadlock ========================================================== 18:10:12 (1786486212) SKIP: sanityn test_81b We need at least 2 MDTs for this test SKIP 81b (1s) == sanityn test 81c: rename revoke LOOKUP lock for remote object ========================================================== 18:10:13 (1786486213) SKIP: sanityn test_81c needs >= 4 MDTs SKIP 81c (1s) == sanityn test 81d: parallel rename file cross-dir on same MDT ========================================================== 18:10:14 (1786486214) total: 2500 open/close in 4.77 seconds: 524.01 ops/second snapshot_time 1786486236.338065098 secs.nsecs start_time 1786486216.138501837 secs.nsecs elapsed_time 20.199563261 secs.nsecs open 2505 samples [usecs] 77 6516 854733 346521215 close 2504 samples [usecs] 59 4391 224957 39716435 mknod 2501 samples [usecs] 199 6447 763061 285357541 unlink 2500 samples [usecs] 315 11435 1440614 1384342258 mkdir 2 samples [usecs] 864 881 1745 1522657 rmdir 2 samples [usecs] 2131 3442 5573 16388525 rename 2500 samples [usecs] 282 5873 2885215 4523154835 getattr 3760 samples [usecs] 6 2863 82020 15067490 parallel_rename_file 2500 samples [usecs] 282 5873 2885215 4523154835 crossdir_rename 2500 samples [usecs] 282 5873 2885215 4523154835 readpage 12 samples [usecs] 1657 13872 53345 353859081 PASS 81d (23s) == sanityn test 82: fsetxattr and fgetxattr on orphan files ========================================================== 18:10:37 (1786486237) multiop /mnt/lustre/f82.sanityn vO_ac TMPPIPE=/tmp/multiop_open_wait_pipe.7632 multiop /mnt/lustre2/f82.sanityn vo_Ac TMPPIPE=/tmp/multiop_open_wait_pipe.7632 PASS 82 (2s) == sanityn test 83: access striped directory while it is being created/unlinked ========================================================== 18:10:39 (1786486239) SKIP: sanityn test_83 needs >= 2 MDTs SKIP 83 (1s) == sanityn test 84: 0-nlink race in lu_object_find() ===== 18:10:40 (1786486240) fail_loc=0x8000060b PASS 84 (7s) == sanityn test 85: Lustre API root cache race =========== 18:10:47 (1786486247) Starting: llapi_root_test /mnt/lustre/f85.sanityn /mnt/lustre2/f85.sanityn Starting test test1 at 1786486247 Finishing test test1 at 1786486248 PASS 85 (2s) == sanityn test 90: open/create and unlink striped directory ========================================================== 18:10:49 (1786486249) SKIP: sanityn test_90 needs >= 2 MDTs SKIP 90 (1s) == sanityn test 91: chmod and unlink striped directory === 18:10:50 (1786486250) SKIP: sanityn test_91 needs >= 2 MDTs SKIP 91 (1s) == sanityn test 92: create remote directory under orphan directory ========================================================== 18:10:51 (1786486251) SKIP: sanityn test_92 needs >= 2 MDTs SKIP 92 (1s) == sanityn test 93: alloc_rr should not allocate on same ost ========================================================== 18:10:52 (1786486252) fail_loc=0x00000163 fail_loc=0x0 /mnt/lustre/f93.sanityn-1/file1 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 6491 0x195b 0x240000400 1 6453 0x1935 0x280000400 /mnt/lustre2/f93.sanityn-2/file2 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 6492 0x195c 0x240000400 1 6454 0x1936 0x280000400 rc1=0 and rc2=0 PASS 93 (9s) == sanityn test 94: signal vs CP callback race =========== 18:11:01 (1786486261) osc.lustre-OST0000-osc-ffff9882c5c7e800.idle_timeout=0 osc.lustre-OST0000-osc-ffff9882ef6cc800.idle_timeout=0 osc.lustre-OST0001-osc-ffff9882c5c7e800.idle_timeout=0 osc.lustre-OST0001-osc-ffff9882ef6cc800.idle_timeout=0 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0173537 s, 472 kB/s fail_val=5 fail_loc=0x80000312 fail_val=6 fail_loc=0x80000329 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 3.0986 s, 1.3 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 6389: 297196 Alarm clock dd if=/dev/zero of=$DIR/$tfile conv=notrunc oflag=append bs=4k count=1 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.010939 s, 374 kB/s osc.lustre-OST0000-osc-ffff9882c5c7e800.idle_timeout=debug osc.lustre-OST0000-osc-ffff9882ef6cc800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9882c5c7e800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9882ef6cc800.idle_timeout=debug PASS 94 (10s) == sanityn test 95a: Check readpage() on a page that was removed from page cache ========================================================== 18:11:11 (1786486271) llite.lustre-ffff9882c5c7e800.fast_read=0 llite.lustre-ffff9882ef6cc800.fast_read=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0170673 s, 123 MB/s fail_loc=0x80001421 fail_val=7 fail_loc=0x80001422 fail_val=10 llite.lustre-ffff9882c5c7e800.fast_read=1 llite.lustre-ffff9882ef6cc800.fast_read=1 PASS 95a (14s) == sanityn test 95b: Check readpage() on a page that is no longer uptodate ========================================================== 18:11:25 (1786486285) llite.lustre-ffff9882c5c7e800.fast_read=0 llite.lustre-ffff9882ef6cc800.fast_read=0 1+0 records in 1+0 records out 12288 bytes (12 kB, 12 KiB) copied, 0.00161191 s, 7.6 MB/s fail_loc=0x80001424 fail_val=5 fail_loc=0 llite.lustre-ffff9882c5c7e800.fast_read=1 llite.lustre-ffff9882ef6cc800.fast_read=1 PASS 95b (8s) == sanityn test 100a: DoM: glimpse RPCs for stat without IO lock (DoM only file) ========================================================== 18:11:34 (1786486294) SKIP: sanityn test_100a Reserved for glimpse-ahead SKIP 100a (2s) == sanityn test 100b: DoM: no glimpse RPC for stat with IO lock (DoM only file) ========================================================== 18:11:35 (1786486295) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00160711 s, 2.5 MB/s /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK PASS 100b (2s) == sanityn test 100c: DoM: write vs stat without IO lock (combined file) ========================================================== 18:11:37 (1786486297) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0346657 s, 60.5 MB/s /mnt/lustre/d100c.sanityn/dom has type file OK /mnt/lustre/d100c.sanityn/dom has size 2097152 OK PASS 100c (2s) == sanityn test 100d: DoM: write+truncate vs stat without IO lock (combined file) ========================================================== 18:11:39 (1786486299) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0354653 s, 59.1 MB/s /mnt/lustre/d100d.sanityn/dom has type file OK /mnt/lustre/d100d.sanityn/dom has size 4096 OK PASS 100d (2s) == sanityn test 100e: DoM: read on open and file size ==== 18:11:41 (1786486301) 1+0 records in 1+0 records out 12000 bytes (12 kB, 12 KiB) copied, 0.00204768 s, 5.9 MB/s PASS 100e (2s) == sanityn test 101a: Discard DoM data on unlink ========= 18:11:43 (1786486303) /mnt/lustre/f101a.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00172998 s, 2.4 MB/s PASS 101a (2s) == sanityn test 101b: Discard DoM data on rename ========= 18:11:45 (1786486305) /mnt/lustre/f101b.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00138789 s, 3.0 MB/s PASS 101b (2s) == sanityn test 101c: Discard DoM data on close-unlink === 18:11:47 (1786486307) /mnt/lustre/f101c.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00154415 s, 2.7 MB/s PASS 101c (3s) == sanityn test 102: Test open by handle of unlinked file ========================================================== 18:11:50 (1786486310) file: /mnt/lustre/f102.sanityn fh_bytes: 32 fh_type: 151 fh_data: d1 0b 00 00 02 00 00 00 f2 11 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 file's parent FID is [0:0x0:0x0] file FID is [0x200000bd1:0x11f2:0x0] access via mount point '/mnt/lustre2' - OK access after unlink - OK check_fhandle_syscalls test Passed! PASS 102 (2s) == sanityn test 103: Test size correctness with lockahead ========================================================== 18:11:52 (1786486312) fail_loc=0x415 fail_loc=0x415 fail_val=2 Incorrect size expected (no glimpse fix): Starting test test23 at 1786486313 lockahead_test: lockahead_test.c:1171: test23: assertion 'sb.st_size == sb2.st_size' failed: size on /mnt/lustre/d103.sanityn/f103.sanityn and /mnt/lustre2/d103.sanityn/f103.sanityn differs: 0 vs 1048576 fail_loc=0 fail_loc=0x214 fail_val=2 Starting test test23 at 1786486315 Finishing test test23 at 1786486316 PASS 103 (7s) == sanityn test 104: Verify that MDS stores atime/mtime/ctime during close ========================================================== 18:11:59 (1786486319) SKIP: sanityn test_104 ldiskfs only test SKIP 104 (1s) == sanityn test 105: Glimpse and lock cancel race ======== 18:12:00 (1786486320) fail_loc=0x416 PASS 105 (18s) == sanityn test 106a: Verify the btime via statx() ======= 18:12:18 (1786486338) SKIP: sanityn test_106a Test only for ldiskfs and statx() supported SKIP 106a (1s) == sanityn test 106b: Glimpse RPCs test for statx ======== 18:12:19 (1786486339) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00347468 s, 295 kB/s File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238826938873 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-11 18:12:19.000000000 -0400 Modify: 2026-08-11 18:12:19.000000000 -0400 Change: 2026-08-11 18:12:19.000000000 -0400 Birth: 2026-08-11 18:12:20.000000000 -0400 /mnt/lustre/f106b.sanityn 144115238826938873 -rw-r--r-- 1 0 0 1786486340 1786486339 1786486339 File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238826938873 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-11 18:12:19.000000000 -0400 Modify: 2026-08-11 18:12:19.000000000 -0400 Change: 2026-08-11 18:12:19.000000000 -0400 Birth: 2026-08-11 18:12:20.000000000 -0400 1786486339 1024 1 PASS 106b (2s) == sanityn test 106c: Verify statx attributes mask ======= 18:12:21 (1786486341) PASS 106c (2s) == sanityn test 107a: Basic grouplock conflict =========== 18:12:23 (1786486343) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.0926048 s, 113 MB/s /mnt/lustre/f107a.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 6501 0x1965 0x240000400 multiop /mnt/lustre/f107a.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7632 multiop /mnt/lustre2/f107a.sanityn vO_G16022000r10g16022000c TMPPIPE=/tmp/multiop_open_wait_pipe.7632 First grouplock blocks second one PASS 107a (4s) == sanityn test 107b: Grouplock is added to the head of waiting list ========================================================== 18:12:27 (1786486347) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.0839604 s, 125 MB/s /mnt/lustre/f107b.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 6461 0x193d 0x280000400 multiop /mnt/lustre/f107b.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7632 Grouplock blocks IO First grouplock blocks second one Second grouplock blocks IO PASS 107b (8s) == sanityn test 108a: lseek: parallel updates ============ 18:12:35 (1786486355) 1+0 records in 1+0 records out 10000 bytes (10 kB, 9.8 KiB) copied, 0.00148858 s, 6.7 MB/s llite.lustre-ffff9882c5c7e800.hybrid_io=0 llite.lustre-ffff9882ef6cc800.hybrid_io=0 fail_val=4 fail_loc=0x80000414 1+0 records in 1+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 4.22262 s, 2.0 MB/s PASS 108a (7s) == sanityn test 109: Race with several mount instances on 1 node ========================================================== 18:12:42 (1786486362) 192.168.201.109@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg109-client.virtnet /mnt/lustre (opts:) 192.168.201.109@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg109-client.virtnet /mnt/lustre2 (opts:) Starting race between client mounts (duration 30) Iteration 0 Starting client: oleg109-client.virtnet: -o user_xattr,flock 192.168.201.109@tcp:/lustre /mnt/lustre Starting client: oleg109-client.virtnet: -o user_xattr,flock 192.168.201.109@tcp:/lustre /mnt/lustre2 192.168.201.109@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.109@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg109-client.virtnet /mnt/lustre (opts:) Stopping client oleg109-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg109-server' oleg109-server: oleg109-server.virtnet: executing load_modules_local oleg109-server: Loading modules from /home/green/git/lustre-release/lustre oleg109-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg109-server: detected 4 online CPUs by sysfs oleg109-server: MODOPTS_LIBCFS= oleg109-server: Force libcfs to create 2 CPU partitions Iteration 1 Starting client: oleg109-client.virtnet: -o user_xattr,flock 192.168.201.109@tcp:/lustre /mnt/lustre Starting client: oleg109-client.virtnet: -o user_xattr,flock 192.168.201.109@tcp:/lustre /mnt/lustre2 192.168.201.109@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.109@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg109-client.virtnet /mnt/lustre (opts:) Stopping client oleg109-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg109-server' oleg109-server: oleg109-server.virtnet: executing load_modules_local oleg109-server: Loading modules from /home/green/git/lustre-release/lustre oleg109-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg109-server: detected 4 online CPUs by sysfs oleg109-server: MODOPTS_LIBCFS= oleg109-server: Force libcfs to create 2 CPU partitions Iteration 2 Starting client: oleg109-client.virtnet: -o user_xattr,flock 192.168.201.109@tcp:/lustre /mnt/lustre Starting client: oleg109-client.virtnet: -o user_xattr,flock 192.168.201.109@tcp:/lustre /mnt/lustre2 192.168.201.109@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.109@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg109-client.virtnet /mnt/lustre (opts:) Stopping client oleg109-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg109-server' oleg109-server: oleg109-server.virtnet: executing load_modules_local oleg109-server: Loading modules from /home/green/git/lustre-release/lustre oleg109-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg109-server: detected 4 online CPUs by sysfs oleg109-server: MODOPTS_LIBCFS= oleg109-server: Force libcfs to create 2 CPU partitions Starting client: oleg109-client.virtnet: -o user_xattr,flock 192.168.201.109@tcp:/lustre /mnt/lustre Starting client: oleg109-client.virtnet: -o user_xattr,flock 192.168.201.109@tcp:/lustre /mnt/lustre2 PASS 109 (34s) == sanityn test 110: do not grant another lock on resend ========================================================== 18:13:16 (1786486396) fail_loc=0x525 fail_val=3 fail_loc=0x534 File: /mnt/lustre/f110.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238860488707 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-11 18:13:17.000000000 -0400 Modify: 2026-08-11 18:13:17.000000000 -0400 Change: 2026-08-11 18:13:17.000000000 -0400 Birth: 2026-08-11 18:13:18.000000000 -0400 fail_loc=0 fail_val=0 Connected clients: oleg109-client.virtnet oleg109-client.virtnet PASS 110 (166s) == sanityn test 111: A racy rename/link an open file should not cause fs corruption ========================================================== 18:16:02 (1786486562) SKIP: sanityn test_111 needs >= 2 MDTs SKIP 111 (1s) == sanityn test 112: update max-inherit in default LMV === 18:16:03 (1786486563) SKIP: sanityn test_112 We need at least 2 MDTs for this test SKIP 112 (1s) == sanityn test 113: check servers of specified fs ======= 18:16:04 (1786486564) lustre-OST0000-osc-ffff9882c5c78800 active. lustre-OST0001-osc-ffff9882c5c78800 active. lustre-OST0000-osc-ffff9882c5c78800 active. lustre-OST0001-osc-ffff9882c5c78800 active. MGC192.168.201.109@tcp active. PASS 113 (3s) == sanityn test 114: implicit default LMV inherit ======== 18:16:07 (1786486567) SKIP: sanityn test_114 We need at least 2 MDTs for this test SKIP 114 (1s) == sanityn test 115: ldiskfs doesn't check direntry for uniqueness ========================================================== 18:16:08 (1786486568) SKIP: sanityn test_115 ldiskfs only test SKIP 115 (1s) == sanityn test 116: DNE: Set default LMV layout from a remote client ========================================================== 18:16:09 (1786486569) SKIP: sanityn test_116 needs >= 2 MDTs SKIP 116 (1s) == sanityn test 121: trunc append race =================== 18:16:10 (1786486570) 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0046419 s, 4.4 MB/s fail_loc=0x80001436 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0128878 s, 1.6 MB/s 0000000 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0005000 0000 0000 0000 0000 0000 0000 0000 0000 * 000c800 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0011800 PASS 121 (5s) == sanityn test 200: service remains healthy while able to process request ========================================================== 18:16:15 (1786486575) at_max=10 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=9 fail_loc=0x1e1 fail_val=20 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0829 s, 0.2 kB/s4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0878 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0847 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0942 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1009 s, 0.2 kB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0983 s, 0.2 kB/s 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0928 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1332 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1396 s, 0.1 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1559 s, 0.1 kB/s 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1624 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1797 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1768 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1703 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2348 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2371 s, 0.1 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2296 s, 0.1 kB/s 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2319 s, 0.1 kB/s at_max=600 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=116 oleg109-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9882c5c78800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9882c5c78800.ost_server_uuid in FULL state after 0 sec PASS 200 (67s) cleanup: ====================================================== == sanityn test complete, duration 6346 sec ============== 18:17:22 (1786486642) === sanityn: start cleanup 18:17:23 (1786486643) === Stopping clients: oleg109-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg109-client.virtnet /mnt/lustre2 opts: === sanityn: finish cleanup 18:18:11 (1786486691) ===