-----============= acceptance-small: sanityn ============----- Fri Sep 11 06:33:38 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg424-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanityn.*ex': No such file or directory excepting tests: 27 40a skipping tests SLOW=no: 33a === sanityn: start setup 06:34:04 (1789122844) === Starting client oleg424-client.virtnet: -o user_xattr,flock 192.168.204.124@tcp:/lustre /mnt/lustre2 Mount client oleg424-client.virtnet: mount -t lustre -o user_xattr,flock 192.168.204.124@tcp:/lustre /mnt/lustre2 Started clients oleg424-client.virtnet: 192.168.204.124@tcp:/lustre on /mnt/lustre2 type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) oleg424-client.virtnet: executing check_config_client /mnt/lustre oleg424-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg424-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8b688b34a000.idle_timeout=debug osc.lustre-OST0000-osc-ffff8b6890c8a800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8b688b34a000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8b6890c8a800.idle_timeout=debug disable quota as required oleg424-server: oleg424-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Setting lod.lustre-MDT0000-mdtlov.enable_compr_rotational from 0 to 1 Waiting 90s for '1' Updated after 2s: want '1' got '1' === sanityn: finish setup 06:34:57 (1789122897) === 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0271151 s, 38.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7649] == sanityn test 0: do_node_vp() and do_facet_vp() do the right thing ========================================================== 06:35:01 (1789122901) PASS 0 (9s) == sanityn test 1: Check attribute updates on 2 mount points ========================================================== 06:35:10 (1789122910) /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0777 OK /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0666 OK /mnt/lustre/f1.sanityn: absent OK PASS 1 (8s) == sanityn test 2a: check cached attribute updates on 2 mtpt's ================================================================== 06:35:19 (1789122919) -rw-r--r-- 1 root root 0 Sep 11 06:35 /mnt/lustre2/f2a /mnt/lustre/f2a has type file OK /mnt/lustre/f2a has perms 0777 OK PASS 2a (8s) == sanityn test 2b: check cached attribute updates on 2 mtpt's ================================================================== 06:35:28 (1789122928) -rw-r--r-- 1 root root 0 Sep 11 06:35 /mnt/lustre2/f2b /mnt/lustre2/f2b has type file OK /mnt/lustre2/f2b has perms 0777 OK PASS 2b (9s) == sanityn test 2c: check cached attribute updates on 2 mtpt's root ============================================================= 06:35:36 (1789122936) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0777 OK PASS 2c (8s) == sanityn test 2d: check cached attribute updates on 2 mtpt's root ============================================================= 06:35:44 (1789122944) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0755 OK PASS 2d (9s) == sanityn test 2e: check chmod on root is propagated to others ========================================================== 06:35:53 (1789122953) total 1 -rwxrwxrwx 1 root root 0 Sep 11 06:35 f2a -rwxrwxrwx 1 root root 0 Sep 11 06:35 f2b total 1 -rwxrwxrwx 1 root root 0 Sep 11 06:35 f2a -rwxrwxrwx 1 root root 0 Sep 11 06:35 f2b running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre2/f2e.sanityn] [count=1] 1+0 records in 1+0 records out 512 bytes copied, 0.0131924 s, 38.8 kB/s PASS 2e (9s) == sanityn test 2f: check attr/owner updates on DNE with 2 mtpt's ========================================================== 06:36:02 (1789122962) SKIP: sanityn test_2f needs >= 2 MDTs SKIP 2f (3s) == sanityn test 2g: check blocks update on sync write ==== 06:36:05 (1789122965) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.574137 s, 3.7 MB/s /mnt/lustre/f2g.sanityn has 4103 blocks /mnt/lustre2/f2g.sanityn has 4103 blocks PASS 2g (8s) == sanityn test 3: symlink on one mtpt, readlink on another ===================================================================== 06:36:13 (1789122973) PASS 3 (8s) == sanityn test 4: fstat validation on multiple mount points ==================================================================== 06:36:22 (1789122982) PASS 4 (8s) == sanityn test 5: create a file on one mount, truncate it on the other ========================================================== 06:36:30 (1789122990) /mnt/lustre/f5 has type file OK /mnt/lustre/f5 has size 100 OK PASS 5 (9s) == sanityn test 6: remove of open file on other node ============================================================================ 06:36:39 (1789122999) opening writing unlinking /mnt/lustre2/f6.sanityn accessing (1) seeking (1) accessing (2) fstat... reading comparing data truncating seeking (2) writing again seeking (3) reading again comparing data again closing SUCCESS - goto beer PASS 6 (10s) == sanityn test 7: remove of open directory on other node ======================================================================= 06:36:49 (1789123009) creating directory /mnt/lustre/d7 opening directory unlinking /mnt/lustre/d7 Ok, everything goes well. PASS 7 (8s) == sanityn test 8: remove of open special file on other node ==================================================================== 06:36:57 (1789123017) creating special file /mnt/lustre/f8.sanityn opening file unlinking /mnt/lustre/f8.sanityn Ok, everything goes well. PASS 8 (8s) == sanityn test 9a: append of file with sub-page size on multiple mounts ========================================================== 06:37:05 (1789123025) PASS 9a (10s) == sanityn test 9b: append to striped sparse file ======== 06:37:15 (1789123035) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0601016 s, 17.4 MB/s 3+0 records in 3+0 records out 3 bytes copied, 0.0446599 s, 0.1 kB/s Data read (expecting 'foo'): foo PASS 9b (8s) == sanityn test 10a: write of file with sub-page size on multiple mounts ========================================================== 06:37:23 (1789123043) 1+0 records in 1+0 records out 1 byte copied, 0.0222688 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0116091 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0170612 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0267584 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0130789 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0163867 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0201657 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0195928 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0107731 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.011762 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0127619 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0257105 s, 0.0 kB/s PASS 10a (9s) == sanityn test 10b: write of file with sub-page size on multiple mounts ========================================================== 06:37:32 (1789123052) 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.0179179 s, 171 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0512775 s, 79.9 kB/s 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.00517921 s, 593 kB/s PASS 10b (9s) == sanityn test 11: execution of file opened for write should return error ============================================================== 06:37:41 (1789123061) multiop /mnt/lustre/d11/f vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7649 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 305: /mnt/lustre2/d11/f: Text file busy PASS 11 (8s) == sanityn test 12: test lock ordering (link, stat, unlink) ========================================================== 06:37:49 (1789123069) warning: '-runtime' deprecated, use '-t runtime' instead - link 294 (time 1789123082.36 total 10.02 last 29.35) - link 566 (time 1789123092.41 total 20.07 last 27.06) - link 819 (time 1789123102.45 total 30.10 last 25.22) - link 1082 (time 1789123112.46 total 40.12 last 26.26) - link 1334 (time 1789123122.50 total 50.16 last 25.10) - link 1602 (time 1789123132.53 total 60.18 last 26.73) - link 1900 (time 1789123142.55 total 70.21 last 29.73) - link 2198 (time 1789123152.56 total 80.22 last 29.77) - link 2473 (time 1789123162.58 total 90.23 last 27.46) total: 2736 link in 100.02 seconds: 27.35 ops/second using seed 2087728888 running for 100 seconds - stat 4510 (time 1789123084 ; total 11 ; last 11) - stat 8828 (time 1789123095 ; total 22 ; last 11) - stat 10000 (time 1789123098 ; total 25 ; last 3) - stat 14042 (time 1789123109 ; total 36 ; last 11) - stat 18532 (time 1789123120 ; total 47 ; last 11) - stat 20000 (time 1789123124 ; total 51 ; last 4) - stat 22360 (time 1789123135 ; total 62 ; last 11) - stat 24763 (time 1789123146 ; total 73 ; last 11) - stat 27120 (time 1789123157 ; total 84 ; last 11) - stat 29288 (time 1789123168 ; total 95 ; last 11) - stat 30000 (time 1789123171 ; total 98 ; last 3) total: 30363 stats in 100 seconds: 303.630005 stats/second - unlinked 0 (time 1789123123 ; total 0 ; last 0) unlink(/mnt/lustre2/lockdir/lockfile2736) error: No such file or directory total: 2735 unlinks in 87 seconds: 31.436781 unlinks/second /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 77: kill: (21669) - No such process /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 78: kill: (21671) - No such process PASS 12 (147s) == sanityn test 13: test directory page revocation ======= 06:40:16 (1789123216) f13 f13 PASS 13 (10s) == sanityn test 14aa: execution of file open for write returns -ETXTBSY ========================================================== 06:40:26 (1789123226) multiop /mnt/lustre/d14aa.sanityn/f14aa.sanityn vOw_c TMPPIPE=/tmp/multiop_open_wait_pipe.7649 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 340: /mnt/lustre2/d14aa.sanityn/f14aa.sanityn: Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5142: 22857 Terminated $MULTIOP_PROG $FILE v$ARGS > $TMPPIPE (wd: ~) PASS 14aa (8s) == sanityn test 14ab: open(RDWR) of executing file returns -ETXTBSY ========================================================== 06:40:34 (1789123234) open(O_RDWR|O_CREAT): Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5142: 23438 Terminated $DIR1/$tdir/sleep 60 (wd: ~) PASS 14ab (8s) == sanityn test 14b: truncate of executing file returns -ETXTBSY ================================================================ 06:40:42 (1789123242) truncate: cannot truncate '/mnt/lustre2/d14b.sanityn/sleep' to length 60: Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 355: 24018 Terminated $DIR1/$tdir/sleep 60 PASS 14b (7s) == sanityn test 14c: open(O_TRUNC) of executing file return -ETXTBSY ============================================================ 06:40:50 (1789123250) cp: cannot create regular file '/mnt/lustre2/d14c.sanityn/sleep': Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 367: 24600 Terminated $DIR1/$tdir/sleep 60 PASS 14c (10s) == sanityn test 14d: chmod of executing file is still possible ================================================================== 06:40:59 (1789123259) chmod /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 378: 25183 Terminated $DIR1/$tdir/sleep 60 PASS 14d (13s) == sanityn test 15: test out-of-space with multiple writers ===================================================================== 06:41:12 (1789123272) PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin: MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky sleep 5 for ZFS MDS Waiting for MDT destroys to complete fallocate on zfs doesn't consume space STRIPECOUNT=2 ORIGFREE=7531520 MAXFREE=800000 SKIP: oos2 test_15 oos2.sh: 7531520KB free > 800000KB, increase MAXFREE (or reduce fs size) sleep 5 for ZFS MDS Waiting for MDT destroys to complete SKIP 15 (57s) == sanityn test 16a: 500 iterations of dual-mount fsx ==== 06:42:09 (1789123329) fallocate on zfs doesn't consume space fallocate not supported 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.712108 s, 11.8 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.436323 s, 19.2 MB/s Chance of close/open is 1 in 50 Seed set to 3334 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read skipping zero size read skipping zero size read skipping zero size read skipping zero size read skipping zero size read truncating to largest ever: 0x66d77a 000014[0] 1789123335.478023 trunc from 00000000 to 0x66d779 (0x66d77a bytes) 000015[1] 1789123335.505759 read 0x1bc88b thru 0x1c7a6c (0xb1e2 bytes) 000021[1] 1789123335.587153 read 0x44aa30 thru 0x4570e9 (0xc6ba bytes) 000022[1] 1789123335.613626 mapread 0x361ab3 thru 0x369049 (0x7597 bytes) 000030[0] 1789123335.647197 write 0x2f737a thru 0x2f9265 (0x1eec bytes) 000031[0] 1789123335.678196 read 0x523684 thru 0x52a70d (0x708a bytes) 000033[0] 1789123335.687380 mapread 0x14f40d thru 0x1507a8 (0x139c bytes) truncating to largest ever: 0x7bfd80 000041[0] 1789123335.784850 trunc from 0x66d77a to 0x7bfd7f (0x152606 bytes) 000043[1] 1789123335.893683 mapread 0x1db2e6 thru 0x1e0a77 (0x5792 bytes) 000044[1] 1789123335.954570 mapread 0x294fdf thru 0x2a114a (0xc16c bytes) 000045[0] 1789123335.991299 read 0x41e291 thru 0x424dc9 (0x6b39 bytes) 000048[1] 1789123336.004824 mapread 0x5f4294 thru 0x5f7e9b (0x3c08 bytes) truncating to largest ever: 0x81b32a 000049[1] 1789123336.099719 trunc from 0x7bfd80 to 0x81b329 (0x5b5aa bytes) 000050[0] 1789123336.165200 write 0x79bcf9 thru 0x79d56a (0x1872 bytes) 000052[0] 1789123336.282144 mapread 0x7efb82 thru 0x7fda03 (0xde82 bytes) 000056[1] 1789123336.412496 read 0x061408 thru 0x067426 (0x601f bytes) 000057[0] 1789123336.438274 mapwrite 0x5c90b4 thru 0x5d7358 (0xe2a5 bytes) 000060[0] 1789123336.587082 mapread 0x211c3e thru 0x21d5e1 (0xb9a4 bytes) 000062[1] 1789123336.658602 mapwrite 0x63b722 thru 0x6467f0 (0xb0cf bytes) 000065[0] 1789123336.821971 mapwrite 0x4e6f90 thru 0x4e742a (0x49b bytes) 000069[1] 1789123336.944995 mapread 0x2971e7 thru 0x29ceaa (0x5cc4 bytes) 000070[0] 1789123336.983147 mapwrite 0x26f0af thru 0x278a19 (0x996b bytes) 000072[0] 1789123337.149255 write 0x065175 thru 0x07460e (0xf49a bytes) 000073[0] 1789123337.166641 trunc from 0x81b32a to 0x66a8d2 (0x1b0a57 bytes) 000078[0] 1789123337.265373 mapread 0x352bef thru 0x355e37 (0x3249 bytes) 000079[1] 1789123337.327166 mapwrite 0x054967 thru 0x059ce4 (0x537e bytes) 000081[0] 1789123337.666416 mapread 0x23b290 thru 0x240252 (0x4fc3 bytes) 000082[0] 1789123337.750811 read 0x3c7126 thru 0x3d2181 (0xb05c bytes) 000083[0] 1789123337.810607 write 0x81df70 thru 0x81f8e3 (0x1974 bytes) 000084[1] 1789123337.917153 mapwrite 0x19d833 thru 0x19f545 (0x1d13 bytes) 000085[0] 1789123338.165235 mapread 0x4b1dda thru 0x4b23ba (0x5e1 bytes) 000086[1] 1789123338.252525 read 0x1914de thru 0x194b14 (0x3637 bytes) 000089[0] 1789123338.327665 read 0x3f0299 thru 0x3fdc40 (0xd9a8 bytes) 000091[0] 1789123338.382009 read 0x512c16 thru 0x5226a7 (0xfa92 bytes) 000096[1] 1789123338.441323 mapread 0x4ad6c9 thru 0x4b40d2 (0x6a0a bytes) 000099[0] 1789123338.499820 trunc from 0x81f8e4 to 0x137d09 (0x6e7bda bytes) 000102[1] 1789123338.556064 mapwrite 0x187bb2 thru 0x18df3e (0x638d bytes) 000103[0] 1789123338.697591 read 0x0566ee thru 0x05dfce (0x78e1 bytes) 000104[1] 1789123338.828831 write 0x9e4fae thru 0x9ef20f (0xa262 bytes) 000107[0] 1789123338.924926 mapwrite 0x919775 thru 0x921f2f (0x87bb bytes) 000108[1] 1789123339.148530 mapwrite 0x865474 thru 0x874f89 (0xfb16 bytes) 000114[1] 1789123339.480740 trunc from 0x9ef210 to 0x5d1b52 (0x41d6bd bytes) 000117[1] 1789123339.633691 trunc from 0x5d1b53 to 0x4dab46 (0xf700c bytes) 000118[1] 1789123339.681632 trunc from 0x4dab47 to 0x2ddbc3 (0x1fcf83 bytes) 000119[1] 1789123339.723229 write 0x96d1f7 thru 0x976ed8 (0x9ce2 bytes) 000120[1] 1789123339.736376 mapwrite 0x91215f thru 0x91839e (0x6240 bytes) 000122[0] 1789123339.872176 mapread 0x67ce95 thru 0x6808d1 (0x3a3d bytes) 000123[1] 1789123340.067699 write 0x2b1c4a thru 0x2b9ce3 (0x809a bytes) 000127[1] 1789123340.184279 read 0x891958 thru 0x89811c (0x67c5 bytes) 000129[0] 1789123340.225181 mapread 0x47cbe5 thru 0x484a4f (0x7e6b bytes) 000130[0] 1789123340.360725 mapwrite 0x64605d thru 0x646910 (0x8b4 bytes) 000131[0] 1789123340.530176 mapwrite 0x93bf47 thru 0x93c8ed (0x9a7 bytes) 000132[1] 1789123340.772300 read 0x5588cf thru 0x55bcbe (0x33f0 bytes) 000134[0] 1789123340.840882 mapread 0x859fd1 thru 0x85d3be (0x33ee bytes) 000139[1] 1789123340.930986 read 0x337670 thru 0x340ba3 (0x9534 bytes) 000140[0] 1789123340.969171 trunc from 0x976ed9 to 0x308ff7 (0x66dee1 bytes) 000141[1] 1789123341.134160 write 0x1bb42a thru 0x1c8695 (0xd26c bytes) 000143[1] 1789123341.290895 write 0x723b05 thru 0x72d494 (0x9990 bytes) 000145[1] 1789123341.340742 read 0x474f19 thru 0x4832c7 (0xe3af bytes) 000147[0] 1789123341.431537 write 0x9e7acf thru 0x9e9429 (0x195b bytes) 000149[0] 1789123341.565038 trunc from 0x9e942a to 0x367a9e (0x68198b bytes) 000153[0] 1789123341.735917 mapwrite 0x73c0b8 thru 0x743188 (0x70d1 bytes) 000157[1] 1789123342.103899 read 0x0dd1cd thru 0x0e2e96 (0x5cca bytes) 000159[0] 1789123342.211316 write 0x38054d thru 0x3821c3 (0x1c77 bytes) 000161[1] 1789123342.306292 write 0x676d28 thru 0x681a79 (0xad52 bytes) 000163[0] 1789123342.429505 trunc from 0x743189 to 0x3683ca (0x3dadbe bytes) 000167[1] 1789123342.563396 trunc from 0x3683cb to 0x643431 (0x2db067 bytes) 000168[0] 1789123342.614654 trunc from 0x643432 to 0x470c7d (0x1d27b4 bytes) 000169[1] 1789123342.670023 mapwrite 0x56dbae thru 0x579761 (0xbbb4 bytes) 000170[1] 1789123342.817480 trunc from 0x579762 to 0x0bb718 (0x4be049 bytes) 000171[1] 1789123342.895404 read 0x0045c5 thru 0x004861 (0x29d bytes) 000173[1] 1789123342.980974 mapread 0x03a44c thru 0x043358 (0x8f0d bytes) 000176[0] 1789123343.074004 read 0x08fd88 thru 0x09be3f (0xc0b8 bytes) 000177[0] 1789123343.124509 trunc from 0x0bb719 to 0x55aa9e (0x49f386 bytes) 000179[0] 1789123343.283128 write 0x15497f thru 0x161dd6 (0xd458 bytes) 000180[1] 1789123343.393271 mapread 0x14777d thru 0x157466 (0xfcea bytes) 000181[1] 1789123343.681874 write 0x2dda5f thru 0x2ea987 (0xcf29 bytes) 000185[1] 1789123343.902078 write 0x8dca4f thru 0x8e04e0 (0x3a92 bytes) 000186[1] 1789123343.949834 read 0x4b8276 thru 0x4c3ce4 (0xba6f bytes) 000187[1] 1789123344.013586 mapread 0x024fb3 thru 0x0319d1 (0xca1f bytes) 000190[0] 1789123344.079042 read 0x06a0d5 thru 0x07312c (0x9058 bytes) 000193[0] 1789123344.132484 trunc from 0x8e04e1 to 0x72db15 (0x1b29cb bytes) 000195[0] 1789123344.249489 write 0x0f12d0 thru 0x0fc2b9 (0xafea bytes) 000196[1] 1789123344.279583 write 0x4bd184 thru 0x4c87df (0xb65c bytes) 000198[0] 1789123344.429267 mapwrite 0x67e18a thru 0x6822e5 (0x415c bytes) 000200[1] 1789123344.626395 mapread 0x190658 thru 0x19ea0e (0xe3b7 bytes) 000201[0] 1789123344.718943 read 0x3397da thru 0x343eef (0xa716 bytes) 000203[1] 1789123344.749638 write 0x580140 thru 0x58a492 (0xa353 bytes) 000206[1] 1789123344.873792 trunc from 0x72db16 to 0x2ce11b (0x45f9fa bytes) 000208[0] 1789123344.929884 write 0x362714 thru 0x364c57 (0x2544 bytes) 000210[1] 1789123344.996665 mapwrite 0x42910e thru 0x4292d4 (0x1c7 bytes) 000211[1] 1789123345.108645 mapwrite 0x44c708 thru 0x4561d5 (0x9ace bytes) 000213[1] 1789123345.319413 trunc from 0x4561d6 to 0x769e58 (0x313c83 bytes) 000214[1] 1789123345.361355 read 0x4dff45 thru 0x4ef334 (0xf3f0 bytes) 000216[0] 1789123345.435338 write 0x996c59 thru 0x9a058e (0x9936 bytes) 000219[1] 1789123345.502559 mapread 0x34ffb6 thru 0x352940 (0x298b bytes) 000220[0] 1789123345.578735 write 0x39b572 thru 0x3a52e8 (0x9d77 bytes) 000222[1] 1789123345.657549 mapread 0x12d144 thru 0x12e588 (0x1445 bytes) 000225[1] 1789123345.714999 trunc from 0x9a058f to 0x6c9064 (0x2d752a bytes) 000226[1] 1789123345.840892 mapwrite 0x04551d thru 0x04c51b (0x6fff bytes) 000227[0] 1789123346.079454 write 0x6a27f3 thru 0x6a9af3 (0x7301 bytes) 000230[1] 1789123346.169181 trunc from 0x6c9065 to 0x2f62bd (0x3d2da7 bytes) 000231[1] 1789123346.251052 write 0x5d83f1 thru 0x5ded3d (0x694d bytes) 000233[1] 1789123346.272714 read 0x45da9d thru 0x46aedc (0xd440 bytes) 000234[1] 1789123346.333302 write 0x440600 thru 0x446062 (0x5a63 bytes) 000235[0] 1789123346.382881 mapread 0x5d5ccb thru 0x5d8c94 (0x2fca bytes) 000236[1] 1789123346.453074 mapwrite 0x1ab9a5 thru 0x1abcdb (0x337 bytes) 000237[1] 1789123346.559044 trunc from 0x5ded3e to 0x142344 (0x49c9f9 bytes) 000239[1] 1789123346.647357 mapwrite 0x1dcdfc thru 0x1e3099 (0x629e bytes) 000240[0] 1789123346.793930 read 0x02e1bb thru 0x034eff (0x6d45 bytes) 000241[1] 1789123346.890102 mapread 0x09b8c1 thru 0x0a9c95 (0xe3d5 bytes) 000242[0] 1789123346.942648 mapwrite 0x0549fc thru 0x05751d (0x2b22 bytes) 000244[1] 1789123347.182590 trunc from 0x1e309a to 0x0600f8 (0x182fa1 bytes) 000247[1] 1789123347.269377 trunc from 0x0600f9 to 0x4cd5cf (0x46d4d7 bytes) 000249[0] 1789123347.320071 write 0x67031c thru 0x680188 (0xfe6d bytes) 000254[0] 1789123347.453227 mapread 0x0ab03b thru 0x0b2385 (0x734b bytes) 000255[1] 1789123347.630068 trunc from 0x680189 to 0x1c3437 (0x4bcd51 bytes) 000256[1] 1789123347.704586 write 0x7efc14 thru 0x7f5df7 (0x61e4 bytes) 000260[1] 1789123347.740358 trunc from 0x7f5df8 to 0x527494 (0x2ce963 bytes) 000261[0] 1789123347.799064 write 0x8b4a08 thru 0x8c17d2 (0xcdcb bytes) 000263[0] 1789123347.870886 mapread 0x421084 thru 0x42a0fc (0x9079 bytes) 000264[0] 1789123347.967475 mapread 0x3e6056 thru 0x3e8fea (0x2f95 bytes) 000266[1] 1789123348.027026 trunc from 0x8c17d3 to 0x131f26 (0x78f8ac bytes) 000268[1] 1789123348.132734 trunc from 0x131f27 to 0x4f27ea (0x3c08c4 bytes) 000270[0] 1789123348.221617 write 0x041fd0 thru 0x049e9a (0x7ecb bytes) 000274[1] 1789123348.382675 mapread 0x322221 thru 0x32f16b (0xcf4b bytes) 000275[0] 1789123348.519551 read 0x46f99f thru 0x477462 (0x7ac4 bytes) 000278[1] 1789123348.602029 mapread 0x473457 thru 0x47d11a (0x9cc4 bytes) 000279[0] 1789123348.639333 mapwrite 0x3b3fd2 thru 0x3b76fa (0x3729 bytes) 000284[0] 1789123348.788813 write 0x35f38b thru 0x3630fe (0x3d74 bytes) 000286[0] 1789123348.808612 read 0x335211 thru 0x343eda (0xecca bytes) 000287[0] 1789123348.824505 mapread 0x15f297 thru 0x163f74 (0x4cde bytes) 000288[0] 1789123348.864716 mapread 0x1971a7 thru 0x19e8ff (0x7759 bytes) 000290[0] 1789123348.906404 read 0x4bd672 thru 0x4c3b34 (0x64c3 bytes) truncating to largest ever: 0x8abd62 000293[1] 1789123348.955667 trunc from 0x4f27eb to 0x8abd61 (0x3b9577 bytes) 000294[0] 1789123349.027032 read 0x005d67 thru 0x01220a (0xc4a4 bytes) 000295[1] 1789123349.056770 read 0x39676d thru 0x39cfdc (0x6870 bytes) 000296[1] 1789123349.067266 mapwrite 0x6ff23e thru 0x70ce42 (0xdc05 bytes) 000297[0] 1789123349.226357 read 0x549cc9 thru 0x54a8af (0xbe7 bytes) 000298[1] 1789123349.271690 write 0x704dae thru 0x70bb7a (0x6dcd bytes) 000299[1] 1789123349.299668 write 0x2104d3 thru 0x21200d (0x1b3b bytes) 000300[0] 1789123349.334095 read 0x3b6408 thru 0x3bb843 (0x543c bytes) 000301[1] 1789123349.381690 mapwrite 0x0db3ef thru 0x0de7c8 (0x33da bytes) 000302[1] 1789123349.491859 mapread 0x052032 thru 0x05fb43 (0xdb12 bytes) 000303[0] 1789123349.513986 mapwrite 0x9b97dc thru 0x9c973c (0xff61 bytes) 000304[0] 1789123349.647223 trunc from 0x9c973d to 0x2c068c (0x7090b0 bytes) 000308[0] 1789123349.735261 mapwrite 0x2484b0 thru 0x25602b (0xdb7c bytes) 000309[1] 1789123349.895296 read 0x0f1c6e thru 0x0f660b (0x499e bytes) 000310[0] 1789123350.066402 mapwrite 0x08702e thru 0x09303a (0xc00d bytes) 000312[1] 1789123350.364567 read 0x034ebd thru 0x037d3e (0x2e82 bytes) 000313[1] 1789123350.513299 write 0x10a3cf thru 0x1105cb (0x61fd bytes) 000314[0] 1789123350.628329 read 0x10b9f3 thru 0x10d556 (0x1b64 bytes) 000317[1] 1789123350.888537 mapread 0x0d3d2f thru 0x0d8271 (0x4543 bytes) 000318[1] 1789123350.993270 trunc from 0x2c068d to 0x66e815 (0x3ae189 bytes) 000322[1] 1789123351.277337 read 0x352941 thru 0x357095 (0x4755 bytes) 000323[1] 1789123351.333226 read 0x63772c thru 0x64090d (0x91e2 bytes) 000324[1] 1789123351.357579 mapread 0x127054 thru 0x130fe4 (0x9f91 bytes) 000325[0] 1789123351.407628 write 0x06f8e2 thru 0x078659 (0x8d78 bytes) 000326[0] 1789123351.541849 mapwrite 0x8f8017 thru 0x903c22 (0xbc0c bytes) 000327[0] 1789123351.813442 write 0x280fad thru 0x28cd81 (0xbdd5 bytes) 000328[0] 1789123351.838514 read 0x12a634 thru 0x1349e7 (0xa3b4 bytes) 000330[0] 1789123351.853956 read 0x868255 thru 0x87504a (0xcdf6 bytes) 000331[1] 1789123351.911212 write 0x7e9435 thru 0x7eedb0 (0x597c bytes) 000338[0] 1789123352.136268 write 0x86a87d thru 0x875ac4 (0xb248 bytes) 000339[0] 1789123352.154562 mapwrite 0x5dad32 thru 0x5dc1fd (0x14cc bytes) 000340[1] 1789123352.285714 read 0x7bf632 thru 0x7cb829 (0xc1f8 bytes) 000341[1] 1789123352.337915 mapwrite 0x2607b2 thru 0x26b912 (0xb161 bytes) 000342[1] 1789123352.484808 write 0x7b23b1 thru 0x7be95a (0xc5aa bytes) 000344[0] 1789123352.516518 write 0x2552e6 thru 0x25791c (0x2637 bytes) 000349[1] 1789123352.705641 mapwrite 0x19e50a thru 0x1ad65c (0xf153 bytes) 000353[0] 1789123353.032824 trunc from 0x903c23 to 0x456e31 (0x4acdf1 bytes) 000355[0] 1789123353.187798 mapread 0x284aa5 thru 0x28cd24 (0x8280 bytes) 000362[0] 1789123353.330854 mapwrite 0x9cb8e5 thru 0x9da040 (0xe75c bytes) 000364[1] 1789123353.515402 trunc from 0x9da041 to 0x09a7f1 (0x93f84f bytes) 000368[1] 1789123353.638915 mapwrite 0x0aa78c thru 0x0b680a (0xc07f bytes) 000369[1] 1789123353.807606 trunc from 0x0b680b to 0x88781b (0x7d1011 bytes) 000370[1] 1789123353.858585 trunc from 0x88781c to 0x1057d9 (0x782042 bytes) 000371[1] 1789123353.897599 read 0x0539b7 thru 0x05d1ac (0x97f6 bytes) 000372[0] 1789123353.899115 read 0x0adaf3 thru 0x0af03b (0x1549 bytes) 000373[0] 1789123353.953995 write 0x417f61 thru 0x425646 (0xd6e6 bytes) 000376[0] 1789123354.054386 read 0x1821a5 thru 0x1843fc (0x2258 bytes) 000377[1] 1789123354.117592 read 0x231556 thru 0x23a1e1 (0x8c8c bytes) 000378[1] 1789123354.196586 mapread 0x22bfe4 thru 0x2366d9 (0xa6f6 bytes) 000380[1] 1789123354.401937 write 0x7e4d2d thru 0x7f4410 (0xf6e4 bytes) 000381[1] 1789123354.473672 read 0x0c83da thru 0x0d4272 (0xbe99 bytes) 000382[0] 1789123354.492149 mapread 0x206bfa thru 0x212555 (0xb95c bytes) 000384[0] 1789123354.759601 trunc from 0x7f4411 to 0x3a499c (0x44fa74 bytes) 000385[1] 1789123354.882661 read 0x1072f6 thru 0x11426b (0xcf76 bytes) 000386[0] 1789123355.000175 read 0x29be4d thru 0x29f19b (0x334f bytes) 000389[0] 1789123355.118345 read 0x1ccd7c thru 0x1d187b (0x4b00 bytes) 000390[0] 1789123355.118947 read 0x1b74e1 thru 0x1bf394 (0x7eb4 bytes) 000392[0] 1789123355.171041 trunc from 0x3a499d to 0x5da947 (0x235fab bytes) 000394[1] 1789123355.319536 trunc from 0x5da948 to 0x808933 (0x22dfec bytes) 000396[0] 1789123355.407488 read 0x5bba4a thru 0x5c00b3 (0x466a bytes) 000397[1] 1789123355.485670 mapread 0x03875a thru 0x0433b5 (0xac5c bytes) 000398[0] 1789123355.567849 trunc from 0x808934 to 0x7c9040 (0x3f8f3 bytes) 000400[0] 1789123355.625581 mapread 0x3135b5 thru 0x31b742 (0x818e bytes) 000403[0] 1789123355.676953 read 0x644966 thru 0x649a8c (0x5127 bytes) 000404[0] 1789123355.689372 mapread 0x0a8c38 thru 0x0b143e (0x8807 bytes) 000405[0] 1789123355.759316 write 0x6248bc thru 0x627d30 (0x3475 bytes) 000410[0] 1789123355.829454 trunc from 0x7c9041 to 0x7f6da5 (0x2dd65 bytes) 000411[0] 1789123355.863754 trunc from 0x7f6da6 to 0x3fafcc (0x3fbdd9 bytes) 000418[0] 1789123356.114655 mapread 0x3767eb thru 0x377a32 (0x1248 bytes) 000419[0] 1789123356.186965 mapread 0x39f533 thru 0x3abde0 (0xc8ae bytes) 000421[1] 1789123356.302310 mapwrite 0x5260ad thru 0x5346f5 (0xe649 bytes) 000427[1] 1789123356.617370 read 0x06258b thru 0x06754b (0x4fc1 bytes) 000429[0] 1789123356.626336 read 0x28ce5c thru 0x2937c3 (0x6968 bytes) 000433[0] 1789123356.666153 trunc from 0x5346f6 to 0x5013d2 (0x33323 bytes) 000434[0] 1789123356.744001 mapread 0x1cfaff thru 0x1d3a53 (0x3f55 bytes) 000435[0] 1789123356.774835 mapwrite 0x261fc2 thru 0x262718 (0x757 bytes) 000436[1] 1789123356.971957 mapwrite 0x65ac03 thru 0x65d3db (0x27d9 bytes) 000438[1] 1789123357.222662 trunc from 0x65d3dc to 0x77ab60 (0x11d785 bytes) 000439[1] 1789123357.296207 write 0x30dadf thru 0x3162c6 (0x87e8 bytes) 000442[0] 1789123357.407116 mapread 0x7790ec thru 0x77ab60 (0x1a75 bytes) 000443[1] 1789123357.475827 mapwrite 0x92db0c thru 0x930546 (0x2a3b bytes) 000444[0] 1789123357.606873 read 0x6da9d1 thru 0x6db2c0 (0x8f0 bytes) 000447[0] 1789123357.714802 mapread 0x54568e thru 0x54bd70 (0x66e3 bytes) 000448[0] 1789123357.777839 write 0x2862b9 thru 0x28ba10 (0x5758 bytes) 000449[0] 1789123357.824861 mapread 0x77e5ce thru 0x784d87 (0x67ba bytes) 000451[0] 1789123357.869151 write 0x41162a thru 0x41e39b (0xcd72 bytes) 000452[1] 1789123357.926204 trunc from 0x930547 to 0x471027 (0x4bf51f bytes) 000455[0] 1789123358.069296 write 0x0cbba9 thru 0x0d2360 (0x67b8 bytes) 000458[1] 1789123358.160090 mapread 0x1c20aa thru 0x1c982f (0x7786 bytes) 000459[0] 1789123358.227307 mapwrite 0x932512 thru 0x935377 (0x2e66 bytes) 000463[0] 1789123358.406705 trunc from 0x935378 to 0x41b537 (0x519e40 bytes) 000464[1] 1789123358.443038 mapread 0x123d9a thru 0x12f776 (0xb9dd bytes) 000465[1] 1789123358.515550 write 0x401a1b thru 0x40670b (0x4cf1 bytes) 000467[1] 1789123358.565298 write 0x3e678b thru 0x3e6bf7 (0x46d bytes) 000469[1] 1789123358.669039 read 0x25c23d thru 0x26607d (0x9e41 bytes) 000470[0] 1789123358.677456 read 0x1ca2a2 thru 0x1d4289 (0x9fe8 bytes) 000472[1] 1789123358.726393 trunc from 0x41b538 to 0x78a2ed (0x36edb6 bytes) 000474[0] 1789123358.815323 write 0x4b5d6b thru 0x4beedd (0x9173 bytes) 000479[1] 1789123358.975832 trunc from 0x78a2ee to 0x39c00d (0x3ee2e0 bytes) 000482[1] 1789123359.070389 trunc from 0x39c00e to 0x5d60a2 (0x23a095 bytes) 000483[1] 1789123359.101631 trunc from 0x5d60a3 to 0x33ba12 (0x29a690 bytes) 000484[1] 1789123359.121195 read 0x3312d6 thru 0x3341f1 (0x2f1c bytes) 000492[1] 1789123359.441210 mapread 0x305d88 thru 0x308979 (0x2bf2 bytes) 000493[1] 1789123359.444095 write 0x649e2d thru 0x659e0a (0xffde bytes) 000498[0] 1789123359.543056 read 0x37f051 thru 0x38344e (0x43fe bytes) 000499[0] 1789123359.606075 mapwrite 0x3a4789 thru 0x3acdbf (0x8637 bytes) All operations completed A-OK! Chance of close/open is 1 in 50 Seed set to 3360 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000001[0] 1789123361.295370 write_OD 0x8a1000 thru 0x8a7fff (0x7000 bytes) 000002[1] 1789123361.395303 write_OD 0x95b000 thru 0x966fff (0xc000 bytes) 000003[0] 1789123361.467280 write_OD 0x556000 thru 0x562fff (0xd000 bytes) 000006[1] 1789123361.533789 mapread 0x875000 thru 0x87d2e6 (0x82e7 bytes) truncating to largest ever: 0x39d350 000010[1] 1789123361.561075 trunc from 0x967000 to 0x39d34f (0x5c9cb0 bytes) 000011[1] 1789123361.613994 write_OD 0x8b0000 thru 0x8b2fff (0x3000 bytes) 000012[0] 1789123361.667282 mapread 0x2b2000 thru 0x2c1ea4 (0xfea5 bytes) 000013[1] 1789123361.707393 write_OD 0x844000 thru 0x850fff (0xd000 bytes) truncating to largest ever: 0x5fb2b7 000015[1] 1789123361.773263 trunc from 0x8b3000 to 0x5fb2b6 (0x2b7d49 bytes) 000016[0] 1789123361.818028 mapread 0x374000 thru 0x377b1b (0x3b1c bytes) 000019[1] 1789123361.866366 write_OD 0x5fd000 thru 0x5fffff (0x3000 bytes) 000020[0] 1789123361.930363 mapwrite 0x385000 thru 0x3869a8 (0x19a9 bytes) 000021[0] 1789123362.056855 trunc from 0x600000 to 0x56aeba (0x95145 bytes) 000025[0] 1789123362.163739 mapwrite 0x3f4000 thru 0x3feea7 (0xaea8 bytes) 000026[0] 1789123362.351417 read_OD 0x37d000 thru 0x388fff (0xc000 bytes) 000027[1] 1789123362.416838 write_OD 0x595000 thru 0x595fff (0x1000 bytes) 000031[1] 1789123362.498190 read_OD 0x4d7000 thru 0x4e1fff (0xb000 bytes) 000036[1] 1789123362.527453 mapwrite 0x729000 thru 0x735e1c (0xce1d bytes) 000037[0] 1789123362.734523 mapread 0x1bb000 thru 0x1c71ba (0xc1bb bytes) 000038[0] 1789123362.844504 mapwrite 0x2a5000 thru 0x2afb5c (0xab5d bytes) 000039[1] 1789123362.962031 read_OD 0x01d000 thru 0x024fff (0x8000 bytes) 000040[0] 1789123363.004312 read_OD 0x2d9000 thru 0x2dbfff (0x3000 bytes) 000043[1] 1789123363.039315 write_OD 0x8e5000 thru 0x8effff (0xb000 bytes) 000046[1] 1789123363.139262 read_OD 0x186000 thru 0x18dfff (0x8000 bytes) 000047[1] 1789123363.149761 read_OD 0x4f5000 thru 0x4fffff (0xb000 bytes) 000048[1] 1789123363.160256 mapwrite 0x399000 thru 0x39eccc (0x5ccd bytes) 000050[1] 1789123363.278318 read_OD 0x4c4000 thru 0x4ccfff (0x9000 bytes) truncating to largest ever: 0x6dba4b 000051[1] 1789123363.318397 trunc from 0x8f0000 to 0x6dba4a (0x2145b5 bytes) 000054[1] 1789123363.408768 write_OD 0x6dc000 thru 0x6e1fff (0x6000 bytes) 000057[1] 1789123363.488405 read_OD 0x06b000 thru 0x074fff (0xa000 bytes) 000059[1] 1789123363.501677 write_OD 0x308000 thru 0x312fff (0xb000 bytes) 000060[1] 1789123363.620616 write_OD 0x6c5000 thru 0x6cafff (0x6000 bytes) 000062[0] 1789123363.708563 mapread 0x182000 thru 0x184af7 (0x2af8 bytes) 000063[0] 1789123363.797270 read_OD 0x319000 thru 0x31afff (0x2000 bytes) 000064[0] 1789123363.804554 mapwrite 0x9cd000 thru 0x9d32a9 (0x62aa bytes) 000072[0] 1789123364.009054 write_OD 0x32f000 thru 0x337fff (0x9000 bytes) 000075[0] 1789123364.084262 write_OD 0x652000 thru 0x660fff (0xf000 bytes) 000077[1] 1789123364.202714 read_OD 0x096000 thru 0x097fff (0x2000 bytes) 000081[0] 1789123364.211746 trunc from 0x9d32aa to 0x4ab1ca (0x5280df bytes) 000082[1] 1789123364.263254 read_OD 0x2d6000 thru 0x2d8fff (0x3000 bytes) 000085[1] 1789123364.293809 mapwrite 0x711000 thru 0x711188 (0x189 bytes) 000087[1] 1789123364.428342 write_OD 0x017000 thru 0x01cfff (0x6000 bytes) 000090[1] 1789123364.512220 mapread 0x136000 thru 0x141e13 (0xbe14 bytes) 000093[1] 1789123364.599382 write_OD 0x6d5000 thru 0x6d9fff (0x5000 bytes) 000096[1] 1789123364.676816 mapread 0x4d4000 thru 0x4db5f6 (0x75f7 bytes) 000097[1] 1789123364.758138 trunc from 0x711189 to 0x1e0ab8 (0x5306d0 bytes) 000104[1] 1789123364.835597 mapread 0x032000 thru 0x039e7a (0x7e7b bytes) 000108[1] 1789123364.897361 read_OD 0x0ac000 thru 0x0acfff (0x1000 bytes) 000109[0] 1789123364.914643 write_OD 0x93b000 thru 0x93cfff (0x2000 bytes) 000113[1] 1789123365.046465 mapwrite 0x3ab000 thru 0x3ab0b2 (0xb3 bytes) 000114[0] 1789123365.291303 read_OD 0x08f000 thru 0x092fff (0x4000 bytes) 000116[1] 1789123365.356392 write_OD 0x5fb000 thru 0x5fdfff (0x3000 bytes) 000117[0] 1789123365.459352 write_OD 0x22b000 thru 0x22cfff (0x2000 bytes) 000118[0] 1789123365.552692 read_OD 0x726000 thru 0x72ffff (0xa000 bytes) 000120[1] 1789123365.568641 mapwrite 0x255000 thru 0x260d14 (0xbd15 bytes) 000122[1] 1789123365.835804 write_OD 0x5ff000 thru 0x606fff (0x8000 bytes) 000123[0] 1789123365.912489 mapwrite 0x7ff000 thru 0x80d135 (0xe136 bytes) 000124[1] 1789123366.051053 mapwrite 0x3b7000 thru 0x3bb89c (0x489d bytes) 000125[0] 1789123366.211613 trunc from 0x93d000 to 0x4f9fd4 (0x44302b bytes) truncating to largest ever: 0x86aab9 000126[1] 1789123366.276317 trunc from 0x4f9fd5 to 0x86aab8 (0x370ae4 bytes) 000129[1] 1789123366.318595 mapwrite 0x820000 thru 0x82dd3f (0xdd40 bytes) 000132[1] 1789123366.573189 read_OD 0x48f000 thru 0x497fff (0x9000 bytes) 000136[0] 1789123366.611987 write_OD 0x956000 thru 0x961fff (0xc000 bytes) 000138[0] 1789123366.677997 trunc from 0x962000 to 0x38de94 (0x5d416b bytes) 000139[1] 1789123366.729928 mapread 0x06a000 thru 0x070237 (0x6238 bytes) 000142[1] 1789123366.784540 read_OD 0x0a1000 thru 0x0a4fff (0x4000 bytes) 000145[1] 1789123366.792789 read_OD 0x0e7000 thru 0x0ebfff (0x5000 bytes) 000146[0] 1789123366.804517 mapread 0x172000 thru 0x17be4b (0x9e4c bytes) 000147[1] 1789123366.822548 mapread 0x071000 thru 0x076790 (0x5791 bytes) 000148[1] 1789123366.911608 write_OD 0x2b3000 thru 0x2bafff (0x8000 bytes) 000149[1] 1789123367.064829 mapwrite 0x656000 thru 0x661072 (0xb073 bytes) 000150[1] 1789123367.325719 mapread 0x51e000 thru 0x526982 (0x8983 bytes) truncating to largest ever: 0x88b4a3 000154[0] 1789123367.363358 trunc from 0x661073 to 0x88b4a2 (0x22a430 bytes) 000161[0] 1789123367.508952 trunc from 0x88b4a3 to 0x086519 (0x804f89 bytes) 000162[1] 1789123367.551403 read_OD 0x008000 thru 0x00bfff (0x4000 bytes) 000164[1] 1789123367.596577 write_OD 0x340000 thru 0x34cfff (0xd000 bytes) 000165[0] 1789123367.664965 mapread 0x283000 thru 0x289831 (0x6832 bytes) 000166[0] 1789123367.696804 read_OD 0x064000 thru 0x069fff (0x6000 bytes) 000167[0] 1789123367.709793 mapwrite 0x2ca000 thru 0x2cbc0e (0x1c0f bytes) 000168[1] 1789123367.997990 mapwrite 0x7f6000 thru 0x800145 (0xa146 bytes) 000169[0] 1789123368.344109 read_OD 0x7ad000 thru 0x7affff (0x3000 bytes) truncating to largest ever: 0x96be89 000170[0] 1789123368.403407 trunc from 0x800146 to 0x96be88 (0x16bd43 bytes) 000171[0] 1789123368.469502 write_OD 0x3e2000 thru 0x3ebfff (0xa000 bytes) 000175[0] 1789123368.588988 trunc from 0x96be89 to 0x6563ad (0x315adb bytes) 000176[1] 1789123368.654136 write_OD 0x30b000 thru 0x315fff (0xb000 bytes) 000177[1] 1789123368.745572 mapread 0x55a000 thru 0x5666cf (0xc6d0 bytes) 000184[0] 1789123368.856048 mapwrite 0x944000 thru 0x95163a (0xd63b bytes) 000188[1] 1789123369.119196 mapwrite 0x591000 thru 0x5960e4 (0x50e5 bytes) 000191[0] 1789123369.303966 write_OD 0x89e000 thru 0x8a2fff (0x5000 bytes) 000193[1] 1789123369.385701 write_OD 0x39f000 thru 0x3a8fff (0xa000 bytes) 000195[0] 1789123369.446308 mapread 0x757000 thru 0x758bc6 (0x1bc7 bytes) 000196[0] 1789123369.457476 read_OD 0x8ad000 thru 0x8b0fff (0x4000 bytes) 000198[1] 1789123369.466204 trunc from 0x95163b to 0x8296d8 (0x127f62 bytes) 000199[0] 1789123369.528190 mapread 0x551000 thru 0x554a9e (0x3a9f bytes) 000201[0] 1789123369.591053 mapwrite 0x476000 thru 0x484801 (0xe802 bytes) 000203[0] 1789123369.764347 mapwrite 0x2f3000 thru 0x2f8fb0 (0x5fb1 bytes) 000204[1] 1789123369.861113 mapread 0x107000 thru 0x115b57 (0xeb58 bytes) 000205[0] 1789123370.005300 write_OD 0x799000 thru 0x799fff (0x1000 bytes) 000207[1] 1789123370.053983 mapwrite 0x354000 thru 0x35bb6e (0x7b6f bytes) 000208[1] 1789123370.175827 mapwrite 0x0ce000 thru 0x0cf5a3 (0x15a4 bytes) 000210[1] 1789123370.259338 mapread 0x65c000 thru 0x668380 (0xc381 bytes) 000212[1] 1789123370.391145 mapwrite 0x54c000 thru 0x5566e8 (0xa6e9 bytes) 000215[1] 1789123370.614724 trunc from 0x8296d9 to 0x82ee8e (0x57b6 bytes) 000216[0] 1789123370.634954 trunc from 0x82ee8f to 0x051eb7 (0x7dcfd7 bytes) 000217[0] 1789123370.682561 mapread 0x003000 thru 0x0093c1 (0x63c2 bytes) 000218[0] 1789123370.736965 mapwrite 0x867000 thru 0x872be3 (0xbbe4 bytes) 000220[0] 1789123370.856734 mapwrite 0x90e000 thru 0x90e043 (0x44 bytes) 000223[0] 1789123370.974104 read_OD 0x830000 thru 0x83afff (0xb000 bytes) 000228[1] 1789123371.004084 mapwrite 0x277000 thru 0x27c8ff (0x5900 bytes) 000229[1] 1789123371.104333 mapwrite 0x04e000 thru 0x05bc38 (0xdc39 bytes) 000231[0] 1789123371.455563 mapread 0x3e6000 thru 0x3f3db4 (0xddb5 bytes) 000232[0] 1789123371.554690 mapwrite 0x17e000 thru 0x18d152 (0xf153 bytes) 000236[1] 1789123371.727923 write_OD 0x87e000 thru 0x882fff (0x5000 bytes) 000237[1] 1789123371.783402 mapread 0x8fe000 thru 0x90ad6f (0xcd70 bytes) 000238[0] 1789123371.821262 mapwrite 0x0e4000 thru 0x0e8e5c (0x4e5d bytes) 000244[0] 1789123371.982600 mapwrite 00000000 thru 0x000142 (0x143 bytes) 000248[1] 1789123372.062405 read_OD 0x65e000 thru 0x666fff (0x9000 bytes) 000249[1] 1789123372.094558 mapwrite 0x401000 thru 0x40843f (0x7440 bytes) 000250[1] 1789123372.192944 trunc from 0x90e044 to 0x0113e5 (0x8fcc5e bytes) 000255[1] 1789123372.317015 trunc from 0x0113e6 to 0x535b46 (0x524761 bytes) 000256[1] 1789123372.376158 mapwrite 0x5e3000 thru 0x5ea935 (0x7936 bytes) 000259[0] 1789123372.651364 mapwrite 0x910000 thru 0x91a3da (0xa3db bytes) 000260[1] 1789123372.858547 mapwrite 0x718000 thru 0x71df6e (0x5f6f bytes) 000261[0] 1789123372.968756 trunc from 0x91a3db to 0x6d7080 (0x24335a bytes) 000262[1] 1789123373.024394 trunc from 0x6d7081 to 0x3429fb (0x394685 bytes) 000268[0] 1789123373.085044 mapwrite 0x6f3000 thru 0x6f5ea7 (0x2ea8 bytes) 000269[0] 1789123373.205868 mapwrite 0x581000 thru 0x58e368 (0xd369 bytes) 000271[1] 1789123373.318832 trunc from 0x6f5ea8 to 0x79d358 (0xa74b1 bytes) 000272[1] 1789123373.367260 mapread 0x242000 thru 0x24422d (0x222e bytes) 000274[1] 1789123373.394971 read_OD 0x082000 thru 0x084fff (0x3000 bytes) 000275[1] 1789123373.414032 read_OD 0x063000 thru 0x069fff (0x7000 bytes) 000278[0] 1789123373.428285 read_OD 0x4ce000 thru 0x4d7fff (0xa000 bytes) 000280[1] 1789123373.437649 trunc from 0x79d359 to 0x363f70 (0x4393e8 bytes) 000282[1] 1789123373.553611 mapwrite 0x40d000 thru 0x40d481 (0x482 bytes) 000286[1] 1789123373.875189 read_OD 0x050000 thru 0x05bfff (0xc000 bytes) 000289[1] 1789123373.976750 write_OD 0x24d000 thru 0x259fff (0xd000 bytes) 000296[1] 1789123374.099597 mapread 0x2d4000 thru 0x2db4de (0x74df bytes) 000297[0] 1789123374.149878 mapread 0x1e1000 thru 0x1e9eb7 (0x8eb8 bytes) 000298[0] 1789123374.175770 mapread 0x376000 thru 0x37bec4 (0x5ec5 bytes) 000299[0] 1789123374.209413 write_OD 0x81f000 thru 0x825fff (0x7000 bytes) 000301[0] 1789123374.292923 trunc from 0x826000 to 0x4235e8 (0x402a17 bytes) 000302[0] 1789123374.360114 mapwrite 0x245000 thru 0x2516b9 (0xc6ba bytes) 000304[0] 1789123374.491991 trunc from 0x4235e9 to 0x6ee0da (0x2caaf2 bytes) 000305[1] 1789123374.547863 trunc from 0x6ee0db to 0x0df544 (0x60eb96 bytes) 000308[1] 1789123374.693293 mapwrite 0x282000 thru 0x283966 (0x1967 bytes) 000310[0] 1789123374.829077 read_OD 0x0b9000 thru 0x0c5fff (0xd000 bytes) 000313[0] 1789123374.870900 read_OD 0x137000 thru 0x137fff (0x1000 bytes) 000314[0] 1789123374.875779 trunc from 0x283967 to 0x40ed93 (0x18b42d bytes) 000318[0] 1789123374.964793 trunc from 0x40ed94 to 0x8c36bd (0x4b492a bytes) 000320[1] 1789123375.014558 read_OD 0x805000 thru 0x810fff (0xc000 bytes) 000321[1] 1789123375.045406 trunc from 0x8c36be to 0x938c4a (0x7558d bytes) 000322[0] 1789123375.073874 read_OD 0x301000 thru 0x30cfff (0xc000 bytes) 000324[1] 1789123375.101063 write_OD 0x6d5000 thru 0x6ddfff (0x9000 bytes) 000325[1] 1789123375.152397 read_OD 0x134000 thru 0x13afff (0x7000 bytes) 000327[0] 1789123375.166776 mapread 0x201000 thru 0x2033ac (0x23ad bytes) truncating to largest ever: 0x9e5e50 000328[0] 1789123375.187505 trunc from 0x938c4b to 0x9e5e4f (0xad205 bytes) 000330[1] 1789123375.276190 read_OD 0x6df000 thru 0x6ebfff (0xd000 bytes) 000331[0] 1789123375.306804 trunc from 0x9e5e50 to 0x38911b (0x65cd34 bytes) 000332[0] 1789123375.342070 read_OD 0x2e3000 thru 0x2e3fff (0x1000 bytes) 000333[0] 1789123375.356887 read_OD 0x077000 thru 0x078fff (0x2000 bytes) 000335[0] 1789123375.363885 read_OD 0x299000 thru 0x2a2fff (0xa000 bytes) 000337[0] 1789123375.377004 mapwrite 0x68b000 thru 0x692bbe (0x7bbf bytes) 000338[0] 1789123375.809777 mapread 0x53c000 thru 0x53f77e (0x377f bytes) 000339[1] 1789123375.810728 mapwrite 0x0f0000 thru 0x0f95b7 (0x95b8 bytes) 000340[0] 1789123375.994148 read_OD 0x43c000 thru 0x449fff (0xe000 bytes) 000342[1] 1789123376.031163 mapwrite 0x3fa000 thru 0x409e13 (0xfe14 bytes) 000345[0] 1789123376.160856 write_OD 0x29d000 thru 0x2a4fff (0x8000 bytes) 000346[1] 1789123376.219769 mapread 0x282000 thru 0x28d184 (0xb185 bytes) 000350[1] 1789123376.292303 write_OD 0x061000 thru 0x065fff (0x5000 bytes) 000351[0] 1789123376.367995 read_OD 0x067000 thru 0x069fff (0x3000 bytes) 000352[0] 1789123376.382368 trunc from 0x692bbf to 0x710044 (0x7d486 bytes) 000354[0] 1789123376.465190 mapwrite 0x1b8000 thru 0x1c147c (0x947d bytes) 000358[1] 1789123376.669583 read_OD 0x4e5000 thru 0x4edfff (0x9000 bytes) 000359[0] 1789123376.730112 write_OD 0x0b5000 thru 0x0bbfff (0x7000 bytes) 000363[1] 1789123376.866245 write_OD 0x9b9000 thru 0x9b9fff (0x1000 bytes) 000364[1] 1789123376.947081 mapread 0x6de000 thru 0x6ec796 (0xe797 bytes) 000365[1] 1789123376.995084 trunc from 0x9ba000 to 0x3d358d (0x5e6a72 bytes) 000366[0] 1789123377.055926 write_OD 0x035000 thru 0x03cfff (0x8000 bytes) 000370[0] 1789123377.113405 mapwrite 0x4f5000 thru 0x4f644d (0x144e bytes) 000374[0] 1789123377.253820 trunc from 0x4f644e to 0x12d32e (0x3c911f bytes) 000376[1] 1789123377.303067 mapread 0x08d000 thru 0x0914aa (0x44ab bytes) 000377[1] 1789123377.387299 mapread 0x042000 thru 0x04a84c (0x884d bytes) 000381[1] 1789123377.411698 read_OD 0x04f000 thru 0x050fff (0x2000 bytes) 000382[1] 1789123377.419304 write_OD 0x93c000 thru 0x94afff (0xf000 bytes) 000384[0] 1789123377.541863 read_OD 0x4d4000 thru 0x4d6fff (0x3000 bytes) 000386[0] 1789123377.564117 trunc from 0x94b000 to 0x127246 (0x823db9 bytes) 000391[0] 1789123377.646375 trunc from 0x127247 to 0x144878 (0x1d632 bytes) 000392[1] 1789123377.662053 trunc from 0x144879 to 0x0977b2 (0xad0c6 bytes) 000394[1] 1789123377.717147 trunc from 0x0977b3 to 0x89b899 (0x8040e7 bytes) 000396[0] 1789123377.814821 trunc from 0x89b89a to 0x7d93ca (0xc24cf bytes) 000397[0] 1789123377.840640 read_OD 0x5df000 thru 0x5e9fff (0xb000 bytes) 000398[1] 1789123377.860347 mapwrite 0x2cc000 thru 0x2d237b (0x637c bytes) 000401[0] 1789123378.081606 mapread 0x324000 thru 0x32ca86 (0x8a87 bytes) 000403[0] 1789123378.159056 trunc from 0x7d93cb to 0x569959 (0x26fa71 bytes) 000405[1] 1789123378.259308 mapwrite 0x3f9000 thru 0x3fce25 (0x3e26 bytes) 000408[0] 1789123378.451772 mapread 0x507000 thru 0x516a78 (0xfa79 bytes) 000409[0] 1789123378.520160 mapwrite 0x848000 thru 0x855d24 (0xdd25 bytes) 000410[0] 1789123378.666332 read_OD 0x163000 thru 0x16efff (0xc000 bytes) 000413[0] 1789123378.702333 write_OD 0x28f000 thru 0x29dfff (0xf000 bytes) 000416[0] 1789123378.838840 mapwrite 0x870000 thru 0x879f35 (0x9f36 bytes) 000417[0] 1789123379.022857 trunc from 0x879f36 to 0x4fe844 (0x37b6f1 bytes) 000419[1] 1789123379.120144 mapwrite 0x104000 thru 0x107585 (0x3586 bytes) 000420[1] 1789123379.293281 read_OD 0x2d6000 thru 0x2dcfff (0x7000 bytes) 000423[1] 1789123379.357331 mapwrite 0x3c6000 thru 0x3d12cb (0xb2cc bytes) 000429[0] 1789123379.536781 mapwrite 0x054000 thru 0x06153d (0xd53e bytes) 000430[0] 1789123379.700718 trunc from 0x4fe845 to 0x74fec2 (0x25167e bytes) 000431[1] 1789123379.755368 write_OD 0x424000 thru 0x432fff (0xf000 bytes) 000433[1] 1789123379.864089 write_OD 0x09d000 thru 0x0a7fff (0xb000 bytes) 000434[1] 1789123379.946318 mapread 0x4d7000 thru 0x4e3fde (0xcfdf bytes) 000435[0] 1789123380.023842 read_OD 0x500000 thru 0x507fff (0x8000 bytes) 000438[1] 1789123380.039021 write_OD 0x794000 thru 0x79bfff (0x8000 bytes) 000445[1] 1789123380.126890 mapread 0x1b4000 thru 0x1b5d34 (0x1d35 bytes) 000447[0] 1789123380.138095 write_OD 0x3cb000 thru 0x3d1fff (0x7000 bytes) 000448[0] 1789123380.217130 mapwrite 0x081000 thru 0x08f687 (0xe688 bytes) 000450[1] 1789123380.394867 read_OD 0x208000 thru 0x216fff (0xf000 bytes) 000452[0] 1789123380.434344 trunc from 0x79c000 to 0x5115f8 (0x28aa07 bytes) 000453[1] 1789123380.501957 trunc from 0x5115f9 to 0x2c309f (0x24e559 bytes) 000454[0] 1789123380.562368 trunc from 0x2c30a0 to 0x4c9bfb (0x206b5c bytes) 000455[1] 1789123380.619044 read_OD 0x032000 thru 0x03ffff (0xe000 bytes) 000456[1] 1789123380.657235 read_OD 0x3d5000 thru 0x3ddfff (0x9000 bytes) 000458[0] 1789123380.682813 trunc from 0x4c9bfc to 0x713a30 (0x249e35 bytes) 000460[1] 1789123380.764189 read_OD 0x5db000 thru 0x5e1fff (0x7000 bytes) 000461[1] 1789123380.790515 mapread 0x343000 thru 0x346078 (0x3079 bytes) 000463[0] 1789123380.833985 trunc from 0x713a31 to 0x5dbcc0 (0x137d70 bytes) 000465[0] 1789123380.937528 mapread 0x15b000 thru 0x1637fb (0x87fc bytes) 000466[1] 1789123380.979748 mapwrite 0x84d000 thru 0x84ef66 (0x1f67 bytes) 000468[1] 1789123381.116680 write_OD 0x318000 thru 0x321fff (0xa000 bytes) 000471[1] 1789123381.194114 mapread 0x576000 thru 0x582215 (0xc216 bytes) 000472[1] 1789123381.252352 read_OD 0x12d000 thru 0x130fff (0x4000 bytes) 000474[0] 1789123381.261809 mapwrite 0x73e000 thru 0x7453e5 (0x73e6 bytes) 000475[0] 1789123381.446151 mapwrite 0x56e000 thru 0x57a321 (0xc322 bytes) 000478[1] 1789123381.576802 read_OD 0x7ad000 thru 0x7b7fff (0xb000 bytes) 000479[0] 1789123381.615293 trunc from 0x84ef67 to 0x6b3553 (0x19ba13 bytes) 000480[0] 1789123381.677473 read_OD 0x2db000 thru 0x2e6fff (0xc000 bytes) 000490[1] 1789123381.717345 mapwrite 0x785000 thru 0x7868e2 (0x18e3 bytes) 000491[0] 1789123381.850480 write_OD 0x9e2000 thru 0x9e8fff (0x7000 bytes) 000493[1] 1789123381.917204 mapread 0x53e000 thru 0x54668e (0x868f bytes) 000494[1] 1789123381.943346 mapwrite 0x886000 thru 0x8958c3 (0xf8c4 bytes) 000499[0] 1789123382.137954 mapwrite 0x762000 thru 0x762ee2 (0xee3 bytes) 000500[1] 1789123382.243113 write_OD 0x355000 thru 0x361fff (0xd000 bytes) All operations completed A-OK! PASS 16a (59s) == sanityn test 16b: 500 iterations of dual-mount fsx at small size ========================================================== 06:43:09 (1789123389) fallocate on zfs doesn't consume space fallocate not supported 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.543322 s, 15.4 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.376494 s, 22.3 MB/s Chance of close/open is 1 in 50 Seed set to 3392 fd 0: /mnt/lustre/f16b.sanityn fd 1: /mnt/lustre2/f16b.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000003[0] 1789123393.554067 mapwrite 0x0882dc thru 0x088a63 (0x788 bytes) 000006[0] 1789123393.679140 read 0x07d847 thru 0x07de60 (0x61a bytes) 000008[1] 1789123393.730185 read 0x06af40 thru 0x06ceb3 (0x1f74 bytes) 000009[1] 1789123393.761175 mapwrite 0x835a63 thru 0x837a38 (0x1fd6 bytes) truncating to largest ever: 0x1af0fe 000011[1] 1789123394.015039 trunc from 0x837a39 to 0x1af0fd (0x68893b bytes) 000014[1] 1789123394.127744 write 0x1695ad thru 0x16a1b4 (0xc08 bytes) 000015[1] 1789123394.137556 read 0x0b340d thru 0x0b4483 (0x1077 bytes) 000016[1] 1789123394.198866 mapwrite 0x142888 thru 0x144697 (0x1e10 bytes) 000018[1] 1789123394.286150 read 0x08dcfc thru 0x08f9d1 (0x1cd6 bytes) 000019[1] 1789123394.303703 read 0x110835 thru 0x11122e (0x9fa bytes) truncating to largest ever: 0x9f20c1 000020[0] 1789123394.304410 trunc from 0x1af0fe to 0x9f20c0 (0x842fc3 bytes) 000021[0] 1789123394.394399 write 0x227597 thru 0x227da0 (0x80a bytes) 000023[0] 1789123394.477278 trunc from 0x9f20c1 to 0x221480 (0x7d0c40 bytes) 000027[0] 1789123394.561169 write 0x9881e2 thru 0x98970e (0x152d bytes) 000030[1] 1789123394.619169 write 0x8856ec thru 0x8869f4 (0x1309 bytes) 000032[1] 1789123394.765374 trunc from 0x98970f to 0x98cc61 (0x3553 bytes) 000033[1] 1789123394.806632 read 0x21ebfa thru 0x21fbca (0xfd1 bytes) 000037[0] 1789123394.861083 trunc from 0x98cc62 to 0x366fd6 (0x625c8b bytes) 000039[0] 1789123394.951345 trunc from 0x366fd7 to 0x3d6799 (0x6f7c3 bytes) 000042[0] 1789123395.061718 read 0x08d2fb thru 0x08ee21 (0x1b27 bytes) 000044[0] 1789123395.143364 mapread 0x04c5c2 thru 0x04d158 (0xb97 bytes) 000045[1] 1789123395.144240 mapwrite 0x1d638c thru 0x1d72a5 (0xf1a bytes) 000047[0] 1789123395.284657 mapread 0x28dec8 thru 0x28ea1b (0xb54 bytes) 000049[0] 1789123395.313190 mapread 0x392905 thru 0x3931d9 (0x8d5 bytes) 000053[0] 1789123395.393819 mapread 0x1a00e7 thru 0x1a1736 (0x1650 bytes) 000054[1] 1789123395.405245 read 0x2356d6 thru 0x236521 (0xe4c bytes) 000056[0] 1789123395.510031 trunc from 0x3d679a to 0x62cac3 (0x25632a bytes) 000057[1] 1789123395.600666 read 0x19c9d6 thru 0x19e3cb (0x19f6 bytes) 000058[0] 1789123395.637553 trunc from 0x62cac4 to 0x60bbb3 (0x20f10 bytes) 000061[1] 1789123395.760826 mapwrite 0x314cae thru 0x31592c (0xc7f bytes) 000063[1] 1789123395.888822 trunc from 0x60bbb4 to 0x8ae059 (0x2a24a6 bytes) 000064[1] 1789123395.918824 read 0x849acb thru 0x84a09a (0x5d0 bytes) 000067[1] 1789123395.959162 write 0x3b06dd thru 0x3b1e2c (0x1750 bytes) 000068[1] 1789123396.001080 mapwrite 0x97d25a thru 0x97d4af (0x256 bytes) 000069[0] 1789123396.094827 write 0x2d54f6 thru 0x2d71a3 (0x1cae bytes) 000070[0] 1789123396.147389 read 0x82ac82 thru 0x82bed2 (0x1251 bytes) 000071[1] 1789123396.172863 read 0x24d9e7 thru 0x24f359 (0x1973 bytes) 000073[1] 1789123396.236915 mapwrite 0x079e1a thru 0x07a1ce (0x3b5 bytes) 000074[1] 1789123396.357477 read 0x46ed8e thru 0x4708e2 (0x1b55 bytes) 000075[1] 1789123396.366360 mapread 0x882d72 thru 0x8845f4 (0x1883 bytes) 000077[1] 1789123396.464720 write 0x3244d2 thru 0x325f09 (0x1a38 bytes) 000078[0] 1789123396.509054 write 0x9c7c2a thru 0x9c8ee0 (0x12b7 bytes) 000081[1] 1789123396.600131 trunc from 0x9c8ee1 to 0x068697 (0x960849 bytes) 000083[0] 1789123396.720664 mapread 0x03f144 thru 0x040afb (0x19b8 bytes) 000084[0] 1789123396.817378 mapread 0x01d013 thru 0x01ef3f (0x1f2d bytes) 000086[1] 1789123396.859969 mapread 0x05cc92 thru 0x05d304 (0x673 bytes) 000089[1] 1789123396.917581 read 0x059bbc thru 0x05b7df (0x1c24 bytes) 000095[1] 1789123396.997095 read 0x0432f8 thru 0x044fdc (0x1ce5 bytes) 000096[0] 1789123396.998346 trunc from 0x068698 to 0x9b438c (0x94bcf5 bytes) 000097[1] 1789123397.061833 trunc from 0x9b438d to 0x89c710 (0x117c7c bytes) 000100[0] 1789123397.162899 read 0x0cbf14 thru 0x0cd3f5 (0x14e2 bytes) 000101[0] 1789123397.202857 mapwrite 0x6f3d5a thru 0x6f58f8 (0x1b9f bytes) 000104[0] 1789123397.412735 read 0x49fc7d thru 0x4a1164 (0x14e8 bytes) 000105[1] 1789123397.467614 write 0x7198b9 thru 0x71b6ea (0x1e32 bytes) 000106[0] 1789123397.700770 read 0x09541e thru 0x096946 (0x1529 bytes) 000108[0] 1789123397.723726 read 0x0701ab thru 0x070be0 (0xa36 bytes) 000111[1] 1789123397.748737 read 0x789282 thru 0x78a747 (0x14c6 bytes) 000113[0] 1789123397.765659 trunc from 0x89c711 to 0x6a7b61 (0x1f4baf bytes) 000114[1] 1789123397.873377 mapwrite 0x63abef thru 0x63c4db (0x18ed bytes) 000117[1] 1789123398.105217 mapread 0x45aac4 thru 0x45b650 (0xb8d bytes) 000118[0] 1789123398.159514 mapwrite 0x21872f thru 0x218875 (0x147 bytes) 000119[0] 1789123398.283247 trunc from 0x6a7b62 to 0x1c429f (0x4e38c2 bytes) 000122[1] 1789123398.353977 trunc from 0x1c42a0 to 0x79b0a6 (0x5d6e07 bytes) 000124[0] 1789123398.441966 trunc from 0x79b0a7 to 0x1fbefc (0x59f1aa bytes) 000125[0] 1789123398.527187 read 0x08695e thru 0x087bd9 (0x127c bytes) 000129[1] 1789123398.632536 write 0x3673d8 thru 0x367668 (0x291 bytes) 000130[0] 1789123398.667958 read 0x09402f thru 0x095fc3 (0x1f95 bytes) 000132[1] 1789123398.817386 read 0x0258cb thru 0x026a4f (0x1185 bytes) 000133[0] 1789123398.919339 write 0x163547 thru 0x16459f (0x1059 bytes) 000138[1] 1789123398.986069 read 0x16632f thru 0x166a46 (0x718 bytes) 000142[1] 1789123399.075718 write 0x9cbf97 thru 0x9cbfb5 (0x1f bytes) 000145[1] 1789123399.167521 mapread 0x83829c thru 0x83965d (0x13c2 bytes) 000146[1] 1789123399.226018 mapread 0x8e4c30 thru 0x8e5edf (0x12b0 bytes) 000148[1] 1789123399.295921 read 0x7b6af7 thru 0x7b759f (0xaa9 bytes) 000149[0] 1789123399.321358 write 0x63f617 thru 0x640907 (0x12f1 bytes) 000152[0] 1789123399.433318 write 0x448285 thru 0x4487ae (0x52a bytes) 000153[1] 1789123399.485413 write 0x4367ee thru 0x436f61 (0x774 bytes) 000159[0] 1789123399.707014 mapwrite 0x8f61ec thru 0x8f81e0 (0x1ff5 bytes) 000161[1] 1789123399.877120 write 0x1e7f36 thru 0x1e996a (0x1a35 bytes) 000162[1] 1789123399.901059 read 0x55dab0 thru 0x55e8f5 (0xe46 bytes) 000164[1] 1789123399.962644 read 0x90a4db thru 0x90beaf (0x19d5 bytes) 000167[1] 1789123400.001317 mapwrite 0x7990d5 thru 0x79a024 (0xf50 bytes) 000168[0] 1789123400.111728 trunc from 0x9cbfb6 to 0x31ee4f (0x6ad166 bytes) 000169[1] 1789123400.195607 mapwrite 0x61b09c thru 0x61c145 (0x10aa bytes) 000172[1] 1789123400.378529 mapread 0x372595 thru 0x373aa3 (0x150f bytes) 000174[1] 1789123400.397709 mapwrite 0x7659b0 thru 0x765d10 (0x361 bytes) 000177[0] 1789123400.531863 mapwrite 0x777045 thru 0x7782e8 (0x12a4 bytes) 000178[1] 1789123400.759488 mapread 0x1c6e04 thru 0x1c8a0a (0x1c07 bytes) 000179[1] 1789123400.835474 trunc from 0x7782e9 to 0x9eb554 (0x27326c bytes) 000180[1] 1789123400.926092 read 0x43b181 thru 0x43bd08 (0xb88 bytes) 000183[0] 1789123400.990940 read 0x85bb27 thru 0x85c6dd (0xbb7 bytes) 000184[0] 1789123401.041005 write 0x5603ad thru 0x5618f0 (0x1544 bytes) 000185[0] 1789123401.079965 read 0x03e37f thru 0x03f89b (0x151d bytes) 000189[1] 1789123401.165011 write 0x1d1d06 thru 0x1d30da (0x13d5 bytes) 000190[0] 1789123401.227139 read 0x463259 thru 0x464d83 (0x1b2b bytes) 000194[1] 1789123401.324496 trunc from 0x9eb555 to 0x3c0cc7 (0x62a88d bytes) 000196[1] 1789123401.369046 mapread 0x000d43 thru 0x002733 (0x19f1 bytes) 000200[0] 1789123401.564968 mapwrite 0x3faf7f thru 0x3fc80b (0x188d bytes) 000202[0] 1789123401.849735 mapwrite 0x6666f0 thru 0x6675f5 (0xf06 bytes) 000207[0] 1789123402.015000 read 0x42fdea thru 0x431316 (0x152d bytes) 000208[0] 1789123402.069791 read 0x202c96 thru 0x2033db (0x746 bytes) 000210[0] 1789123402.114683 trunc from 0x6675f6 to 0x6e654e (0x7ef59 bytes) 000211[1] 1789123402.171691 read 0x42b78c thru 0x42bfff (0x874 bytes) 000212[0] 1789123402.221994 mapwrite 0x94d72e thru 0x94d883 (0x156 bytes) 000213[1] 1789123402.382710 read 0x5f5568 thru 0x5f68e7 (0x1380 bytes) 000214[0] 1789123402.441185 read 0x15b27e thru 0x15c845 (0x15c8 bytes) 000218[0] 1789123402.467740 read 0x749eac thru 0x74a99b (0xaf0 bytes) 000219[1] 1789123402.479050 read 0x34c0eb thru 0x34cf01 (0xe17 bytes) 000220[0] 1789123402.501157 read 0x7a27eb thru 0x7a2ea4 (0x6ba bytes) 000221[1] 1789123402.518794 mapwrite 0x184d6e thru 0x185365 (0x5f8 bytes) 000222[1] 1789123402.620189 mapwrite 0x6715b2 thru 0x672afd (0x154c bytes) 000224[1] 1789123402.762419 read 0x271597 thru 0x271c83 (0x6ed bytes) 000228[0] 1789123402.927560 mapwrite 0x876b41 thru 0x87751b (0x9db bytes) 000231[1] 1789123403.130552 mapread 0x6a707c thru 0x6a7175 (0xfa bytes) 000235[1] 1789123403.249737 mapwrite 0x8b5cfb thru 0x8b7c73 (0x1f79 bytes) 000237[0] 1789123403.377515 trunc from 0x94d884 to 0x60cfd3 (0x3408b0 bytes) 000238[1] 1789123403.487842 mapwrite 0x1e2173 thru 0x1e2fab (0xe39 bytes) 000243[0] 1789123403.724396 mapwrite 0x0400dd thru 0x040812 (0x736 bytes) 000248[1] 1789123403.948756 read 0x3d6991 thru 0x3d7632 (0xca2 bytes) 000249[0] 1789123403.980726 trunc from 0x60cfd4 to 0x28ccae (0x380325 bytes) 000252[0] 1789123404.054853 write 0x1afb0e thru 0x1b191c (0x1e0f bytes) 000254[1] 1789123404.066383 read 0x21e5c8 thru 0x21f396 (0xdcf bytes) 000256[0] 1789123404.155317 trunc from 0x28ccaf to 0x5c8b02 (0x33be54 bytes) 000259[0] 1789123404.263749 mapwrite 0x69a0b1 thru 0x69b001 (0xf51 bytes) 000261[1] 1789123404.397789 read 0x1f604b thru 0x1f6952 (0x908 bytes) 000263[0] 1789123404.418048 mapwrite 0x2a7e2c thru 0x2a9188 (0x135d bytes) 000264[0] 1789123404.572070 read 0x00ba9c thru 0x00c2a9 (0x80e bytes) 000265[1] 1789123404.640964 write 0x5cbedb thru 0x5cc966 (0xa8c bytes) 000266[1] 1789123404.708704 mapread 0x55eda8 thru 0x55f80f (0xa68 bytes) 000272[0] 1789123404.888718 write 0x367d05 thru 0x369812 (0x1b0e bytes) 000273[0] 1789123404.918192 trunc from 0x69b002 to 0x891267 (0x1f6266 bytes) 000277[1] 1789123405.049974 mapwrite 0x92729b thru 0x9273e9 (0x14f bytes) 000281[0] 1789123405.189682 write 0x003652 thru 0x003786 (0x135 bytes) 000282[1] 1789123405.233417 trunc from 0x9273ea to 0x29218d (0x69525c bytes) 000285[0] 1789123405.374554 write 0x77007c thru 0x770c9a (0xc1f bytes) 000286[1] 1789123405.406781 trunc from 0x770c9b to 0x883e74 (0x1131da bytes) 000287[1] 1789123405.485798 trunc from 0x883e75 to 0x8a287d (0x1ea09 bytes) 000290[1] 1789123405.538982 mapread 0x46d5de thru 0x46e7bd (0x11e0 bytes) 000291[0] 1789123405.573930 mapwrite 0x71a52c thru 0x71ac44 (0x719 bytes) 000292[0] 1789123405.650161 trunc from 0x8a287e to 0x2de75e (0x5c411f bytes) 000295[0] 1789123405.689183 read 0x217c41 thru 0x217da3 (0x163 bytes) 000296[0] 1789123405.801644 mapread 0x212f91 thru 0x214ed5 (0x1f45 bytes) 000299[1] 1789123405.806469 read 0x04092b thru 0x040fc6 (0x69c bytes) 000300[1] 1789123405.907650 write 0x19eb25 thru 0x19fc39 (0x1115 bytes) 000301[0] 1789123405.949854 mapwrite 0x655b2f thru 0x656d67 (0x1239 bytes) 000302[1] 1789123406.090915 trunc from 0x656d68 to 0x2d8b82 (0x37e1e5 bytes) 000303[0] 1789123406.167339 mapread 0x17e1aa thru 0x17f9a2 (0x17f9 bytes) 000304[0] 1789123406.226862 trunc from 0x2d8b83 to 0x31d702 (0x44b80 bytes) 000306[1] 1789123406.314038 write 0x88a21e thru 0x88a802 (0x5e5 bytes) 000307[1] 1789123406.357468 read 0x3fc7ed thru 0x3fe0fc (0x1910 bytes) 000308[0] 1789123406.384853 mapread 0x5a2745 thru 0x5a4642 (0x1efe bytes) 000309[0] 1789123406.465227 mapwrite 0x38f665 thru 0x3900f5 (0xa91 bytes) 000310[0] 1789123406.659202 mapread 0x3d10f6 thru 0x3d2632 (0x153d bytes) 000311[0] 1789123406.671937 trunc from 0x88a803 to 0x4cbec6 (0x3be93c bytes) 000317[1] 1789123406.755034 write 0x401f20 thru 0x40345c (0x153d bytes) 000321[0] 1789123406.864185 write 0x4eb06d thru 0x4ec57a (0x150e bytes) 000322[0] 1789123406.889765 trunc from 0x4ec57b to 0x393d82 (0x1587f8 bytes) 000323[1] 1789123406.945990 read 0x147f61 thru 0x1483fa (0x49a bytes) 000324[1] 1789123407.072780 mapread 0x10086e thru 0x100e7d (0x610 bytes) 000326[0] 1789123407.151292 read 0x241b8e thru 0x243085 (0x14f8 bytes) 000329[0] 1789123407.296747 mapwrite 0x8da645 thru 0x8dc201 (0x1bbd bytes) 000331[0] 1789123407.577741 read 0x71756c thru 0x718c13 (0x16a8 bytes) 000333[1] 1789123407.630250 mapwrite 0x30c187 thru 0x30c691 (0x50b bytes) 000334[0] 1789123407.829736 mapwrite 0x2d3fea thru 0x2d5713 (0x172a bytes) 000336[0] 1789123407.966877 write 0x8549b1 thru 0x8554e3 (0xb33 bytes) 000337[1] 1789123407.976162 write 0x7298e3 thru 0x72b2c7 (0x19e5 bytes) 000338[1] 1789123408.097929 read 0x48f376 thru 0x48fc6b (0x8f6 bytes) 000339[1] 1789123408.124471 trunc from 0x8dc202 to 0x45f61f (0x47cbe2 bytes) 000345[0] 1789123408.223292 mapwrite 0x7bd9b1 thru 0x7be7be (0xe0e bytes) 000350[1] 1789123408.422802 write 0x1d13eb thru 0x1d2f76 (0x1b8c bytes) 000354[0] 1789123408.522223 write 0x954f18 thru 0x955b95 (0xc7e bytes) 000356[1] 1789123408.604774 mapwrite 0x5ea3c3 thru 0x5eaf38 (0xb76 bytes) 000358[1] 1789123408.742984 mapwrite 0x8979b2 thru 0x89809e (0x6ed bytes) 000359[1] 1789123408.944788 mapwrite 0x09dfec thru 0x09e9c6 (0x9db bytes) 000360[1] 1789123409.065524 mapwrite 0x6f6473 thru 0x6f6c15 (0x7a3 bytes) 000361[1] 1789123409.257104 trunc from 0x955b96 to 0x489427 (0x4cc76e bytes) 000363[0] 1789123409.365245 trunc from 0x489428 to 0x05c71e (0x42cd09 bytes) 000364[1] 1789123409.442330 mapwrite 0x7e01e2 thru 0x7e155c (0x137b bytes) 000365[1] 1789123409.706610 mapread 0x46bf3c thru 0x46c828 (0x8ed bytes) 000369[1] 1789123409.871932 write 0x1af5dc thru 0x1b12c4 (0x1ce9 bytes) 000370[0] 1789123409.933030 trunc from 0x7e155d to 0x1330e1 (0x6ae47b bytes) 000371[1] 1789123410.071696 mapwrite 0x69b809 thru 0x69c3d2 (0xbca bytes) 000372[0] 1789123410.278954 mapwrite 0x7b04a3 thru 0x7b141d (0xf7b bytes) 000373[1] 1789123410.441109 mapread 0x3bad86 thru 0x3bae08 (0x83 bytes) 000374[0] 1789123410.487616 write 0x06699d thru 0x066afb (0x15f bytes) 000375[0] 1789123410.532689 write 0x81f167 thru 0x81f83f (0x6d9 bytes) 000377[0] 1789123410.566164 read 0x6f6437 thru 0x6f65f7 (0x1c1 bytes) 000381[0] 1789123410.607003 trunc from 0x81f840 to 0x8a8aa1 (0x89262 bytes) 000382[0] 1789123410.632920 trunc from 0x8a8aa2 to 0x80e5c6 (0x9a4db bytes) 000384[1] 1789123410.648661 mapread 0x59c7c3 thru 0x59cad5 (0x313 bytes) 000387[1] 1789123410.728410 trunc from 0x80e5c7 to 0x9863c1 (0x177dfb bytes) 000391[1] 1789123410.885840 mapread 0x3e8253 thru 0x3e9972 (0x1720 bytes) 000394[0] 1789123410.946109 trunc from 0x9863c2 to 0x8c9e67 (0xbc55a bytes) 000397[1] 1789123411.129507 mapwrite 0x9c335b thru 0x9c4905 (0x15ab bytes) 000399[1] 1789123411.341105 write 0x1f132f thru 0x1f1acb (0x79d bytes) 000401[1] 1789123411.409844 read 0x624f75 thru 0x6260f6 (0x1182 bytes) 000402[1] 1789123411.440246 write 0x35d324 thru 0x35ef22 (0x1bff bytes) 000404[0] 1789123411.475959 read 0x2f62e5 thru 0x2f6b71 (0x88d bytes) 000405[0] 1789123411.510485 read 0x423afc thru 0x423d44 (0x249 bytes) 000406[1] 1789123411.539151 trunc from 0x9c4906 to 0x680270 (0x344695 bytes) 000407[0] 1789123411.627171 write 0x7bea53 thru 0x7bf320 (0x8ce bytes) 000408[1] 1789123411.658363 write 0x885acc thru 0x886062 (0x597 bytes) 000409[0] 1789123411.684720 write 0x12143a thru 0x122d5f (0x1926 bytes) 000419[0] 1789123412.056922 read 0x25cac4 thru 0x25e41a (0x1957 bytes) 000420[1] 1789123412.065973 read 0x292a67 thru 0x29471c (0x1cb6 bytes) 000421[1] 1789123412.157229 trunc from 0x886063 to 0x2327a0 (0x6538c2 bytes) 000424[0] 1789123412.308032 trunc from 0x2327a1 to 0x754959 (0x5221b9 bytes) 000426[0] 1789123412.376838 mapread 0x001e1e thru 0x003d2c (0x1f0f bytes) 000429[1] 1789123412.532241 mapread 0x64f8e8 thru 0x65070e (0xe27 bytes) 000430[1] 1789123412.663303 trunc from 0x75495a to 0x0bd057 (0x697902 bytes) 000431[0] 1789123412.767913 trunc from 0x0bd058 to 0x138b17 (0x7bac0 bytes) 000432[1] 1789123412.841783 write 0x77a240 thru 0x77ae95 (0xc56 bytes) 000433[0] 1789123412.942063 mapread 0x646b7b thru 0x64872b (0x1bb1 bytes) 000434[1] 1789123413.054045 read 0x2ccb93 thru 0x2cd65a (0xac8 bytes) 000435[1] 1789123413.120754 mapwrite 0x2b4022 thru 0x2b5deb (0x1dca bytes) 000437[0] 1789123413.218370 mapwrite 0x41a430 thru 0x41a70a (0x2db bytes) 000438[0] 1789123413.308110 write 0x166ca4 thru 0x168b13 (0x1e70 bytes) 000440[0] 1789123413.394149 mapread 0x5748c5 thru 0x576001 (0x173d bytes) 000441[0] 1789123413.414359 read 0x745489 thru 0x746144 (0xcbc bytes) 000442[0] 1789123413.420373 mapwrite 0x98a1f1 thru 0x98a825 (0x635 bytes) 000448[0] 1789123413.576120 read 0x586d8b thru 0x5886ad (0x1923 bytes) 000449[0] 1789123413.584631 mapwrite 0x6bbee4 thru 0x6bdb4b (0x1c68 bytes) 000453[0] 1789123413.697136 write 0x3bf41a thru 0x3c08d4 (0x14bb bytes) 000454[1] 1789123413.714631 write 0x740ced thru 0x74107c (0x390 bytes) 000456[0] 1789123413.790162 mapwrite 0x2b5ed7 thru 0x2b6ff8 (0x1122 bytes) 000457[0] 1789123413.947275 read 0x405524 thru 0x406b58 (0x1635 bytes) 000460[1] 1789123414.015593 write 0x8ca88e thru 0x8cc50c (0x1c7f bytes) 000464[1] 1789123414.072747 mapwrite 0x9bd5d4 thru 0x9be01c (0xa49 bytes) 000465[0] 1789123414.180661 mapwrite 0x4e72b4 thru 0x4e81a8 (0xef5 bytes) 000466[0] 1789123414.280418 trunc from 0x9be01d to 0x1938b2 (0x82a76a bytes) 000467[0] 1789123414.315604 mapwrite 0x69b8d0 thru 0x69bc98 (0x3c9 bytes) 000468[0] 1789123414.470602 mapread 0x1f4fb1 thru 0x1f6e5a (0x1eaa bytes) 000470[1] 1789123414.515489 mapwrite 0x451a9d thru 0x45228f (0x7f3 bytes) 000471[1] 1789123414.631836 mapread 0x12666f thru 0x128355 (0x1ce7 bytes) 000473[1] 1789123414.770932 write 0x0215ef thru 0x0228c7 (0x12d9 bytes) 000474[1] 1789123414.835707 read 0x567727 thru 0x568dc3 (0x169d bytes) 000475[1] 1789123414.898531 mapwrite 0x41fd3c thru 0x420f12 (0x11d7 bytes) 000479[1] 1789123415.186538 mapread 0x4bdd49 thru 0x4bedc7 (0x107f bytes) 000482[1] 1789123415.296311 trunc from 0x69bc99 to 0x218048 (0x483c50 bytes) 000483[0] 1789123415.349042 trunc from 0x218049 to 0x500bb0 (0x2e8b68 bytes) 000488[1] 1789123415.400010 write 0x4fb1c2 thru 0x4fbfa0 (0xddf bytes) 000492[1] 1789123415.497715 mapread 0x2c1c52 thru 0x2c3971 (0x1d20 bytes) 000495[1] 1789123415.623852 mapread 0x36d335 thru 0x36d432 (0xfe bytes) 000496[0] 1789123415.648564 write 0x90a27a thru 0x90adc8 (0xb4f bytes) 000497[1] 1789123415.683514 mapwrite 0x7b05e7 thru 0x7b0ad8 (0x4f2 bytes) 000498[1] 1789123415.772180 write 0x9f4839 thru 0x9f60ce (0x1896 bytes) All operations completed A-OK! run again with default file layout /mnt/lustre2/f16b.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 18 0x12 0x280000400 Chance of close/open is 1 in 50 Seed set to 3418 fd 0: /mnt/lustre/f16b.sanityn fd 1: /mnt/lustre2/f16b.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000001[1] 1789123419.418077 mapwrite 0x5c63d0 thru 0x5c7b05 (0x1736 bytes) 000003[1] 1789123419.554390 mapwrite 0x7b2328 thru 0x7b260f (0x2e8 bytes) 000004[1] 1789123419.735748 write 0x8476ed thru 0x848532 (0xe46 bytes) 000005[1] 1789123419.777537 write 0x258d25 thru 0x25a43c (0x1718 bytes) 000006[0] 1789123419.845780 read 0x28d38e thru 0x28dfc8 (0xc3b bytes) 000007[1] 1789123419.917261 read 0x2365a7 thru 0x236dd6 (0x830 bytes) 000009[1] 1789123419.951536 mapread 0x5ce248 thru 0x5cfe78 (0x1c31 bytes) 000010[0] 1789123419.973931 mapwrite 0x3c7e66 thru 0x3c919c (0x1337 bytes) 000020[0] 1789123420.137017 write 0x2b83bc thru 0x2b8668 (0x2ad bytes) 000021[1] 1789123420.144481 mapwrite 0x4e3193 thru 0x4e3d1f (0xb8d bytes) 000022[1] 1789123420.234545 read 0x2908fc thru 0x2917d3 (0xed8 bytes) truncating to largest ever: 0x8146ec 000023[0] 1789123420.240644 trunc from 0x848533 to 0x8146eb (0x33e47 bytes) 000024[0] 1789123420.296772 read 0x3cbd8b thru 0x3ccf47 (0x11bd bytes) 000027[1] 1789123420.337786 mapread 0x4d7b45 thru 0x4d887c (0xd38 bytes) 000028[1] 1789123420.367246 trunc from 0x8146ec to 0x04af6e (0x7c977d bytes) 000029[1] 1789123420.401665 read 0x03ad05 thru 0x03afbb (0x2b7 bytes) 000030[1] 1789123420.439677 mapwrite 0x359abe thru 0x35a2f4 (0x837 bytes) 000032[0] 1789123420.602197 trunc from 0x35a2f5 to 0x1684dc (0x1f1e18 bytes) 000033[1] 1789123420.673734 write 0x3484fa thru 0x349f76 (0x1a7d bytes) 000034[1] 1789123420.708709 mapwrite 0x74320a thru 0x7432ae (0xa5 bytes) 000037[1] 1789123420.896118 trunc from 0x7432af to 0x1d71c0 (0x56c0ee bytes) 000039[0] 1789123420.932573 trunc from 0x1d71c1 to 0x047645 (0x18fb7b bytes) 000041[1] 1789123421.039982 mapwrite 0x789b5e thru 0x78a547 (0x9ea bytes) 000046[1] 1789123421.225869 read 0x78a461 thru 0x78a547 (0xe7 bytes) 000048[1] 1789123421.249114 read 0x07fec5 thru 0x081ab7 (0x1bf3 bytes) 000052[1] 1789123421.323202 trunc from 0x78a548 to 0x72119a (0x693ad bytes) 000053[0] 1789123421.392976 write 0x49ddf9 thru 0x49ecee (0xef6 bytes) 000054[1] 1789123421.447094 trunc from 0x72119b to 0x17a0db (0x5a70bf bytes) 000056[1] 1789123421.588536 write 0x48297e thru 0x482eeb (0x56e bytes) 000057[0] 1789123421.626252 write 0x30d2aa thru 0x30df5f (0xcb6 bytes) 000058[1] 1789123421.692476 mapwrite 0x57c8ec thru 0x57d8d2 (0xfe7 bytes) 000059[0] 1789123421.826387 read 0x3f3b82 thru 0x3f4c41 (0x10c0 bytes) 000060[0] 1789123421.894141 mapread 0x2bbe2e thru 0x2bce52 (0x1025 bytes) 000061[1] 1789123421.920167 read 0x3a14f4 thru 0x3a2c9f (0x17ac bytes) 000064[0] 1789123421.939330 read 0x47a680 thru 0x47a8b4 (0x235 bytes) 000065[1] 1789123421.961328 read 0x290c88 thru 0x2923a2 (0x171b bytes) 000066[0] 1789123421.977729 mapwrite 0x764d42 thru 0x7654ab (0x76a bytes) 000067[0] 1789123422.188996 read 0x6f6a65 thru 0x6f77b9 (0xd55 bytes) 000068[0] 1789123422.241196 write 0x4a28f6 thru 0x4a2f8b (0x696 bytes) 000071[1] 1789123422.310035 read 0x48cc4d thru 0x48e8ef (0x1ca3 bytes) 000074[0] 1789123422.359192 mapread 0x0bdd1d thru 0x0be480 (0x764 bytes) 000075[0] 1789123422.379677 mapread 0x37af6f thru 0x37cc9b (0x1d2d bytes) 000078[0] 1789123422.456195 read 0x45e73c thru 0x45fdb3 (0x1678 bytes) 000086[1] 1789123422.482249 trunc from 0x7654ac to 0x7d6155 (0x70caa bytes) 000087[1] 1789123422.566493 mapread 0x35e96d thru 0x35ff93 (0x1627 bytes) 000093[0] 1789123422.620068 write 0x830b3e thru 0x830df3 (0x2b6 bytes) 000095[1] 1789123422.651732 mapwrite 0x54017d thru 0x541cc0 (0x1b44 bytes) 000096[0] 1789123422.749185 mapread 0x558d43 thru 0x559373 (0x631 bytes) 000098[1] 1789123422.788745 trunc from 0x830df4 to 0x71fb1f (0x1112d4 bytes) 000099[1] 1789123422.877210 mapwrite 0x0df4e4 thru 0x0e0eb9 (0x19d6 bytes) 000102[1] 1789123422.993943 read 0x456a6d thru 0x456cde (0x272 bytes) 000103[0] 1789123423.021976 read 0x25784f thru 0x2584f5 (0xca7 bytes) 000104[1] 1789123423.052506 trunc from 0x71fb20 to 0x1d94cf (0x546650 bytes) 000105[1] 1789123423.114641 trunc from 0x1d94d0 to 0x31dd09 (0x14483a bytes) 000107[0] 1789123423.142241 read 0x2a7010 thru 0x2a7ad2 (0xac3 bytes) 000110[0] 1789123423.258881 mapread 0x0e53bb thru 0x0e579f (0x3e5 bytes) 000111[1] 1789123423.259525 write 0x341832 thru 0x341d68 (0x537 bytes) 000114[1] 1789123423.311598 read 0x2dd89b thru 0x2ddbbc (0x322 bytes) 000115[0] 1789123423.420969 write 0x1ebfff thru 0x1ed19a (0x119c bytes) 000116[1] 1789123423.487884 mapwrite 0x4fc011 thru 0x4fd0d9 (0x10c9 bytes) 000120[0] 1789123423.644584 mapread 0x0b42db thru 0x0b61a2 (0x1ec8 bytes) 000121[0] 1789123423.719449 mapwrite 0x3af6aa thru 0x3b0336 (0xc8d bytes) 000127[1] 1789123423.876087 read 0x27a4f5 thru 0x27ac9a (0x7a6 bytes) 000129[0] 1789123423.903196 write 0x17d1b6 thru 0x17e259 (0x10a4 bytes) 000130[1] 1789123423.961408 mapread 0x0ce8d6 thru 0x0d0462 (0x1b8d bytes) 000132[1] 1789123424.026984 write 0x0424c6 thru 0x0436e5 (0x1220 bytes) 000134[0] 1789123424.065079 mapread 0x3760b0 thru 0x376f0f (0xe60 bytes) 000137[0] 1789123424.113285 mapwrite 0x0d4d35 thru 0x0d543f (0x70b bytes) 000139[1] 1789123424.276975 read 0x28d370 thru 0x28ee23 (0x1ab4 bytes) 000140[1] 1789123424.302274 mapwrite 0x4e60b1 thru 0x4e7057 (0xfa7 bytes) 000141[1] 1789123424.489523 mapwrite 0x2093b7 thru 0x209e47 (0xa91 bytes) 000142[0] 1789123424.629498 mapwrite 0x48b2ff thru 0x48c9ba (0x16bc bytes) 000145[1] 1789123424.779322 mapread 0x19a358 thru 0x19b108 (0xdb1 bytes) 000148[0] 1789123424.827978 trunc from 0x4fd0da to 0x54b151 (0x4e078 bytes) 000152[0] 1789123424.917306 mapread 0x1e269d thru 0x1e3c74 (0x15d8 bytes) 000153[0] 1789123424.929679 mapread 0x22f161 thru 0x230d24 (0x1bc4 bytes) 000154[1] 1789123424.958187 mapread 0x3ee227 thru 0x3ee90b (0x6e5 bytes) 000157[0] 1789123424.991566 write 0x108a9b thru 0x109146 (0x6ac bytes) 000158[1] 1789123425.040000 write 0x35bdf1 thru 0x35cc32 (0xe42 bytes) 000162[0] 1789123425.079084 write 0x525659 thru 0x526702 (0x10aa bytes) 000164[0] 1789123425.132836 mapwrite 0x3d57d5 thru 0x3d66f8 (0xf24 bytes) 000166[0] 1789123425.210015 read 0x1e1638 thru 0x1e1eaf (0x878 bytes) 000168[1] 1789123425.254107 mapread 0x4f7152 thru 0x4f8576 (0x1425 bytes) 000169[1] 1789123425.303651 trunc from 0x54b152 to 0x314acb (0x236686 bytes) 000172[0] 1789123425.377907 mapread 0x12fb43 thru 0x131353 (0x1811 bytes) 000174[0] 1789123425.440291 write 0x63cef4 thru 0x63ed4c (0x1e59 bytes) 000175[1] 1789123425.490029 mapread 0x4252fd thru 0x426934 (0x1638 bytes) 000177[1] 1789123425.570836 trunc from 0x63ed4d to 0x6f462d (0xb58e1 bytes) 000178[0] 1789123425.699770 read 0x3f72dc thru 0x3f9077 (0x1d9c bytes) 000179[0] 1789123425.755501 mapread 0x4ff84a thru 0x501675 (0x1e2c bytes) 000181[0] 1789123425.801842 mapwrite 0x85f564 thru 0x860f57 (0x19f4 bytes) 000187[0] 1789123426.034996 write 0x1ad479 thru 0x1af2aa (0x1e32 bytes) 000188[0] 1789123426.071156 write 0x8ca186 thru 0x8caa09 (0x884 bytes) 000189[0] 1789123426.073660 trunc from 0x8caa0a to 0x7d9fd5 (0xf0a34 bytes) 000190[0] 1789123426.113105 read 0x208650 thru 0x208b48 (0x4f9 bytes) 000196[0] 1789123426.264160 write 0x2db06d thru 0x2dceef (0x1e83 bytes) 000201[0] 1789123426.355191 trunc from 0x7d9fd6 to 0x62939d (0x1b0c38 bytes) 000202[1] 1789123426.418134 read 0x20764a thru 0x2088d3 (0x128a bytes) 000203[1] 1789123426.467160 trunc from 0x62939e to 0x7ad88c (0x1844ef bytes) 000204[0] 1789123426.529196 trunc from 0x7ad88d to 0x27d7e8 (0x5300a4 bytes) 000205[1] 1789123426.596584 mapwrite 0x87fe67 thru 0x880831 (0x9cb bytes) 000206[1] 1789123426.800402 mapwrite 0x58981c thru 0x58add7 (0x15bc bytes) 000209[0] 1789123426.931053 write 0x2f5033 thru 0x2f57ed (0x7bb bytes) 000211[0] 1789123427.064379 write 0x946c71 thru 0x9482ce (0x165e bytes) 000214[0] 1789123427.157501 write 0x292959 thru 0x2944ee (0x1b96 bytes) 000215[1] 1789123427.187154 write 0x56f9bf thru 0x5704df (0xb21 bytes) truncating to largest ever: 0x9e4b4f 000216[0] 1789123427.318086 trunc from 0x9482cf to 0x9e4b4e (0x9c880 bytes) 000217[1] 1789123427.419047 trunc from 0x9e4b4f to 0x374309 (0x670845 bytes) 000219[0] 1789123427.481179 read 0x35294d thru 0x3538b2 (0xf66 bytes) 000220[0] 1789123427.513114 mapread 0x2cc89f thru 0x2cdfd4 (0x1736 bytes) 000223[0] 1789123427.530486 read 0x0ad8f5 thru 0x0af048 (0x1754 bytes) 000226[0] 1789123427.641303 write 0x53c0c7 thru 0x53c88b (0x7c5 bytes) 000227[1] 1789123427.686168 read 0x142bcc thru 0x142c37 (0x6c bytes) 000230[0] 1789123427.719792 trunc from 0x53c88c to 0x84b7fd (0x30ef72 bytes) 000233[1] 1789123427.811000 trunc from 0x84b7fe to 0x6a375d (0x1a80a0 bytes) 000237[0] 1789123427.909090 mapwrite 0x6ee53a thru 0x6eea49 (0x510 bytes) 000240[1] 1789123428.141893 mapwrite 0x24c1a8 thru 0x24d9eb (0x1844 bytes) 000242[0] 1789123428.264550 mapread 0x6c2401 thru 0x6c27f1 (0x3f1 bytes) 000245[1] 1789123428.298377 write 0x6e16a8 thru 0x6e279b (0x10f4 bytes) 000246[1] 1789123428.373179 write 0x08eb00 thru 0x08f067 (0x568 bytes) 000247[1] 1789123428.402005 write 0x059ada thru 0x05b410 (0x1937 bytes) 000248[0] 1789123428.433672 read 0x407e35 thru 0x409c33 (0x1dff bytes) 000250[0] 1789123428.523407 mapwrite 0x4bdd0d thru 0x4bfa5a (0x1d4e bytes) 000251[1] 1789123428.747157 read 0x5670df thru 0x56745c (0x37e bytes) 000252[1] 1789123428.784263 read 0x261af7 thru 0x262139 (0x643 bytes) 000255[1] 1789123428.802028 mapread 0x073cbd thru 0x0742c2 (0x606 bytes) 000259[1] 1789123428.820129 mapwrite 0x05bc9b thru 0x05c9f3 (0xd59 bytes) 000261[0] 1789123429.002612 mapwrite 0x86a8a3 thru 0x86c440 (0x1b9e bytes) 000264[1] 1789123429.214831 write 0x16d248 thru 0x16d387 (0x140 bytes) 000265[0] 1789123429.265308 mapwrite 0x134659 thru 0x136630 (0x1fd8 bytes) 000272[1] 1789123429.482311 write 0x4f41d9 thru 0x4f43dd (0x205 bytes) 000273[0] 1789123429.512942 read 0x0ae7d4 thru 0x0afcff (0x152c bytes) 000275[0] 1789123429.537252 read 0x2c35f3 thru 0x2c3770 (0x17e bytes) 000276[1] 1789123429.543019 write 0x5612d4 thru 0x562492 (0x11bf bytes) 000277[1] 1789123429.578830 mapwrite 0x60798c thru 0x607e4e (0x4c3 bytes) 000279[1] 1789123429.674573 write 0x384ef2 thru 0x3855fe (0x70d bytes) 000282[1] 1789123429.736394 read 0x4bf628 thru 0x4c0a5b (0x1434 bytes) truncating to largest ever: 0x9f17ad 000285[0] 1789123429.778945 trunc from 0x86c441 to 0x9f17ac (0x18536c bytes) 000286[1] 1789123429.834282 read 0x375a38 thru 0x376b2b (0x10f4 bytes) 000292[0] 1789123429.866992 mapwrite 0x876824 thru 0x876cbc (0x499 bytes) 000293[0] 1789123429.922206 write 0x0d3af8 thru 0x0d51bf (0x16c8 bytes) 000297[1] 1789123429.977850 read 0x8ab99b thru 0x8ac651 (0xcb7 bytes) 000299[1] 1789123430.002885 mapwrite 0x2acfb7 thru 0x2aded4 (0xf1e bytes) 000300[0] 1789123430.078957 mapwrite 0x056aec thru 0x05744d (0x962 bytes) 000301[0] 1789123430.165119 write 0x5dc078 thru 0x5ddc18 (0x1ba1 bytes) 000302[1] 1789123430.175504 write 0x05aa93 thru 0x05c0c6 (0x1634 bytes) 000303[1] 1789123430.206313 trunc from 0x9f17ad to 0x0ac026 (0x945786 bytes) 000304[1] 1789123430.239349 mapread 0x033277 thru 0x0334e8 (0x272 bytes) 000305[0] 1789123430.260756 read 0x066919 thru 0x0674e9 (0xbd1 bytes) 000306[0] 1789123430.306692 write 0x586d1e thru 0x587b1c (0xdff bytes) 000310[1] 1789123430.385224 write 0x8c0df3 thru 0x8c25a8 (0x17b6 bytes) 000311[0] 1789123430.422030 trunc from 0x8c25a9 to 0x44d4aa (0x4750fe bytes) 000313[1] 1789123430.519829 mapread 0x01e8de thru 0x01ff6e (0x1691 bytes) 000316[1] 1789123430.595718 mapread 0x0c2296 thru 0x0c2f66 (0xcd1 bytes) 000318[1] 1789123430.605802 read 0x3690f6 thru 0x369efa (0xe05 bytes) 000320[1] 1789123430.612165 write 0x6a8275 thru 0x6a9385 (0x1111 bytes) 000324[0] 1789123430.648104 trunc from 0x6a9386 to 0x6dc825 (0x334a0 bytes) 000329[1] 1789123430.779926 write 0x2d5110 thru 0x2d5cbe (0xbaf bytes) 000332[0] 1789123430.824719 trunc from 0x6dc826 to 0x282976 (0x459eaf bytes) 000334[1] 1789123430.891517 read 0x2138c5 thru 0x214cf5 (0x1431 bytes) 000336[0] 1789123430.977086 mapread 0x08e6ee thru 0x08f6c2 (0xfd5 bytes) 000340[0] 1789123431.072270 mapread 0x0b14bf thru 0x0b17d7 (0x319 bytes) 000347[1] 1789123431.096087 mapread 0x0b5430 thru 0x0b61d3 (0xda4 bytes) 000349[0] 1789123431.100098 mapwrite 0x9bfd0e thru 0x9c1931 (0x1c24 bytes) 000350[0] 1789123431.340405 mapread 0x89c349 thru 0x89ca8f (0x747 bytes) 000351[1] 1789123431.351982 mapwrite 0x35a0ea thru 0x35b97e (0x1895 bytes) 000353[1] 1789123431.550785 mapread 0x295709 thru 0x2957e2 (0xda bytes) 000355[0] 1789123431.568990 write 0x5c5a7e thru 0x5c5dc7 (0x34a bytes) 000356[0] 1789123431.626848 trunc from 0x9c1932 to 0x108017 (0x8b991a bytes) 000358[1] 1789123431.659380 trunc from 0x108018 to 0x16646b (0x5e454 bytes) 000359[0] 1789123431.703902 trunc from 0x16646c to 0x27a26b (0x113e00 bytes) 000360[0] 1789123431.780260 mapwrite 0x6951a3 thru 0x6963b4 (0x1212 bytes) 000361[0] 1789123431.966201 mapwrite 0x5a9dfd thru 0x5ab74c (0x1950 bytes) 000362[1] 1789123432.082294 mapwrite 0x8fd901 thru 0x8fe66a (0xd6a bytes) 000363[1] 1789123432.271666 read 0x8f8104 thru 0x8f9410 (0x130d bytes) 000366[1] 1789123432.278730 read 0x340487 thru 0x3423a9 (0x1f23 bytes) 000367[1] 1789123432.285037 mapwrite 0x677bab thru 0x67910a (0x1560 bytes) 000369[0] 1789123432.359213 mapwrite 0x48276c thru 0x482d30 (0x5c5 bytes) 000370[1] 1789123432.447099 trunc from 0x8fe66b to 0x3c38f8 (0x53ad72 bytes) 000373[1] 1789123432.531026 write 0x3dd750 thru 0x3de980 (0x1231 bytes) 000374[1] 1789123432.552235 trunc from 0x3de981 to 0x9190a7 (0x53a727 bytes) 000375[0] 1789123432.587550 mapread 0x8d49f7 thru 0x8d5ddf (0x13e9 bytes) 000376[0] 1789123432.631794 trunc from 0x9190a8 to 0x8a76ae (0x719f9 bytes) 000377[1] 1789123432.692058 write 0x7dbe5b thru 0x7dc7ab (0x951 bytes) 000381[0] 1789123432.828147 read 0x04d63a thru 0x04ea41 (0x1408 bytes) 000383[0] 1789123432.892290 read 0x802ad9 thru 0x803179 (0x6a1 bytes) 000385[0] 1789123432.908267 read 0x0b7b59 thru 0x0b91f4 (0x169c bytes) 000386[0] 1789123432.926010 trunc from 0x8a76af to 0x8dade6 (0x33738 bytes) 000389[0] 1789123433.008597 read 0x395624 thru 0x396c2b (0x1608 bytes) 000390[0] 1789123433.016910 mapread 0x41d223 thru 0x41eab4 (0x1892 bytes) 000391[0] 1789123433.062476 mapread 0x8166b3 thru 0x818696 (0x1fe4 bytes) 000392[1] 1789123433.103161 mapread 0x2f54ca thru 0x2f6792 (0x12c9 bytes) 000395[0] 1789123433.140788 mapwrite 0x5c0c76 thru 0x5c220a (0x1595 bytes) 000396[0] 1789123433.238665 read 0x1ba98b thru 0x1badfe (0x474 bytes) 000398[1] 1789123433.287396 mapwrite 0x6f9b49 thru 0x6fa20b (0x6c3 bytes) 000401[1] 1789123433.453935 mapwrite 0x8db8ad thru 0x8dd66e (0x1dc2 bytes) 000406[1] 1789123433.644052 read 0x7ce481 thru 0x7cf552 (0x10d2 bytes) 000407[1] 1789123433.654750 write 0x803004 thru 0x803577 (0x574 bytes) 000408[0] 1789123433.661918 read 0x754eb7 thru 0x755f69 (0x10b3 bytes) 000412[0] 1789123433.700343 mapread 0x5a8651 thru 0x5a8d72 (0x722 bytes) 000413[0] 1789123433.708077 mapwrite 0x852cea thru 0x853b75 (0xe8c bytes) 000415[0] 1789123433.791758 mapread 0x69a113 thru 0x69b62a (0x1518 bytes) 000416[0] 1789123433.826895 trunc from 0x8dd66f to 0x65bf02 (0x28176c bytes) 000418[0] 1789123433.880682 mapread 0x45ea61 thru 0x45f481 (0xa21 bytes) 000419[1] 1789123433.895950 trunc from 0x65bf03 to 0x09bce9 (0x5c0219 bytes) 000421[1] 1789123433.956420 write 0x67d412 thru 0x67d680 (0x26f bytes) 000423[1] 1789123434.056590 write 0x52eb41 thru 0x530705 (0x1bc5 bytes) 000424[1] 1789123434.093171 trunc from 0x67d681 to 0x08b771 (0x5f1f0f bytes) 000425[1] 1789123434.136056 trunc from 0x08b772 to 0x0bfe3f (0x346ce bytes) 000426[0] 1789123434.156157 mapwrite 0x4d0cd9 thru 0x4d213a (0x1462 bytes) 000427[0] 1789123434.270106 read 0x3982e2 thru 0x3983d3 (0xf2 bytes) 000428[0] 1789123434.311168 write 0x7f8f2f thru 0x7fa244 (0x1316 bytes) 000431[0] 1789123434.320772 trunc from 0x7fa245 to 0x27ebbd (0x57b687 bytes) 000433[0] 1789123434.411026 trunc from 0x27ebbe to 0x428634 (0x1a9a77 bytes) 000434[1] 1789123434.456741 mapwrite 0x6e0890 thru 0x6e09f5 (0x166 bytes) 000435[1] 1789123434.619205 trunc from 0x6e09f6 to 0x38ee36 (0x351bbf bytes) 000437[1] 1789123434.655961 write 0x6e4972 thru 0x6e6519 (0x1ba8 bytes) 000438[0] 1789123434.708297 mapread 0x65ba8f thru 0x65be4c (0x3be bytes) 000439[0] 1789123434.754270 read 0x3ad0cd thru 0x3ad5ae (0x4e2 bytes) 000440[1] 1789123434.759980 write 0x1a1a12 thru 0x1a2a1f (0x100e bytes) 000442[0] 1789123434.796705 mapwrite 0x5fbeaf thru 0x5fd8f7 (0x1a49 bytes) 000444[0] 1789123434.904198 trunc from 0x6e651a to 0x9bbe79 (0x2d5960 bytes) 000445[0] 1789123434.946980 read 0x2f80a7 thru 0x2f82f4 (0x24e bytes) 000446[1] 1789123434.957745 mapwrite 0x84b634 thru 0x84d3ad (0x1d7a bytes) 000448[0] 1789123435.056959 write 0x05a169 thru 0x05c07c (0x1f14 bytes) 000449[0] 1789123435.094149 mapread 0x3512af thru 0x351a9f (0x7f1 bytes) 000451[0] 1789123435.115111 mapread 0x90c08e thru 0x90ce05 (0xd78 bytes) 000453[1] 1789123435.124977 mapread 0x96c720 thru 0x96d688 (0xf69 bytes) 000454[0] 1789123435.194242 mapwrite 0x2efeac thru 0x2f1732 (0x1887 bytes) 000455[0] 1789123435.317933 trunc from 0x9bbe7a to 0x7d6c17 (0x1e5262 bytes) 000456[1] 1789123435.366750 write 0x966988 thru 0x967f17 (0x1590 bytes) 000457[0] 1789123435.388296 trunc from 0x967f18 to 0x462fa5 (0x504f72 bytes) 000460[1] 1789123435.448108 mapread 0x2bec23 thru 0x2c0282 (0x1660 bytes) 000461[1] 1789123435.585129 mapread 0x1598fe thru 0x15ab77 (0x127a bytes) 000465[0] 1789123435.619632 mapread 0x3ee5ac thru 0x3ef10c (0xb61 bytes) 000467[0] 1789123435.654925 mapread 0x32c2da thru 0x32d777 (0x149e bytes) 000469[0] 1789123435.752574 mapwrite 0x60d3a5 thru 0x60ecb4 (0x1910 bytes) 000471[0] 1789123436.016737 write 0x8424a3 thru 0x84333b (0xe99 bytes) 000473[0] 1789123436.023789 mapwrite 0x14cd1f thru 0x14dc73 (0xf55 bytes) 000477[1] 1789123436.106059 mapread 0x50cd3b thru 0x50e540 (0x1806 bytes) 000479[0] 1789123436.153364 mapread 0x038ce0 thru 0x038ff1 (0x312 bytes) 000482[1] 1789123436.169809 mapread 0x5b62f1 thru 0x5b6831 (0x541 bytes) 000483[1] 1789123436.195895 read 0x009e05 thru 0x00ac2d (0xe29 bytes) 000485[1] 1789123436.205700 trunc from 0x84333c to 0x61d58e (0x225dad bytes) 000487[0] 1789123436.275230 mapwrite 0x513d98 thru 0x5148e7 (0xb50 bytes) 000488[0] 1789123436.380582 write 0x03d3ec thru 0x03f09c (0x1cb1 bytes) 000489[1] 1789123436.396728 mapwrite 0x987e83 thru 0x987eb4 (0x32 bytes) 000490[0] 1789123436.576151 trunc from 0x987eb5 to 0x1eed89 (0x79912b bytes) 000491[0] 1789123436.692026 trunc from 0x1eed8a to 0x864b95 (0x675e0c bytes) 000492[1] 1789123436.726186 mapread 0x26b571 thru 0x26ce3c (0x18cc bytes) 000493[0] 1789123436.791853 mapread 0x457462 thru 0x45863c (0x11db bytes) 000494[1] 1789123436.820699 mapread 0x4e33b6 thru 0x4e4e4b (0x1a96 bytes) 000495[0] 1789123436.852187 mapread 0x7dbcd6 thru 0x7dc80c (0xb37 bytes) 000498[1] 1789123436.869170 read 0x54c37f thru 0x54cd42 (0x9c4 bytes) 000499[1] 1789123436.874267 trunc from 0x864b96 to 0x6a9dc2 (0x1badd3 bytes) 000500[0] 1789123436.934964 read 0x1adf6f thru 0x1ae0c3 (0x155 bytes) All operations completed A-OK! PASS 16b (54s) == sanityn test 16c: verify data consistency on ldiskfs with cache disabled (b=17397) ========================================================== 06:44:03 (1789123443) SKIP: sanityn test_16c dio on ldiskfs only SKIP 16c (5s) == sanityn test 16d: Verify DIO and buffer IO with two clients ========================================================== 06:44:08 (1789123448) 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.439194 s, 19.1 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.319768 s, 26.2 MB/s ldlm.namespaces.MGC192.168.204.124@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8b688b34a000.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8b6890c8a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8b688b34a000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8b6890c8a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8b688b34a000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8b6890c8a800.lru_size=clear 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.4126 s, 9.2 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 3.81953 s, 27.5 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 2.32958 s, 45.0 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.67041 s, 12.1 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0946 s, 10.4 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.10834 s, 20.5 MB/s PASS 16d (56s) == sanityn test 16e: Verify size consistency for O_DIRECT write ========================================================== 06:45:05 (1789123505) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.35958 s, 29.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199296 s, 5.3 MB/s PASS 16e (8s) == sanityn test 16f: rw sequential consistency vs drop_caches ========================================================== 06:45:13 (1789123513) TIMEOUT PASS 16f (28s) == sanityn test 16g: mmap rw sequential consistency vs drop_caches ========================================================== 06:45:41 (1789123541) TIMEOUT PASS 16g (29s) == sanityn test 16h: mmap read after truncate file ======= 06:46:11 (1789123571) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16h (9s) == sanityn test 16i: read after truncate file ============ 06:46:19 (1789123579) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16i (8s) == sanityn test 16j: race dio with buffered i/o ========== 06:46:27 (1789123587) 1+0 records in 1+0 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.20635 s, 11.9 MB/s bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.39353 s, 18.8 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.78424 s, 9.4 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.04401 s, 12.8 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.69552 s, 9.7 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.44531 s, 10.7 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.97142 s, 8.8 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.65426 s, 9.9 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.71169 s, 9.7 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.39605 s, 7.7 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.62187 s, 7.2 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.22569 s, 11.8 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.32076 s, 11.3 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.95708 s, 8.9 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.09894 s, 8.5 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.45112 s, 10.7 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.64439 s, 9.9 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.18491 s, 8.2 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.35825 s, 7.8 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK PASS 16j (40s) == sanityn test 16k: Parallel FSX and drop caches should not panic ========================================================== 06:47:07 (1789123627) + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.1 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.2 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.3 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.4 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.5 -l 67108864 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read skipping zero size read truncating to largest ever: 0x1a7062c truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x1a7062c truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! + fsx 1 done: rc=0 All operations completed A-OK! Fri Sep 11 06:47:58 EDT 2026 + fsx 2 done: rc=0 Fri Sep 11 06:47:58 EDT 2026 + fsx 3 done: rc=0 Fri Sep 11 06:47:58 EDT 2026 + fsx 4 done: rc=0 Fri Sep 11 06:47:58 EDT 2026 + fsx 5 done: rc=0 Fri Sep 11 06:47:58 EDT 2026 PASS 16k (60s) == sanityn test 17: resource creation/LVB creation race ========================================================================= 06:48:07 (1789123687) fail_loc=0x8000030a PASS 17 (12s) == sanityn test 18: mmap sanity check =========================================================================================== 06:48:20 (1789123700) mmap test1: basic mmap operation (PASS, 0.1856s) mmap test2: MAP_PRIVATE not write back (PASS, 0.17672s) mmap test3: concurrent mmap ops on two nodes (PASS, 8.0325s) mmap test4: c1 write to f1 from mmapped f2, c2 write to f1 from mmapped f1 (PASS, 4.7496s) mmap test5: read/write file to/from the buffer which mmapped to just this file (PASS, 0.22547s) mmap test6: check mmap write/read content on two nodes (PASS, 0.25541s) mmap test7: file i/o with an unmapped buffer (SKIPPED, 0s) mmap test8: SIGBUS for beyond file size (SKIPPED, 0s) mmap test9: SIGBUS for negative file offset (SKIPPED, 0s) mmap test10: multi-thread mmap access (PASS, 1.4505s) mmap test11: multi-thread shared mmap (PASS, 1.0959s) mmap test10: mtime not change for readonly mmap access (PASS, 1.1578s) PASS 18 (27s) == sanityn test 19: test concurrent uncached read races ========================================================================= 06:48:46 (1789123726) oleg424-server: error: get_param: param_path 'osd-*/lustre-OST*/read_cache_enable': No such file or directory pdsh@oleg424-client: oleg424-server: ssh exited with exit code 2 SKIP: sanityn test_19 not cache-capable obdfilter SKIP 19 (5s) == sanityn test 20: test extra readahead page left in cache ============================================================== 06:48:51 (1789123731) PASS 20 (7s) == sanityn test 21: Try to remove mountpoint on another dir ============================================================== 06:48:59 (1789123739) rmdir: removing directory, '/mnt/lustre/d21.sanityn' rmdir: failed to remove '/mnt/lustre/d21.sanityn': Device or resource busy rmdir: removing directory, '/mnt/lustre2/d21.sanityn' Removed mounted directory from another mountpoint, needs to be fixed PASS 21 (6s) == sanityn test 23: others should see updated atime while another read============================================================== 06:49:06 (1789123746) now is 1789123748 starting reads multiop /mnt/lustre/f23.sanityn vor20_c TMPPIPE=/tmp/multiop_open_wait_pipe.7649 new atime is 1789123809 PASS 23 (70s) == sanityn test 24a: lfs df [-ih] [path] test =================================================================================== 06:50:15 (1789123815) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 31744 3737600 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 7168 3762176 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 31744 3737600 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 7168 3762176 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre2 UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 114.5K 403 114.1K 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 115.3K 399 114.9K 1% /mnt/lustre[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 114.5K 403 114.1K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 115.3K 399 114.9K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre2 UUID bytes Used Available Use% Mounted on lustre-MDT0000_UUID 2.1G 2.1M 2.1G 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3.6G 31.0M 3.6G 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3.6G 7.0M 3.6G 1% /mnt/lustre[OST:1] filesystem_summary: 7.2G 38.0M 7.2G 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 117267 403 116864 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 118031 399 117632 1% /mnt/lustre2[OST:1] filesystem_summary: 234882 386 234496 1% /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 31744 3737600 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 7168 3762176 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 114.5K 403 114.1K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 115.3K 399 114.9K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 117267 403 116864 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 118031 399 117632 1% /mnt/lustre[OST:1] filesystem_summary: 234882 386 234496 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 117267 403 116864 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 118031 399 117632 1% /mnt/lustre2[OST:1] filesystem_summary: 234882 386 234496 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 31744 3737600 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 7168 3762176 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 31744 3737600 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 7168 3762176 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre2 PASS 24a (8s) == sanityn test 24b: lfs df should show both filesystems ========================================================================= 06:50:23 (1789123823) PASS 24b (7s) == sanityn test 25a: change ACL on one mountpoint be seen on another ============================================================= 06:50:30 (1789123830) running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] PASS 25a (7s) == sanityn test 25b: change ACL under remote dir on one mountpoint be seen on another ========================================================== 06:50:37 (1789123837) SKIP: sanityn test_25b needs >= 2 MDTs SKIP 25b (4s) == sanityn test 26a: allow mtime to get older ============ 06:50:41 (1789123841) utime: good mknod times 1789123842 <= 1789123843 <= 1789123843 for /mnt/lustre/f26a utime: good mknod times 1789123842 <= 1789123843 <= 1789123843 for /mnt/lustre2/f26a utime: good utime mtimes 100000, atime 200000 utime: updated times for /mnt/lustre2/f26a PASS 26a (8s) == sanityn test 26b: sync mtime between ost and mds ====== 06:50:50 (1789123850) PASS 26b (10s) == sanityn test 26c: set-in-past on open file is not lost on close ========================================================== 06:50:59 (1789123859) multiop /mnt/lustre/f26c.sanityn vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7649 File: /mnt/lustre/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502400 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-09-11 06:51:00.000000000 -0400 Birth: 2026-09-11 06:51:00.000000000 -0400 File: /mnt/lustre2/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502400 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-09-11 06:51:00.000000000 -0400 Birth: 2026-09-11 06:51:00.000000000 -0400 PASS 26c (7s) SKIP: sanityn test_27 skipping excluded test 27 == sanityn test 30: recreate file race =================== 06:51:08 (1789123868) 0 PASS 30 (8s) == sanityn test 31a: voluntary cancel / blocking ast race======================================================================== 06:51:16 (1789123876) fail_loc=0x314 PASS 31a (8s) == sanityn test 31b: voluntary OST cancel / blocking ast race======================================================================== 06:51:24 (1789123884) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg424-server mds-ost sync done. sleep 5 for ZFS MDS Waiting for MDT destroys to complete fail_loc=0x314 fail_loc=0x316 fail_loc=0 fail_loc=0 Filesystem 1K-blocks Used Available Use% Mounted on 192.168.204.124@tcp:/lustre 7542784 39936 7498752 1% /mnt/lustre2 PASS 31b (26s) == sanityn test 31r: open-rename(replace) race =========== 06:51:50 (1789123910) -rw-r--r-- 1 root root 0 Sep 11 06:51 /mnt/lustre/f31r.sanityn.target fail_loc=0x1419 fail_val=3 PASS 31r (11s) == sanityn test 31s: open should not revalidate invalid dentry ========================================================== 06:52:01 (1789123921) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 1225: 43523 Terminated tail -f $tfile PASS 31s (9s) == sanityn test 31t: getattr should not revalidate invalid dentry ========================================================== 06:52:10 (1789123930) PASS 31t (9s) == sanityn test 32b: lockless i/o ======================== 06:52:19 (1789123939) SKIP: sanityn test_32b max_nolock_bytes is removed >= 2.17.53 SKIP 32b (3s) == sanityn test 32c: contention detection ================ 06:52:22 (1789123942) ldlm.namespaces.filter-lustre-OST0000_UUID.contended_locks=2 ldlm.namespaces.filter-lustre-OST0001_UUID.contended_locks=2 ldlm.namespaces.filter-lustre-OST0000_UUID.contention_events=0 ldlm.namespaces.filter-lustre-OST0001_UUID.contention_events=0 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.73401 s, 30.2 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.68607 s, 31.1 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.33773 s, 22.4 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.67057 s, 19.6 MB/s contention_seconds 4, contention_events 17 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.9676 s, 26.6 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.51748 s, 20.8 MB/s ldlm.namespaces.filter-lustre-OST0000_UUID.contention_events=0 ldlm.namespaces.filter-lustre-OST0001_UUID.contention_events=0 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.33721 s, 39.2 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.35647 s, 38.7 MB/s PASS 32c (33s) SKIP: sanityn test_33a skipping SLOW test 33a == sanityn test 33b: COS: cross create/delete, 2 clients, benchmark under remote dir ========================================================== 06:52:57 (1789123977) SKIP: sanityn test_33b Need two or more clients, have 1 SKIP 33b (4s) == sanityn test 33c: Cancel cross-MDT lock should trigger Sync-on-Lock-Cancel ========================================================== 06:53:02 (1789123982) SKIP: sanityn test_33c needs >= 2 MDTs SKIP 33c (3s) == sanityn test 33d: dependent transactions should trigger COS ========================================================== 06:53:05 (1789123985) SKIP: sanityn test_33d needs >= 2 MDTs SKIP 33d (3s) == sanityn test 33e: independent transactions shouldn't trigger COS ========================================================== 06:53:09 (1789123989) SKIP: sanityn test_33e needs >= 2 MDTs SKIP 33e (4s) == sanityn test 34: no lock timeout under IO ============= 06:53:13 (1789123993) fail_loc=0x512 fail_loc=0x512 lock should not expire writing on client1 reading on client2 success fail_loc=0x511 fail_loc=0x511 lock should expire writing on client1 reading on client2 success oleg424-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8b688b34a000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8b688b34a000.ost_server_uuid in FULL state after 0 sec oleg424-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8b688b34a000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8b688b34a000.ost_server_uuid in IDLE state after 0 sec oleg424-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8b688b34a000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8b688b34a000.ost_server_uuid in IDLE state after 0 sec oleg424-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8b688b34a000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8b688b34a000.ost_server_uuid in IDLE state after 0 sec oleg424-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8b688b34a000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8b688b34a000.ost_server_uuid in IDLE state after 0 sec oleg424-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8b688b34a000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8b688b34a000.ost_server_uuid in IDLE state after 0 sec PASS 34 (146s) == sanityn test 35: -EINTR cp_ast vs. bl_ast race does not evict client ========================================================== 06:55:40 (1789124140) fail_loc=0x80000317 Race attempt 0 Wait for 50824 50850 for 60 sec... /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 1875: 50824 Killed createmany -o $MOUNT2/$tdir/a 4000 fail_loc=0x0 Filesystem Size Used Avail Use% Mounted on 192.168.204.124@tcp:/lustre 7.2G 39M 7.2G 1% /mnt/lustre 192.168.204.124@tcp:/lustre 7.2G 39M 7.2G 1% /mnt/lustre2 - open/close 511 (time 1789124158.41 total 10.01 last 51.06) - open/close 1154 (time 1789124168.42 total 20.02 last 64.24) - open/close 1779 (time 1789124178.43 total 30.03 last 62.44) - open/close 2461 (time 1789124188.44 total 40.03 last 68.15) - open/close 3084 (time 1789124198.45 total 50.05 last 62.24) - open/close 3752 (time 1789124208.45 total 60.05 last 66.75) total: 4000 open/close in 66.40 seconds: 60.24 ops/second PASS 35 (75s) == sanityn test 36: handle ESTALE/open-unlink correctly == 06:56:55 (1789124215) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg424-server mds-ost sync done. Waiting for MDT destroys to complete 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.899405 s, 58.3 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7649 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(0) *** before(7498752) after_dd(7435264) after(7498752) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.65784 s, 79.7 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7649 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(1) *** before(7498752) after_dd(7438336) after(7489536) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.650646 s, 80.6 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7649 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(2) *** before(7489536) after_dd(7425024) after(7476224) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.664513 s, 78.9 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7649 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(3) *** before(7476224) after_dd(7442432) after(7493632) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.582158 s, 90.1 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7649 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(4) *** before(7493632) after_dd(7440384) after(7491584) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.69948 s, 75.0 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7649 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(5) *** before(7491584) after_dd(7440384) after(7491584) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.921131 s, 56.9 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7649 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(6) *** before(7491584) after_dd(7440384) after(7491584) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.784618 s, 66.8 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7649 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(7) *** before(7491584) after_dd(7440384) after(7498752) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.12881 s, 46.4 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7649 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(8) *** before(7498752) after_dd(7440384) after(7491584) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.660983 s, 79.3 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7649 sleep 5 for ZFS MDS Waiting for MDT destroys to complete *** cycle(9) *** before(7491584) after_dd(7428096) after(7479296) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.934501 s, 56.1 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7649 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(10) *** before(7479296) after_dd(7440384) after(7491584) PASS 36 (318s) == sanityn test 37: check i_size is not updated for directory on close (bug 18695) ======================================================================== 07:02:13 (1789124533) multiop /mnt/lustre/d37.sanityn vD_c TMPPIPE=/tmp/multiop_open_wait_pipe.7649 - create 1148 (time 1789124548.35 total 10.01 last 114.73) - create 2233 (time 1789124558.36 total 20.01 last 108.41) - create 3477 (time 1789124568.36 total 30.01 last 124.40) - create 4656 (time 1789124578.36 total 40.02 last 117.83) - create 5905 (time 1789124588.37 total 50.02 last 124.88) - create 6979 (time 1789124598.37 total 60.03 last 107.33) - create 8064 (time 1789124608.38 total 70.03 last 108.47) - create 9155 (time 1789124618.38 total 80.04 last 109.05) total: 10000 create in 88.27 seconds: 113.29 ops/second PASS 37 (103s) == sanityn test 39a: file mtime does not change after rename ========================================================== 07:03:56 (1789124636) repeat after cancel_lru_locks PASS 39a (9s) == sanityn test 39b: file mtime the same on clients with/out lock ========================================================== 07:04:05 (1789124645) repeat after cancel_lru_locks PASS 39b (9s) == sanityn test 39c: check truncate mtime update ================================================================================ 07:04:14 (1789124654) repeat after cancel_lru_locks PASS 39c (9s) == sanityn test 39d: sync write should update mtime ====== 07:04:23 (1789124663) fail_loc=0x411 fail_loc=0 PASS 39d (6s) SKIP: sanityn test_40a skipping ALWAYS excluded test 40a == sanityn test 40b: pdirops: open|create and others ======================================================================== 07:04:31 (1789124671) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40b (17s) == sanityn test 40c: pdirops: link and others ======================================================================== 07:04:49 (1789124689) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40c (17s) == sanityn test 40d: pdirops: unlink and others ======================================================================== 07:05:05 (1789124705) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40d (18s) == sanityn test 40e: pdirops: rename and others ======================================================================== 07:05:24 (1789124724) No conflict No conflict No conflict No conflict No conflict No conflict PASS 40e (19s) == sanityn test 41a: pdirops: create vs mkdir ======================================================================== 07:05:43 (1789124743) mkdir: cannot create directory '/mnt/lustre2/f41a.sanityn': File exists Conflict PASS 41a (15s) == sanityn test 41b: pdirops: create vs create ======================================================================== 07:05:59 (1789124759) open: File exists Conflict PASS 41b (13s) == sanityn test 41c: pdirops: create vs link ======================================================================== 07:06:12 (1789124772) link: cannot create link '/mnt/lustre2/f41c.sanityn' to '/mnt/lustre2/f41c.sanityn-2': File exists Conflict PASS 41c (14s) == sanityn test 41d: pdirops: create vs unlink ======================================================================== 07:06:26 (1789124786) Conflict PASS 41d (13s) == sanityn test 41e: pdirops: create and rename (tgt) ======================================================================== 07:06:39 (1789124799) Conflict PASS 41e (14s) == sanityn test 41f: pdirops: create and rename (src) ======================================================================== 07:06:53 (1789124813) Conflict PASS 41f (14s) == sanityn test 41g: pdirops: create vs getattr ======================================================================== 07:07:08 (1789124828) Conflict PASS 41g (14s) == sanityn test 41h: pdirops: create vs readdir ======================================================================== 07:07:21 (1789124841) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 2447: lctl set_param -n fail_loc=0 2>/dev/null || true: No such file or directory Conflict PASS 41h (19s) == sanityn test 41i: reint_open: create vs create ======== 07:07:40 (1789124860) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 41i (1114s) == sanityn test 42a: pdirops: mkdir vs mkdir ======================================================================== 07:26:15 (1789125975) mkdir: cannot create directory '/mnt/lustre2/f42a.sanityn': File exists Conflict PASS 42a (14s) == sanityn test 42b: pdirops: mkdir vs create ======================================================================== 07:26:28 (1789125988) open: File exists Conflict PASS 42b (13s) == sanityn test 42c: pdirops: mkdir vs link ======================================================================== 07:26:41 (1789126001) link: cannot create link '/mnt/lustre2/f42c.sanityn' to '/mnt/lustre2/f42c.sanityn-2': File exists Conflict PASS 42c (12s) == sanityn test 42d: pdirops: mkdir vs unlink ======================================================================== 07:26:53 (1789126013) Conflict PASS 42d (14s) == sanityn test 42e: pdirops: mkdir and rename (tgt) ======================================================================== 07:27:07 (1789126027) mv: cannot overwrite directory '/mnt/lustre2/f42e.sanityn' with non-directory Conflict PASS 42e (14s) == sanityn test 42f: pdirops: mkdir and rename (src) ======================================================================== 07:27:21 (1789126041) Conflict PASS 42f (15s) == sanityn test 42g: pdirops: mkdir vs getattr ======================================================================== 07:27:36 (1789126056) Conflict PASS 42g (13s) == sanityn test 42h: pdirops: mkdir vs readdir ======================================================================== 07:27:49 (1789126069) Conflict PASS 42h (15s) == sanityn test 43a: rmdir,mkdir doesn't return -EEXIST ======================================================================== 07:28:04 (1789126084) PASS 43a (76s) == sanityn test 43b: pdirops: unlink vs create ======================================================================== 07:29:20 (1789126160) Conflict PASS 43b (14s) == sanityn test 43c: pdirops: unlink vs link ======================================================================== 07:29:35 (1789126175) Conflict PASS 43c (16s) == sanityn test 43d: pdirops: unlink vs unlink ======================================================================== 07:29:50 (1789126190) rm: cannot remove '/mnt/lustre2/f43d.sanityn': No such file or directory Conflict PASS 43d (16s) == sanityn test 43e: pdirops: unlink and rename (tgt) ======================================================================== 07:30:06 (1789126206) Conflict PASS 43e (16s) == sanityn test 43f: pdirops: unlink and rename (src) ======================================================================== 07:30:22 (1789126222) mv: cannot stat '/mnt/lustre2/f43f.sanityn': No such file or directory Conflict PASS 43f (17s) == sanityn test 43g: pdirops: unlink vs getattr ======================================================================== 07:30:39 (1789126239) stat: cannot statx '/mnt/lustre2/f43g.sanityn': No such file or directory Conflict PASS 43g (16s) == sanityn test 43h: pdirops: unlink vs readdir ======================================================================== 07:30:55 (1789126255) Conflict PASS 43h (17s) == sanityn test 43i: pdirops: unlink vs remote mkdir ===== 07:31:12 (1789126272) SKIP: sanityn test_43i needs >= 2 MDTs SKIP 43i (4s) == sanityn test 43j: racy mkdir return EEXIST ======================================================================== 07:31:16 (1789126276) mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists PASS 43j (153s) == sanityn test 43k: unlink vs create ==================== 07:33:49 (1789126429) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 43k (1305s) == sanityn test 44a: pdirops: rename tgt vs mkdir ======================================================================== 07:55:35 (1789127735) mkdir: cannot create directory '/mnt/lustre2/f44a.sanityn': File exists Conflict Fri Sep 11 07:55:43 EDT 2026 PASS 44a (16s) == sanityn test 44b: pdirops: rename tgt vs create ======================================================================== 07:55:50 (1789127750) open: File exists Conflict PASS 44b (18s) == sanityn test 44c: pdirops: rename tgt vs link ======================================================================== 07:56:08 (1789127768) link: cannot create link '/mnt/lustre2/f44c.sanityn' to '/mnt/lustre2/f44c.sanityn-3': File exists Conflict PASS 44c (16s) == sanityn test 44d: pdirops: rename tgt vs unlink ======================================================================== 07:56:24 (1789127784) Conflict PASS 44d (15s) == sanityn test 44e: pdirops: rename tgt and rename (tgt) ======================================================================== 07:56:39 (1789127799) Conflict PASS 44e (15s) == sanityn test 44f: pdirops: rename tgt and rename (src) ======================================================================== 07:56:55 (1789127815) Conflict PASS 44f (19s) == sanityn test 44g: pdirops: rename tgt vs getattr ======================================================================== 07:57:13 (1789127833) Conflict PASS 44g (17s) == sanityn test 44h: pdirops: rename tgt vs readdir ======================================================================== 07:57:30 (1789127850) Conflict PASS 44h (16s) == sanityn test 44i: pdirops: rename tgt vs remote mkdir ========================================================== 07:57:46 (1789127866) SKIP: sanityn test_44i needs >= 2 MDTs SKIP 44i (4s) == sanityn test 45a: rename,mkdir doesn't return -EEXIST ======================================================================== 07:57:50 (1789127870) PASS 45a (143s) == sanityn test 45b: pdirops: rename src vs create ======================================================================== 08:00:13 (1789128013) Conflict PASS 45b (13s) == sanityn test 45c: pdirops: rename src vs link ======================================================================== 08:00:26 (1789128026) Conflict PASS 45c (14s) == sanityn test 45d: pdirops: rename src vs unlink ======================================================================== 08:00:40 (1789128040) rm: cannot remove '/mnt/lustre2/f45d.sanityn': No such file or directory Conflict PASS 45d (13s) == sanityn test 45e: pdirops: rename src and rename (tgt) ======================================================================== 08:00:54 (1789128054) Conflict PASS 45e (14s) == sanityn test 45f: pdirops: rename src and rename (src) ======================================================================== 08:01:09 (1789128069) mv: cannot stat '/mnt/lustre2/f45f.sanityn': No such file or directory Conflict PASS 45f (15s) == sanityn test 45g: pdirops: rename src vs getattr ======================================================================== 08:01:24 (1789128084) stat: cannot statx '/mnt/lustre2/f45g.sanityn': No such file or directory Conflict PASS 45g (13s) == sanityn test 45h: pdirops: unlink vs readdir ======================================================================== 08:01:38 (1789128098) Conflict PASS 45h (14s) == sanityn test 45i: pdirops: rename src vs remote mkdir ========================================================== 08:01:52 (1789128112) SKIP: sanityn test_45i needs >= 2 MDTs SKIP 45i (3s) == sanityn test 45j: read vs rename ====================== 08:01:56 (1789128116) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 45j (1237s) == sanityn test 46a: pdirops: link vs mkdir ======================================================================== 08:22:32 (1789129352) mkdir: cannot create directory '/mnt/lustre2/f46a.sanityn': File exists Conflict PASS 46a (14s) == sanityn test 46b: pdirops: link vs create ======================================================================== 08:22:46 (1789129366) open: File exists Conflict PASS 46b (15s) == sanityn test 46c: pdirops: link vs link ======================================================================== 08:23:01 (1789129381) link: cannot create link '/mnt/lustre2/f46c.sanityn' to '/mnt/lustre2/f46c.sanityn': File exists Conflict PASS 46c (13s) == sanityn test 46d: pdirops: link vs unlink ======================================================================== 08:23:14 (1789129394) Conflict PASS 46d (14s) == sanityn test 46e: pdirops: link and rename (tgt) ======================================================================== 08:23:28 (1789129408) Conflict PASS 46e (14s) == sanityn test 46f: pdirops: link and rename (src) ======================================================================== 08:23:42 (1789129422) Conflict PASS 46f (14s) == sanityn test 46g: pdirops: link vs getattr ======================================================================== 08:23:56 (1789129436) Conflict PASS 46g (16s) == sanityn test 46h: pdirops: link vs readdir ======================================================================== 08:24:12 (1789129452) Conflict PASS 46h (18s) == sanityn test 46i: pdirops: link vs remote mkdir ======= 08:24:30 (1789129470) SKIP: sanityn test_46i needs >= 2 MDTs SKIP 46i (4s) == sanityn test 47a: pdirops: remote mkdir vs mkdir ====== 08:24:34 (1789129474) SKIP: sanityn test_47a needs >= 2 MDTs SKIP 47a (4s) == sanityn test 47b: pdirops: remote mkdir vs create ===== 08:24:38 (1789129478) SKIP: sanityn test_47b needs >= 2 MDTs SKIP 47b (3s) == sanityn test 47c: pdirops: remote mkdir vs link ======= 08:24:41 (1789129481) SKIP: sanityn test_47c needs >= 2 MDTs SKIP 47c (5s) == sanityn test 47d: pdirops: remote mkdir vs unlink ===== 08:24:46 (1789129486) SKIP: sanityn test_47d needs >= 2 MDTs SKIP 47d (3s) == sanityn test 47e: pdirops: remote mkdir and rename (tgt) ========================================================== 08:24:50 (1789129490) SKIP: sanityn test_47e needs >= 2 MDTs SKIP 47e (4s) == sanityn test 47f: pdirops: remote mkdir and rename (src) ========================================================== 08:24:54 (1789129494) SKIP: sanityn test_47f needs >= 2 MDTs SKIP 47f (4s) == sanityn test 47g: pdirops: remote mkdir vs getattr ==== 08:24:59 (1789129499) SKIP: sanityn test_47g needs >= 2 MDTs SKIP 47g (4s) == sanityn test 50: osc lvb attrs: enqueue vs. CP AST ======================================================================== 08:25:02 (1789129502) 10+0 records in 10+0 records out 10240 bytes (10 kB, 10 KiB) copied, 0.0715024 s, 143 kB/s fail_loc=0x410 fail_loc=0x0 PASS 50 (14s) == sanityn test 51a: layout lock: refresh layout should work ========================================================== 08:25:16 (1789129516) 0+1 records in 0+1 records out 158 bytes copied, 0.0303728 s, 5.2 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 3710: kill: (218945) - No such process PASS 51a (10s) == sanityn test 51b: layout lock: glimpse should be able to restart if layout changed ========================================================== 08:25:26 (1789129526) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0492733 s, 20.8 kB/s fail_loc=0x1404 fail_val=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141698 s, 7.4 MB/s 3145728 PASS 51b (21s) == sanityn test 51c: layout lock: IT_LAYOUT blocked and correct layout can be returned ========================================================== 08:25:47 (1789129547) fail_loc=0x172 Setting layout to have 2 stripes ... 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0300374 s, 34.1 kB/s PASS 51c (12s) == sanityn test 51d: layout lock: losing layout lock should clean up memory map region ========================================================== 08:26:00 (1789129560) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0224169 s, 46.8 MB/s Before revoking layout lock: 1024 KB mapped PASS 51d (9s) == sanityn test 51e: lfs getstripe does not break leases, part 2 ========================================================== 08:26:09 (1789129569) /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 2451 0x993 0x280000400 write lease(2) has applied. /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 2451 0x993 0x280000400 read lease(1) has applied. PASS 51e (9s) == sanityn test 54: rename locking ======================= 08:26:18 (1789129578) llite.lustre-ffff8b688b34a000.xattr_cache=1 llite.lustre-ffff8b6890c8a800.xattr_cache=1 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 7680 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505017 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-11 08:26:21.000000000 -0400 Modify: 2026-09-11 08:26:21.000000000 -0400 Change: 2026-09-11 08:26:21.000000000 -0400 Birth: 2026-09-11 08:26:20.000000000 -0400 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000154 File: /mnt/lustre/d1/d2 Size: 7680 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505020 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-11 08:26:27.000000000 -0400 Modify: 2026-09-11 08:26:28.000000000 -0400 Change: 2026-09-11 08:26:28.000000000 -0400 Birth: 2026-09-11 08:26:26.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 512 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505023 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-11 08:26:33.000000000 -0400 Modify: 2026-09-11 08:26:33.000000000 -0400 Change: 2026-09-11 08:26:33.000000000 -0400 Birth: 2026-09-11 08:26:33.000000000 -0400 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1 Size: 7680 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505022 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-11 08:26:33.000000000 -0400 Modify: 2026-09-11 08:26:34.000000000 -0400 Change: 2026-09-11 08:26:34.000000000 -0400 Birth: 2026-09-11 08:26:32.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000154 File: /mnt/lustre/d1/d2 Size: 7680 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505026 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-11 08:26:39.000000000 -0400 Modify: 2026-09-11 08:26:40.000000000 -0400 Change: 2026-09-11 08:26:40.000000000 -0400 Birth: 2026-09-11 08:26:39.000000000 -0400 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1 Size: 7680 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505025 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-11 08:26:39.000000000 -0400 Modify: 2026-09-11 08:26:40.000000000 -0400 Change: 2026-09-11 08:26:40.000000000 -0400 Birth: 2026-09-11 08:26:39.000000000 -0400 PASS 54 (34s) == sanityn test 55a: rename vs unlink target dir ========= 08:26:53 (1789129613) fail_loc=0x80000156 mv: cannot move '/mnt/lustre/d55a.sanityn/d1/d2' to '/mnt/lustre/d55a.sanityn/d3/d2': No such file or directory PASS 55a (16s) == sanityn test 55b: rename vs unlink source dir ========= 08:27:08 (1789129628) fail_loc=0x80000156 mv: cannot stat '/mnt/lustre/d55b.sanityn/d1/d2': No such file or directory PASS 55b (14s) == sanityn test 55c: rename vs unlink orphan target dir == 08:27:22 (1789129642) fail_loc=0x156 mv: cannot move '/mnt/lustre/d55c.sanityn/d1/d2' to '/mnt/lustre/d55c.sanityn/d3/d2': No such file or directory PASS 55c (20s) == sanityn test 55d: rename file vs link ================= 08:27:43 (1789129663) fail_loc=0x155 mv: '/mnt/lustre/d55d.sanityn/f1' and '/mnt/lustre/d55d.sanityn/d55d.sanityn/f1' are the same file PASS 55d (15s) == sanityn test 55e: rename race AB/BA under the same parent dir ========================================================== 08:27:58 (1789129678) SKIP: sanityn test_55e needs >= 2 MDTs SKIP 55e (4s) == sanityn test 55f: rename: (P1/A -> P2/B) race with (P2/B -> P1/A) ========================================================== 08:28:02 (1789129682) fail_loc=0x155 We survived after P1A_P2B/P2B_P1A race test fail_loc=0x0 PASS 55f (23s) == sanityn test 55g: rename: race with trylock =========== 08:28:25 (1789129705) mdt.lustre-MDT0000.enable_rename_trylock=1 fail_loc=0x155 mv: cannot move '/mnt/lustre2/d55g.sanityn/b1/b2' to a subdirectory of itself, '/mnt/lustre2/d55g.sanityn/a1/a2/a3/a4/b2' We survived race with trylock test fail_loc=0x0 PASS 55g (28s) == sanityn test 56a: test llverdev with single large stripe ========================================================== 08:28:53 (1789129733) run_llverdev /mnt/lustre/f56a.sanityn -p -s 16000000 -c 4k llverdev: /mnt/lustre/f56a.sanityn is 16777216000000 bytes (15625 GB) in size timestamp: 1789129735 chunksize: 4096 size: 16777216000000 write offset: 16384000000kB 0.521701 MB/s read offset: 16384000000kB inf MB/s llverdev: data verified successfully PASS 56a (124s) == sanityn test 56b: test llverdev and partial verify of wide stripe file ========================================================== 08:30:57 (1789129857) run_llverdev /mnt/lustre/f56b.sanityn -p -s 32000000 -c 4k llverdev: /mnt/lustre/f56b.sanityn is 33554432000000 bytes (31250 GB) in size timestamp: 1789129865 chunksize: 4096 size: 33554432000000 write offset: 32768000000kB 1.14088 MB/s read offset: 32768000000kB inf MB/s llverdev: data verified successfully PASS 56b (126s) == sanityn test 60: Verify data_version behaviour ======== 08:33:03 (1789129983) PASS 60 (10s) == sanityn test 70a: cd directory && rm directory ======== 08:33:14 (1789129994) PASS 70a (9s) == sanityn test 70b: remove files after calling rm_entry ========================================================== 08:33:23 (1789130003) PASS 70b (11s) == sanityn test 71a: correct file map just after write operation is finished ========================================================== 08:33:34 (1789130014) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0138656 s, 4.7 MB/s SKIP: sanityn test_71a ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71a (4s) == sanityn test 71b: check fiemap support for stripecount > 1 ========================================================== 08:33:39 (1789130019) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0214456 s, 3.1 MB/s SKIP: sanityn test_71b ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71b (5s) == sanityn test 71c: check FIEMAP_EXTENT_LAST flag with different extents number ========================================================== 08:33:43 (1789130023) SKIP: sanityn test_71c support only ldiskfs ost SKIP 71c (3s) == sanityn test 71d: fiemap corruption test with fm_extent_count=0 ========================================================== 08:33:47 (1789130027) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0181449 s, 3.6 MB/s SKIP: sanityn test_71d ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71d (4s) == sanityn test 72: getxattr/setxattr cache should be consistent between nodes ========================================================== 08:33:51 (1789130031) llite.lustre-ffff8b688b34a000.xattr_cache=1 llite.lustre-ffff8b6890c8a800.xattr_cache=1 getfattr: Removing leading '/' from absolute path names user.attr1="value1" getfattr: Removing leading '/' from absolute path names user.attr1="value2" getfattr: Removing leading '/' from absolute path names getfattr: Removing leading '/' from absolute path names b362740ad371eae580e7f85452e8830b - 870e072da1ebfc20ea8c0cd3d6bdf3b6 - PASS 72 (8s) == sanityn test 73: getxattr should not cause xattr lock cancellation ========================================================== 08:33:59 (1789130039) llite.lustre-ffff8b688b34a000.xattr_cache=1 llite.lustre-ffff8b6890c8a800.xattr_cache=1 getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" PASS 73 (7s) == sanityn test 74: flock deadlock: different mounts ======================================================================== 08:34:07 (1789130047) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0113055 s, 90.6 kB/s 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0201677 s, 50.8 kB/s 231776: get lock2 231777: get lock1 231776: done 231776: get lock1 231777: done 231777: get lock2 fcntl cmd 7 failed: Resource deadlock avoided 231777: cannot get lock2: Resource deadlock avoided 231777: exit rc=0 231776: done 231776: exit rc=0 PASS 74 (11s) == sanityn test 75: osc: upcall after unuse lock============================================================================= 08:34:18 (1789130058) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0813368 s, 25.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0273714 s, 38.3 MB/s fail_loc=0x31d /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 4393: 232358 Killed stat -c %s $DIR1/$tfile fail_loc=0 PASS 75 (14s) == sanityn test 76: Verify MDT open_files listing ======== 08:34:32 (1789130072) opening files: ........ closing files: ........ PASS 76 (75s) == sanityn test 77a: check FIFO NRS policy =============== 08:35:47 (1789130147) ost.OSS.ost.nrs_policies=fifo ost.OSS.ost_create.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_out.nrs_policies=fifo ost.OSS.ost_seq.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.2571 s, 65.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.021752 s, 48.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0181184 s, 57.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0316048 s, 33.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016116 s, 65.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132194 s, 79.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0397541 s, 26.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157533 s, 66.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0530209 s, 19.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0345449 s, 30.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110346 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120716 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0992075 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134843 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0975094 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126285 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23379 s, 4.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0537056 s, 19.5 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0542505 s, 19.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0567748 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0600623 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0990223 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0673762 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0772574 s, 13.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0784522 s, 13.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0816524 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0841754 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0915748 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0894985 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100469 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104357 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107223 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104739 s, 10.0 MB/s removed '/mnt/lustre/d77a.sanityn/nrs_r_oleg424-client.virtnet' removed '/mnt/lustre/d77a.sanityn/nrs_w_oleg424-client.virtnet' removed directory '/mnt/lustre/d77a.sanityn' PASS 77a (11s) == sanityn test 77b: check CRR-N NRS policy ============== 08:35:58 (1789130158) ost.OSS.ost.nrs_policies=crrn ost.OSS.ost_create.nrs_policies=crrn ost.OSS.ost_io.nrs_policies=crrn ost.OSS.ost_out.nrs_policies=crrn ost.OSS.ost_seq.nrs_policies=crrn ost.OSS.ost.nrs_crrn_quantum=1 ost.OSS.ost_create.nrs_crrn_quantum=1 ost.OSS.ost_io.nrs_crrn_quantum=1 ost.OSS.ost_out.nrs_crrn_quantum=1 ost.OSS.ost_seq.nrs_crrn_quantum=1 policy: crr-n, crrn_quantum 1 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.235116 s, 71.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0526222 s, 19.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0650357 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0521663 s, 20.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118876 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0784688 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0944551 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0339591 s, 30.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169638 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0653211 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0916902 s, 11.4 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8b688b34a000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.317608 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.308946 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.300684 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.35041 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.304481 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.464791 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.053858 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0044665 s, 235 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0735598 s, 14.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8b688b34a000.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff8b688b34a000.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0106432 s, 98.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00247149 s, 424 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0104216 s, 101 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00364686 s, 288 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00720328 s, 146 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195067 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00952067 s, 110 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0101258 s, 104 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14298 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206843 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120889 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116195 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0819097 s, 12.8 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg424-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg424-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost.nrs_crrn_quantum=64 ost.OSS.ost_create.nrs_crrn_quantum=64 ost.OSS.ost_io.nrs_crrn_quantum=64 ost.OSS.ost_out.nrs_crrn_quantum=64 ost.OSS.ost_seq.nrs_crrn_quantum=64 policy: crr-n, crrn_quantum 64 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.264817 s, 63.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.066938 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0870728 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135454 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13835 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165923 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163057 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186482 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175066 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191304 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203996 s, 5.1 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.239979 s, 4.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194708 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.214563 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.24274 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.219144 s, 4.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8b688b34a000.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff8b688b34a000.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.320391 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0794775 s, 13.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0829973 s, 12.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0533461 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0395513 s, 26.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117213 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0517287 s, 20.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0622914 s, 16.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0662062 s, 15.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0625004 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0884914 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0900794 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0729871 s, 14.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0958252 s, 10.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0706204 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0803108 s, 13.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0896181 s, 11.7 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg424-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg424-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77b (16s) == sanityn test 77c: check ORR NRS policy ================ 08:36:14 (1789130174) ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_orr_quantum=1 ost.OSS.ost_io.nrs_orr_offset_type=physical ost.OSS.ost_io.nrs_orr_supported=reads policy: orr, orr_quantum 1, orr_offset_type physical, orr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.306273 s, 54.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.025427 s, 41.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0335792 s, 31.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0430398 s, 24.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0707286 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101207 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107488 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149126 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161902 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149484 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169946 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171641 s, 6.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192681 s, 5.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207882 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184931 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190491 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.26224 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0415582 s, 25.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00719971 s, 146 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0557281 s, 18.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0530729 s, 19.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0455496 s, 23.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0546802 s, 19.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0692134 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00444986 s, 236 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0597152 s, 17.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.07139 s, 14.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00894379 s, 117 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0506753 s, 20.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00995155 s, 105 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127467 s, 82.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0516421 s, 20.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0251095 s, 41.8 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg424-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg424-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=writes ost.OSS.ost_io.nrs_orr_quantum=64 policy: orr, orr_quantum 64, orr_offset_type physical, orr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.320974 s, 52.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0709501 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0270398 s, 38.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0407415 s, 25.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.091376 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168288 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0345792 s, 30.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159803 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143533 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195197 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.050955 s, 20.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192701 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.238303 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0327354 s, 32.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143133 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0597479 s, 17.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8b688b34a000.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff8b688b34a000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114016 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143227 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147719 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184326 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149125 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137126 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119517 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0222948 s, 47.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.230029 s, 4.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120924 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0957124 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0736702 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0900764 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0944674 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0621947 s, 16.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0915946 s, 11.4 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0598369 s, 17.5 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg424-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg424-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=reads_and_writes ost.OSS.ost_io.nrs_orr_offset_type=logical policy: orr, orr_quantum 64, orr_offset_type logical, orr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.359838 s, 46.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0513066 s, 20.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0588086 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131256 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0631469 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147055 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11594 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0595656 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16994 s, 6.2 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8b688b34a000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.036446 s, 28.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.264548 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.224569 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212062 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.299693 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127358 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.287735 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.348403 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0962471 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0399932 s, 26.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0902005 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0610851 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0705513 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0713396 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0692029 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0741698 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0782243 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0921195 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0895101 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0986175 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0986161 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0996256 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100807 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0992391 s, 10.6 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg424-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg424-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77c (17s) == sanityn test 77d: check TRR nrs policy ================ 08:36:32 (1789130192) ost.OSS.ost_io.nrs_policies=trr ost.OSS.ost_io.nrs_trr_quantum=1 ost.OSS.ost_io.nrs_trr_offset_type=physical ost.OSS.ost_io.nrs_trr_supported=reads policy: trr, trr_quantum 1, trr_offset_type physical, trr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.385985 s, 43.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.042348 s, 24.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.053412 s, 19.6 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110841 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122267 s, 8.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0317019 s, 33.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121255 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105762 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195445 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130868 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215224 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0296451 s, 35.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0188781 s, 55.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0673444 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183964 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190131 s, 5.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8b688b34a000.lock_unused_count=5 ldlm.namespaces.lustre-OST0001-osc-ffff8b688b34a000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199813 s, 5.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0485906 s, 21.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0515335 s, 20.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0400325 s, 26.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0525072 s, 20.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.08601 s, 12.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127062 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127453 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0633906 s, 16.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.073892 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0546991 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0557873 s, 18.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0527275 s, 19.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125192 s, 8.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0803155 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0869424 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0884487 s, 11.9 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg424-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg424-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=writes ost.OSS.ost_io.nrs_trr_quantum=64 policy: trr, trr_quantum 64, trr_offset_type physical, trr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.230607 s, 72.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0518591 s, 20.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.084878 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114371 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0938012 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131633 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17983 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108646 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177679 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191972 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199349 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.257331 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.231781 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220081 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.251844 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.25435 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.34482 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117284 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00702513 s, 149 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19555 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00520357 s, 202 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142958 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16097 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180404 s, 5.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110391 s, 9.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106429 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0851516 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182766 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.061062 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0703086 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103345 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0495022 s, 21.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0365857 s, 28.7 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg424-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg424-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=reads_and_writes ost.OSS.ost_io.nrs_trr_offset_type=logical policy: trr, trr_quantum 64, trr_offset_type logical, trr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.320597 s, 52.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0418664 s, 25.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.055035 s, 19.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0666375 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0367531 s, 28.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0884849 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184143 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121941 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151868 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19755 s, 5.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8b688b34a000.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223601 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19969 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157361 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223284 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.338161 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.340069 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.366742 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0683561 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0545764 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0466896 s, 22.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138803 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0711702 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0533442 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0479128 s, 21.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113536 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0595139 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0797691 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102141 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0893253 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0772643 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0835239 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0959036 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103065 s, 10.2 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg424-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg424-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77d (16s) == sanityn test 77e: check TBF NID nrs policy ============ 08:36:49 (1789130209) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_tbf_rule=start localhost nid={0@lo} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start clients nid={192.168.204.24@tcp} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start others nid={*.*.*.*@tcp} rate=50 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.231303 s, 72.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0269139 s, 39.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0554271 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0467027 s, 22.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0698056 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165591 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.077405 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120512 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190657 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119849 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148168 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210661 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188025 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193925 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149766 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.226082 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.225453 s, 4.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.065098 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0468721 s, 22.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118304 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0965504 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0533812 s, 19.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0560793 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0558332 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0606826 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11598 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0523439 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0637522 s, 16.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0585222 s, 17.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0676725 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0735198 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0682218 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0699491 s, 15.0 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg424-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg424-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change localhost rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change clients rate=101 ost.OSS.ost_io.nrs_tbf_rule=change others rate=51 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.276345 s, 60.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0468474 s, 22.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0193513 s, 54.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0489516 s, 21.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147254 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16935 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128476 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179422 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186011 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191846 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191513 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184433 s, 5.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227545 s, 4.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215735 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213422 s, 4.9 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.234238 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.255909 s, 4.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8b688b34a000.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff8b688b34a000.lock_unused_count=6 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173856 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132679 s, 7.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.079667 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126057 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106827 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19826 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161486 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111446 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211835 s, 4.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205579 s, 5.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213518 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160339 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187016 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.236574 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135597 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178009 s, 5.9 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg424-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg424-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop localhost ost.OSS.ost_io.nrs_tbf_rule=stop clients ost.OSS.ost_io.nrs_tbf_rule=stop others 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.252081 s, 66.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.053208 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0848119 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0849631 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175884 s, 6.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8b688b34a000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129815 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184653 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190101 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181492 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210391 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164555 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.244173 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167695 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192156 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.224145 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16911 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.311514 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0919882 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.010013 s, 105 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00703141 s, 149 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00633564 s, 166 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0464074 s, 22.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11054 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0263911 s, 39.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00741558 s, 141 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00888896 s, 118 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0419498 s, 25.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0115234 s, 91.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00286961 s, 365 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0534033 s, 19.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00947277 s, 111 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0093147 s, 113 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00578942 s, 181 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg424-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg424-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.214397 s, 78.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0562675 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0491712 s, 21.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139942 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0889725 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144145 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137284 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205749 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.219949 s, 4.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.225381 s, 4.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194659 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208966 s, 5.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.266137 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.249665 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.290761 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.262282 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.303683 s, 3.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8b688b34a000.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff8b688b34a000.lock_unused_count=7 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0607965 s, 17.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0337916 s, 31.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0453665 s, 23.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166837 s, 62.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0807474 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0485898 s, 21.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0428569 s, 24.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0647417 s, 16.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0174124 s, 60.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00801759 s, 131 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0258996 s, 40.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147359 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0512028 s, 20.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140445 s, 74.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0686328 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0410894 s, 25.5 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg424-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg424-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' PASS 77e (24s) == sanityn test 77f: check TBF JobID nrs policy ========== 08:37:13 (1789130233) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start runas jobid={iozone.500 dd.500 tiotest.500} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start iozone_runas jobid={iozone.500} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg424-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.292438 s, 57.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0516827 s, 20.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.094735 s, 11.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132733 s, 7.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119431 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0647862 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0899995 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126264 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19487 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153116 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195965 s, 5.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124529 s, 8.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159843 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167745 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192992 s, 5.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8b688b34a000.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff8b688b34a000.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.245822 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227064 s, 4.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164868 s, 63.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102848 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0411861 s, 25.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00255054 s, 411 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00241261 s, 435 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.271248 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.318189 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22024 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15329 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.38729 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132039 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111143 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0906922 s, 11.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149789 s, 7.0 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0316418 s, 33.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0445996 s, 23.5 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg424-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg424-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change runas rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change iozone_runas rate=101 ost.OSS.ost_io.nrs_tbf_rule=change dd_runas rate=51 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg424-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.331749 s, 50.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109988 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0349748 s, 30.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104523 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0812847 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0715916 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212209 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203289 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.269767 s, 3.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0847913 s, 12.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.287257 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.294835 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.219858 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170152 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.323516 s, 3.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8b688b34a000.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffff8b688b34a000.lock_unused_count=1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.403978 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.537676 s, 2.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8b688b34a000.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.083618 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137854 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0962326 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0332142 s, 31.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0334158 s, 31.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0576447 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0485174 s, 21.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0523857 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0552047 s, 19.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0918292 s, 11.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0697895 s, 15.0 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0718149 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0865715 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0801227 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11432 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114948 s, 9.1 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg424-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg424-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop runas ost.OSS.ost_io.nrs_tbf_rule=stop iozone_runas ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg424-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.517242 s, 32.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0545516 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0462163 s, 22.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0802928 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129744 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197574 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0975678 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.233168 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.245902 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175075 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155801 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.257374 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133796 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195392 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213806 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.294259 s, 3.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8b688b34a000.lock_unused_count=5 ldlm.namespaces.lustre-OST0001-osc-ffff8b688b34a000.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.404507 s, 2.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0851453 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0322275 s, 32.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0472648 s, 22.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0118746 s, 88.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00692945 s, 151 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0596828 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108592 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139297 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166135 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173355 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137014 s, 76.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.020807 s, 50.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138519 s, 75.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0380003 s, 27.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141876 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162322 s, 64.6 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg424-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg424-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_policies=fifo running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg424-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.502937 s, 33.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0690583 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0932187 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0510347 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0542901 s, 19.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163853 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206658 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143551 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156606 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.230018 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178368 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.253957 s, 4.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8b688b34a000.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.219506 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.313023 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.24304 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220464 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207084 s, 5.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155543 s, 67.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00244127 s, 430 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00248726 s, 422 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160479 s, 65.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00251973 s, 416 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.36121 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.289455 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.311236 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.252629 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.233877 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.394702 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.214818 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178307 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156872 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0948013 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136269 s, 7.7 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg424-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg424-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' PASS 77f (31s) == sanityn test 77g: Change TBF type directly ============ 08:37:45 (1789130265) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 ost.OSS.ost_io.nrs_policies=fifo PASS 77g (14s) == sanityn test 77h: Wrong policy name should report error, not LBUG ========================================================== 08:37:58 (1789130278) oleg424-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=abc: No such file or directory oleg424-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='abc': No such file or directory pdsh@oleg424-client: oleg424-server: ssh exited with exit code 2 oleg424-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc: Invalid argument oleg424-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc': Invalid argument pdsh@oleg424-client: oleg424-server: ssh exited with exit code 22 oleg424-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf reg abc: Invalid argument oleg424-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf reg abc': Invalid argument pdsh@oleg424-client: oleg424-server: ssh exited with exit code 22 oleg424-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc efg: Invalid argument oleg424-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc efg': Invalid argument pdsh@oleg424-client: oleg424-server: ssh exited with exit code 22 PASS 77h (14s) == sanityn test 77i: Change rank of TBF rule ============= 08:38:12 (1789130292) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start before jobid={jobid} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start after jobid={jobid} rate=1000 rank=default ost.OSS.ost_io.nrs_tbf_rule=start target jobid={jobid} rate=1000 rank=after Move before itself ost.OSS.ost_io.nrs_tbf_rule=change target rank=target Move to higher rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=before Move to lower rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=after Move before default ost.OSS.ost_io.nrs_tbf_rule=change target rank=default ost.OSS.ost_io.nrs_policies=fifo PASS 77i (26s) == sanityn test 77j: check TBF-OPCode NRS policy ========= 08:38:38 (1789130318) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf opcode ost.OSS.ost_io.nrs_tbf_rule=start ost_r opcode={ost_read} rate=5 ost.OSS.ost_io.nrs_tbf_rule=start ost_w opcode={ost_write} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.404769 s, 41.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0625002 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0513494 s, 20.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0834448 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134085 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131662 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126826 s, 8.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171629 s, 6.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17594 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222244 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200923 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.247808 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.257668 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.241962 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242741 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.282948 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.281177 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00253599 s, 413 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0895925 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.090106 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0956758 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0919524 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115153 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102082 s, 10.3 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0661501 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0689955 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115149 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158443 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116556 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118201 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119508 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121847 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129456 s, 8.1 MB/s removed '/mnt/lustre/d77j.sanityn/nrs_w_oleg424-client.virtnet' removed '/mnt/lustre/d77j.sanityn/nrs_r_oleg424-client.virtnet' removed directory '/mnt/lustre/d77j.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 5 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.51413 s, 12.3 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.6141 s, 5.3 MB/s Read runtime is 20 s, speed is 5.000000 IOPS cleanup_tbf /mnt/lustre/d77j.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_r ost.OSS.ost_io.nrs_tbf_rule=stop ost_w ost.OSS.ost_io.nrs_policies=fifo PASS 77j (70s) == sanityn test 77ja: check TBF-UID/GID NRS policy ======= 08:39:48 (1789130388) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg424-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg424-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg424-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg424-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg424-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.28465 s, 13.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173892 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142499 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187097 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124198 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186473 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22995 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23627 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.285983 s, 3.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237435 s, 4.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.267011 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.229616 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.293836 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.311228 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.329944 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.289193 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.406757 s, 2.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0387758 s, 27.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0904958 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183549 s, 57.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11363 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117844 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109997 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143376 s, 73.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110913 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168999 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0371162 s, 28.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107758 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0313108 s, 33.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179173 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0251998 s, 41.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0254648 s, 41.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0216771 s, 48.4 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg424-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg424-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.2044 s, 5.2 MB/s Write runtime is 22 s, speed is 4.545454 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.7641 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg424-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg424-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg424-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg424-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg424-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.28127 s, 13.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0672321 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102933 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0907019 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0524058 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0371135 s, 28.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213905 s, 4.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21071 s, 5.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0924716 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143103 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21468 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0808087 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117957 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22399 s, 4.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105077 s, 10.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0813956 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.236023 s, 4.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0896268 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159569 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155405 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189017 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210652 s, 5.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19802 s, 5.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164392 s, 6.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0576361 s, 18.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.228106 s, 4.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174526 s, 6.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0197116 s, 53.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118202 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.229251 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0458092 s, 22.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0523565 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220859 s, 4.7 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg424-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg424-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.0334 s, 5.2 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9857 s, 5.2 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77ja (164s) == sanityn test 77jb: check TBF-UID/GID NRS policy on files that don't belong to us ========================================================== 08:42:32 (1789130552) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg424-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg424-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg424-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg424-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg424-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.28794 s, 13.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0705759 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130002 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0852611 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130521 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0823576 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0514145 s, 20.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0733722 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116802 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180495 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0696092 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192582 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113787 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.305966 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.238185 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.295278 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.3218 s, 3.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0518521 s, 20.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0710232 s, 14.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.033644 s, 31.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0509734 s, 20.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.044516 s, 23.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0849852 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.082397 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0670383 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0970888 s, 10.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0737232 s, 14.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0724849 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0628979 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0680222 s, 15.4 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10804 s, 9.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0683722 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0794706 s, 13.2 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg424-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg424-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.3195 s, 5.2 MB/s Write runtime is 22 s, speed is 4.545454 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9788 s, 5.2 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg424-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg424-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg424-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg424-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg424-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.26233 s, 13.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0955748 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0887459 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106924 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123031 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227534 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.218228 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221045 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.240581 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.236624 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201732 s, 5.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.268085 s, 3.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.316893 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.249404 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.276781 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23198 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.313912 s, 3.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.028992 s, 36.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0570993 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0590833 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0514333 s, 20.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0241646 s, 43.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0454739 s, 23.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0381482 s, 27.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0712582 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0614247 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0531762 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0897193 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0548589 s, 19.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0642602 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0985774 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102956 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110414 s, 9.5 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg424-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg424-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9797 s, 5.2 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9597 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77jb (156s) == sanityn test 77k: check TBF policy with UID/GID/JobID/OPCode expression ========================================================== 08:45:09 (1789130709) ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg424-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.44759 s, 37.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0422091 s, 24.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0438239 s, 23.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0777622 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102686 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0989311 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109228 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0697006 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110355 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153633 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145153 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0759086 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125518 s, 8.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0983545 s, 10.7 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0991443 s, 10.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0933431 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116996 s, 9.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0219918 s, 47.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0303603 s, 34.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0348769 s, 30.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.039405 s, 26.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.055178 s, 19.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0605652 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0616813 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0595964 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0657726 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136113 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0894091 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130078 s, 8.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121201 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0795748 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0927324 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0977403 s, 10.7 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg424-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg424-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.6786 s, 13.7 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.97678 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.204.24@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.204.24@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.673404 s, 24.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0215685 s, 48.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0346674 s, 30.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0341448 s, 30.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0673797 s, 15.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0700267 s, 15.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.080787 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0503278 s, 20.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0542087 s, 19.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0510972 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0650036 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0882047 s, 11.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0245435 s, 42.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11373 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0739219 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0467483 s, 22.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165666 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.071442 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.055683 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0820571 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0868651 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0690226 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0514868 s, 20.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0546123 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0816333 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0552855 s, 19.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.069285 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0792458 s, 13.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00288173 s, 364 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0630776 s, 16.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0688307 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.076182 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0037457 s, 280 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg424-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg424-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.37165 s, 12.5 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.97965 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.204.24@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg424-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.425985 s, 39.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0479808 s, 21.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0373058 s, 28.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0470917 s, 22.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140108 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1686 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191837 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114248 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15428 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138054 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223545 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.279224 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.279055 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.225607 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174105 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.306746 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.269383 s, 3.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.008281 s, 127 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0824526 s, 12.7 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152964 s, 68.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013688 s, 76.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00243002 s, 432 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0912643 s, 11.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186764 s, 5.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0920089 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00543636 s, 193 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0619076 s, 16.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.048569 s, 21.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0669487 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0430693 s, 24.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0230285 s, 45.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0608903 s, 17.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.06537 s, 16.0 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg424-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg424-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.4351 s, 12.4 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.3214 s, 19.7 MB/s Read runtime is 7 s, speed is 14.285714 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg424-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.569764 s, 29.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0449822 s, 23.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0432458 s, 24.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0779956 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.044051 s, 23.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0589866 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211044 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183227 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.226338 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.231719 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.261466 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.265833 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.273784 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.284774 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.282084 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.299506 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.298797 s, 3.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0409835 s, 25.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0586394 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.044417 s, 23.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0618605 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0632345 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0457269 s, 22.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101004 s, 10.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0470138 s, 22.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0506091 s, 20.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.077066 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0671266 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0732348 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.06941 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0782175 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0823852 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0870829 s, 12.0 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg424-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg424-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.3118 s, 9.3 MB/s Write runtime is 13 s, speed is 7.692307 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.92973 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.02052 s, 13.1 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.98956 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg424-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.358649 s, 46.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0760661 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.073207 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157183 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0582413 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148945 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103373 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186441 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197657 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175626 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203094 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153674 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143165 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202245 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.263079 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242159 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20555 s, 5.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0821415 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0658403 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0361347 s, 29.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145718 s, 72.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0471419 s, 22.2 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00674675 s, 155 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0106786 s, 98.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167177 s, 62.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0104167 s, 101 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122185 s, 8.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103021 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00612315 s, 171 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00270165 s, 388 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0694501 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0721874 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.028763 s, 36.5 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg424-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg424-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8587 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.6085 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_ug ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg424-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.230743 s, 72.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0293013 s, 35.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0381986 s, 27.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0828976 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0548077 s, 19.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013087 s, 80.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0586098 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0640681 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126632 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118441 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15092 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175999 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0845425 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162663 s, 6.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123828 s, 8.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201457 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181177 s, 5.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0884117 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105848 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0655565 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0680361 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0704385 s, 14.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103071 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102879 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118213 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0809611 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0974593 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0891289 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0917921 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133229 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0966178 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107727 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107481 s, 9.8 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg424-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg424-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.6636 s, 13.7 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.94091 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg424-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.25981 s, 64.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0281864 s, 37.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0320178 s, 32.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0671428 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0674399 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157637 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0980166 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113134 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0917206 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180705 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0760125 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139795 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115212 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170439 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208815 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.262497 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.278498 s, 3.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0348644 s, 30.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0497546 s, 21.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0498608 s, 21.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0387684 s, 27.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0426107 s, 24.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0643957 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0620262 s, 16.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0495365 s, 21.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0542586 s, 19.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0542124 s, 19.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0593056 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0604121 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.075031 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0816424 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0810616 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0837108 s, 12.5 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg424-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg424-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.7835 s, 9.7 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.87442 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.95 s, 9.6 MB/s Write runtime is 13 s, speed is 7.692307 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.88185 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_policies=fifo PASS 77k (436s) == sanityn test 77kb: Check different granular TBF type combination: jobid+opcode ========================================================== 08:52:25 (1789131145) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/nrs_r_oleg424-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.390101 s, 43.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.029447 s, 35.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0290394 s, 36.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0507722 s, 20.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.051873 s, 20.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136996 s, 76.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0291691 s, 35.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0359778 s, 29.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0404151 s, 25.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0312656 s, 33.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169785 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1315 s, 8.0 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.214468 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.248988 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.273019 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23002 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.29056 s, 3.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0713191 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112005 s, 9.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0452043 s, 23.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0846334 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118446 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0627093 s, 16.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105745 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0667325 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0684057 s, 15.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0930222 s, 11.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0583213 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123184 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0833147 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0857852 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101473 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117875 s, 8.9 MB/s removed '/mnt/lustre/d77kb.sanityn/nrs_w_oleg424-client.virtnet' removed '/mnt/lustre/d77kb.sanityn/nrs_r_oleg424-client.virtnet' removed directory '/mnt/lustre/d77kb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.56085 s, 12.2 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0061 s, 10.5 MB/s Read runtime is 12 s, speed is 8.333333 IOPS cleanup_tbf /mnt/lustre/d77kb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kb (52s) == sanityn test 77kc: Check different granular TBF type combination: nid+opcode ========================================================== 08:53:18 (1789131198) ost.OSS.ost_io.nrs_policies=tbf nid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.204.24@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.204.24@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.33494 s, 50.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0368019 s, 28.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0411987 s, 25.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0929216 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0831972 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10402 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0816935 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.094198 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113639 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115408 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108013 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11751 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129431 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.240086 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.245754 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.288471 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.246737 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0334931 s, 31.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00681096 s, 154 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134797 s, 77.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00785599 s, 133 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0627473 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0828506 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0958776 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126392 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.066665 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0539754 s, 19.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0885482 s, 11.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0666268 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0269398 s, 38.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0322335 s, 32.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0333298 s, 31.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0220106 s, 47.6 MB/s removed '/mnt/lustre/d77kc.sanityn/nrs_r_oleg424-client.virtnet' removed '/mnt/lustre/d77kc.sanityn/nrs_w_oleg424-client.virtnet' removed directory '/mnt/lustre/d77kc.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.78553 s, 13.5 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.076 s, 10.4 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kc.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kc (49s) == sanityn test 77kd: Check different granular TBF type combination: nid+jobid ========================================================== 08:54:07 (1789131247) ost.OSS.ost_io.nrs_policies=tbf nid+jobid ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.204.24@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/nrs_r_oleg424-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.394555 s, 42.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0399987 s, 26.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0442507 s, 23.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0626551 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0377644 s, 27.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0543195 s, 19.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0987928 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0500436 s, 21.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119922 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134866 s, 7.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135355 s, 7.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146416 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156113 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0699086 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112472 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177533 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132954 s, 7.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0260999 s, 40.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.07359 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0557432 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0310712 s, 33.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0829803 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0439117 s, 23.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0482336 s, 21.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0799881 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0511068 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0523532 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0576307 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0579133 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0624654 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0640897 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0688729 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0781249 s, 13.4 MB/s removed '/mnt/lustre/d77kd.sanityn/nrs_w_oleg424-client.virtnet' removed '/mnt/lustre/d77kd.sanityn/nrs_r_oleg424-client.virtnet' removed directory '/mnt/lustre/d77kd.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.88842 s, 15.2 MB/s Write runtime is 7 s, speed is 14.285714 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kd.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.10786 s, 20.5 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kd.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kd (47s) == sanityn test 77ke: Check different granular TBF type combination: jobid+opcode ========================================================== 08:54:55 (1789131295) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/nrs_r_oleg424-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.590451 s, 28.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0459126 s, 22.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0404282 s, 25.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0180146 s, 58.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0613131 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0699316 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.051413 s, 20.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0474024 s, 22.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0787784 s, 13.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.258128 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.347801 s, 3.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.329891 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.278017 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.366334 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.36486 s, 2.9 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.347514 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.437492 s, 2.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0698489 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0670367 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0371336 s, 28.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0760512 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.093397 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0635283 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0947288 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.05207 s, 20.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0589679 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0592771 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0834948 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0918626 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0784757 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0938385 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104158 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111509 s, 9.4 MB/s removed '/mnt/lustre/d77ke.sanityn/nrs_w_oleg424-client.virtnet' removed '/mnt/lustre/d77ke.sanityn/nrs_r_oleg424-client.virtnet' removed directory '/mnt/lustre/d77ke.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.7301 s, 9.8 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ke.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.99447 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.35105 s, 14.3 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.98576 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77ke (96s) == sanityn test 77kf: Check different granular TBF type combination: uid+gid ========================================================== 08:56:32 (1789131392) ost.OSS.ost_io.nrs_policies=tbf uid+gid ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/nrs_r_oleg424-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.04626 s, 16.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0407208 s, 25.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0466015 s, 22.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11126 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124283 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121518 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0246125 s, 42.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.011759 s, 89.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0753589 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187434 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.232318 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152076 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.274675 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.331758 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.225634 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210932 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.28737 s, 3.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0380731 s, 27.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0450869 s, 23.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0485985 s, 21.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0495285 s, 21.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0855512 s, 12.3 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0800446 s, 13.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00904902 s, 116 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0115046 s, 91.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0724641 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110752 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.080783 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0745645 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114404 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.068513 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0693338 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0435157 s, 24.1 MB/s removed '/mnt/lustre/d77kf.sanityn/nrs_w_oleg424-client.virtnet' removed '/mnt/lustre/d77kf.sanityn/nrs_r_oleg424-client.virtnet' removed directory '/mnt/lustre/d77kf.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.0639 s, 5.2 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kf.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9488 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77kf.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kf (77s) == sanityn test 77kg: check different granular TBF type combination: uid+opcode ========================================================== 08:57:49 (1789131469) ost.OSS.ost_io.nrs_policies=tbf uid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg424-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.432839 s, 38.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0620752 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0524125 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0703478 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0519798 s, 20.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164472 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191909 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183767 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197052 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194539 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193999 s, 5.4 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200732 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209105 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165886 s, 6.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.224876 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166919 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.252607 s, 4.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0746794 s, 14.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147381 s, 71.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.038046 s, 27.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0561198 s, 18.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0898291 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.022055 s, 47.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0197981 s, 53.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00562008 s, 187 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123824 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00344765 s, 304 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0753766 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0225141 s, 46.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140871 s, 74.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0109122 s, 96.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.027529 s, 38.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00859716 s, 122 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg424-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg424-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.38216 s, 12.5 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.9251 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg424-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.581288 s, 28.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0639724 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0769162 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0553683 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0781926 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116974 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124091 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0500187 s, 21.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101208 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182746 s, 5.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110407 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.279336 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.26792 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.27224 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210369 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.287798 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.293967 s, 3.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0536353 s, 19.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.065644 s, 16.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0690322 s, 15.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0751473 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.074856 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0498688 s, 21.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0893001 s, 11.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0561262 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0865049 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.096764 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0679092 s, 15.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0898889 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119573 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154363 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0732779 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0755137 s, 13.9 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg424-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg424-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.9772 s, 9.6 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.89766 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.2573 s, 10.2 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.868 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kg (150s) == sanityn test 77kh: Verify that Project ID support for NRS TBF rule ========================================================== 09:00:19 (1789131619) using SAVE_PROJECT_SUPPORTED=0 Stopping clients: oleg424-client.virtnet /mnt/lustre (opts:) Stopping client oleg424-client.virtnet /mnt/lustre opts: Stopping clients: oleg424-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg424-client.virtnet /mnt/lustre2 opts: Stopping /mnt/lustre-mds1 (opts:-f) on oleg424-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg424-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg424-server sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime) proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) devtmpfs on /dev type devtmpfs (rw,nosuid,mode=755) securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime) tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev) devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000) tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755) tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755) cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd) pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime) bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700) cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct) cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma) cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event) cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices) cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio) cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids) cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer) cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset) cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb) cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio) cgroup on /sys/fs/cgroup/misc type cgroup (rw,nosuid,nodev,noexec,relatime,misc) cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory) none on /sys/kernel/tracing type tracefs (rw,relatime) configfs on /sys/kernel/config type configfs (rw,relatime) /dev/nbd0 on / type ext4 (ro,relatime,stripe=32) rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M) debugfs on /sys/kernel/debug type debugfs (rw,relatime) mqueue on /dev/mqueue type mqueue (rw,relatime) none on /mnt type ramfs (rw,relatime) none on /var/lib/stateless/writable type tmpfs (rw,relatime) /dev/vda on /home/green/git/lustre-release type squashfs (ro,relatime) none on /var/cache/man type tmpfs (rw,relatime) none on /var/log type tmpfs (rw,relatime) none on /var/lib/dbus type tmpfs (rw,relatime) none on /tmp type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) none on /var/tmp type tmpfs (rw,relatime) none on /var/lib/NetworkManager type tmpfs (rw,relatime) none on /var/lib/systemd/random-seed type tmpfs (rw,relatime) none on /var/spool type tmpfs (rw,relatime) none on /var/lib/nfs type tmpfs (rw,relatime) none on /var/lib/gssproxy type tmpfs (rw,relatime) none on /var/lib/logrotate type tmpfs (rw,relatime) none on /etc type tmpfs (rw,relatime) none on /var/lib/rsyslog type tmpfs (rw,relatime) none on /root/.ssh type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) 192.168.200.253:/exports/state/oleg424-client.virtnet on /var/lib/stateless/state type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.204.24,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg424-client.virtnet/boot on /boot type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.204.24,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg424-client.virtnet/etc/kdump.conf on /etc/kdump.conf type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.204.24,local_lock=none,addr=192.168.200.253) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=378180k,mode=700) 192.168.200.253:/exports/testreports/70058/testresults/sanityn-zfs-rocky8.10_x86_64-rocky8.10_x86_64 on /tmp/testlogs type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.204.24,local_lock=none,addr=192.168.200.253) /dev/vda on /usr/sbin/mount.lustre type squashfs (ro,relatime) Checking servers environments Checking clients oleg424-client.virtnet environments /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg424-server' oleg424-server: oleg424-server.virtnet: executing load_modules_local oleg424-server: Loading modules from /home/green/git/lustre-release/lustre oleg424-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy oleg424-server: detected 4 online CPUs by sysfs oleg424-server: MODOPTS_LIBCFS= oleg424-server: Force libcfs to create 2 CPU partitions Setup mgs, mdt, osts Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg424-server: oleg424-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg424-client: oleg424-server: ssh exited with exit code 1 Started lustre-MDT0000 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg424-server: oleg424-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg424-client: oleg424-server: ssh exited with exit code 1 Started lustre-OST0000 Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg424-server: oleg424-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg424-client: oleg424-server: ssh exited with exit code 1 Started lustre-OST0001 Starting client: oleg424-client.virtnet: -o user_xattr,flock 192.168.204.124@tcp:/lustre /mnt/lustre Starting client oleg424-client.virtnet: -o user_xattr,flock 192.168.204.124@tcp:/lustre /mnt/lustre Started clients oleg424-client.virtnet: 192.168.204.124@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Starting client: oleg424-client.virtnet: -o user_xattr,flock 192.168.204.124@tcp:/lustre /mnt/lustre2 Starting client oleg424-client.virtnet: -o user_xattr,flock 192.168.204.124@tcp:/lustre /mnt/lustre2 Started clients oleg424-client.virtnet: 192.168.204.124@tcp:/lustre on /mnt/lustre2 type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8b6882ed5800.idle_timeout=debug osc.lustre-OST0000-osc-ffff8b68a1ad9800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8b6882ed5800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8b68a1ad9800.idle_timeout=debug disable quota as required ost.OSS.ost_io.nrs_policies=tbf projid ost.OSS.ost_io.nrs_tbf_rule=start ext_proj_rw projid={100} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.431011 s, 38.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0973381 s, 10.8 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113658 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212271 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160665 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.232776 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.217027 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203277 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.230872 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.310873 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.262886 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.348887 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.311884 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.329138 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.351263 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.364007 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.394054 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0577584 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0546801 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0342463 s, 30.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0820632 s, 12.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0611589 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0870616 s, 12.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0734206 s, 14.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0784669 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100139 s, 10.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0728799 s, 14.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.064493 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0531809 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0783633 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0643003 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0859491 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0915604 s, 11.5 MB/s removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg424-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg424-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.57312 s, 12.2 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.38267 s, 19.5 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_proj_rw ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf projid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w projid={100}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r projid={100}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.579768 s, 28.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.066781 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0730962 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0894944 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121823 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126164 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.06618 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148762 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187359 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153961 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.280067 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.322027 s, 3.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.306248 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.31003 s, 3.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.333135 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.32742 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.353937 s, 3.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8b68a1ad9800.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212075 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165086 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168612 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182996 s, 5.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192449 s, 5.4 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162121 s, 6.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.229506 s, 4.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181951 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187411 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17192 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220401 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195989 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186273 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206397 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216845 s, 4.8 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8b68a1ad9800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215497 s, 4.9 MB/s removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg424-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg424-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.65361 s, 13.7 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.117 s, 10.4 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77kh (218s) == sanityn test 77ki: Add rule with unsupported TBF type should fail for generic TBF ========================================================== 09:03:58 (1789131838) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 oleg424-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_fail projid={100}&opcode={ost_write} rate=20: Invalid argument oleg424-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_fail projid={100}&opcode={ost_write} rate=20': Invalid argument pdsh@oleg424-client: oleg424-server: ssh exited with exit code 22 ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77ki (23s) == sanityn test 77kj: Verify nodemap support for NRS TBF rule ========================================================== 09:04:20 (1789131860) On MGS 192.168.204.124, active = nodemap.active=1 On MGS 192.168.204.124, TBF = ost.OSS.ost_io.nrs_policies=tbf nodemap ost.OSS.ost_io.nrs_tbf_rule=start ext_nm_rw nodemap={TBF} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.646098 s, 26.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.045307 s, 23.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0218147 s, 48.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.05716 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0777845 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.033164 s, 31.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19795 s, 5.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.281738 s, 3.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.266974 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.274709 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.266435 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.372463 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.298556 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.38712 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.343125 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.319996 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.356048 s, 2.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8b68a1ad9800.lock_unused_count=5 ldlm.namespaces.lustre-OST0001-osc-ffff8b68a1ad9800.lock_unused_count=2 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.248329 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.249861 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.250027 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.249376 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.309588 s, 3.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.251499 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.259898 s, 4.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.335335 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.263317 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.265147 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.324198 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.292006 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.334602 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.272983 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.274957 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.314913 s, 3.3 MB/s removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg424-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg424-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.33299 s, 11.2 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.15679 s, 20.3 MB/s Read runtime is 7 s, speed is 14.285714 IOPS cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete On MGS 192.168.204.124, TBF.id = On MGS 192.168.204.124, active = nodemap.active=0 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.557237 s, 30.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0848434 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120161 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102546 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122068 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118362 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137965 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197801 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.25581 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.25204 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222945 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.271471 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.257119 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.244113 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.297716 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.233516 s, 4.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8b68a1ad9800.lock_unused_count=3 ldlm.namespaces.lustre-OST0001-osc-ffff8b68a1ad9800.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.255793 s, 4.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0372012 s, 28.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0400756 s, 26.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0694882 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0624515 s, 16.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0351292 s, 29.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0511742 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0520979 s, 20.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0793657 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0623309 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0613833 s, 17.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0598295 s, 17.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0646908 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0551647 s, 19.0 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0703903 s, 14.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0965917 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0801118 s, 13.1 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8b68a1ad9800.lock_unused_count=1 removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg424-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg424-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 1 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.48093 s, 14.0 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 2.80569 s, 37.4 MB/s Read runtime is 4 s, speed is 25.000000 IOPS read rate (25.000000) should exceed limit (20 * 1) cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_nm_rw ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf nodemap+opcode oleg424-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_w nodemap={TBF}&opcode={ost_write} rate=20: Invalid argument oleg424-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_w nodemap={TBF}&opcode={ost_write} rate=20': Invalid argument oleg424-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_r nodemap={TBF}&opcode={ost_read} rate=10: Invalid argument oleg424-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_r nodemap={TBF}&opcode={ost_read} rate=10': Invalid argument pdsh@oleg424-client: oleg424-server: ssh exited with exit code 22 On MGS 192.168.204.124, active = nodemap.active=1 On MGS 192.168.204.124, TBF = ost.OSS.ost_io.nrs_tbf_rule=start ext_w nodemap={TBF}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nodemap={TBF}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.784351 s, 21.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0606289 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.099638 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0512958 s, 20.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123056 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0638913 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146248 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139233 s, 7.5 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0895984 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175854 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.255063 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176236 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.25462 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.282826 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.348992 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22448 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.24781 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133687 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181987 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139259 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20299 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154805 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186164 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196622 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16733 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216467 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190989 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187403 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183831 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186607 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192236 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193409 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20388 s, 5.1 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8b68a1ad9800.lock_unused_count=5 removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg424-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg424-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.03752 s, 13.0 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.91571 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo On MGS 192.168.204.124, TBF.id = On MGS 192.168.204.124, active = nodemap.active=0 PASS 77kj (193s) == sanityn test 77l: check the output of NRS policies for generic TBF ========================================================== 09:07:33 (1789132053) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_policies=fifo PASS 77l (11s) == sanityn test 77m: check NRS Delay slows write RPC processing ========================================================== 09:07:44 (1789132064) ost.OSS.ost_io.nrs_policies=delay ost.OSS.ost_io.nrs_delay_min=4 ost.OSS.ost_io.nrs_delay_max=4 ost.OSS.ost_io.nrs_delay_pct=100 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg424-client.virtnet] [bs=1M] [count=1] [oflag=direct] [conv=fdatasync] 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 3.39564 s, 309 kB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg424-client.virtnet] [bs=1M] [count=10] [oflag=direct] [conv=fdatasync] 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 40.1007 s, 261 kB/s ost.OSS.ost_io.nrs_policies=fifo PASS 77m (56s) == sanityn test 77n: check wildcard support for TBF JobID NRS policy ========================================================== 09:08:40 (1789132120) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={*.500} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.369943 s, 45.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0584442 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103164 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118651 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166538 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13248 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125006 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194594 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.217869 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142362 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.249038 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.241037 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208753 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178241 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.243888 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.257745 s, 4.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8b68a1ad9800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff8b68a1ad9800.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133724 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0756046 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00325376 s, 322 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.070075 s, 15.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128361 s, 8.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0112747 s, 93.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0589224 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136068 s, 7.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0883478 s, 11.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0534286 s, 19.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119273 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0872834 s, 12.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0352364 s, 29.8 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0332867 s, 31.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0937629 s, 11.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150752 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0285488 s, 36.7 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg424-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg424-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77n.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.24368 s, 14.5 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77n.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.20216 s, 20.2 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.*} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.470896 s, 35.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0634879 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0947122 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0984859 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0829425 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0812125 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0922617 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137181 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121845 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212932 s, 4.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18968 s, 5.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101782 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223012 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196868 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.245988 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.260716 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199739 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0711423 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0448527 s, 23.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0429233 s, 24.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0424391 s, 24.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0873114 s, 12.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0719327 s, 14.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110362 s, 9.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108335 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0616238 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0780415 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121834 s, 8.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0820822 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0573326 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0865738 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.070352 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104281 s, 10.1 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg424-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg424-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.0937 s, 14.8 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.3491 s, 19.6 MB/s Read runtime is 7 s, speed is 14.285714 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_policies=fifo PASS 77n (89s) == sanityn test 77o: Changing rank should not panic ====== 09:10:10 (1789132210) mds.MDS.mdt.nrs_policies=tbf nid mds.MDS.mdt.nrs_tbf_rule=start name nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=start name1 nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=change name1 rank=name mds.MDS.mdt.nrs_tbf_rule=stop name mds.MDS.mdt.nrs_policies=fifo PASS 77o (10s) == sanityn test 77q: Parallel TBF rule definitions should not panic ========================================================== 09:10:21 (1789132221) mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_policies=fifo PASS 77q (110s) == sanityn test 77p: Check validity of rule names for TBF policies ========================================================== 09:12:11 (1789132331) mds.MDS.mdt.nrs_policies=tbf oleg424-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_overflo uid={500} rate=500: File name too long oleg424-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_overflo uid={500} rate=500': File name too long pdsh@oleg424-client: oleg424-server: ssh exited with exit code 36 oleg424-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start uid={500} rate=500: Invalid argument oleg424-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start uid={500} rate=500': Invalid argument pdsh@oleg424-client: oleg424-server: ssh exited with exit code 22 oleg424-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_empty: Invalid argument oleg424-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_empty': Invalid argument pdsh@oleg424-client: oleg424-server: ssh exited with exit code 22 oleg424-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p@spec uid={500} rate=500: Invalid argument oleg424-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p@spec uid={500} rate=500': Invalid argument pdsh@oleg424-client: oleg424-server: ssh exited with exit code 22 oleg424-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p.spec uid={500} rate=500: Invalid argument oleg424-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p.spec uid={500} rate=500': Invalid argument pdsh@oleg424-client: oleg424-server: ssh exited with exit code 22 oleg424-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p~spec uid={500} rate=500: Invalid argument oleg424-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p~spec uid={500} rate=500': Invalid argument pdsh@oleg424-client: oleg424-server: ssh exited with exit code 22 oleg424-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p#spec uid={500} rate=500: Invalid argument oleg424-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p#spec uid={500} rate=500': Invalid argument pdsh@oleg424-client: oleg424-server: ssh exited with exit code 22 oleg424-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p/spec uid={500} rate=500: Invalid argument oleg424-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p/spec uid={500} rate=500': Invalid argument pdsh@oleg424-client: oleg424-server: ssh exited with exit code 22 oleg424-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p^spec uid={500} rate=500: Invalid argument oleg424-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p^spec uid={500} rate=500': Invalid argument pdsh@oleg424-client: oleg424-server: ssh exited with exit code 22 oleg424-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p%spec uid={500} rate=500: Invalid argument oleg424-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p%spec uid={500} rate=500': Invalid argument pdsh@oleg424-client: oleg424-server: ssh exited with exit code 22 oleg424-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p*spec uid={500} rate=500: Invalid argument oleg424-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p*spec uid={500} rate=500': Invalid argument pdsh@oleg424-client: oleg424-server: ssh exited with exit code 22 oleg424-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p;spec uid={500} rate=500: Invalid argument oleg424-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p;spec uid={500} rate=500': Invalid argument pdsh@oleg424-client: oleg424-server: ssh exited with exit code 22 oleg424-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p,spec uid={500} rate=500: Invalid argument oleg424-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p,spec uid={500} rate=500': Invalid argument pdsh@oleg424-client: oleg424-server: ssh exited with exit code 22 oleg424-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p?spec uid={500} rate=500: Invalid argument oleg424-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p?spec uid={500} rate=500': Invalid argument pdsh@oleg424-client: oleg424-server: ssh exited with exit code 22 oleg424-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77pspec uid={500} rate=500: Invalid argument oleg424-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p>spec uid={500} rate=500': Invalid argument pdsh@oleg424-client: oleg424-server: ssh exited with exit code 22 oleg424-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p:spec uid={500} rate=500: Invalid argument oleg424-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p:spec uid={500} rate=500': Invalid argument pdsh@oleg424-client: oleg424-server: ssh exited with exit code 22 oleg424-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p+spec uid={500} rate=500: Invalid argument oleg424-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p+spec uid={500} rate=500': Invalid argument pdsh@oleg424-client: oleg424-server: ssh exited with exit code 22 oleg424-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p=spec uid={500} rate=500: Invalid argument oleg424-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p=spec uid={500} rate=500': Invalid argument pdsh@oleg424-client: oleg424-server: ssh exited with exit code 22 oleg424-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p)spec uid={500} rate=500: Invalid argument oleg424-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p)spec uid={500} rate=500': Invalid argument pdsh@oleg424-client: oleg424-server: ssh exited with exit code 22 oleg424-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p(spec uid={500} rate=500: Invalid argument oleg424-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p(spec uid={500} rate=500': Invalid argument pdsh@oleg424-client: oleg424-server: ssh exited with exit code 22 oleg424-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p{spec uid={500} rate=500: Invalid argument oleg424-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p{spec uid={500} rate=500': Invalid argument pdsh@oleg424-client: oleg424-server: ssh exited with exit code 22 oleg424-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p}spec uid={500} rate=500: Invalid argument oleg424-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p}spec uid={500} rate=500': Invalid argument pdsh@oleg424-client: oleg424-server: ssh exited with exit code 22 oleg424-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p|spec uid={500} rate=500: Invalid argument oleg424-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p|spec uid={500} rate=500': Invalid argument pdsh@oleg424-client: oleg424-server: ssh exited with exit code 22 oleg424-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p[spec uid={500} rate=500: Invalid argument oleg424-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p[spec uid={500} rate=500': Invalid argument pdsh@oleg424-client: oleg424-server: ssh exited with exit code 22 oleg424-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p]spec uid={500} rate=500: Invalid argument oleg424-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p]spec uid={500} rate=500': Invalid argument pdsh@oleg424-client: oleg424-server: ssh exited with exit code 22 oleg424-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p!spec uid={500} rate=500: Invalid argument oleg424-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p!spec uid={500} rate=500': Invalid argument pdsh@oleg424-client: oleg424-server: ssh exited with exit code 22 oleg424-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p&spec uid={500} rate=500: Invalid argument oleg424-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p&spec uid={500} rate=500': Invalid argument pdsh@oleg424-client: oleg424-server: ssh exited with exit code 22 oleg424-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p$spec uid={500} rate=500: Invalid argument oleg424-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p$spec uid={500} rate=500': Invalid argument pdsh@oleg424-client: oleg424-server: ssh exited with exit code 22 oleg424-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p`spec uid={500} rate=500: Invalid argument oleg424-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p`spec uid={500} rate=500': Invalid argument pdsh@oleg424-client: oleg424-server: ssh exited with exit code 22 oleg424-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p\spec uid={500} rate=500: Invalid argument oleg424-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p\spec uid={500} rate=500': Invalid argument pdsh@oleg424-client: oleg424-server: ssh exited with exit code 22 mds.MDS.mdt.nrs_policies=fifo PASS 77p (35s) == sanityn test 77r: Change type of tbf policy at run time ========================================================== 09:12:47 (1789132367) mds.MDS.mdt.nrs_policies=tbf uid mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf gid mds.MDS.mdt.nrs_tbf_rule=start md_rule gid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start md_rule jobid={*.500} rate=1 mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=fifo check the number of requests in queue: fifo: 0 crrn: 0 tbf: 0 delay: 0 fifo: 0 crrn: 0 tbf: 0 delay: 0 cleanup 77r 274341 kill md_thread xargs PASS 77r (55s) == sanityn test 77s: Check TBF LRU shrinker ============== 09:13:41 (1789132421) jobid_name=%e.%p jobid_var=TEST_77s_JOBVAR mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start touch jobid={touch.*} rate=1000 mds.MDS.mdt.nrs_tbf_rule=start unlink jobid={unlink.*} rate=1000 touch jobid={touch.*} 1000, ref 8 touch jobid={touch.*} 1000, ref 0 unlink jobid={unlink.*} 1000, ref 8 unlink jobid={unlink.*} 1000, ref 0 mds.MDS.mdt.nrs_policies=fifo jobid_var=procname_uid jobid_name=%e.%u PASS 77s (59s) == sanityn test 78: Enable policy and specify tunings right away ========================================================== 09:14:40 (1789132480) oleg424-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_orr_quantum=1: No such device oleg424-server: error: set_param: setting 'ost/OSS/*/nrs_orr_quantum'='1': No such device pdsh@oleg424-client: oleg424-server: ssh exited with exit code 19 ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_policies=fifo PASS 78 (9s) == sanityn test 79: xattr: intent error ================== 09:14:49 (1789132489) File: /mnt/lustre/d79.sanityn Size: 512 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115238826935667 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-11 09:14:51.000000000 -0400 Modify: 2026-09-11 09:14:51.000000000 -0400 Change: 2026-09-11 09:14:51.000000000 -0400 Birth: 2026-09-11 09:14:50.000000000 -0400 fail_val=0 fail_loc=0x80000160 fail_val=0 fail_loc=0x80000131 # file: mnt/lustre/d79.sanityn trusted.name1="value1" PASS 79 (18s) == sanityn test 80a: migrate directory when some children is being opened ========================================================== 09:15:08 (1789132508) SKIP: sanityn test_80a needs >= 2 MDTs SKIP 80a (3s) == sanityn test 80b: Accessing directory during migration ========================================================== 09:15:11 (1789132511) SKIP: sanityn test_80b needs >= 2 MDTs SKIP 80b (4s) == sanityn test 81a: rename and stat under striped directory ========================================================== 09:15:14 (1789132514) SKIP: sanityn test_81a needs >= 2 MDTs SKIP 81a (3s) == sanityn test 81b: rename under striped directory doesn't deadlock ========================================================== 09:15:18 (1789132518) SKIP: sanityn test_81b We need at least 2 MDTs for this test SKIP 81b (3s) == sanityn test 81c: rename revoke LOOKUP lock for remote object ========================================================== 09:15:21 (1789132521) SKIP: sanityn test_81c needs >= 4 MDTs SKIP 81c (4s) == sanityn test 81d: parallel rename file cross-dir on same MDT ========================================================== 09:15:25 (1789132525) - open/close 726 (time 1789132541.31 total 10.00 last 72.57) - open/close 1456 (time 1789132551.33 total 20.02 last 72.89) - open/close 2239 (time 1789132561.34 total 30.03 last 78.19) total: 2500 open/close in 33.36 seconds: 74.93 ops/second snapshot_time 1789132645.559507102 secs.nsecs start_time 1789132528.338388289 secs.nsecs elapsed_time 117.221118813 secs.nsecs open 2505 samples [usecs] 149 22495 4049471 10441636555 close 2504 samples [usecs] 163 10121 1005990 897063674 mknod 2501 samples [usecs] 586 22374 3551676 8250961258 unlink 2500 samples [usecs] 756 58184 4948080 25898503250 mkdir 2 samples [usecs] 965 1022 1987 1975709 rmdir 2 samples [usecs] 16550 18807 35357 627605749 rename 2500 samples [usecs] 528 40865 15176975 171814889513 getattr 3763 samples [usecs] 10 9263 223533 256642597 parallel_rename_file 2500 samples [usecs] 528 40865 15176975 171814889513 crossdir_rename 2500 samples [usecs] 528 40865 15176975 171814889513 readpage 15 samples [usecs] 4833 48461 253448 6155818056 PASS 81d (125s) == sanityn test 82: fsetxattr and fgetxattr on orphan files ========================================================== 09:17:31 (1789132651) multiop /mnt/lustre/f82.sanityn vO_ac TMPPIPE=/tmp/multiop_open_wait_pipe.7649 multiop /mnt/lustre2/f82.sanityn vo_Ac TMPPIPE=/tmp/multiop_open_wait_pipe.7649 PASS 82 (7s) == sanityn test 83: access striped directory while it is being created/unlinked ========================================================== 09:17:38 (1789132658) SKIP: sanityn test_83 needs >= 2 MDTs SKIP 83 (3s) == sanityn test 84: 0-nlink race in lu_object_find() ===== 09:17:41 (1789132661) fail_loc=0x8000060b PASS 84 (11s) == sanityn test 85: Lustre API root cache race =========== 09:17:52 (1789132672) Starting: llapi_root_test /mnt/lustre/f85.sanityn /mnt/lustre2/f85.sanityn Starting test test1 at 1789132674 Finishing test test1 at 1789132678 PASS 85 (11s) == sanityn test 90: open/create and unlink striped directory ========================================================== 09:18:03 (1789132683) SKIP: sanityn test_90 needs >= 2 MDTs SKIP 90 (3s) == sanityn test 91: chmod and unlink striped directory === 09:18:06 (1789132686) SKIP: sanityn test_91 needs >= 2 MDTs SKIP 91 (3s) == sanityn test 92: create remote directory under orphan directory ========================================================== 09:18:09 (1789132689) SKIP: sanityn test_92 needs >= 2 MDTs SKIP 92 (3s) == sanityn test 93: alloc_rr should not allocate on same ost ========================================================== 09:18:12 (1789132692) fail_loc=0x00000163 fail_loc=0x0 /mnt/lustre/d93.sanityn-1/file1 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5792 0x16a0 0x240000400 1 5754 0x167a 0x280000400 /mnt/lustre2/d93.sanityn-2/file2 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5793 0x16a1 0x240000400 1 5755 0x167b 0x280000400 rc1=0 and rc2=0 PASS 93 (15s) == sanityn test 94: signal vs CP callback race =========== 09:18:27 (1789132707) osc.lustre-OST0000-osc-ffff8b6882ed5800.idle_timeout=0 osc.lustre-OST0000-osc-ffff8b68a1ad9800.idle_timeout=0 osc.lustre-OST0001-osc-ffff8b6882ed5800.idle_timeout=0 osc.lustre-OST0001-osc-ffff8b68a1ad9800.idle_timeout=0 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.214281 s, 38.2 kB/s fail_val=5 fail_loc=0x80000312 fail_val=6 fail_loc=0x80000329 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 2.96019 s, 1.4 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 6414: 291466 Alarm clock dd if=/dev/zero of=$DIR/$tfile conv=notrunc oflag=append bs=4k count=1 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.058771 s, 69.7 kB/s osc.lustre-OST0000-osc-ffff8b6882ed5800.idle_timeout=debug osc.lustre-OST0000-osc-ffff8b68a1ad9800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8b6882ed5800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8b68a1ad9800.idle_timeout=debug PASS 94 (15s) == sanityn test 95a: Check readpage() on a page that was removed from page cache ========================================================== 09:18:42 (1789132722) llite.lustre-ffff8b6882ed5800.fast_read=0 llite.lustre-ffff8b68a1ad9800.fast_read=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0784995 s, 26.7 MB/s fail_loc=0x80001421 fail_val=7 fail_loc=0x80001422 fail_val=10 llite.lustre-ffff8b6882ed5800.fast_read=1 llite.lustre-ffff8b68a1ad9800.fast_read=1 PASS 95a (19s) == sanityn test 95b: Check readpage() on a page that is no longer uptodate ========================================================== 09:19:01 (1789132741) llite.lustre-ffff8b6882ed5800.fast_read=0 llite.lustre-ffff8b68a1ad9800.fast_read=0 1+0 records in 1+0 records out 12288 bytes (12 kB, 12 KiB) copied, 0.0220852 s, 556 kB/s fail_loc=0x80001424 fail_val=5 fail_loc=0 llite.lustre-ffff8b6882ed5800.fast_read=1 llite.lustre-ffff8b68a1ad9800.fast_read=1 PASS 95b (14s) == sanityn test 100a: DoM: glimpse RPCs for stat without IO lock (DoM only file) ========================================================== 09:19:15 (1789132755) SKIP: sanityn test_100a Reserved for glimpse-ahead SKIP 100a (3s) == sanityn test 100b: DoM: no glimpse RPC for stat with IO lock (DoM only file) ========================================================== 09:19:18 (1789132758) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00362615 s, 1.1 MB/s /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK PASS 100b (7s) == sanityn test 100c: DoM: write vs stat without IO lock (combined file) ========================================================== 09:19:25 (1789132765) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.152865 s, 13.7 MB/s /mnt/lustre/d100c.sanityn/dom has type file OK /mnt/lustre/d100c.sanityn/dom has size 2097152 OK PASS 100c (7s) == sanityn test 100d: DoM: write+truncate vs stat without IO lock (combined file) ========================================================== 09:19:32 (1789132772) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.158247 s, 13.3 MB/s /mnt/lustre/d100d.sanityn/dom has type file OK /mnt/lustre/d100d.sanityn/dom has size 4096 OK PASS 100d (7s) == sanityn test 100e: DoM: read on open and file size ==== 09:19:39 (1789132779) 1+0 records in 1+0 records out 12000 bytes (12 kB, 12 KiB) copied, 0.0109998 s, 1.1 MB/s PASS 100e (8s) == sanityn test 101a: Discard DoM data on unlink ========= 09:19:47 (1789132787) /mnt/lustre/f101a.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0173179 s, 237 kB/s PASS 101a (7s) == sanityn test 101b: Discard DoM data on rename ========= 09:19:54 (1789132794) /mnt/lustre/f101b.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.01722 s, 238 kB/s PASS 101b (7s) == sanityn test 101c: Discard DoM data on close-unlink === 09:20:02 (1789132802) /mnt/lustre/f101c.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00817936 s, 501 kB/s PASS 101c (7s) == sanityn test 102: Test open by handle of unlinked file ========================================================== 09:20:09 (1789132809) file: /mnt/lustre/f102.sanityn fh_bytes: 32 fh_type: 151 fh_data: d1 0b 00 00 02 00 00 00 54 0f 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 file's parent FID is [0:0x0:0x0] file FID is [0x200000bd1:0xf54:0x0] access via mount point '/mnt/lustre2' - OK access after unlink - OK check_fhandle_syscalls test Passed! PASS 102 (8s) == sanityn test 103: Test size correctness with lockahead ========================================================== 09:20:17 (1789132817) fail_loc=0x415 fail_loc=0x415 fail_val=2 Incorrect size expected (no glimpse fix): Starting test test23 at 1789132820 lockahead_test: lockahead_test.c:1171: test23: assertion 'sb.st_size == sb2.st_size' failed: size on /mnt/lustre/d103.sanityn/f103.sanityn and /mnt/lustre2/d103.sanityn/f103.sanityn differs: 0 vs 1048576 fail_loc=0 fail_loc=0x214 fail_val=2 Starting test test23 at 1789132824 Finishing test test23 at 1789132824 PASS 103 (15s) == sanityn test 104: Verify that MDS stores atime/mtime/ctime during close ========================================================== 09:20:32 (1789132832) SKIP: sanityn test_104 ldiskfs only test SKIP 104 (4s) == sanityn test 105: Glimpse and lock cancel race ======== 09:20:36 (1789132836) fail_loc=0x416 PASS 105 (24s) == sanityn test 106a: Verify the btime via statx() ======= 09:21:00 (1789132860) SKIP: sanityn test_106a Test only for ldiskfs and statx() supported SKIP 106a (4s) == sanityn test 106b: Glimpse RPCs test for statx ======== 09:21:04 (1789132864) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0128968 s, 79.4 kB/s File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238826938203 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-11 09:21:06.000000000 -0400 Modify: 2026-09-11 09:21:06.000000000 -0400 Change: 2026-09-11 09:21:06.000000000 -0400 Birth: 2026-09-11 09:21:05.000000000 -0400 /mnt/lustre/f106b.sanityn 144115238826938203 -rw-r--r-- 1 0 0 1789132865 1789132866 1789132866 File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238826938203 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-11 09:21:06.000000000 -0400 Modify: 2026-09-11 09:21:06.000000000 -0400 Change: 2026-09-11 09:21:06.000000000 -0400 Birth: 2026-09-11 09:21:05.000000000 -0400 1789132866 1024 11 PASS 106b (8s) == sanityn test 106c: Verify statx attributes mask ======= 09:21:12 (1789132872) PASS 106c (6s) == sanityn test 107a: Basic grouplock conflict =========== 09:21:19 (1789132879) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.241704 s, 43.4 MB/s /mnt/lustre/f107a.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5802 0x16aa 0x240000400 multiop /mnt/lustre/f107a.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7649 multiop /mnt/lustre2/f107a.sanityn vO_G16022000r10g16022000c TMPPIPE=/tmp/multiop_open_wait_pipe.7649 First grouplock blocks second one PASS 107a (8s) == sanityn test 107b: Grouplock is added to the head of waiting list ========================================================== 09:21:26 (1789132886) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.182913 s, 57.3 MB/s /mnt/lustre/f107b.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 5762 0x1682 0x280000400 multiop /mnt/lustre/f107b.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7649 Grouplock blocks IO First grouplock blocks second one Second grouplock blocks IO PASS 107b (12s) == sanityn test 108a: lseek: parallel updates ============ 09:21:39 (1789132899) 1+0 records in 1+0 records out 10000 bytes (10 kB, 9.8 KiB) copied, 0.00873798 s, 1.1 MB/s llite.lustre-ffff8b6882ed5800.hybrid_io=0 llite.lustre-ffff8b68a1ad9800.hybrid_io=0 fail_val=4 fail_loc=0x80000414 1+0 records in 1+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 4.44529 s, 1.9 MB/s PASS 108a (10s) == sanityn test 109: Race with several mount instances on 1 node ========================================================== 09:21:49 (1789132909) 192.168.204.124@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg424-client.virtnet /mnt/lustre (opts:) 192.168.204.124@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg424-client.virtnet /mnt/lustre2 (opts:) Starting race between client mounts (duration 30) Iteration 0 Starting client: oleg424-client.virtnet: -o user_xattr,flock 192.168.204.124@tcp:/lustre /mnt/lustre Starting client: oleg424-client.virtnet: -o user_xattr,flock 192.168.204.124@tcp:/lustre /mnt/lustre2 192.168.204.124@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.124@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg424-client.virtnet /mnt/lustre (opts:) Stopping client oleg424-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg424-server' oleg424-server: oleg424-server.virtnet: executing load_modules_local oleg424-server: Loading modules from /home/green/git/lustre-release/lustre oleg424-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy oleg424-server: detected 4 online CPUs by sysfs oleg424-server: MODOPTS_LIBCFS= oleg424-server: Force libcfs to create 2 CPU partitions Iteration 1 Starting client: oleg424-client.virtnet: -o user_xattr,flock 192.168.204.124@tcp:/lustre /mnt/lustre2 Starting client: oleg424-client.virtnet: -o user_xattr,flock 192.168.204.124@tcp:/lustre /mnt/lustre 192.168.204.124@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.124@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg424-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg424-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg424-server' oleg424-server: oleg424-server.virtnet: executing load_modules_local oleg424-server: Loading modules from /home/green/git/lustre-release/lustre oleg424-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy oleg424-server: detected 4 online CPUs by sysfs oleg424-server: MODOPTS_LIBCFS= oleg424-server: Force libcfs to create 2 CPU partitions Starting client: oleg424-client.virtnet: -o user_xattr,flock 192.168.204.124@tcp:/lustre /mnt/lustre Starting client: oleg424-client.virtnet: -o user_xattr,flock 192.168.204.124@tcp:/lustre /mnt/lustre2 PASS 109 (62s) == sanityn test 110: do not grant another lock on resend ========================================================== 09:22:51 (1789132971) fail_loc=0x525 fail_val=3 fail_loc=0x534 File: /mnt/lustre/f110.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238860488707 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-11 09:22:53.000000000 -0400 Modify: 2026-09-11 09:22:53.000000000 -0400 Change: 2026-09-11 09:22:53.000000000 -0400 Birth: 2026-09-11 09:22:52.000000000 -0400 fail_loc=0 fail_val=0 Connected clients: oleg424-client.virtnet oleg424-client.virtnet PASS 110 (171s) == sanityn test 111: A racy rename/link an open file should not cause fs corruption ========================================================== 09:25:42 (1789133142) SKIP: sanityn test_111 needs >= 2 MDTs SKIP 111 (4s) == sanityn test 112: update max-inherit in default LMV === 09:25:46 (1789133146) SKIP: sanityn test_112 We need at least 2 MDTs for this test SKIP 112 (4s) == sanityn test 113: check servers of specified fs ======= 09:25:50 (1789133150) lustre-OST0000-osc-ffff8b68912be000 active. lustre-OST0001-osc-ffff8b68912be000 active. lustre-OST0000-osc-ffff8b68912be000 active. lustre-OST0001-osc-ffff8b68912be000 active. MGC192.168.204.124@tcp active. PASS 113 (8s) == sanityn test 114: implicit default LMV inherit ======== 09:25:58 (1789133158) SKIP: sanityn test_114 We need at least 2 MDTs for this test SKIP 114 (4s) == sanityn test 115: ldiskfs doesn't check direntry for uniqueness ========================================================== 09:26:03 (1789133163) SKIP: sanityn test_115 ldiskfs only test SKIP 115 (3s) == sanityn test 116: DNE: Set default LMV layout from a remote client ========================================================== 09:26:06 (1789133166) SKIP: sanityn test_116 needs >= 2 MDTs SKIP 116 (3s) == sanityn test 121: trunc append race =================== 09:26:10 (1789133170) 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0243727 s, 840 kB/s fail_loc=0x80001436 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.120669 s, 170 kB/s 0000000 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0005000 0000 0000 0000 0000 0000 0000 0000 0000 * 000c800 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0011800 PASS 121 (12s) == sanityn test 200: service remains healthy while able to process request ========================================================== 09:26:22 (1789133182) at_max=10 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=12 fail_loc=0x1e1 fail_val=20 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.2058 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.2288 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.2975 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.2205 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.2642 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.2477 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1958 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.2488 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.2368 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.0678 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1271 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.0499 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.0675 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.0978 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1989 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.2164 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1073 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.2188 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.1946 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.1918 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.3614 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.264 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.3229 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.3349 s, 0.1 kB/s at_max=600 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=116 oleg424-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8b6882b93800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8b6882b93800.ost_server_uuid in IDLE state after 0 sec PASS 200 (83s) cleanup: ====================================================== == sanityn test complete, duration 10448 sec ============= 09:27:48 (1789133268) osc.lustre-OST0000-osc-ffff8b6882b93800.stats_compr= write_pages_compressible: 0 write_pages_incompressible: 0 write_chunks_compressible: 0 write_chunks_incompressible: 0 write_bytes_compr: 0 write_bytes_raw: 0 write_bytes_incompr: 0 read_pages_compressed: 0 read_chunks_compressed: 0 read_bytes_compr: 0 read_bytes_raw: 0 osc.lustre-OST0000-osc-ffff8b68912be000.stats_compr= write_pages_compressible: 0 write_pages_incompressible: 0 write_chunks_compressible: 0 write_chunks_incompressible: 0 write_bytes_compr: 0 write_bytes_raw: 0 write_bytes_incompr: 0 read_pages_compressed: 0 read_chunks_compressed: 0 read_bytes_compr: 0 read_bytes_raw: 0 osc.lustre-OST0001-osc-ffff8b6882b93800.stats_compr= write_pages_compressible: 0 write_pages_incompressible: 0 write_chunks_compressible: 0 write_chunks_incompressible: 0 write_bytes_compr: 0 write_bytes_raw: 0 write_bytes_incompr: 0 read_pages_compressed: 0 read_chunks_compressed: 0 read_bytes_compr: 0 read_bytes_raw: 0 osc.lustre-OST0001-osc-ffff8b68912be000.stats_compr= write_pages_compressible: 0 write_pages_incompressible: 0 write_chunks_compressible: 0 write_chunks_incompressible: 0 write_bytes_compr: 0 write_bytes_raw: 0 write_bytes_incompr: 0 read_pages_compressed: 0 read_chunks_compressed: 0 read_bytes_compr: 0 read_bytes_raw: 0 obdfilter.lustre-OST0000.stats_compr= snapshot_time 1789133270.969169820 secs.nsecs start_time 1789131696.899749787 secs.nsecs elapsed_time 1574.069420033 secs.nsecs uncompressed_objects 1515 samples [reqs] obdfilter.lustre-OST0001.stats_compr= snapshot_time 1789133270.969234471 secs.nsecs start_time 1789131712.300840474 secs.nsecs elapsed_time 1558.668393997 secs.nsecs uncompressed_objects 1519 samples [reqs] === sanityn: start cleanup 09:27:51 (1789133271) === Stopping clients: oleg424-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg424-client.virtnet /mnt/lustre2 opts: === sanityn: finish cleanup 09:32:05 (1789133525) ===