-----============= acceptance-small: sanityn ============----- Sun Jun 7 00:51:26 EDT 2026 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg350-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanityn.*ex': No such file or directory excepting tests: 27 40a 102 skipping tests SLOW=no: 33a === sanityn: start setup 00:51:35 (1780807895) === Starting client oleg350-client.virtnet: -o user_xattr,flock 192.168.203.150@tcp:/lustre /mnt/lustre2 Mount client oleg350-client.virtnet: mount -t lustre -o user_xattr,flock 192.168.203.150@tcp:/lustre /mnt/lustre2 Started clients oleg350-client.virtnet: 192.168.203.150@tcp:/lustre on /mnt/lustre2 type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) oleg350-client.virtnet: executing check_config_client /mnt/lustre oleg350-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg350-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8f15073b7800.idle_timeout=debug osc.lustre-OST0000-osc-ffff8f150925a000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8f15073b7800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8f150925a000.idle_timeout=debug disable quota as required oleg350-server: oleg350-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanityn: finish setup 00:51:48 (1780807908) === 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142067 s, 73.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8387] == sanityn test 0: do_node_vp() and do_facet_vp() do the right thing ========================================================== 00:51:49 (1780807909) PASS 0 (3s) == sanityn test 1: Check attribute updates on 2 mount points ========================================================== 00:51:52 (1780807912) /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0777 OK /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0666 OK /mnt/lustre/f1.sanityn: absent OK PASS 1 (3s) == sanityn test 2a: check cached attribute updates on 2 mtpt's ================================================================== 00:51:55 (1780807915) -rw-r--r-- 1 root root 0 Jun 7 00:51 /mnt/lustre2/f2a /mnt/lustre/f2a has type file OK /mnt/lustre/f2a has perms 0777 OK PASS 2a (2s) == sanityn test 2b: check cached attribute updates on 2 mtpt's ================================================================== 00:51:57 (1780807917) -rw-r--r-- 1 root root 0 Jun 7 00:51 /mnt/lustre2/f2b /mnt/lustre2/f2b has type file OK /mnt/lustre2/f2b has perms 0777 OK PASS 2b (3s) == sanityn test 2c: check cached attribute updates on 2 mtpt's root ============================================================= 00:52:00 (1780807920) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0777 OK PASS 2c (2s) == sanityn test 2d: check cached attribute updates on 2 mtpt's root ============================================================= 00:52:02 (1780807922) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0755 OK PASS 2d (2s) == sanityn test 2e: check chmod on root is propagated to others ========================================================== 00:52:04 (1780807924) total 1 -rwxrwxrwx 1 root root 0 Jun 7 00:51 f2a -rwxrwxrwx 1 root root 0 Jun 7 00:51 f2b total 1 -rwxrwxrwx 1 root root 0 Jun 7 00:51 f2a -rwxrwxrwx 1 root root 0 Jun 7 00:51 f2b running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre2/f2e.sanityn] [count=1] 1+0 records in 1+0 records out 512 bytes copied, 0.00390798 s, 131 kB/s PASS 2e (3s) == sanityn test 2f: check attr/owner updates on DNE with 2 mtpt's ========================================================== 00:52:07 (1780807927) /mnt/lustre2/d2f.sanityn/remote_dir/f2f.sanityn has type file OK /mnt/lustre2/d2f.sanityn/remote_dir/f2f.sanityn has perms 0777 OK /mnt/lustre2/d2f.sanityn/remote_dir/f2f.sanityn is owned by user #500 OK /mnt/lustre2/d2f.sanityn/remote_dir/f2f.sanityn is owned by group #500 OK Can't lstat /mnt/lustre2/d2f.sanityn/remote_dir/f2f.sanityn: No such file or directory PASS 2f (2s) == sanityn test 2g: check blocks update on sync write ==== 00:52:09 (1780807929) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0929143 s, 22.6 MB/s /mnt/lustre/f2g.sanityn has 4103 blocks /mnt/lustre2/f2g.sanityn has 4103 blocks PASS 2g (3s) == sanityn test 3: symlink on one mtpt, readlink on another ===================================================================== 00:52:12 (1780807932) PASS 3 (2s) == sanityn test 4: fstat validation on multiple mount points ==================================================================== 00:52:14 (1780807934) PASS 4 (3s) == sanityn test 5: create a file on one mount, truncate it on the other ========================================================== 00:52:17 (1780807937) /mnt/lustre/f5 has type file OK /mnt/lustre/f5 has size 100 OK PASS 5 (3s) == sanityn test 6: remove of open file on other node ============================================================================ 00:52:20 (1780807940) opening writing unlinking /mnt/lustre2/f6.sanityn accessing (1) seeking (1) accessing (2) fstat... reading comparing data truncating seeking (2) writing again seeking (3) reading again comparing data again closing SUCCESS - goto beer PASS 6 (2s) == sanityn test 7: remove of open directory on other node ======================================================================= 00:52:22 (1780807942) creating directory /mnt/lustre/d7 opening directory unlinking /mnt/lustre/d7 Ok, everything goes well. PASS 7 (3s) == sanityn test 8: remove of open special file on other node ==================================================================== 00:52:25 (1780807945) creating special file /mnt/lustre/f8.sanityn opening file unlinking /mnt/lustre/f8.sanityn Ok, everything goes well. PASS 8 (2s) == sanityn test 9a: append of file with sub-page size on multiple mounts ========================================================== 00:52:27 (1780807947) PASS 9a (3s) == sanityn test 9b: append to striped sparse file ======== 00:52:30 (1780807950) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0248564 s, 42.2 MB/s 3+0 records in 3+0 records out 3 bytes copied, 0.0137286 s, 0.2 kB/s Data read (expecting 'foo'): foo PASS 9b (2s) == sanityn test 10a: write of file with sub-page size on multiple mounts ========================================================== 00:52:32 (1780807952) 1+0 records in 1+0 records out 1 byte copied, 0.005264 s, 0.2 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00648528 s, 0.2 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0036294 s, 0.3 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00358028 s, 0.3 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00402701 s, 0.2 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00497373 s, 0.2 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00715038 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00305026 s, 0.3 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00369754 s, 0.3 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00460764 s, 0.2 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00382116 s, 0.3 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00386913 s, 0.3 kB/s PASS 10a (3s) == sanityn test 10b: write of file with sub-page size on multiple mounts ========================================================== 00:52:35 (1780807955) 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.00352897 s, 871 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0112043 s, 366 kB/s 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.000230512 s, 13.3 MB/s PASS 10b (2s) == sanityn test 11: execution of file opened for write should return error ============================================================== 00:52:37 (1780807957) striped dir -i1 -c2 -H crush2 /mnt/lustre/d11 multiop /mnt/lustre/d11/f vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.8387 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 300: /mnt/lustre2/d11/f: Text file busy PASS 11 (3s) == sanityn test 12: test lock ordering (link, stat, unlink) ========================================================== 00:52:40 (1780807960) warning: '-runtime' deprecated, use '-t runtime' instead - link 1409 (time 1780807970.97 total 10.01 last 140.78) - link 2502 (time 1780807980.97 total 20.01 last 109.24) - link 3997 (time 1780807990.97 total 30.01 last 149.50) - link 5419 (time 1780808000.98 total 40.02 last 142.14) - link 6963 (time 1780808010.98 total 50.02 last 154.32) - link 8535 (time 1780808020.98 total 60.03 last 157.17) - link 10000 (time 1780808030.42 total 69.46 last 155.31) - link 11558 (time 1780808040.42 total 79.46 last 155.74) - link 13042 (time 1780808050.42 total 89.46 last 148.40) - link 14238 (time 1780808060.42 total 99.46 last 119.58) total: 14311 link in 100.00 seconds: 143.10 ops/second using seed 2025450090 running for 100 seconds - stat 10000 (time 1780807967 ; total 6 ; last 6) - stat 20000 (time 1780807973 ; total 12 ; last 6) - stat 30000 (time 1780807979 ; total 18 ; last 6) - stat 40000 (time 1780807984 ; total 23 ; last 5) - stat 50000 (time 1780807990 ; total 29 ; last 6) - stat 60000 (time 1780807995 ; total 34 ; last 5) - stat 70000 (time 1780808001 ; total 40 ; last 6) - stat 80000 (time 1780808006 ; total 45 ; last 5) - stat 90000 (time 1780808012 ; total 51 ; last 6) - stat 96375 (time 1780808023 ; total 62 ; last 11) - stat 100000 (time 1780808028 ; total 67 ; last 5) - stat 106162 (time 1780808039 ; total 78 ; last 11) - stat 110000 (time 1780808045 ; total 84 ; last 6) - stat 116801 (time 1780808056 ; total 95 ; last 11) total: 119786 stats in 100 seconds: 1197.859985 stats/second - unlinked 0 (time 1780808011 ; total 0 ; last 0) - unlinked 10000 (time 1780808087 ; total 76 ; last 76) unlink(/mnt/lustre2/lockdir/lockfile14311) error: No such file or directory total: 14310 unlinks in 96 seconds: 149.062500 unlinks/second /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 77: kill: (22723) - No such process /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 78: kill: (22725) - No such process PASS 12 (149s) == sanityn test 13: test directory page revocation ======= 00:55:09 (1780808109) striped dir -i1 -c2 -H crush /mnt/lustre/d13 f13 f13 PASS 13 (2s) == sanityn test 14aa: execution of file open for write returns -ETXTBSY ========================================================== 00:55:11 (1780808111) striped dir -i0 -c2 -H crush2 /mnt/lustre/d14aa.sanityn multiop /mnt/lustre/d14aa.sanityn/f14aa.sanityn vOw_c TMPPIPE=/tmp/multiop_open_wait_pipe.8387 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 335: /mnt/lustre2/d14aa.sanityn/f14aa.sanityn: Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5002: 23935 Terminated $MULTIOP_PROG $FILE v$ARGS > $TMPPIPE (wd: ~) PASS 14aa (3s) == sanityn test 14ab: open(RDWR) of executing file returns -ETXTBSY ========================================================== 00:55:14 (1780808114) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d14ab.sanityn open(O_RDWR|O_CREAT): Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5002: 24527 Terminated $DIR1/$tdir/sleep 60 (wd: ~) PASS 14ab (2s) == sanityn test 14b: truncate of executing file returns -ETXTBSY ================================================================ 00:55:16 (1780808116) striped dir -i0 -c2 -H all_char /mnt/lustre/d14b.sanityn truncate: cannot truncate '/mnt/lustre2/d14b.sanityn/sleep' to length 60: Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 350: 25117 Terminated $DIR1/$tdir/sleep 60 PASS 14b (2s) == sanityn test 14c: open(O_TRUNC) of executing file return -ETXTBSY ============================================================ 00:55:18 (1780808118) striped dir -i0 -c2 -H crush2 /mnt/lustre/d14c.sanityn cp: cannot create regular file '/mnt/lustre2/d14c.sanityn/sleep': Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 362: 25710 Terminated $DIR1/$tdir/sleep 60 PASS 14c (3s) == sanityn test 14d: chmod of executing file is still possible ================================================================== 00:55:21 (1780808121) striped dir -i0 -c2 -H all_char /mnt/lustre/d14d.sanityn chmod /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 373: 26305 Terminated $DIR1/$tdir/sleep 60 PASS 14d (3s) == sanityn test 15: test out-of-space with multiple writers ===================================================================== 00:55:24 (1780808124) PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin: mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky sleep 5 for ZFS MDS Waiting for MDT destroys to complete fallocate on zfs doesn't consume space STRIPECOUNT=2 ORIGFREE=7526400 MAXFREE=800000 SKIP: oos2 test_15 oos2.sh: 7526400KB free > 800000KB, increase MAXFREE (or reduce fs size) sleep 5 for ZFS MDS Waiting for MDT destroys to complete SKIP 15 (30s) == sanityn test 16a: 500 iterations of dual-mount fsx ==== 00:55:55 (1780808155) fallocate on zfs doesn't consume space fallocate not supported 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.11794 s, 71.1 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.0827251 s, 101 MB/s Chance of close/open is 1 in 50 Seed set to 8156 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read skipping zero size read truncating to largest ever: 0x60d3c8 000007[0] 1780808156.467923 trunc from 00000000 to 0x60d3c7 (0x60d3c8 bytes) 000009[1] 1780808156.488375 mapwrite 0x8b9da9 thru 0x8c33b0 (0x9608 bytes) 000010[0] 1780808156.530199 write 0x8e6d47 thru 0x8f66fe (0xf9b8 bytes) 000013[0] 1780808156.549176 mapread 0x246343 thru 0x252a60 (0xc71e bytes) 000017[1] 1780808156.579194 write 0x1ef33e thru 0x1f9e3c (0xaaff bytes) 000019[1] 1780808156.596113 trunc from 0x8f66ff to 0x462e32 (0x4938cc bytes) 000022[0] 1780808156.611152 trunc from 0x462e33 to 0x340df2 (0x122040 bytes) 000023[1] 1780808156.631313 mapread 0x1a8e00 thru 0x1b889e (0xfa9f bytes) 000024[1] 1780808156.646107 write 0x4649cd thru 0x465b0d (0x1141 bytes) 000025[0] 1780808156.650217 mapread 0x0c2091 thru 0x0c5cc6 (0x3c36 bytes) 000029[0] 1780808156.666437 read 0x192e6b thru 0x1a285c (0xf9f2 bytes) 000030[0] 1780808156.676809 mapwrite 0x027bff thru 0x031761 (0x9b63 bytes) 000037[1] 1780808156.744709 mapwrite 0x6ff996 thru 0x70b7a7 (0xbe12 bytes) 000038[1] 1780808156.803113 write 0x4d9488 thru 0x4e0db2 (0x792b bytes) 000039[0] 1780808156.813822 read 0x666a85 thru 0x66f1cc (0x8748 bytes) 000041[0] 1780808156.826722 trunc from 0x70b7a8 to 0x59e3d4 (0x16d3d3 bytes) 000043[0] 1780808156.842427 read 0x59b0af thru 0x59e3d4 (0x3326 bytes) 000044[0] 1780808156.845651 write 0x4d71c7 thru 0x4e4c03 (0xda3d bytes) 000045[0] 1780808156.849910 mapread 0x03490e thru 0x03e14f (0x9842 bytes) 000048[1] 1780808156.865164 read 0x503cdf thru 0x50bdcf (0x80f1 bytes) 000049[0] 1780808156.878372 write 0x501cb1 thru 0x50de31 (0xc181 bytes) 000051[0] 1780808156.892273 mapread 0x18b461 thru 0x198d43 (0xd8e3 bytes) 000052[0] 1780808156.910565 trunc from 0x59e3d5 to 0x04b781 (0x552c53 bytes) 000054[0] 1780808156.918212 mapread 0x02c4f8 thru 0x02ceda (0x9e3 bytes) 000057[1] 1780808156.943685 mapwrite 0x228cd5 thru 0x233c49 (0xaf75 bytes) 000059[0] 1780808157.002221 mapread 0x0df719 thru 0x0e8bb0 (0x9498 bytes) 000060[1] 1780808157.016617 read 0x1e6169 thru 0x1e9461 (0x32f9 bytes) 000061[0] 1780808157.030023 mapread 0x12a245 thru 0x12cc91 (0x2a4d bytes) 000062[0] 1780808157.036110 read 0x0c7649 thru 0x0cb920 (0x42d8 bytes) 000065[1] 1780808157.054479 write 0x2330ff thru 0x236ec2 (0x3dc4 bytes) 000067[1] 1780808157.074457 trunc from 0x236ec3 to 0x0eeff8 (0x147eca bytes) 000070[1] 1780808157.092366 read 0x00e8ae thru 0x014046 (0x5799 bytes) 000071[0] 1780808157.104226 read 0x0e5f49 thru 0x0ee110 (0x81c8 bytes) truncating to largest ever: 0x81a4e3 000072[0] 1780808157.119383 trunc from 0x0eeff9 to 0x81a4e2 (0x72b4ea bytes) 000073[1] 1780808157.133253 mapread 0x40455b thru 0x411f48 (0xd9ee bytes) 000074[0] 1780808157.164516 write 0x873300 thru 0x875bd4 (0x28d5 bytes) 000075[1] 1780808157.172313 mapread 0x2e24e4 thru 0x2e9762 (0x727f bytes) 000076[0] 1780808157.198277 trunc from 0x875bd5 to 0x081eb8 (0x7f3d1c bytes) 000080[1] 1780808157.228475 read 0x033a2f thru 0x041f7a (0xe54c bytes) 000081[0] 1780808157.240344 mapwrite 0x2d6746 thru 0x2e1ea4 (0xb75f bytes) 000082[0] 1780808157.282552 mapread 0x22ecf6 thru 0x23acf4 (0xbfff bytes) 000083[0] 1780808157.285006 mapwrite 0x929042 thru 0x932fd5 (0x9f94 bytes) 000085[1] 1780808157.345620 mapread 0x159c14 thru 0x15a734 (0xb21 bytes) 000087[0] 1780808157.373657 mapread 0x2f6a64 thru 0x301d9a (0xb337 bytes) 000088[1] 1780808157.409317 mapwrite 0x814698 thru 0x818701 (0x406a bytes) 000089[1] 1780808157.454572 trunc from 0x932fd6 to 0x5284af (0x40ab26 bytes) 000091[0] 1780808157.486010 mapread 0x139a8d thru 0x13ed2e (0x52a2 bytes) 000095[1] 1780808157.521927 mapread 0x029bf1 thru 0x038dbe (0xf1ce bytes) 000096[0] 1780808157.552335 write 0x40c6f3 thru 0x414175 (0x7a83 bytes) 000097[1] 1780808157.562490 mapread 0x31cf0e thru 0x328049 (0xb13c bytes) 000100[1] 1780808157.605920 read 0x08cdc7 thru 0x08fbbc (0x2df6 bytes) 000102[1] 1780808157.609206 write 0x4f63d5 thru 0x4fd5c2 (0x71ee bytes) 000103[1] 1780808157.620549 mapread 0x3ce4f9 thru 0x3cee25 (0x92d bytes) 000104[1] 1780808157.628676 mapwrite 0x947c9f thru 0x956347 (0xe6a9 bytes) 000105[1] 1780808157.671771 read 0x411537 thru 0x4193fa (0x7ec4 bytes) 000106[1] 1780808157.675333 mapwrite 0x550682 thru 0x5535d3 (0x2f52 bytes) 000108[1] 1780808157.704923 mapread 0x71b99b thru 0x726bae (0xb214 bytes) 000111[0] 1780808157.739396 read 0x171f18 thru 0x17456a (0x2653 bytes) 000114[0] 1780808157.757134 write 0x0826b8 thru 0x085a03 (0x334c bytes) 000115[0] 1780808157.773562 mapread 0x2462ef thru 0x2497c0 (0x34d2 bytes) 000116[0] 1780808157.794364 mapread 0x253dae thru 0x256ff4 (0x3247 bytes) 000118[0] 1780808157.810250 trunc from 0x956348 to 0x68fe3e (0x2c6509 bytes) 000119[0] 1780808157.834184 trunc from 0x68fe3f to 0x649ec3 (0x45f7b bytes) 000121[0] 1780808157.852970 read 0x4ee112 thru 0x4ee3e4 (0x2d3 bytes) 000123[0] 1780808157.863430 mapread 0x397edc thru 0x39953c (0x1661 bytes) 000124[1] 1780808157.871561 mapread 0x511f5c thru 0x51b9fd (0x9aa2 bytes) 000125[1] 1780808157.890125 mapwrite 0x2e4ea8 thru 0x2ee640 (0x9799 bytes) 000127[1] 1780808157.929400 mapwrite 0x83cefd thru 0x84cda2 (0xfea6 bytes) 000129[0] 1780808157.978220 trunc from 0x84cda3 to 0x36e35d (0x4dea45 bytes) 000133[0] 1780808158.013725 read 0x0978ac thru 0x0a6835 (0xef8a bytes) 000134[1] 1780808158.033277 write 0x325dc2 thru 0x326472 (0x6b1 bytes) truncating to largest ever: 0x9c24d7 000135[1] 1780808158.049382 trunc from 0x36e35e to 0x9c24d6 (0x654179 bytes) 000137[0] 1780808158.072760 write 0x68d713 thru 0x69a78e (0xd07c bytes) 000139[1] 1780808158.089524 mapread 0x851f7d thru 0x86005b (0xe0df bytes) 000145[1] 1780808158.127777 write 0x939d10 thru 0x94076b (0x6a5c bytes) 000147[1] 1780808158.140887 read 0x7c6c05 thru 0x7c9e13 (0x320f bytes) 000148[1] 1780808158.149964 write 0x7f5a0b thru 0x7f9607 (0x3bfd bytes) 000152[1] 1780808158.167156 write 0x5100a8 thru 0x517757 (0x76b0 bytes) 000153[1] 1780808158.171760 mapread 0x7117c1 thru 0x712f5e (0x179e bytes) 000155[0] 1780808158.177189 mapwrite 0x8a6404 thru 0x8adee8 (0x7ae5 bytes) 000159[0] 1780808158.253597 mapwrite 0x16f784 thru 0x1786ed (0x8f6a bytes) 000160[0] 1780808158.306821 mapwrite 0x8ff5b6 thru 0x903346 (0x3d91 bytes) 000161[0] 1780808158.342362 read 0x91582d thru 0x9214f8 (0xbccc bytes) 000163[1] 1780808158.364271 mapread 0x47f94b thru 0x48b5f8 (0xbcae bytes) 000164[1] 1780808158.388923 trunc from 0x9c24d7 to 0x90a5f4 (0xb7ee2 bytes) 000165[1] 1780808158.405426 mapread 0x5eed31 thru 0x5eef4d (0x21d bytes) 000166[0] 1780808158.413481 mapread 0x5ed739 thru 0x5f6d0b (0x95d3 bytes) 000168[0] 1780808158.436613 write 0x61910a thru 0x628727 (0xf61e bytes) 000169[0] 1780808158.450920 write 0x8ea7e5 thru 0x8ebf70 (0x178c bytes) 000170[1] 1780808158.459577 trunc from 0x90a5f5 to 0x14a3d3 (0x7c0221 bytes) 000171[1] 1780808158.479609 read 0x02cd67 thru 0x032bc6 (0x5e60 bytes) 000172[0] 1780808158.488508 mapread 0x071cb8 thru 0x078ddc (0x7125 bytes) 000175[0] 1780808158.515316 read 0x1434b5 thru 0x14a3d3 (0x6f1f bytes) 000176[1] 1780808158.531610 mapwrite 0x0df9c9 thru 0x0edce1 (0xe319 bytes) 000178[1] 1780808158.615453 mapwrite 0x58db9b thru 0x59ae9a (0xd300 bytes) 000179[0] 1780808158.676709 mapwrite 0x0e8c83 thru 0x0f2aeb (0x9e69 bytes) 000180[0] 1780808158.762445 trunc from 0x59ae9b to 0x086532 (0x514968 bytes) 000181[1] 1780808158.779725 read 0x052c45 thru 0x05c5c8 (0x9984 bytes) 000182[0] 1780808158.795512 mapread 0x005e7b thru 0x00c515 (0x669b bytes) 000183[0] 1780808158.822619 write 0x007073 thru 0x0071c4 (0x152 bytes) 000184[1] 1780808158.841564 read 0x001afb thru 0x008836 (0x6d3c bytes) 000191[1] 1780808158.888825 mapwrite 0x7d9546 thru 0x7da81b (0x12d6 bytes) 000192[1] 1780808158.965997 read 0x5e9747 thru 0x5e9f38 (0x7f2 bytes) 000193[0] 1780808158.975684 mapread 0x19280f thru 0x1a099b (0xe18d bytes) 000194[0] 1780808159.024646 read 0x73a41b thru 0x73f5a0 (0x5186 bytes) 000198[1] 1780808159.043189 read 0x270fe0 thru 0x27bcc0 (0xace1 bytes) 000199[0] 1780808159.047681 write 0x003f3a thru 0x008bd2 (0x4c99 bytes) 000200[1] 1780808159.057811 read 0x4c5c04 thru 0x4ce310 (0x870d bytes) 000201[0] 1780808159.061162 mapwrite 0x718a81 thru 0x71a82f (0x1daf bytes) 000202[1] 1780808159.093655 trunc from 0x7da81c to 0x7e5170 (0xa955 bytes) 000203[0] 1780808159.110175 write 0x2ab1cc thru 0x2b3d82 (0x8bb7 bytes) 000205[1] 1780808159.128541 write 0x714492 thru 0x716a52 (0x25c1 bytes) 000206[1] 1780808159.136507 mapread 0x25a109 thru 0x261a60 (0x7958 bytes) 000209[0] 1780808159.152790 mapwrite 0x20426e thru 0x20d49f (0x9232 bytes) 000212[1] 1780808159.211233 mapread 0x1a76a9 thru 0x1b4e22 (0xd77a bytes) 000215[0] 1780808159.245480 mapwrite 0x9a8971 thru 0x9a9c41 (0x12d1 bytes) 000217[1] 1780808159.306303 read 0x0cca20 thru 0x0d7112 (0xa6f3 bytes) 000218[1] 1780808159.314542 mapwrite 0x42ce0c thru 0x42dff7 (0x11ec bytes) 000219[0] 1780808159.349425 mapwrite 0x82935f thru 0x83307d (0x9d1f bytes) 000222[0] 1780808159.400175 mapread 0x2eee3f thru 0x2fa89f (0xba61 bytes) 000223[0] 1780808159.418890 mapread 0x51c5bc thru 0x528be2 (0xc627 bytes) 000224[1] 1780808159.439732 mapread 0x30139f thru 0x30c645 (0xb2a7 bytes) 000228[1] 1780808159.458187 mapread 0x815ced thru 0x818bbc (0x2ed0 bytes) 000231[0] 1780808159.465309 mapread 0x46f784 thru 0x477151 (0x79ce bytes) 000234[1] 1780808159.469635 mapread 0x55ae8b thru 0x5632ad (0x8423 bytes) 000237[1] 1780808159.477007 mapread 0x5b86a1 thru 0x5b981d (0x117d bytes) 000241[0] 1780808159.481521 read 0x4fd109 thru 0x509a9d (0xc995 bytes) 000243[0] 1780808159.485162 mapread 0x927567 thru 0x935a11 (0xe4ab bytes) 000245[1] 1780808159.494844 read 0x23d0d4 thru 0x23db4c (0xa79 bytes) 000248[0] 1780808159.497959 write 0x256803 thru 0x260d98 (0xa596 bytes) 000249[1] 1780808159.506215 write 0x85a629 thru 0x85d0ad (0x2a85 bytes) 000250[1] 1780808159.517699 trunc from 0x9a9c42 to 0x120493 (0x8897ae bytes) 000253[0] 1780808159.530766 trunc from 0x120494 to 0x13b0dd (0x1ac4a bytes) 000254[1] 1780808159.542752 read 0x027adc thru 0x0330f5 (0xb61a bytes) 000255[1] 1780808159.557263 write 0x7be587 thru 0x7c6345 (0x7dbf bytes) 000256[1] 1780808159.564849 write 0x21c355 thru 0x2202e2 (0x3f8e bytes) 000257[0] 1780808159.576535 read 0x211697 thru 0x21af84 (0x98ee bytes) 000259[1] 1780808159.607970 mapread 0x26f228 thru 0x279fbc (0xad95 bytes) 000269[1] 1780808159.692863 mapread 0x506525 thru 0x50b4bf (0x4f9b bytes) 000272[0] 1780808159.714766 mapwrite 0x63e2a2 thru 0x6497cd (0xb52c bytes) 000276[1] 1780808159.788594 trunc from 0x7c6346 to 0x12e50f (0x697e36 bytes) 000277[0] 1780808159.811275 trunc from 0x12e510 to 0x15a430 (0x2bf21 bytes) 000279[1] 1780808159.823954 trunc from 0x15a431 to 0x3a20a5 (0x247c75 bytes) 000284[0] 1780808159.865036 mapwrite 0x8aa0b7 thru 0x8adcb3 (0x3bfd bytes) 000285[0] 1780808159.924359 write 0x4fa71f thru 0x4ff35f (0x4c41 bytes) 000288[1] 1780808159.933855 mapwrite 0x6cd728 thru 0x6d620a (0x8ae3 bytes) 000291[0] 1780808160.023790 read 0x39e42c thru 0x3a9241 (0xae16 bytes) 000294[1] 1780808160.045698 read 0x4b2661 thru 0x4b357a (0xf1a bytes) 000295[0] 1780808160.055849 write 0x7812bc thru 0x7906ba (0xf3ff bytes) 000297[0] 1780808160.066626 trunc from 0x8adcb4 to 0x621c7b (0x28c038 bytes) 000298[1] 1780808160.082900 read 0x56c600 thru 0x575fa2 (0x99a3 bytes) 000301[1] 1780808160.103770 trunc from 0x621c7c to 0x592c4d (0x8f02e bytes) 000304[1] 1780808160.127743 mapwrite 0x96eaa7 thru 0x973d2e (0x5288 bytes) 000306[1] 1780808160.169354 write 0x06bc30 thru 0x073648 (0x7a19 bytes) 000309[0] 1780808160.187968 mapread 0x63599e thru 0x642cdc (0xd33f bytes) 000311[1] 1780808160.212064 write 0x21dbc6 thru 0x229c84 (0xc0bf bytes) 000313[0] 1780808160.225835 mapread 0x282347 thru 0x28ccb0 (0xa96a bytes) 000314[1] 1780808160.240900 mapwrite 0x857a2b thru 0x859508 (0x1ade bytes) 000315[1] 1780808160.273429 read 0x546407 thru 0x5476e4 (0x12de bytes) 000316[1] 1780808160.276975 read 0x22d973 thru 0x23b961 (0xdfef bytes) 000324[1] 1780808160.315355 read 0x31c791 thru 0x31eeb6 (0x2726 bytes) 000325[0] 1780808160.318941 mapread 0x03ef1d thru 0x04ba6b (0xcb4f bytes) 000326[1] 1780808160.345074 read 0x5a563a thru 0x5a840f (0x2dd6 bytes) 000328[1] 1780808160.348479 read 0x6fa85a thru 0x706916 (0xc0bd bytes) 000329[1] 1780808160.352634 read 0x5bcf8e thru 0x5c0594 (0x3607 bytes) 000330[1] 1780808160.356699 write 0x9b30eb thru 0x9b3d4f (0xc65 bytes) 000332[1] 1780808160.374340 read 0x4aeebe thru 0x4b64be (0x7601 bytes) 000335[1] 1780808160.384919 trunc from 0x9b3d50 to 0x3278f1 (0x68c45e bytes) 000336[1] 1780808160.410404 mapwrite 0x098be6 thru 0x09f588 (0x69a3 bytes) 000337[0] 1780808160.449553 write 0x806fc5 thru 0x8161f7 (0xf233 bytes) 000338[1] 1780808160.463486 trunc from 0x8161f8 to 0x243263 (0x5d2f94 bytes) 000341[0] 1780808160.495385 trunc from 0x243264 to 0x7b9f62 (0x576cff bytes) 000345[0] 1780808160.533810 write 0x67d3e2 thru 0x68af41 (0xdb60 bytes) 000346[0] 1780808160.538976 write 0x937aff thru 0x93c4c1 (0x49c3 bytes) 000347[1] 1780808160.548960 read 0x369011 thru 0x36da22 (0x4a12 bytes) 000350[0] 1780808160.580384 read 0x091426 thru 0x09f817 (0xe3f2 bytes) 000354[1] 1780808160.608262 read 0x87bb34 thru 0x881e0d (0x62da bytes) 000357[1] 1780808160.628130 mapwrite 0x5aec4d thru 0x5bc434 (0xd7e8 bytes) 000358[1] 1780808160.668678 mapread 0x246f1d thru 0x24b341 (0x4425 bytes) 000359[0] 1780808160.681269 mapread 0x530b72 thru 0x5315e6 (0xa75 bytes) 000360[1] 1780808160.695070 mapread 0x38320e thru 0x38b5f3 (0x83e6 bytes) 000361[1] 1780808160.705438 mapread 0x8bb4a5 thru 0x8c4950 (0x94ac bytes) 000364[1] 1780808160.717050 read 0x10d93e thru 0x10db27 (0x1ea bytes) 000365[1] 1780808160.721326 write 0x545542 thru 0x54e55a (0x9019 bytes) 000367[0] 1780808160.749181 mapread 0x2a3b1e thru 0x2a57be (0x1ca1 bytes) 000368[0] 1780808160.770702 mapread 0x2aed61 thru 0x2b8b01 (0x9da1 bytes) 000370[1] 1780808160.794464 write 0x2e799c thru 0x2f0fbf (0x9624 bytes) 000374[0] 1780808160.809518 mapread 0x8e4bdc thru 0x8f13ed (0xc812 bytes) 000377[0] 1780808160.829445 write 0x6d3abb thru 0x6dead0 (0xb016 bytes) 000378[0] 1780808160.843460 mapwrite 0x882495 thru 0x88476c (0x22d8 bytes) 000380[1] 1780808160.882353 mapread 0x39d250 thru 0x3a159c (0x434d bytes) 000381[0] 1780808160.900540 mapread 0x152c32 thru 0x1593cd (0x679c bytes) 000382[0] 1780808160.914802 mapwrite 0x70d3fe thru 0x7134eb (0x60ee bytes) 000384[0] 1780808160.958600 write 0x672550 thru 0x68137c (0xee2d bytes) 000385[1] 1780808160.968230 read 0x769d94 thru 0x772457 (0x86c4 bytes) 000386[1] 1780808160.981708 mapwrite 0x19077b thru 0x194a74 (0x42fa bytes) 000387[0] 1780808161.013400 trunc from 0x93c4c2 to 0x4b1c80 (0x48a841 bytes) 000390[1] 1780808161.040627 read 0x263225 thru 0x269d60 (0x6b3c bytes) 000392[0] 1780808161.048539 read 0x494d8a thru 0x495aee (0xd65 bytes) 000393[1] 1780808161.050901 trunc from 0x4b1c81 to 0x3939e7 (0x11e299 bytes) 000394[0] 1780808161.064027 read 0x14bc9e thru 0x1523e9 (0x674c bytes) 000395[0] 1780808161.074708 mapread 0x31af2e thru 0x321c80 (0x6d53 bytes) 000396[0] 1780808161.096477 mapread 0x09a73b thru 0x0a527f (0xab45 bytes) 000397[1] 1780808161.112695 mapwrite 0x945713 thru 0x94a8f3 (0x51e1 bytes) 000399[1] 1780808161.158481 mapwrite 0x359aa6 thru 0x369051 (0xf5ac bytes) 000400[1] 1780808161.185415 mapwrite 0x3f136a thru 0x3fcfc0 (0xbc57 bytes) 000403[1] 1780808161.214230 trunc from 0x94a8f4 to 0x544830 (0x4060c3 bytes) 000404[0] 1780808161.221030 mapwrite 0x3c6dad thru 0x3d0a01 (0x9c55 bytes) 000405[0] 1780808161.258666 trunc from 0x544831 to 0x1d90af (0x36b781 bytes) 000406[0] 1780808161.276781 write 0x75c003 thru 0x75e7b9 (0x27b7 bytes) 000414[1] 1780808161.306959 write 0x6c19f2 thru 0x6d044e (0xea5d bytes) 000415[1] 1780808161.322362 trunc from 0x75e7ba to 0x6d860c (0x861ad bytes) 000416[0] 1780808161.331905 trunc from 0x6d860d to 0x8b3738 (0x1db12c bytes) 000419[1] 1780808161.353619 write 0x972ad2 thru 0x978f5f (0x648e bytes) 000425[1] 1780808161.386737 read 0x350000 thru 0x352461 (0x2462 bytes) 000428[0] 1780808161.408244 write 0x03cb05 thru 0x048902 (0xbdfe bytes) 000431[0] 1780808161.430852 read 0x764ba6 thru 0x773e20 (0xf27b bytes) 000432[0] 1780808161.439196 write 0x7afd1d thru 0x7bb37c (0xb660 bytes) 000439[0] 1780808161.457686 write 0x2c2020 thru 0x2c7619 (0x55fa bytes) 000442[1] 1780808161.471584 mapread 0x19bc49 thru 0x1a96b6 (0xda6e bytes) 000445[0] 1780808161.495954 mapwrite 0x24ba42 thru 0x252861 (0x6e20 bytes) 000446[0] 1780808161.517274 mapread 0x87316b thru 0x878295 (0x512b bytes) 000447[0] 1780808161.530782 read 0x7bf272 thru 0x7c83f8 (0x9187 bytes) 000448[1] 1780808161.539314 write 0x2e422e thru 0x2eb458 (0x722b bytes) 000450[1] 1780808161.555001 write 0x305f76 thru 0x30f9b4 (0x9a3f bytes) 000451[0] 1780808161.566769 trunc from 0x978f60 to 0x841534 (0x137a2b bytes) 000456[0] 1780808161.596165 mapwrite 0x7cd4a6 thru 0x7d3677 (0x61d2 bytes) 000457[0] 1780808161.624577 mapwrite 0x153891 thru 0x16332c (0xfa9c bytes) 000458[1] 1780808161.658392 read 0x5c1074 thru 0x5c6dd0 (0x5d5d bytes) 000461[0] 1780808161.670064 mapwrite 0x06f9f6 thru 0x076c61 (0x726c bytes) 000463[1] 1780808161.703580 trunc from 0x841535 to 0x632d9f (0x20e795 bytes) 000466[0] 1780808161.736577 trunc from 0x632da0 to 0x32438e (0x30ea11 bytes) 000469[0] 1780808161.755838 mapread 0x09daa0 thru 0x0a64e1 (0x8a42 bytes) 000474[0] 1780808161.779432 read 0x1c8001 thru 0x1d38d3 (0xb8d3 bytes) 000477[1] 1780808161.800239 read 0x0cabba thru 0x0d26e4 (0x7b2b bytes) 000479[0] 1780808161.831385 mapwrite 0x6681a6 thru 0x66b365 (0x31c0 bytes) 000480[1] 1780808161.877913 trunc from 0x66b366 to 0x116f73 (0x5543f2 bytes) 000483[0] 1780808161.891798 write 0x2bcfad thru 0x2c38df (0x6933 bytes) 000485[0] 1780808161.903352 trunc from 0x2c38e0 to 0x3074af (0x43bd0 bytes) 000489[0] 1780808161.927220 write 0x7161b6 thru 0x71d03b (0x6e86 bytes) 000490[1] 1780808161.929380 write 0x1d3348 thru 0x1d3e24 (0xadd bytes) 000492[0] 1780808161.951366 trunc from 0x71d03c to 0x6d4768 (0x488d3 bytes) 000496[1] 1780808161.998039 mapread 0x490191 thru 0x4902cb (0x13b bytes) 000497[0] 1780808162.011368 write 0x8e925b thru 0x8ef85b (0x6601 bytes) 000500[1] 1780808162.020610 read 0x2387d1 thru 0x239656 (0xe86 bytes) All operations completed A-OK! Chance of close/open is 1 in 50 Seed set to 8162 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported truncating to largest ever: 0x263ced 000001[0] 1780808162.353292 trunc from 00000000 to 0x263cec (0x263ced bytes) 000002[1] 1780808162.380781 mapread 0x10f000 thru 0x119201 (0xa202 bytes) 000005[1] 1780808162.411551 mapread 0x188000 thru 0x1918a4 (0x98a5 bytes) 000007[0] 1780808162.423508 read_OD 0x082000 thru 0x084fff (0x3000 bytes) 000008[1] 1780808162.430457 write_OD 0x792000 thru 0x798fff (0x7000 bytes) 000009[0] 1780808162.466583 mapwrite 0x415000 thru 0x41c45c (0x745d bytes) 000010[0] 1780808162.517375 mapwrite 0x763000 thru 0x764701 (0x1702 bytes) 000011[0] 1780808162.542466 mapread 0x152000 thru 0x154410 (0x2411 bytes) 000012[1] 1780808162.553631 mapwrite 0x3fc000 thru 0x408d28 (0xcd29 bytes) 000017[0] 1780808162.591914 mapwrite 0x8c6000 thru 0x8ca8ca (0x48cb bytes) 000019[0] 1780808162.627359 read_OD 0x730000 thru 0x736fff (0x7000 bytes) 000020[0] 1780808162.634188 write_OD 0x80a000 thru 0x80afff (0x1000 bytes) 000021[0] 1780808162.649988 read_OD 0x471000 thru 0x47cfff (0xc000 bytes) 000027[1] 1780808162.656925 mapwrite 0x8cf000 thru 0x8d649b (0x749c bytes) 000028[0] 1780808162.702370 read_OD 0x54c000 thru 0x54ffff (0x4000 bytes) 000029[0] 1780808162.710555 write_OD 0x01a000 thru 0x01ffff (0x6000 bytes) truncating to largest ever: 0x28433e 000031[0] 1780808162.733809 trunc from 0x8d649c to 0x28433d (0x65215e bytes) 000034[1] 1780808162.758411 read_OD 0x250000 thru 0x257fff (0x8000 bytes) 000036[1] 1780808162.771364 trunc from 0x28433e to 0x178bd7 (0x10b766 bytes) truncating to largest ever: 0x77f487 000038[1] 1780808162.789652 trunc from 0x178bd8 to 0x77f486 (0x6068af bytes) 000040[0] 1780808162.806321 write_OD 0x292000 thru 0x29ffff (0xe000 bytes) 000043[1] 1780808162.829057 mapwrite 0x6f5000 thru 0x6f77e0 (0x27e1 bytes) 000044[0] 1780808162.860281 write 0x781000 thru 0x782a51 (0x1a52 bytes) 000050[1] 1780808162.890233 read_OD 0x372000 thru 0x375fff (0x4000 bytes) 000051[1] 1780808162.902997 read_OD 0x3b1000 thru 0x3b7fff (0x7000 bytes) 000052[1] 1780808162.908893 write_OD 0x9cc000 thru 0x9d2fff (0x7000 bytes) 000053[1] 1780808162.929020 mapread 0x8cd000 thru 0x8d284e (0x584f bytes) truncating to largest ever: 0x8e764f 000055[0] 1780808162.937718 trunc from 0x9d3000 to 0x8e764e (0xeb9b1 bytes) 000056[0] 1780808162.948961 mapwrite 0x0d8000 thru 0x0e5e2f (0xde30 bytes) 000058[0] 1780808162.978423 read 0x704000 thru 0x70d196 (0x9197 bytes) 000060[0] 1780808162.989232 read 0x2bc000 thru 0x2c6375 (0xa376 bytes) 000063[1] 1780808162.996486 write_OD 0x569000 thru 0x575fff (0xd000 bytes) 000064[0] 1780808163.014872 write 0x6da000 thru 0x6e6702 (0xc703 bytes) 000065[1] 1780808163.026730 mapwrite 0x6ec000 thru 0x6f0ffd (0x4ffe bytes) 000066[1] 1780808163.064325 read_OD 0x190000 thru 0x194fff (0x5000 bytes) 000068[1] 1780808163.070279 read_OD 0x8e5000 thru 0x8e6fff (0x2000 bytes) 000070[0] 1780808163.073495 mapwrite 0x5eb000 thru 0x5eb01c (0x1d bytes) 000071[1] 1780808163.093482 read_OD 0x0fc000 thru 0x106fff (0xb000 bytes) 000073[1] 1780808163.102640 trunc from 0x8e764f to 0x7decf2 (0x10895c bytes) 000074[1] 1780808163.116843 read_OD 0x09c000 thru 0x0a8fff (0xd000 bytes) 000076[1] 1780808163.127204 read_OD 0x377000 thru 0x382fff (0xc000 bytes) 000078[0] 1780808163.133898 read 0x28e000 thru 0x294aab (0x6aac bytes) 000079[0] 1780808163.137436 write 0x406000 thru 0x40d880 (0x7881 bytes) 000081[1] 1780808163.143419 write_OD 0x914000 thru 0x921fff (0xe000 bytes) 000082[0] 1780808163.169226 write 0x486000 thru 0x48877b (0x277c bytes) 000084[0] 1780808163.184914 mapwrite 0x83b000 thru 0x83bf41 (0xf42 bytes) 000085[1] 1780808163.217337 trunc from 0x922000 to 0x0981f0 (0x889e0f bytes) 000086[0] 1780808163.238392 trunc from 0x0981f1 to 0x13b280 (0xa3090 bytes) 000087[0] 1780808163.249093 mapwrite 0x4b2000 thru 0x4b6426 (0x4427 bytes) 000088[0] 1780808163.275677 mapread 0x10b000 thru 0x10ca32 (0x1a33 bytes) 000089[0] 1780808163.279247 mapwrite 0x10c000 thru 0x1162e2 (0xa2e3 bytes) truncating to largest ever: 0x97132d 000091[0] 1780808163.308198 trunc from 0x4b6427 to 0x97132c (0x4baf06 bytes) 000094[0] 1780808163.339477 mapread 0x7ce000 thru 0x7d245a (0x445b bytes) 000095[1] 1780808163.355637 write_OD 0x1fa000 thru 0x1fdfff (0x4000 bytes) 000097[1] 1780808163.375618 write_OD 0x2d4000 thru 0x2d4fff (0x1000 bytes) 000098[1] 1780808163.393334 write_OD 0x302000 thru 0x30bfff (0xa000 bytes) 000102[0] 1780808163.416999 read 0x86b000 thru 0x874c2a (0x9c2b bytes) 000103[0] 1780808163.421582 mapwrite 0x11f000 thru 0x124fe4 (0x5fe5 bytes) 000104[1] 1780808163.456842 mapwrite 0x0be000 thru 0x0c8340 (0xa341 bytes) 000107[0] 1780808163.506620 write 0x5df000 thru 0x5e4294 (0x5295 bytes) 000110[0] 1780808163.521348 mapread 0x1bf000 thru 0x1c9316 (0xa317 bytes) 000113[1] 1780808163.541998 trunc from 0x97132d to 0x0b1e16 (0x8bf516 bytes) 000115[0] 1780808163.554920 trunc from 0x0b1e17 to 0x2e4d81 (0x232f6b bytes) 000116[0] 1780808163.572372 write 0x01d000 thru 0x02a5dc (0xd5dd bytes) 000119[1] 1780808163.587363 trunc from 0x2e4d82 to 0x7f0540 (0x50b7bf bytes) 000120[1] 1780808163.606832 write_OD 0x467000 thru 0x46cfff (0x6000 bytes) 000121[1] 1780808163.628362 mapwrite 0x85e000 thru 0x86a9e5 (0xc9e6 bytes) 000122[0] 1780808163.666189 trunc from 0x86a9e6 to 0x6fafcb (0x16fa1a bytes) 000124[0] 1780808163.679755 write 0x78e000 thru 0x797603 (0x9604 bytes) 000125[1] 1780808163.685500 mapwrite 0x7c0000 thru 0x7cc980 (0xc981 bytes) 000127[1] 1780808163.721425 mapread 0x25c000 thru 0x260ad3 (0x4ad4 bytes) 000129[1] 1780808163.730212 trunc from 0x7cc981 to 0x5282d4 (0x2a46ac bytes) 000130[0] 1780808163.738493 read 0x0c7000 thru 0x0ca86f (0x3870 bytes) 000135[1] 1780808163.748475 mapread 0x3b8000 thru 0x3c2e85 (0xae86 bytes) 000138[1] 1780808163.755986 mapread 0x1df000 thru 0x1e67ed (0x77ee bytes) 000139[0] 1780808163.763475 mapwrite 0x387000 thru 0x38c596 (0x5597 bytes) 000141[1] 1780808163.802245 mapread 0x42b000 thru 0x437482 (0xc483 bytes) 000143[1] 1780808163.821066 mapread 0x181000 thru 0x182f19 (0x1f1a bytes) 000144[1] 1780808163.825518 trunc from 0x5282d5 to 0x73e694 (0x2163c0 bytes) 000146[1] 1780808163.840599 read_OD 0x6c2000 thru 0x6c9fff (0x8000 bytes) 000148[0] 1780808163.849974 trunc from 0x73e695 to 0x842de4 (0x104750 bytes) 000149[1] 1780808163.873444 write_OD 0x682000 thru 0x685fff (0x4000 bytes) 000150[0] 1780808163.895333 write 0x74f000 thru 0x759453 (0xa454 bytes) 000151[1] 1780808163.902306 trunc from 0x842de5 to 0x6ca409 (0x1789db bytes) 000153[0] 1780808163.924754 mapread 0x5dc000 thru 0x5e2d03 (0x6d04 bytes) 000154[0] 1780808163.937193 read 0x1dd000 thru 0x1eb661 (0xe662 bytes) 000156[1] 1780808163.941147 read_OD 0x07c000 thru 0x07cfff (0x1000 bytes) 000160[1] 1780808163.946721 mapread 0x31e000 thru 0x31e04c (0x4d bytes) 000161[1] 1780808163.949349 read_OD 0x644000 thru 0x64cfff (0x9000 bytes) 000163[1] 1780808163.954022 mapwrite 0x517000 thru 0x51b34b (0x434c bytes) 000165[0] 1780808163.992294 read 0x109000 thru 0x10d9b9 (0x49ba bytes) 000166[0] 1780808164.000174 write 0x524000 thru 0x526d80 (0x2d81 bytes) 000171[1] 1780808164.035977 mapread 0x612000 thru 0x614108 (0x2109 bytes) 000172[1] 1780808164.051997 mapwrite 0x451000 thru 0x45dad7 (0xcad8 bytes) 000176[0] 1780808164.093820 mapread 0x104000 thru 0x104a25 (0xa26 bytes) 000177[0] 1780808164.101950 write 0x10f000 thru 0x1131bf (0x41c0 bytes) 000178[0] 1780808164.114274 trunc from 0x6ca40a to 0x112a0d (0x5b79fc bytes) 000179[0] 1780808164.122930 write 0x47c000 thru 0x48765b (0xb65c bytes) 000181[0] 1780808164.136327 write 0x87b000 thru 0x8876f3 (0xc6f4 bytes) 000184[1] 1780808164.139412 mapread 0x1ea000 thru 0x1f78b8 (0xd8b9 bytes) 000185[1] 1780808164.166589 mapread 0x20d000 thru 0x21507f (0x8080 bytes) 000186[0] 1780808164.173923 mapwrite 0x288000 thru 0x28d46e (0x546f bytes) 000187[0] 1780808164.225221 write 0x209000 thru 0x212ed3 (0x9ed4 bytes) 000188[1] 1780808164.231356 write_OD 0x278000 thru 0x27dfff (0x6000 bytes) 000189[1] 1780808164.261904 read_OD 0x228000 thru 0x22efff (0x7000 bytes) 000190[1] 1780808164.267466 trunc from 0x8876f4 to 0x06584f (0x821ea4 bytes) 000193[1] 1780808164.289755 write_OD 0x813000 thru 0x814fff (0x2000 bytes) 000196[1] 1780808164.312481 read_OD 0x302000 thru 0x306fff (0x5000 bytes) 000197[0] 1780808164.316990 write 0x74b000 thru 0x7515b5 (0x65b6 bytes) 000200[0] 1780808164.331882 mapwrite 0x0e9000 thru 0x0edc92 (0x4c93 bytes) 000201[0] 1780808164.359280 mapwrite 0x2c5000 thru 0x2cf333 (0xa334 bytes) 000205[0] 1780808164.391578 mapread 0x232000 thru 0x241096 (0xf097 bytes) 000206[0] 1780808164.399457 trunc from 0x815000 to 0x85cb9e (0x47b9f bytes) 000208[1] 1780808164.410508 mapwrite 0x451000 thru 0x45193f (0x940 bytes) 000210[0] 1780808164.449681 trunc from 0x85cb9f to 0x3243ec (0x5387b2 bytes) 000212[1] 1780808164.466721 trunc from 0x3243ed to 0x563805 (0x23f419 bytes) 000213[1] 1780808164.479873 mapread 0x3a7000 thru 0x3b0e24 (0x9e25 bytes) 000215[1] 1780808164.495615 write_OD 0x31a000 thru 0x31ffff (0x6000 bytes) 000216[1] 1780808164.519119 read_OD 0x422000 thru 0x426fff (0x5000 bytes) 000217[1] 1780808164.524988 mapread 0x497000 thru 0x4a4634 (0xd635 bytes) 000220[1] 1780808164.531329 write_OD 0x8c2000 thru 0x8cbfff (0xa000 bytes) 000221[0] 1780808164.551148 read 0x81a000 thru 0x827037 (0xd038 bytes) 000222[1] 1780808164.559617 mapread 0x319000 thru 0x31a065 (0x1066 bytes) 000224[0] 1780808164.564680 mapread 0x0f4000 thru 0x0f8150 (0x4151 bytes) 000225[1] 1780808164.585886 mapread 0x71d000 thru 0x72438f (0x7390 bytes) 000227[1] 1780808164.597397 write_OD 0x3e7000 thru 0x3e9fff (0x3000 bytes) 000228[1] 1780808164.621396 trunc from 0x8cc000 to 0x9648fa (0x988fb bytes) 000229[0] 1780808164.633166 read 0x185000 thru 0x18af69 (0x5f6a bytes) 000230[0] 1780808164.645448 read 0x1d2000 thru 0x1d5dd1 (0x3dd2 bytes) truncating to largest ever: 0x9dd609 000231[0] 1780808164.650397 trunc from 0x9648fb to 0x9dd608 (0x78d0e bytes) 000232[1] 1780808164.676810 trunc from 0x9dd609 to 0x9028c6 (0xdad42 bytes) 000234[0] 1780808164.694687 read 0x2db000 thru 0x2deb01 (0x3b02 bytes) 000236[1] 1780808164.702211 write_OD 0x95e000 thru 0x96bfff (0xe000 bytes) 000237[1] 1780808164.720169 mapread 0x86e000 thru 0x87ae89 (0xce8a bytes) 000239[0] 1780808164.732235 read 0x765000 thru 0x76e9a4 (0x99a5 bytes) 000240[0] 1780808164.735430 write 0x987000 thru 0x991fc2 (0xafc3 bytes) 000242[1] 1780808164.745420 mapwrite 0x905000 thru 0x90eeeb (0x9eec bytes) 000244[0] 1780808164.790516 write 0x9eb000 thru 0x9ecc12 (0x1c13 bytes) 000246[1] 1780808164.807934 write_OD 0x6b5000 thru 0x6b8fff (0x4000 bytes) 000247[0] 1780808164.828450 mapread 0x089000 thru 0x091af9 (0x8afa bytes) 000248[1] 1780808164.835994 mapread 0x68b000 thru 0x690bf9 (0x5bfa bytes) 000251[1] 1780808164.848032 write_OD 0x3f5000 thru 0x3fefff (0xa000 bytes) 000253[1] 1780808164.867502 read_OD 0x287000 thru 0x292fff (0xc000 bytes) 000255[1] 1780808164.871270 read_OD 0x3a0000 thru 0x3a1fff (0x2000 bytes) 000256[1] 1780808164.873515 trunc from 0x9ecc13 to 0x3d3ec7 (0x618d4b bytes) 000258[1] 1780808164.893387 mapwrite 0x6bd000 thru 0x6c5da1 (0x8da2 bytes) 000259[0] 1780808164.935264 mapread 0x4bc000 thru 0x4c67d9 (0xa7da bytes) 000260[1] 1780808164.956108 mapread 0x1cb000 thru 0x1dae18 (0xfe19 bytes) 000261[1] 1780808164.970733 mapwrite 0x430000 thru 0x430341 (0x342 bytes) 000264[0] 1780808164.990310 read 0x226000 thru 0x22e384 (0x8385 bytes) 000265[1] 1780808164.998067 mapread 0x277000 thru 0x27f113 (0x8114 bytes) 000266[0] 1780808165.005216 mapwrite 0x8d9000 thru 0x8ddb33 (0x4b34 bytes) 000267[1] 1780808165.043758 mapread 0x1df000 thru 0x1ecb8f (0xdb90 bytes) 000269[1] 1780808165.055773 write_OD 0x536000 thru 0x539fff (0x4000 bytes) 000270[1] 1780808165.075355 mapread 0x639000 thru 0x64327b (0xa27c bytes) 000273[1] 1780808165.091570 trunc from 0x8ddb34 to 0x6d2104 (0x20ba2f bytes) 000276[1] 1780808165.122577 trunc from 0x6d2105 to 0x3a3e94 (0x32e270 bytes) 000277[1] 1780808165.132012 mapread 0x2d4000 thru 0x2d616d (0x216e bytes) 000281[1] 1780808165.157400 read_OD 0x036000 thru 0x03efff (0x9000 bytes) 000288[1] 1780808165.169640 trunc from 0x3a3e95 to 0x533ed2 (0x19003e bytes) 000290[0] 1780808165.196248 mapread 0x1c8000 thru 0x1cbf49 (0x3f4a bytes) 000291[0] 1780808165.211669 trunc from 0x533ed3 to 0x2f516e (0x23ed64 bytes) 000292[1] 1780808165.227413 mapwrite 0x64f000 thru 0x654eab (0x5eac bytes) 000295[1] 1780808165.300457 mapwrite 0x3b3000 thru 0x3c0a89 (0xda8a bytes) 000296[1] 1780808165.324040 write_OD 0x834000 thru 0x83afff (0x7000 bytes) 000298[0] 1780808165.365882 mapread 0x05e000 thru 0x05ff60 (0x1f61 bytes) 000300[0] 1780808165.372253 write 0x2d3000 thru 0x2d96d1 (0x66d2 bytes) 000304[0] 1780808165.384769 mapread 0x809000 thru 0x8101fe (0x71ff bytes) 000305[1] 1780808165.398202 read_OD 0x1c6000 thru 0x1cbfff (0x6000 bytes) 000307[0] 1780808165.411168 mapwrite 0x156000 thru 0x156052 (0x53 bytes) 000308[0] 1780808165.434870 mapwrite 0x29d000 thru 0x2a9859 (0xc85a bytes) 000309[1] 1780808165.471003 mapread 0x07a000 thru 0x084421 (0xa422 bytes) 000310[0] 1780808165.487887 read 0x2b5000 thru 0x2c105d (0xc05e bytes) 000312[1] 1780808165.492471 write_OD 0x034000 thru 0x040fff (0xd000 bytes) 000313[1] 1780808165.520343 write_OD 0x65b000 thru 0x65dfff (0x3000 bytes) 000315[1] 1780808165.541486 trunc from 0x83b000 to 0x5e94b6 (0x251b49 bytes) 000318[0] 1780808165.557441 write 0x2fe000 thru 0x30a353 (0xc354 bytes) 000320[0] 1780808165.567265 read 0x2fe000 thru 0x303abe (0x5abf bytes) 000322[0] 1780808165.575661 mapread 0x539000 thru 0x542762 (0x9763 bytes) 000323[0] 1780808165.589516 write 0x06d000 thru 0x0762e2 (0x92e3 bytes) 000324[0] 1780808165.592144 mapwrite 0x066000 thru 0x0675a8 (0x15a9 bytes) 000325[0] 1780808165.611429 read 0x3f2000 thru 0x401012 (0xf013 bytes) 000326[1] 1780808165.614735 mapwrite 0x7a8000 thru 0x7b0c34 (0x8c35 bytes) 000328[0] 1780808165.654695 write 0x77e000 thru 0x78380f (0x5810 bytes) 000332[1] 1780808165.674397 read_OD 0x15b000 thru 0x161fff (0x7000 bytes) 000334[1] 1780808165.689561 write_OD 0x53d000 thru 0x545fff (0x9000 bytes) 000336[1] 1780808165.714407 write_OD 0x863000 thru 0x865fff (0x3000 bytes) 000342[1] 1780808165.741578 write_OD 0x382000 thru 0x383fff (0x2000 bytes) 000343[1] 1780808165.777380 mapwrite 0x1a1000 thru 0x1afbac (0xebad bytes) 000345[0] 1780808165.823942 mapread 0x4ee000 thru 0x4effc6 (0x1fc7 bytes) 000349[0] 1780808165.841468 trunc from 0x866000 to 0x389b37 (0x4dc4c8 bytes) 000352[1] 1780808165.876325 mapwrite 0x681000 thru 0x68b731 (0xa732 bytes) 000353[0] 1780808165.922353 mapread 0x336000 thru 0x33fdac (0x9dad bytes) 000354[1] 1780808165.947969 mapread 0x17b000 thru 0x182a18 (0x7a19 bytes) 000356[0] 1780808165.963238 mapread 0x540000 thru 0x545a3d (0x5a3e bytes) 000357[1] 1780808165.975900 trunc from 0x68b732 to 0x4e98e0 (0x1a1e51 bytes) 000358[1] 1780808165.999635 mapwrite 0x919000 thru 0x927a2d (0xea2e bytes) 000361[1] 1780808166.071764 trunc from 0x927a2e to 0x0c7e5e (0x85fbcf bytes) 000362[0] 1780808166.100598 mapread 0x05e000 thru 0x05e59e (0x59f bytes) 000363[0] 1780808166.119546 mapwrite 0x345000 thru 0x345655 (0x656 bytes) 000364[1] 1780808166.179345 mapwrite 0x827000 thru 0x82f3ae (0x83af bytes) 000365[1] 1780808166.250389 mapwrite 0x830000 thru 0x8363a5 (0x63a6 bytes) 000366[1] 1780808166.277966 mapwrite 0x5fc000 thru 0x6077c7 (0xb7c8 bytes) 000367[1] 1780808166.311509 trunc from 0x8363a6 to 0x530567 (0x305e3e bytes) 000368[0] 1780808166.320325 read 0x043000 thru 0x04c704 (0x9705 bytes) 000369[1] 1780808166.353238 trunc from 0x530568 to 0x33d85b (0x1f2d0c bytes) 000372[0] 1780808166.379270 mapread 0x0de000 thru 0x0eadf6 (0xcdf7 bytes) 000374[0] 1780808166.412887 write 0x9b6000 thru 0x9b8ad1 (0x2ad2 bytes) 000376[1] 1780808166.425829 mapread 0x0ca000 thru 0x0d34b4 (0x94b5 bytes) 000378[0] 1780808166.446465 mapwrite 0x046000 thru 0x047f11 (0x1f12 bytes) 000380[0] 1780808166.484862 mapread 0x865000 thru 0x86e0b7 (0x90b8 bytes) 000381[1] 1780808166.521888 mapwrite 0x7a3000 thru 0x7b1f86 (0xef87 bytes) 000382[1] 1780808166.572356 write_OD 0x552000 thru 0x55efff (0xd000 bytes) 000385[0] 1780808166.602284 write 0x5fc000 thru 0x606545 (0xa546 bytes) 000387[1] 1780808166.613012 write_OD 0x0c2000 thru 0x0cafff (0x9000 bytes) 000389[0] 1780808166.634319 mapread 0x868000 thru 0x86a95b (0x295c bytes) 000392[1] 1780808166.640434 write_OD 0x2c0000 thru 0x2c2fff (0x3000 bytes) 000393[1] 1780808166.659959 mapwrite 0x2d7000 thru 0x2dbe23 (0x4e24 bytes) 000394[1] 1780808166.699380 mapwrite 0x40a000 thru 0x40b8fc (0x18fd bytes) 000395[1] 1780808166.729489 write_OD 0x89f000 thru 0x8a2fff (0x4000 bytes) 000397[1] 1780808166.751267 mapwrite 0x43a000 thru 0x43adaa (0xdab bytes) 000399[1] 1780808166.779523 mapread 0x08e000 thru 0x097b3d (0x9b3e bytes) 000400[1] 1780808166.794504 trunc from 0x9b8ad2 to 0x1e2ff6 (0x7d5adb bytes) 000401[0] 1780808166.806174 mapread 0x0fe000 thru 0x1087d8 (0xa7d9 bytes) 000402[1] 1780808166.861636 mapread 0x1b6000 thru 0x1c5b12 (0xfb13 bytes) 000403[0] 1780808166.882544 read 0x09a000 thru 0x0a9b38 (0xfb39 bytes) 000404[0] 1780808166.897803 mapwrite 0x991000 thru 0x99550c (0x450d bytes) 000406[0] 1780808166.987115 mapread 0x442000 thru 0x450351 (0xe352 bytes) 000409[1] 1780808166.988533 read_OD 0x38e000 thru 0x38ffff (0x2000 bytes) 000411[0] 1780808167.005852 read 0x8d8000 thru 0x8e69ec (0xe9ed bytes) 000412[1] 1780808167.010625 read_OD 0x794000 thru 0x798fff (0x5000 bytes) 000413[0] 1780808167.013717 mapread 0x349000 thru 0x358c03 (0xfc04 bytes) 000414[0] 1780808167.023162 trunc from 0x99550d to 0x8a3b18 (0xf19f4 bytes) 000415[1] 1780808167.041823 write_OD 0x9e8000 thru 0x9f5fff (0xe000 bytes) 000418[0] 1780808167.066033 trunc from 0x9f6000 to 0x560427 (0x495bd8 bytes) 000420[1] 1780808167.093391 read_OD 0x450000 thru 0x450fff (0x1000 bytes) 000426[1] 1780808167.107286 mapread 0x51a000 thru 0x5206a0 (0x66a1 bytes) 000427[0] 1780808167.116152 mapwrite 0x590000 thru 0x59b318 (0xb319 bytes) 000428[0] 1780808167.160403 mapread 0x3bd000 thru 0x3c3207 (0x6208 bytes) 000430[0] 1780808167.181907 write 0x1d7000 thru 0x1dd1ad (0x61ae bytes) 000432[1] 1780808167.188839 write_OD 0x186000 thru 0x192fff (0xd000 bytes) 000434[0] 1780808167.207028 read 0x034000 thru 0x038028 (0x4029 bytes) 000436[1] 1780808167.210012 mapwrite 0x264000 thru 0x2671cb (0x31cc bytes) 000439[0] 1780808167.247038 trunc from 0x59b319 to 0x3b8ce9 (0x1e262f bytes) 000440[0] 1780808167.263464 mapwrite 0x363000 thru 0x365855 (0x2856 bytes) 000441[1] 1780808167.289815 mapwrite 0x158000 thru 0x15aea9 (0x2eaa bytes) 000444[0] 1780808167.324547 write 0x5cc000 thru 0x5ccf17 (0xf18 bytes) 000445[1] 1780808167.332562 write_OD 0x211000 thru 0x21efff (0xe000 bytes) 000446[0] 1780808167.349135 trunc from 0x5ccf18 to 0x66b77c (0x9e865 bytes) 000447[1] 1780808167.363015 mapread 0x01d000 thru 0x01f98f (0x2990 bytes) 000448[1] 1780808167.375359 trunc from 0x66b77d to 0x5fa380 (0x713fc bytes) 000449[1] 1780808167.395001 mapread 0x014000 thru 0x019e3f (0x5e40 bytes) 000457[1] 1780808167.425005 read_OD 0x4ea000 thru 0x4f0fff (0x7000 bytes) 000459[1] 1780808167.433858 mapread 0x364000 thru 0x36f80a (0xb80b bytes) 000464[1] 1780808167.451876 trunc from 0x5fa381 to 0x81a12a (0x21fdaa bytes) 000465[1] 1780808167.470730 read_OD 0x376000 thru 0x37efff (0x9000 bytes) 000469[1] 1780808167.481640 read_OD 0x715000 thru 0x717fff (0x3000 bytes) 000470[0] 1780808167.484499 mapread 0x810000 thru 0x819c5d (0x9c5e bytes) 000471[0] 1780808167.491612 write 0x1c1000 thru 0x1cbc9d (0xac9e bytes) 000477[1] 1780808167.504809 read_OD 0x789000 thru 0x78bfff (0x3000 bytes) 000478[1] 1780808167.514531 write_OD 0x55c000 thru 0x562fff (0x7000 bytes) 000481[1] 1780808167.532183 trunc from 0x81a12b to 0x16dc92 (0x6ac498 bytes) 000483[1] 1780808167.546891 mapread 0x02c000 thru 0x031d1c (0x5d1d bytes) 000484[0] 1780808167.558463 read 0x023000 thru 0x024980 (0x1981 bytes) 000485[1] 1780808167.573458 read_OD 0x116000 thru 0x11efff (0x9000 bytes) 000491[1] 1780808167.578345 read_OD 0x069000 thru 0x077fff (0xf000 bytes) 000493[1] 1780808167.583217 trunc from 0x16dc93 to 0x2bc9ce (0x14ed3c bytes) 000499[0] 1780808167.609334 write 0x0d3000 thru 0x0e1ccb (0xeccc bytes) All operations completed A-OK! PASS 16a (14s) == sanityn test 16b: 500 iterations of dual-mount fsx at small size ========================================================== 00:56:09 (1780808169) fallocate on zfs doesn't consume space fallocate not supported 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.118341 s, 70.9 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.0809897 s, 104 MB/s Chance of close/open is 1 in 50 Seed set to 8170 fd 0: /mnt/lustre/f16b.sanityn fd 1: /mnt/lustre2/f16b.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported truncating to largest ever: 0x51decc 000001[0] 1780808170.706920 trunc from 00000000 to 0x51decb (0x51decc bytes) 000002[1] 1780808170.713629 mapwrite 0x4c2cc2 thru 0x4c491d (0x1c5c bytes) 000003[0] 1780808170.748512 write 0x1c35db thru 0x1c3b6b (0x591 bytes) 000006[0] 1780808170.763211 write 0x67b327 thru 0x67b610 (0x2ea bytes) 000009[1] 1780808170.781590 mapwrite 0x677fa7 thru 0x677feb (0x45 bytes) 000014[0] 1780808170.829004 read 0x0a9643 thru 0x0a9673 (0x31 bytes) 000016[0] 1780808170.845819 mapread 0x2511f0 thru 0x25299a (0x17ab bytes) 000018[0] 1780808170.876787 mapread 0x32a02e thru 0x32b726 (0x16f9 bytes) 000020[1] 1780808170.896625 mapwrite 0x446bb6 thru 0x4477f6 (0xc41 bytes) 000021[1] 1780808170.920887 read 0x5ae951 thru 0x5aec63 (0x313 bytes) 000022[1] 1780808170.928562 write 0x4179a3 thru 0x418e7f (0x14dd bytes) 000026[0] 1780808170.961124 mapread 0x5c3476 thru 0x5c4ac4 (0x164f bytes) 000027[0] 1780808170.983214 mapread 0x3d90eb thru 0x3daa15 (0x192b bytes) 000030[0] 1780808171.017303 mapwrite 0x5b0e2f thru 0x5b1114 (0x2e6 bytes) 000032[1] 1780808171.049013 trunc from 0x67b611 to 0x40f910 (0x26bd00 bytes) 000033[0] 1780808171.069837 read 0x2b12b7 thru 0x2b1ec7 (0xc11 bytes) 000036[1] 1780808171.087677 mapwrite 0x0070a0 thru 0x007a13 (0x974 bytes) 000040[1] 1780808171.111870 mapread 0x33a766 thru 0x33adfe (0x699 bytes) 000042[1] 1780808171.125359 write 0x58785f thru 0x588af9 (0x129b bytes) 000043[0] 1780808171.132295 write 0x91fdc1 thru 0x9208d1 (0xb11 bytes) truncating to largest ever: 0x774eb3 000046[0] 1780808171.154256 trunc from 0x9208d2 to 0x774eb2 (0x1aba1f bytes) 000047[1] 1780808171.172893 mapwrite 0x2ced18 thru 0x2cf1c6 (0x4af bytes) 000051[1] 1780808171.229168 read 0x27e920 thru 0x27f9cb (0x10ac bytes) 000052[1] 1780808171.236501 mapread 0x53eca2 thru 0x53f0de (0x43d bytes) 000056[0] 1780808171.270561 read 0x0e124c thru 0x0e3145 (0x1efa bytes) 000057[1] 1780808171.277036 mapwrite 0x8c732e thru 0x8c7413 (0xe6 bytes) truncating to largest ever: 0x7cc013 000058[1] 1780808171.311622 trunc from 0x8c7414 to 0x7cc012 (0xfb401 bytes) 000060[0] 1780808171.331972 read 0x1f787f thru 0x1f88f0 (0x1072 bytes) 000069[1] 1780808171.368562 mapread 0x56c07f thru 0x56c639 (0x5bb bytes) 000070[1] 1780808171.371158 write 0x6bab17 thru 0x6bc82f (0x1d19 bytes) 000072[0] 1780808171.382263 mapread 0x032a2d thru 0x033fd2 (0x15a6 bytes) 000079[1] 1780808171.412252 trunc from 0x7cc013 to 0x3e294e (0x3e96c4 bytes) 000082[0] 1780808171.444923 write 0x150c93 thru 0x1517f2 (0xb60 bytes) 000084[1] 1780808171.463855 mapwrite 0x627be8 thru 0x6293cb (0x17e4 bytes) 000087[1] 1780808171.502723 mapwrite 0x8cf261 thru 0x8d0727 (0x14c7 bytes) 000090[1] 1780808171.535030 write 0x6f3fac thru 0x6f49b9 (0xa0e bytes) 000092[0] 1780808171.551217 mapread 0x691429 thru 0x6919d9 (0x5b1 bytes) 000093[1] 1780808171.563943 mapread 0x67280b thru 0x6741af (0x19a5 bytes) 000094[0] 1780808171.571462 write 0x1eba1e thru 0x1ecf43 (0x1526 bytes) 000095[1] 1780808171.578779 write 0x80dce8 thru 0x80fad5 (0x1dee bytes) 000102[1] 1780808171.598846 mapread 0x273ced thru 0x273e27 (0x13b bytes) 000104[1] 1780808171.607394 read 0x304526 thru 0x305622 (0x10fd bytes) 000105[1] 1780808171.610499 mapwrite 0x0bf905 thru 0x0c108a (0x1786 bytes) 000106[1] 1780808171.637677 mapwrite 0x8a2b83 thru 0x8a426f (0x16ed bytes) 000108[1] 1780808171.666377 mapread 0x783d03 thru 0x785785 (0x1a83 bytes) 000109[1] 1780808171.684476 mapread 0x53ee5b thru 0x540876 (0x1a1c bytes) 000110[0] 1780808171.698633 trunc from 0x8d0728 to 0x151e81 (0x77e8a6 bytes) 000112[1] 1780808171.725594 mapread 0x0c0212 thru 0x0c2110 (0x1eff bytes) 000113[1] 1780808171.741464 mapread 0x04df1c thru 0x04f670 (0x1755 bytes) 000115[1] 1780808171.762248 mapread 0x0fe43b thru 0x0feb45 (0x70b bytes) 000116[1] 1780808171.770765 mapwrite 0x2cb242 thru 0x2cb956 (0x715 bytes) 000117[0] 1780808171.812284 write 0x5d51a5 thru 0x5d5c5e (0xaba bytes) 000118[0] 1780808171.823387 trunc from 0x5d5c5f to 0x370c18 (0x265046 bytes) 000119[1] 1780808171.834456 write 0x893334 thru 0x894c70 (0x193d bytes) truncating to largest ever: 0x92865f 000122[0] 1780808171.862482 trunc from 0x894c71 to 0x92865e (0x939ee bytes) 000123[0] 1780808171.879407 mapwrite 0x5fb0c5 thru 0x5fb76d (0x6a9 bytes) 000124[1] 1780808171.903234 read 0x3c35b1 thru 0x3c4eda (0x192a bytes) 000125[1] 1780808171.913393 mapread 0x6f173d thru 0x6f2ac0 (0x1384 bytes) 000129[0] 1780808171.928018 mapwrite 0x12b89a thru 0x12ce97 (0x15fe bytes) 000139[0] 1780808171.991602 write 0x2fc942 thru 0x2fd516 (0xbd5 bytes) 000141[0] 1780808172.002998 read 0x0e5364 thru 0x0e57eb (0x488 bytes) 000143[1] 1780808172.015551 read 0x430fb8 thru 0x432762 (0x17ab bytes) 000144[1] 1780808172.018316 trunc from 0x92865f to 0x53a476 (0x3ee1e8 bytes) 000145[1] 1780808172.036205 trunc from 0x53a477 to 0x1ac627 (0x38de4f bytes) 000147[0] 1780808172.053048 mapread 0x089e82 thru 0x08be41 (0x1fc0 bytes) 000152[1] 1780808172.081998 trunc from 0x1ac628 to 0x5911b2 (0x3e4b8b bytes) 000153[0] 1780808172.098725 read 0x082845 thru 0x08433d (0x1af9 bytes) 000154[0] 1780808172.107419 trunc from 0x5911b3 to 0x85a6f3 (0x2c9541 bytes) 000156[1] 1780808172.131251 trunc from 0x85a6f4 to 0x2ac773 (0x5adf80 bytes) 000158[1] 1780808172.146334 trunc from 0x2ac774 to 0x89a03e (0x5ed8cb bytes) 000162[1] 1780808172.170998 write 0x652252 thru 0x65328a (0x1039 bytes) 000166[0] 1780808172.212344 mapwrite 0x5d27d4 thru 0x5d3455 (0xc82 bytes) 000167[0] 1780808172.260807 trunc from 0x89a03f to 0x198af1 (0x70154d bytes) 000169[1] 1780808172.295695 mapread 0x18c374 thru 0x18e09d (0x1d2a bytes) truncating to largest ever: 0x9a2050 000171[0] 1780808172.317186 trunc from 0x198af2 to 0x9a204f (0x80955e bytes) 000172[1] 1780808172.342288 write 0x52db52 thru 0x52f661 (0x1b10 bytes) 000173[0] 1780808172.362350 mapwrite 0x09fc23 thru 0x09ffd3 (0x3b1 bytes) 000174[1] 1780808172.386908 write 0x4dee04 thru 0x4e0839 (0x1a36 bytes) 000175[1] 1780808172.395005 read 0x57b8a3 thru 0x57c392 (0xaf0 bytes) 000176[0] 1780808172.404490 trunc from 0x9a2050 to 0x22d2ec (0x774d63 bytes) 000177[0] 1780808172.423781 read 0x0a1a80 thru 0x0a36f9 (0x1c7a bytes) 000178[0] 1780808172.452028 write 0x9ed0fd thru 0x9ede41 (0xd45 bytes) 000180[0] 1780808172.462391 write 0x68c7c3 thru 0x68d11b (0x959 bytes) 000181[1] 1780808172.499898 read 0x3ed1aa thru 0x3ed347 (0x19e bytes) 000182[1] 1780808172.523056 read 0x307896 thru 0x30829d (0xa08 bytes) 000185[1] 1780808172.542971 trunc from 0x9ede42 to 0x73bff8 (0x2b1e49 bytes) 000186[1] 1780808172.577134 read 0x60186e thru 0x601a7d (0x210 bytes) 000188[0] 1780808172.582921 mapwrite 0x62f665 thru 0x6302b2 (0xc4e bytes) 000190[0] 1780808172.623491 mapread 0x620783 thru 0x6212b6 (0xb34 bytes) 000192[0] 1780808172.640612 write 0x7729f9 thru 0x772fe6 (0x5ee bytes) 000196[0] 1780808172.671057 write 0x382dcd thru 0x3848d9 (0x1b0d bytes) 000201[1] 1780808172.704899 write 0x46ed9e thru 0x46f40c (0x66f bytes) 000202[1] 1780808172.724874 trunc from 0x772fe7 to 0x536bc5 (0x23c421 bytes) 000203[1] 1780808172.739228 trunc from 0x536bc6 to 0x564c30 (0x2e06b bytes) 000204[0] 1780808172.751189 mapread 0x4b5a28 thru 0x4b739c (0x1975 bytes) 000205[0] 1780808172.767600 trunc from 0x564c31 to 0x121396 (0x44389a bytes) 000206[0] 1780808172.783277 write 0x4169ff thru 0x418541 (0x1b43 bytes) 000210[0] 1780808172.799139 mapread 0x059bbe thru 0x059f99 (0x3dc bytes) 000211[0] 1780808172.801785 mapwrite 0x29c820 thru 0x29e64d (0x1e2e bytes) 000212[1] 1780808172.844644 mapwrite 0x1801a5 thru 0x1815e4 (0x1440 bytes) 000213[0] 1780808172.891124 mapwrite 0x0ecec0 thru 0x0ee4f1 (0x1632 bytes) 000215[1] 1780808172.932171 trunc from 0x418542 to 0x3565e2 (0xc1f5f bytes) 000216[0] 1780808172.946025 mapread 0x18e20d thru 0x190206 (0x1ffa bytes) 000217[1] 1780808172.967944 mapread 0x21f2b8 thru 0x2209a6 (0x16ef bytes) 000218[1] 1780808172.982369 write 0x5039c0 thru 0x505411 (0x1a52 bytes) 000219[1] 1780808172.993646 read 0x2cdcf3 thru 0x2ce037 (0x345 bytes) 000223[1] 1780808173.014435 trunc from 0x505412 to 0x663127 (0x15dd16 bytes) 000224[1] 1780808173.027402 mapread 0x178f00 thru 0x17a4a8 (0x15a9 bytes) 000226[0] 1780808173.054864 mapwrite 0x0d199f thru 0x0d283b (0xe9d bytes) 000229[0] 1780808173.099523 mapread 0x06adfb thru 0x06c548 (0x174e bytes) 000230[0] 1780808173.113225 trunc from 0x663128 to 0x1f2139 (0x470fee bytes) 000232[1] 1780808173.147650 mapwrite 0x09abeb thru 0x09c30d (0x1723 bytes) 000234[0] 1780808173.187969 trunc from 0x1f213a to 0x198a9d (0x5969c bytes) 000235[0] 1780808173.199813 read 0x042880 thru 0x0436ff (0xe80 bytes) 000236[0] 1780808173.215041 trunc from 0x198a9e to 0x4130a6 (0x27a609 bytes) 000237[1] 1780808173.230232 write 0x4577c1 thru 0x4586fe (0xf3e bytes) 000241[0] 1780808173.276138 mapread 0x3b4528 thru 0x3b5d12 (0x17eb bytes) 000243[0] 1780808173.325058 mapread 0x3677f6 thru 0x3696ad (0x1eb8 bytes) 000249[0] 1780808173.365128 write 0x5b5e15 thru 0x5b62ea (0x4d6 bytes) 000252[0] 1780808173.380834 mapread 0x3350f0 thru 0x335560 (0x471 bytes) 000253[0] 1780808173.388934 read 0x5a832a thru 0x5a9e52 (0x1b29 bytes) 000255[1] 1780808173.398562 trunc from 0x5b62eb to 0x001607 (0x5b4ce3 bytes) 000256[1] 1780808173.422263 write 0x9da8e5 thru 0x9da982 (0x9e bytes) 000257[1] 1780808173.427318 mapwrite 0x7441a5 thru 0x744461 (0x2bd bytes) 000258[1] 1780808173.445791 read 0x7033fe thru 0x703c58 (0x85b bytes) 000259[0] 1780808173.454930 mapread 0x7653a2 thru 0x766a1a (0x1679 bytes) 000262[1] 1780808173.469320 trunc from 0x9da983 to 0x26e011 (0x76c971 bytes) 000263[1] 1780808173.478947 mapwrite 0x3f07e5 thru 0x3f10e5 (0x901 bytes) 000264[1] 1780808173.506513 mapwrite 0x04bdb9 thru 0x04d855 (0x1a9d bytes) 000265[1] 1780808173.532852 mapread 0x35d6d2 thru 0x35db71 (0x4a0 bytes) 000266[0] 1780808173.541511 write 0x6f556c thru 0x6f60a2 (0xb37 bytes) 000267[1] 1780808173.552690 mapread 0x5f666d thru 0x5f7efb (0x188f bytes) 000268[0] 1780808173.566809 mapread 0x631647 thru 0x632de1 (0x179b bytes) 000270[0] 1780808173.577821 mapread 0x1ba1d7 thru 0x1bad85 (0xbaf bytes) 000272[0] 1780808173.586512 write 0x391e83 thru 0x3932df (0x145d bytes) 000273[1] 1780808173.592345 write 0x79ec60 thru 0x79fb60 (0xf01 bytes) 000274[0] 1780808173.602243 mapread 0x6dcd69 thru 0x6dd11c (0x3b4 bytes) 000275[0] 1780808173.615348 mapwrite 0x3874be thru 0x3887b0 (0x12f3 bytes) 000277[0] 1780808173.637365 read 0x6022ab thru 0x602e48 (0xb9e bytes) 000279[1] 1780808173.647241 trunc from 0x79fb61 to 0x172069 (0x62daf7 bytes) 000280[0] 1780808173.659662 mapwrite 0x88146f thru 0x8829b6 (0x1548 bytes) 000284[0] 1780808173.716804 write 0x2c0ce6 thru 0x2c0e9d (0x1b8 bytes) 000288[0] 1780808173.720550 trunc from 0x8829b7 to 0x73c711 (0x1462a5 bytes) 000289[1] 1780808173.730837 trunc from 0x73c712 to 0x4f0a66 (0x24bcab bytes) 000290[0] 1780808173.746193 read 0x29e660 thru 0x29fec2 (0x1863 bytes) 000291[1] 1780808173.754430 trunc from 0x4f0a67 to 0x047933 (0x4a9133 bytes) 000292[1] 1780808173.766304 mapread 0x01b0d3 thru 0x01b691 (0x5bf bytes) 000294[0] 1780808173.775980 mapread 0x0162fa thru 0x0180de (0x1de5 bytes) 000295[1] 1780808173.787425 read 0x012d0b thru 0x0135cc (0x8c2 bytes) 000296[0] 1780808173.791850 write 0x1bd7ce thru 0x1bdf5f (0x792 bytes) 000297[1] 1780808173.799386 mapwrite 0x248b9b thru 0x24a5c7 (0x1a2d bytes) 000302[1] 1780808173.871731 mapwrite 0x5cf1d3 thru 0x5cf23a (0x68 bytes) 000305[1] 1780808173.929852 trunc from 0x5cf23b to 0x49545e (0x139ddc bytes) 000306[0] 1780808173.948480 write 0x412816 thru 0x412d88 (0x573 bytes) 000307[1] 1780808173.958490 write 0x201495 thru 0x201a8a (0x5f6 bytes) 000309[0] 1780808173.969766 write 0x837bb1 thru 0x8384ef (0x93f bytes) 000310[1] 1780808173.983030 mapread 0x7519af thru 0x752a81 (0x10d3 bytes) 000314[0] 1780808174.003144 mapread 0x0a2aa2 thru 0x0a3fc7 (0x1526 bytes) 000318[0] 1780808174.023322 mapread 0x8332d2 thru 0x833366 (0x95 bytes) 000319[0] 1780808174.025628 trunc from 0x8384f0 to 0x528371 (0x31017e bytes) 000323[1] 1780808174.058427 mapwrite 0x3a5ea5 thru 0x3a782a (0x1986 bytes) 000324[0] 1780808174.090530 mapwrite 0x6b0a17 thru 0x6b1b80 (0x116a bytes) 000330[0] 1780808174.161146 read 0x5818d6 thru 0x58350a (0x1c35 bytes) 000331[1] 1780808174.164597 mapread 0x04e951 thru 0x04ecca (0x37a bytes) 000335[0] 1780808174.201494 read 0x2dbdab thru 0x2dd1c2 (0x1418 bytes) 000336[0] 1780808174.210222 trunc from 0x6b1b81 to 0x2723eb (0x43f795 bytes) 000338[0] 1780808174.231209 mapwrite 0x447652 thru 0x448253 (0xc02 bytes) 000341[0] 1780808174.261142 trunc from 0x448254 to 0x847962 (0x3ff70f bytes) 000343[1] 1780808174.268693 write 0x44ac60 thru 0x44c6ea (0x1a8b bytes) 000344[1] 1780808174.281773 read 0x0b7dbf thru 0x0b8d8e (0xfd0 bytes) 000345[1] 1780808174.292645 write 0x879164 thru 0x8795b9 (0x456 bytes) 000347[0] 1780808174.307365 mapwrite 0x6e9133 thru 0x6e97d6 (0x6a4 bytes) 000348[0] 1780808174.347381 trunc from 0x8795ba to 0x58fa3a (0x2e9b7f bytes) 000350[1] 1780808174.378022 mapwrite 0x05599a thru 0x05716f (0x17d6 bytes) 000351[1] 1780808174.420529 mapread 0x343ca8 thru 0x3447d5 (0xb2e bytes) 000354[1] 1780808174.445309 write 0x4d0271 thru 0x4d0568 (0x2f8 bytes) 000357[0] 1780808174.458394 read 0x2dbed0 thru 0x2dd1a2 (0x12d3 bytes) 000358[0] 1780808174.470274 write 0x109daa thru 0x10a352 (0x5a9 bytes) 000360[1] 1780808174.491922 mapwrite 0x55d1ed thru 0x55d45b (0x26f bytes) 000361[1] 1780808174.519418 mapwrite 0x65bf87 thru 0x65dd9b (0x1e15 bytes) 000363[0] 1780808174.554355 trunc from 0x65dd9c to 0x69d564 (0x3f7c9 bytes) 000364[1] 1780808174.565761 read 0x0dfbbf thru 0x0e153d (0x197f bytes) 000368[0] 1780808174.588235 mapread 0x5abe2c thru 0x5ac959 (0xb2e bytes) truncating to largest ever: 0x9fd0c4 000372[1] 1780808174.603308 trunc from 0x69d565 to 0x9fd0c3 (0x35fb5f bytes) 000375[0] 1780808174.631928 read 0x7a10fd thru 0x7a17f8 (0x6fc bytes) 000381[0] 1780808174.653784 mapwrite 0x2566b3 thru 0x256a51 (0x39f bytes) 000384[0] 1780808174.680380 write 0x4353f9 thru 0x436cb4 (0x18bc bytes) 000386[1] 1780808174.692930 trunc from 0x9fd0c4 to 0x737a66 (0x2c565d bytes) 000388[0] 1780808174.714490 write 0x6e880b thru 0x6ea24c (0x1a42 bytes) 000389[0] 1780808174.726943 mapwrite 0x7619af thru 0x7625b1 (0xc03 bytes) 000390[1] 1780808174.754596 mapread 0x4d52aa thru 0x4d6b8e (0x18e5 bytes) 000391[0] 1780808174.767006 mapread 0x63a980 thru 0x63b95d (0xfde bytes) 000395[0] 1780808174.781041 mapwrite 0x2b8596 thru 0x2b960b (0x1076 bytes) 000397[0] 1780808174.814226 mapwrite 0x76a16d thru 0x76a812 (0x6a6 bytes) 000398[0] 1780808174.849504 mapwrite 0x09e781 thru 0x09ec3d (0x4bd bytes) 000400[1] 1780808174.871478 mapread 0x3619e1 thru 0x36314e (0x176e bytes) 000402[0] 1780808174.891202 mapwrite 0x024dc2 thru 0x0259b0 (0xbef bytes) 000403[0] 1780808174.916986 trunc from 0x76a813 to 0x47b9c5 (0x2eee4d bytes) 000404[0] 1780808174.928005 mapwrite 0x0bcc6a thru 0x0bd914 (0xcab bytes) 000406[0] 1780808174.948864 trunc from 0x47b9c6 to 0x3978d6 (0xe40ef bytes) 000410[0] 1780808174.969576 trunc from 0x3978d7 to 0x57a62d (0x1e2d57 bytes) 000411[1] 1780808174.975640 mapwrite 0x34d470 thru 0x34d78b (0x31c bytes) 000413[1] 1780808175.005667 mapwrite 0x1c5575 thru 0x1c6f44 (0x19d0 bytes) 000414[0] 1780808175.032047 mapread 0x3a4916 thru 0x3a4d8f (0x47a bytes) 000415[0] 1780808175.042447 trunc from 0x57a62e to 0x7be11a (0x243aed bytes) 000416[0] 1780808175.051012 mapread 0x4c0906 thru 0x4c13ef (0xaea bytes) 000418[1] 1780808175.066350 trunc from 0x7be11b to 0x5a6900 (0x21781a bytes) 000419[1] 1780808175.081169 trunc from 0x5a6901 to 0x88516f (0x2de86f bytes) 000420[0] 1780808175.091196 read 0x17be65 thru 0x17de61 (0x1ffd bytes) 000421[1] 1780808175.098400 mapread 0x5613cc thru 0x5625a4 (0x11d9 bytes) 000423[0] 1780808175.103594 mapwrite 0x133da0 thru 0x1358ed (0x1b4e bytes) 000424[1] 1780808175.129428 read 0x5c234f thru 0x5c3fe8 (0x1c9a bytes) 000425[0] 1780808175.137646 read 0x534e89 thru 0x536c81 (0x1df9 bytes) 000427[0] 1780808175.150503 mapread 0x01d7de thru 0x01e6c5 (0xee8 bytes) 000432[0] 1780808175.168640 mapread 0x3e9e8c thru 0x3ea11f (0x294 bytes) 000435[0] 1780808175.177804 mapread 0x0477ea thru 0x047b76 (0x38d bytes) 000437[0] 1780808175.189377 trunc from 0x885170 to 0x57e84b (0x306924 bytes) 000441[1] 1780808175.217160 read 0x2efe3e thru 0x2f1545 (0x1708 bytes) 000442[0] 1780808175.224890 mapread 0x3108ab thru 0x31270c (0x1e62 bytes) 000444[0] 1780808175.240249 trunc from 0x57e84c to 0x21ef27 (0x35f924 bytes) 000450[0] 1780808175.278039 write 0x564456 thru 0x56481a (0x3c5 bytes) 000451[1] 1780808175.286674 trunc from 0x56481b to 0x61962d (0xb4e13 bytes) 000452[1] 1780808175.312506 mapwrite 0x0cd982 thru 0x0cf321 (0x19a0 bytes) 000453[0] 1780808175.361799 mapread 0x35c5da thru 0x35e56e (0x1f95 bytes) 000454[1] 1780808175.385734 mapwrite 0x1bd1d7 thru 0x1bd986 (0x7b0 bytes) 000455[0] 1780808175.414436 mapread 0x0260a5 thru 0x02713b (0x1097 bytes) 000456[1] 1780808175.427246 mapwrite 0x05aeb0 thru 0x05be2a (0xf7b bytes) 000457[0] 1780808175.455665 mapwrite 0x9b911d thru 0x9ba151 (0x1035 bytes) 000458[0] 1780808175.489220 write 0x1101e2 thru 0x110367 (0x186 bytes) 000459[1] 1780808175.500471 write 0x1ce47c thru 0x1cf363 (0xee8 bytes) 000460[1] 1780808175.512585 mapread 0x16387f thru 0x164df0 (0x1572 bytes) 000463[0] 1780808175.535725 read 0x1f259e thru 0x1f3033 (0xa96 bytes) 000468[0] 1780808175.551035 write 0x9746b4 thru 0x974979 (0x2c6 bytes) 000469[1] 1780808175.556765 mapread 0x37e09d thru 0x37f6da (0x163e bytes) 000470[1] 1780808175.572153 write 0x634586 thru 0x635540 (0xfbb bytes) 000474[0] 1780808175.590020 write 0x0f97e5 thru 0x0fb4d6 (0x1cf2 bytes) 000476[0] 1780808175.604182 mapread 0x31b85b thru 0x31cf40 (0x16e6 bytes) 000479[1] 1780808175.619951 read 0x5380b8 thru 0x539cd4 (0x1c1d bytes) 000480[1] 1780808175.625739 read 0x73658a thru 0x736fb3 (0xa2a bytes) 000481[1] 1780808175.631850 read 0x883e5b thru 0x885296 (0x143c bytes) 000483[1] 1780808175.643381 mapread 0x4a88d8 thru 0x4aa464 (0x1b8d bytes) 000484[1] 1780808175.653303 read 0x35763e thru 0x358766 (0x1129 bytes) 000488[1] 1780808175.676343 mapwrite 0x5ed31e thru 0x5edf3d (0xc20 bytes) 000489[1] 1780808175.696973 mapread 0x55db7c thru 0x55de85 (0x30a bytes) 000490[1] 1780808175.699827 mapread 0x56117f thru 0x56185d (0x6df bytes) 000495[1] 1780808175.723631 trunc from 0x9ba152 to 0x0c6b92 (0x8f35bf bytes) 000496[1] 1780808175.738493 mapwrite 0x8b5f91 thru 0x8b6bd0 (0xc40 bytes) 000499[0] 1780808175.783011 mapread 0x7f6029 thru 0x7f6a84 (0xa5c bytes) 000500[0] 1780808175.798503 write 0x42a3eb thru 0x42bd96 (0x19ac bytes) All operations completed A-OK! PASS 16b (8s) == sanityn test 16c: verify data consistency on ldiskfs with cache disabled (b=17397) ========================================================== 00:56:17 (1780808177) SKIP: sanityn test_16c dio on ldiskfs only SKIP 16c (1s) == sanityn test 16d: Verify DIO and buffer IO with two clients ========================================================== 00:56:18 (1780808178) 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.124614 s, 67.3 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.0860241 s, 97.5 MB/s ldlm.namespaces.MGC192.168.203.150@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8f15073b7800.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8f150925a000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8f15073b7800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8f150925a000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8f15073b7800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8f150925a000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8f15073b7800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8f150925a000.lru_size=clear 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 2.95252 s, 35.5 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 0.852206 s, 123 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 0.946587 s, 111 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 2.01087 s, 52.1 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 2.37796 s, 44.1 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 1.33104 s, 78.8 MB/s PASS 16d (15s) == sanityn test 16e: Verify size consistency for O_DIRECT write ========================================================== 00:56:33 (1780808193) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.117489 s, 89.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0311376 s, 33.7 MB/s PASS 16e (3s) == sanityn test 16f: rw sequential consistency vs drop_caches ========================================================== 00:56:36 (1780808196) TIMEOUT PASS 16f (22s) == sanityn test 16g: mmap rw sequential consistency vs drop_caches ========================================================== 00:56:58 (1780808218) TIMEOUT PASS 16g (23s) == sanityn test 16h: mmap read after truncate file ======= 00:57:21 (1780808241) striped dir -i0 -c2 -H all_char /mnt/lustre/d16h.sanityn ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16h (2s) == sanityn test 16i: read after truncate file ============ 00:57:23 (1780808243) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d16i.sanityn ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16i (2s) == sanityn test 16j: race dio with buffered i/o ========== 00:57:25 (1780808245) 1+0 records in 1+0 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.611076 s, 42.9 MB/s bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.442962 s, 59.2 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.627615 s, 41.8 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.406846 s, 64.4 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.544332 s, 48.2 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.463996 s, 56.5 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.659637 s, 39.7 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.506894 s, 51.7 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.608934 s, 43.0 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.792598 s, 33.1 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.811975 s, 32.3 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.549847 s, 47.7 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.612731 s, 42.8 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.745682 s, 35.2 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.778142 s, 33.7 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.642488 s, 40.8 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.726641 s, 36.1 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.831872 s, 31.5 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.860569 s, 30.5 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK PASS 16j (11s) == sanityn test 16k: Parallel FSX and drop caches should not panic ========================================================== 00:57:36 (1780808256) + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.1 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.2 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.3 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.4 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.5 -l 67108864 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read skipping zero size read truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported truncating to largest ever: 0x1a7062c fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported truncating to largest ever: 0x32cce75 skipping zero size read truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x3c569db truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x3c569db truncating to largest ever: 0x32cce75 truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! + fsx 1 done: rc=0 Sun Jun 7 00:57:47 EDT 2026 + fsx 2 done: rc=0 Sun Jun 7 00:57:47 EDT 2026 All operations completed A-OK! + fsx 3 done: rc=0 Sun Jun 7 00:57:47 EDT 2026 + fsx 4 done: rc=0 Sun Jun 7 00:57:47 EDT 2026 + fsx 5 done: rc=0 Sun Jun 7 00:57:47 EDT 2026 PASS 16k (16s) == sanityn test 17: resource creation/LVB creation race ========================================================================= 00:57:52 (1780808272) fail_loc=0x8000030a PASS 17 (5s) == sanityn test 18: mmap sanity check =========================================================================================== 00:57:57 (1780808277) mmap test1: basic mmap operation (PASS, 0.043152s) mmap test2: MAP_PRIVATE not write back (PASS, 0.038504s) mmap test3: concurrent mmap ops on two nodes (PASS, 3.7591s) mmap test4: c1 write to f1 from mmapped f2, c2 write to f1 from mmapped f1 (PASS, 4.1666s) mmap test5: read/write file to/from the buffer which mmapped to just this file (PASS, 0.059557s) mmap test6: check mmap write/read content on two nodes (PASS, 0.06409s) mmap test7: file i/o with an unmapped buffer (SKIPPED, 0s) mmap test8: SIGBUS for beyond file size (SKIPPED, 0s) mmap test9: SIGBUS for negative file offset (SKIPPED, 0s) mmap test10: multi-thread mmap access (PASS, 0.49278s) mmap test11: multi-thread shared mmap (PASS, 0.49154s) mmap test10: mtime not change for readonly mmap access (PASS, 1.0394s) PASS 18 (14s) == sanityn test 19: test concurrent uncached read races ========================================================================= 00:58:11 (1780808291) oleg350-server: error: get_param: param_path 'osd-*/lustre-OST*/read_cache_enable': No such file or directory pdsh@oleg350-client: oleg350-server: ssh exited with exit code 2 SKIP: sanityn test_19 not cache-capable obdfilter SKIP 19 (2s) == sanityn test 20: test extra readahead page left in cache ============================================================== 00:58:13 (1780808293) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d20.sanityn PASS 20 (2s) == sanityn test 21: Try to remove mountpoint on another dir ============================================================== 00:58:15 (1780808295) striped dir -i1 -c2 -H all_char /mnt/lustre/d21.sanityn rmdir: removing directory, '/mnt/lustre/d21.sanityn' rmdir: failed to remove '/mnt/lustre/d21.sanityn': Device or resource busy rmdir: removing directory, '/mnt/lustre2/d21.sanityn' Removed mounted directory from another mountpoint, needs to be fixed PASS 21 (3s) == sanityn test 23: others should see updated atime while another read============================================================== 00:58:18 (1780808298) now is 1780808299 starting reads multiop /mnt/lustre/f23.sanityn vor20_c TMPPIPE=/tmp/multiop_open_wait_pipe.8387 new atime is 1780808360 PASS 23 (64s) == sanityn test 24a: lfs df [-ih] [path] test =================================================================================== 00:59:22 (1780808362) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2432 2206208 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 2210304 2176 2206080 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3771392 31744 3737600 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 7168 3762176 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2432 2206208 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 2210304 2176 2206080 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3771392 31744 3737600 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 7168 3762176 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre2 UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 516.1K 404 515.7K 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 518.9K 337 518.6K 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 114.6K 468 114.1K 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 115.3K 466 114.9K 1% /mnt/lustre[OST:1] filesystem_summary: 229.7K 741 229.0K 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 516.1K 404 515.7K 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 518.9K 337 518.6K 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 114.6K 468 114.1K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 115.3K 466 114.9K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.7K 741 229.0K 1% /mnt/lustre2 UUID bytes Used Available Use% Mounted on lustre-MDT0000_UUID 2.1G 2.4M 2.1G 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 2.1G 2.1M 2.1G 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3.6G 31.0M 3.6G 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3.6G 7.0M 3.6G 1% /mnt/lustre[OST:1] filesystem_summary: 7.2G 38.0M 7.2G 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 528476 404 528072 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 531344 337 531007 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 117332 468 116864 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 118098 466 117632 1% /mnt/lustre2[OST:1] filesystem_summary: 235237 741 234496 1% /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2432 2206208 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 2210304 2176 2206080 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3771392 31744 3737600 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 7168 3762176 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 516.1K 404 515.7K 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 518.9K 337 518.6K 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 114.6K 468 114.1K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 115.3K 466 114.9K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.7K 741 229.0K 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 528476 404 528072 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 531344 337 531007 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 117332 468 116864 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 118098 466 117632 1% /mnt/lustre[OST:1] filesystem_summary: 235237 741 234496 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 528476 404 528072 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 531344 337 531007 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 117332 468 116864 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 118098 466 117632 1% /mnt/lustre2[OST:1] filesystem_summary: 235237 741 234496 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2432 2206208 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 2210304 2176 2206080 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3771392 31744 3737600 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 7168 3762176 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2432 2206208 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 2210304 2176 2206080 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3771392 31744 3737600 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 7168 3762176 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre2 PASS 24a (2s) == sanityn test 24b: lfs df should show both filesystems ========================================================================= 00:59:24 (1780808364) PASS 24b (3s) == sanityn test 25a: change ACL on one mountpoint be seen on another ============================================================= 00:59:27 (1780808367) running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] PASS 25a (2s) == sanityn test 25b: change ACL under remote dir on one mountpoint be seen on another ========================================================== 00:59:29 (1780808369) running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25b.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25b.sanityn/f1] Can't lstat /mnt/lustre2/d25b.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25b.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25b.sanityn/f1] Can't lstat /mnt/lustre2/d25b.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25b.sanityn/f1] PASS 25b (3s) == sanityn test 26a: allow mtime to get older ============ 00:59:32 (1780808372) utime: good mknod times 1780808371 <= 1780808372 <= 1780808372 for /mnt/lustre/f26a utime: good mknod times 1780808371 <= 1780808372 <= 1780808372 for /mnt/lustre2/f26a utime: good utime mtimes 100000, atime 200000 utime: updated times for /mnt/lustre2/f26a PASS 26a (3s) == sanityn test 26b: sync mtime between ost and mds ====== 00:59:35 (1780808375) PASS 26b (4s) == sanityn test 26c: set-in-past on open file is not lost on close ========================================================== 00:59:39 (1780808379) multiop /mnt/lustre/f26c.sanityn vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.8387 File: /mnt/lustre/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205289279600 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-06-07 00:59:40.000000000 -0400 Birth: 2026-06-07 00:59:39.000000000 -0400 File: /mnt/lustre2/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205289279600 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-06-07 00:59:40.000000000 -0400 Birth: 2026-06-07 00:59:39.000000000 -0400 PASS 26c (4s) SKIP: sanityn test_27 skipping excluded test 27 == sanityn test 30: recreate file race =================== 00:59:43 (1780808383) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d30.sanityn 0 PASS 30 (4s) == sanityn test 31a: voluntary cancel / blocking ast race======================================================================== 00:59:47 (1780808387) striped dir -i1 -c2 -H crush2 /mnt/lustre/d31a.sanityn fail_loc=0x314 PASS 31a (4s) == sanityn test 31b: voluntary OST cancel / blocking ast race======================================================================== 00:59:51 (1780808391) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg350-server mds-ost sync done. sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d31b.sanityn fail_loc=0x314 fail_loc=0x316 fail_loc=0 fail_loc=0 Filesystem 1K-blocks Used Available Use% Mounted on 192.168.203.150@tcp:/lustre 7542784 39936 7498752 1% /mnt/lustre2 PASS 31b (21s) == sanityn test 31r: open-rename(replace) race =========== 01:00:12 (1780808412) -rw-r--r-- 1 root root 0 Jun 7 01:00 /mnt/lustre/f31r.sanityn.target fail_loc=0x1419 fail_val=3 PASS 31r (5s) == sanityn test 31s: open should not revalidate invalid dentry ========================================================== 01:00:17 (1780808417) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 1207: 45177 Terminated tail -f $tfile PASS 31s (3s) == sanityn test 31t: getattr should not revalidate invalid dentry ========================================================== 01:00:20 (1780808420) PASS 31t (4s) SKIP: sanityn test_33a skipping SLOW test 33a == sanityn test 33b: COS: cross create/delete, 2 clients, benchmark under remote dir ========================================================== 01:00:24 (1780808424) SKIP: sanityn test_33b Need two or more clients, have 1 SKIP 33b (2s) == sanityn test 33c: Cancel cross-MDT lock should trigger Sync-on-Lock-Cancel ========================================================== 01:00:26 (1780808426) Stopping /mnt/lustre-mds1 (opts:) on oleg350-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg350-server: oleg350-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0000 create file on 2nd stripe under striped directory /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 0 -c 2 /mnt/lustre/d33c.sanityn/striped touch /mnt/lustre2/d33c.sanityn/striped/subfile SoLC count 2 Below operations shouldn't trigger Solc: create file on 2nd stripe after setattr chmod 777 /mnt/lustre/d33c.sanityn/striped; touch /mnt/lustre2/d33c.sanityn/striped/subfile create remote dir and local dir /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 1 /mnt/lustre/d33c.sanityn/remote create local dir after remote dir creation transaction commit /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 1 /mnt/lustre/d33c.sanityn/remote.3; do_facet mds2 /home/green/git/lustre-release/lustre/utils/lctl set_param -n osd*.*MDT0001.force_sync 1; mkdir /mnt/lustre/d33c.sanityn/local.3 Solc count 0 PASS 33c (21s) == sanityn test 33d: dependent transactions should trigger COS ========================================================== 01:00:47 (1780808447) remote directory create /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 0 /mnt/lustre/d33d.sanityn /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 1 /mnt/lustre2/d33d.sanityn/subdir CoS count 2 cross-MDT rename mkdir /mnt/lustre/d33d.sanityn; /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 0 /mnt/lustre/d33d.sanityn/d1; /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 1 /mnt/lustre/d33d.sanityn/d2; touch /mnt/lustre/d33d.sanityn/d1/src mv /mnt/lustre2/d33d.sanityn/d1/src /mnt/lustre2/d33d.sanityn/d2/tgt CoS count 2 migrate /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 0 /mnt/lustre/d33d.sanityn /home/green/git/lustre-release/lustre/utils/lfs migrate -m 1 /mnt/lustre2/d33d.sanityn CoS count 2 PASS 33d (22s) == sanityn test 33e: independent transactions shouldn't trigger COS ========================================================== 01:01:09 (1780808469) plain dir creation mkdir /mnt/lustre2/d33e.sanityn/plain open file and write echo abc > /mnt/lustre2/d33e.sanityn/f33e.sanityn append write echo dfg >> /mnt/lustre2/d33e.sanityn/f33e.sanityn setattr touch /mnt/lustre2/d33e.sanityn/f33e.sanityn file unlink rm /mnt/lustre2/d33e.sanityn/f33e.sanityn plain dir unlink rmdir /mnt/lustre2/d33e.sanityn/plain striped directory creation /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 0 -c 2 /mnt/lustre2/d33e.sanityn/striped set default LMV to create striped subdir /home/green/git/lustre-release/lustre/utils/lfs setdirstripe -D -c 2 /mnt/lustre/d33e.sanityn striped subdir creation createmany -d /mnt/lustre/d33e.sanityn/subdir 100 total: 100 mkdir in 0.50 seconds: 198.82 ops/second sub file creation and write createmany -o /mnt/lustre/d33e.sanityn/subfile 100; echo abc > /mnt/lustre/d33e.sanityn/subfile1 total: 100 open/close in 0.40 seconds: 251.54 ops/second sub file append write echo dfg >> /mnt/lustre2/d33e.sanityn/subfile2 subdir setatttr touch /mnt/lustre2/d33e.sanityn/subdir1 subdir unlink unlinkmany -d /mnt/lustre/d33e.sanityn/subdir 100 - unlinked 0 (time 1780808471 ; total 0 ; last 0) total: 100 unlinks in 3 seconds: 33.333332 unlinks/second sub file unlink unlinkmany /mnt/lustre2/d33e.sanityn/subfile 100 - unlinked 0 (time 1780808474 ; total 0 ; last 0) total: 100 unlinks in 1 seconds: 100.000000 unlinks/second sub file creation follows striped dir chmod chmod 777 /mnt/lustre/d33e.sanityn/striped; touch /mnt/lustre/d33e.sanityn/striped/subfile striped directory unlink rm -rf /mnt/lustre2/d33e.sanityn/striped directory unlink rm -rf /mnt/lustre2/d33e.sanityn CoS count 0 PASS 33e (8s) == sanityn test 34: no lock timeout under IO ============= 01:01:17 (1780808477) fail_loc=0x512 fail_loc=0x512 lock should not expire writing on client1 reading on client2 success fail_loc=0x511 fail_loc=0x511 lock should expire writing on client1 reading on client2 success oleg350-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8f15073b7800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8f15073b7800.ost_server_uuid in IDLE state after 0 sec oleg350-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8f15073b7800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8f15073b7800.ost_server_uuid in FULL state after 0 sec oleg350-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8f15073b7800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8f15073b7800.ost_server_uuid in IDLE state after 0 sec oleg350-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8f15073b7800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8f15073b7800.ost_server_uuid in FULL state after 0 sec oleg350-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8f15073b7800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8f15073b7800.ost_server_uuid in IDLE state after 0 sec oleg350-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8f15073b7800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8f15073b7800.ost_server_uuid in FULL state after 0 sec PASS 34 (87s) == sanityn test 35: -EINTR cp_ast vs. bl_ast race does not evict client ========================================================== 01:02:44 (1780808564) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d35.sanityn fail_loc=0x80000317 Race attempt 0 Wait for 55925 56057 for 60 sec... total: 4000 open/close in 8.96 seconds: 446.25 ops/second fail_loc=0x0 Filesystem Size Used Avail Use% Mounted on 192.168.203.150@tcp:/lustre 7.2G 40M 7.2G 1% /mnt/lustre 192.168.203.150@tcp:/lustre 7.2G 40M 7.2G 1% /mnt/lustre2 PASS 35 (65s) == sanityn test 36: handle ESTALE/open-unlink correctly == 01:03:49 (1780808629) striped dir -i0 -c2 -H crush /mnt/lustre/d36.sanityn Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg350-server mds-ost sync done. Waiting for MDT destroys to complete 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.442957 s, 118 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.8387 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(0) *** before(7497728) after_dd(7446528) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.446726 s, 117 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.8387 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(1) *** before(7497728) after_dd(7446528) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.41953 s, 125 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.8387 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(2) *** before(7497728) after_dd(7446528) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.455871 s, 115 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.8387 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(3) *** before(7497728) after_dd(7446528) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.501995 s, 104 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.8387 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(4) *** before(7497728) after_dd(7446528) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.449997 s, 117 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.8387 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(5) *** before(7497728) after_dd(7446528) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.44911 s, 117 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.8387 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(6) *** before(7497728) after_dd(7446528) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.465422 s, 113 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.8387 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(7) *** before(7497728) after_dd(7444480) after(7495680) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.425194 s, 123 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.8387 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(8) *** before(7495680) after_dd(7446528) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.434119 s, 121 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.8387 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(9) *** before(7497728) after_dd(7444480) after(7495680) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.403879 s, 130 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.8387 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(10) *** before(7495680) after_dd(7446528) after(7497728) PASS 36 (232s) == sanityn test 37: check i_size is not updated for directory on close (bug 18695) ======================================================================== 01:07:41 (1780808861) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d37.sanityn multiop /mnt/lustre/d37.sanityn vD_c TMPPIPE=/tmp/multiop_open_wait_pipe.8387 - create 8836 (time 1780808873.08 total 10.00 last 883.59) total: 10000 create in 11.56 seconds: 864.93 ops/second PASS 37 (15s) == sanityn test 39a: file mtime does not change after rename ========================================================== 01:07:56 (1780808876) repeat after cancel_lru_locks PASS 39a (2s) == sanityn test 39b: file mtime the same on clients with/out lock ========================================================== 01:07:58 (1780808878) repeat after cancel_lru_locks PASS 39b (3s) == sanityn test 39c: check truncate mtime update ================================================================================ 01:08:01 (1780808881) repeat after cancel_lru_locks PASS 39c (3s) == sanityn test 39d: sync write should update mtime ====== 01:08:04 (1780808884) fail_loc=0x411 fail_loc=0 PASS 39d (2s) SKIP: sanityn test_40a skipping ALWAYS excluded test 40a == sanityn test 40b: pdirops: open|create and others ======================================================================== 01:08:07 (1780808887) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=3 No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40b (8s) == sanityn test 40c: pdirops: link and others ======================================================================== 01:08:15 (1780808895) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40c (7s) == sanityn test 40d: pdirops: unlink and others ======================================================================== 01:08:22 (1780808902) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=8 No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40d (8s) == sanityn test 40e: pdirops: rename and others ======================================================================== 01:08:30 (1780808910) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=3 No conflict No conflict No conflict No conflict No conflict No conflict PASS 40e (7s) == sanityn test 41a: pdirops: create vs mkdir ======================================================================== 01:08:37 (1780808917) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=6 mkdir: cannot create directory '/mnt/lustre2/f41a.sanityn': File exists Conflict PASS 41a (5s) == sanityn test 41b: pdirops: create vs create ======================================================================== 01:08:42 (1780808922) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 open: File exists Conflict PASS 41b (5s) == sanityn test 41c: pdirops: create vs link ======================================================================== 01:08:48 (1780808928) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 link: cannot create link '/mnt/lustre2/f41c.sanityn' to '/mnt/lustre2/f41c.sanityn-2': File exists Conflict PASS 41c (5s) == sanityn test 41d: pdirops: create vs unlink ======================================================================== 01:08:53 (1780808933) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=2 Conflict PASS 41d (5s) == sanityn test 41e: pdirops: create and rename (tgt) ======================================================================== 01:08:58 (1780808938) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 41e (5s) == sanityn test 41f: pdirops: create and rename (src) ======================================================================== 01:09:03 (1780808943) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 41f (5s) == sanityn test 41g: pdirops: create vs getattr ======================================================================== 01:09:08 (1780808948) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=2 Conflict PASS 41g (5s) == sanityn test 41h: pdirops: create vs readdir ======================================================================== 01:09:13 (1780808953) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=2 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 2318: lctl set_param -n fail_loc=0 2>/dev/null || true: No such file or directory Conflict PASS 41h (5s) == sanityn test 41i: reint_open: create vs create ======== 01:09:18 (1780808958) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 41i (712s) == sanityn test 42a: pdirops: mkdir vs mkdir ======================================================================== 01:21:10 (1780809670) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 mkdir: cannot create directory '/mnt/lustre2/f42a.sanityn': File exists Conflict PASS 42a (5s) == sanityn test 42b: pdirops: mkdir vs create ======================================================================== 01:21:15 (1780809675) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 open: File exists Conflict PASS 42b (5s) == sanityn test 42c: pdirops: mkdir vs link ======================================================================== 01:21:20 (1780809680) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=2 ldlm.namespaces.mdt-lustre-MDT0001_UUID.lock_count=1 link: cannot create link '/mnt/lustre2/f42c.sanityn' to '/mnt/lustre2/f42c.sanityn-2': File exists Conflict PASS 42c (5s) == sanityn test 42d: pdirops: mkdir vs unlink ======================================================================== 01:21:25 (1780809685) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=3 Conflict PASS 42d (5s) == sanityn test 42e: pdirops: mkdir and rename (tgt) ======================================================================== 01:21:30 (1780809690) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=2 ldlm.namespaces.mdt-lustre-MDT0001_UUID.lock_count=1 mv: cannot overwrite directory '/mnt/lustre2/f42e.sanityn' with non-directory Conflict PASS 42e (5s) == sanityn test 42f: pdirops: mkdir and rename (src) ======================================================================== 01:21:35 (1780809695) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=5 Conflict PASS 42f (5s) == sanityn test 42g: pdirops: mkdir vs getattr ======================================================================== 01:21:40 (1780809700) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=4 ldlm.namespaces.mdt-lustre-MDT0001_UUID.lock_count=1 Conflict PASS 42g (5s) == sanityn test 42h: pdirops: mkdir vs readdir ======================================================================== 01:21:45 (1780809705) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 42h (5s) == sanityn test 43a: rmdir,mkdir doesn't return -EEXIST ======================================================================== 01:21:50 (1780809710) PASS 43a (36s) == sanityn test 43b: pdirops: unlink vs create ======================================================================== 01:22:26 (1780809746) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=4 ldlm.namespaces.mdt-lustre-MDT0001_UUID.lock_count=20 Conflict PASS 43b (6s) == sanityn test 43c: pdirops: unlink vs link ======================================================================== 01:22:32 (1780809752) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=2 Conflict PASS 43c (5s) == sanityn test 43d: pdirops: unlink vs unlink ======================================================================== 01:22:37 (1780809757) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=3 rm: cannot remove '/mnt/lustre2/f43d.sanityn': No such file or directory Conflict PASS 43d (5s) == sanityn test 43e: pdirops: unlink and rename (tgt) ======================================================================== 01:22:42 (1780809762) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 43e (5s) == sanityn test 43f: pdirops: unlink and rename (src) ======================================================================== 01:22:47 (1780809767) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=6 mv: cannot stat '/mnt/lustre2/f43f.sanityn': No such file or directory Conflict PASS 43f (5s) == sanityn test 43g: pdirops: unlink vs getattr ======================================================================== 01:22:52 (1780809772) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=2 stat: cannot statx '/mnt/lustre2/f43g.sanityn': No such file or directory Conflict PASS 43g (5s) == sanityn test 43h: pdirops: unlink vs readdir ======================================================================== 01:22:57 (1780809777) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 43h (5s) == sanityn test 43i: pdirops: unlink vs remote mkdir ===== 01:23:02 (1780809782) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 43i (6s) == sanityn test 43j: racy mkdir return EEXIST ======================================================================== 01:23:08 (1780809788) mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists PASS 43j (37s) == sanityn test 43k: unlink vs create ==================== 01:23:45 (1780809825) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 43k (423s) == sanityn test 44a: pdirops: rename tgt vs mkdir ======================================================================== 01:30:48 (1780810248) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=2 ldlm.namespaces.mdt-lustre-MDT0001_UUID.lock_count=2 mkdir: cannot create directory '/mnt/lustre2/f44a.sanityn': File exists Conflict Sun Jun 7 01:30:52 EDT 2026 PASS 44a (6s) == sanityn test 44b: pdirops: rename tgt vs create ======================================================================== 01:30:54 (1780810254) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 open: File exists Conflict PASS 44b (5s) == sanityn test 44c: pdirops: rename tgt vs link ======================================================================== 01:30:59 (1780810259) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 link: cannot create link '/mnt/lustre2/f44c.sanityn' to '/mnt/lustre2/f44c.sanityn-3': File exists Conflict PASS 44c (5s) == sanityn test 44d: pdirops: rename tgt vs unlink ======================================================================== 01:31:04 (1780810264) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 44d (5s) == sanityn test 44e: pdirops: rename tgt and rename (tgt) ======================================================================== 01:31:09 (1780810269) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 44e (5s) == sanityn test 44f: pdirops: rename tgt and rename (src) ======================================================================== 01:31:14 (1780810274) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 44f (5s) == sanityn test 44g: pdirops: rename tgt vs getattr ======================================================================== 01:31:19 (1780810279) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 44g (6s) == sanityn test 44h: pdirops: rename tgt vs readdir ======================================================================== 01:31:25 (1780810285) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 44h (5s) == sanityn test 44i: pdirops: rename tgt vs remote mkdir ========================================================== 01:31:30 (1780810290) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 lfs mkdir: dirstripe error on '/mnt/lustre2/f44i.sanityn': stripe already set lfs setdirstripe: cannot create dir '/mnt/lustre2/f44i.sanityn': File exists Conflict PASS 44i (5s) == sanityn test 45a: rename,mkdir doesn't return -EEXIST ======================================================================== 01:31:35 (1780810295) PASS 45a (45s) == sanityn test 45b: pdirops: rename src vs create ======================================================================== 01:32:20 (1780810340) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=72 Conflict PASS 45b (6s) == sanityn test 45c: pdirops: rename src vs link ======================================================================== 01:32:26 (1780810346) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 45c (5s) == sanityn test 45d: pdirops: rename src vs unlink ======================================================================== 01:32:31 (1780810351) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 rm: cannot remove '/mnt/lustre2/f45d.sanityn': No such file or directory Conflict PASS 45d (5s) == sanityn test 45e: pdirops: rename src and rename (tgt) ======================================================================== 01:32:36 (1780810356) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 45e (5s) == sanityn test 45f: pdirops: rename src and rename (src) ======================================================================== 01:32:41 (1780810361) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 mv: cannot stat '/mnt/lustre2/f45f.sanityn': No such file or directory Conflict PASS 45f (5s) == sanityn test 45g: pdirops: rename src vs getattr ======================================================================== 01:32:46 (1780810366) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 stat: cannot statx '/mnt/lustre2/f45g.sanityn': No such file or directory Conflict PASS 45g (5s) == sanityn test 45h: pdirops: unlink vs readdir ======================================================================== 01:32:51 (1780810371) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 45h (5s) == sanityn test 45i: pdirops: rename src vs remote mkdir ========================================================== 01:32:56 (1780810376) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 45i (5s) == sanityn test 45j: read vs rename ====================== 01:33:01 (1780810381) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 45j (428s) == sanityn test 46a: pdirops: link vs mkdir ======================================================================== 01:40:09 (1780810809) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=3 mkdir: cannot create directory '/mnt/lustre2/f46a.sanityn': File exists Conflict PASS 46a (4s) == sanityn test 46b: pdirops: link vs create ======================================================================== 01:40:14 (1780810814) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 open: File exists Conflict PASS 46b (5s) == sanityn test 46c: pdirops: link vs link ======================================================================== 01:40:18 (1780810818) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 link: cannot create link '/mnt/lustre2/f46c.sanityn' to '/mnt/lustre2/f46c.sanityn': File exists Conflict PASS 46c (5s) == sanityn test 46d: pdirops: link vs unlink ======================================================================== 01:40:23 (1780810823) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 46d (5s) == sanityn test 46e: pdirops: link and rename (tgt) ======================================================================== 01:40:28 (1780810828) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 46e (5s) == sanityn test 46f: pdirops: link and rename (src) ======================================================================== 01:40:33 (1780810833) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 46f (5s) == sanityn test 46g: pdirops: link vs getattr ======================================================================== 01:40:38 (1780810838) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 46g (4s) == sanityn test 46h: pdirops: link vs readdir ======================================================================== 01:40:42 (1780810842) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 46h (5s) == sanityn test 46i: pdirops: link vs remote mkdir ======= 01:40:47 (1780810847) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 lfs mkdir: dirstripe error on '/mnt/lustre2/f46i.sanityn': stripe already set lfs setdirstripe: cannot create dir '/mnt/lustre2/f46i.sanityn': File exists Conflict PASS 46i (5s) == sanityn test 47a: pdirops: remote mkdir vs mkdir ====== 01:40:52 (1780810852) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=7 mkdir: cannot create directory '/mnt/lustre2/f47a.sanityn': File exists Conflict PASS 47a (5s) == sanityn test 47b: pdirops: remote mkdir vs create ===== 01:40:57 (1780810857) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=2 ldlm.namespaces.mdt-lustre-MDT0001_UUID.lock_count=1 open: File exists Conflict PASS 47b (5s) == sanityn test 47c: pdirops: remote mkdir vs link ======= 01:41:02 (1780810862) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 link: cannot create link '/mnt/lustre2/f47c.sanityn' to '/mnt/lustre2/f47c.sanityn-2': File exists Conflict PASS 47c (6s) == sanityn test 47d: pdirops: remote mkdir vs unlink ===== 01:41:08 (1780810868) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=2 Conflict PASS 47d (4s) == sanityn test 47e: pdirops: remote mkdir and rename (tgt) ========================================================== 01:41:12 (1780810872) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 mv: cannot overwrite directory '/mnt/lustre2/f47e.sanityn' with non-directory Conflict PASS 47e (5s) == sanityn test 47f: pdirops: remote mkdir and rename (src) ========================================================== 01:41:17 (1780810877) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=2 Conflict PASS 47f (5s) == sanityn test 47g: pdirops: remote mkdir vs getattr ==== 01:41:22 (1780810882) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 47g (7s) == sanityn test 50: osc lvb attrs: enqueue vs. CP AST ======================================================================== 01:41:29 (1780810889) 10+0 records in 10+0 records out 10240 bytes (10 kB, 10 KiB) copied, 0.00498883 s, 2.1 MB/s fail_loc=0x410 fail_loc=0x0 PASS 50 (7s) == sanityn test 51a: layout lock: refresh layout should work ========================================================== 01:41:36 (1780810896) 0+1 records in 0+1 records out 158 bytes copied, 0.00366181 s, 43.1 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 3581: kill: (235807) - No such process PASS 51a (4s) == sanityn test 51b: layout lock: glimpse should be able to restart if layout changed ========================================================== 01:41:40 (1780810900) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00429863 s, 238 kB/s fail_loc=0x1404 fail_val=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155133 s, 67.6 MB/s 3145728 PASS 51b (14s) == sanityn test 51c: layout lock: IT_LAYOUT blocked and correct layout can be returned ========================================================== 01:41:54 (1780810914) fail_loc=0x172 Setting layout to have 2 stripes ... 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00403614 s, 254 kB/s PASS 51c (7s) == sanityn test 51d: layout lock: losing layout lock should clean up memory map region ========================================================== 01:42:01 (1780810921) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133021 s, 78.8 MB/s Before revoking layout lock: 1024 KB mapped PASS 51d (3s) == sanityn test 51e: lfs getstripe does not break leases, part 2 ========================================================== 01:42:04 (1780810924) /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 1290 0x50a 0x2c0000401 write lease(2) has applied. /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 1290 0x50a 0x2c0000401 read lease(1) has applied. PASS 51e (4s) == sanityn test 54: rename locking ======================= 01:42:08 (1780810928) llite.lustre-ffff8f15073b7800.xattr_cache=1 llite.lustre-ffff8f150925a000.xattr_cache=1 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 7680 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 162129603798762680 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-06-07 01:42:09.000000000 -0400 Modify: 2026-06-07 01:42:09.000000000 -0400 Change: 2026-06-07 01:42:09.000000000 -0400 Birth: 2026-06-07 01:42:08.000000000 -0400 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000154 File: /mnt/lustre/d1/d2 Size: 7680 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205289281262 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-06-07 01:42:14.000000000 -0400 Modify: 2026-06-07 01:42:14.000000000 -0400 Change: 2026-06-07 01:42:14.000000000 -0400 Birth: 2026-06-07 01:42:14.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 7680 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 162129603798762683 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-06-07 01:42:20.000000000 -0400 Modify: 2026-06-07 01:42:20.000000000 -0400 Change: 2026-06-07 01:42:20.000000000 -0400 Birth: 2026-06-07 01:42:19.000000000 -0400 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1 Size: 7680 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205289281263 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-06-07 01:42:20.000000000 -0400 Modify: 2026-06-07 01:42:20.000000000 -0400 Change: 2026-06-07 01:42:20.000000000 -0400 Birth: 2026-06-07 01:42:19.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000154 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1/d2 Size: 7680 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205289281265 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-06-07 01:42:25.000000000 -0400 Modify: 2026-06-07 01:42:26.000000000 -0400 Change: 2026-06-07 01:42:26.000000000 -0400 Birth: 2026-06-07 01:42:25.000000000 -0400 File: /mnt/lustre/d1 Size: 7680 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 162129603798762684 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-06-07 01:42:25.000000000 -0400 Modify: 2026-06-07 01:42:26.000000000 -0400 Change: 2026-06-07 01:42:26.000000000 -0400 Birth: 2026-06-07 01:42:25.000000000 -0400 PASS 54 (24s) == sanityn test 55a: rename vs unlink target dir ========= 01:42:32 (1780810952) fail_loc=0x80000156 mv: cannot move '/mnt/lustre/d55a.sanityn/d1/d2' to '/mnt/lustre/d55a.sanityn/d3/d2': No such file or directory PASS 55a (7s) == sanityn test 55b: rename vs unlink source dir ========= 01:42:39 (1780810959) fail_loc=0x80000156 mv: cannot stat '/mnt/lustre/d55b.sanityn/d1/d2': No such file or directory PASS 55b (8s) == sanityn test 55c: rename vs unlink orphan target dir == 01:42:47 (1780810967) fail_loc=0x156 mv: cannot move '/mnt/lustre/d55c.sanityn/d1/d2' to '/mnt/lustre/d55c.sanityn/d3/d2': No such file or directory PASS 55c (12s) == sanityn test 55d: rename file vs link ================= 01:42:59 (1780810979) fail_loc=0x155 mv: '/mnt/lustre/d55d.sanityn/f1' and '/mnt/lustre/d55d.sanityn/d55d.sanityn/f1' are the same file PASS 55d (10s) == sanityn test 55e: rename race AB/BA under the same parent dir ========================================================== 01:43:09 (1780810989) fail_loc=0x155 We survived after AB/BA race test fail_loc=0x0 PASS 55e (13s) == sanityn test 55f: rename: (P1/A -> P2/B) race with (P2/B -> P1/A) ========================================================== 01:43:22 (1780811002) fail_loc=0x155 We survived after P1A_P2B/P2B_P1A race test fail_loc=0x0 PASS 55f (13s) == sanityn test 55g: rename: race with trylock =========== 01:43:35 (1780811015) mdt.lustre-MDT0000.enable_rename_trylock=1 mdt.lustre-MDT0001.enable_rename_trylock=1 fail_loc=0x155 mv: cannot move '/mnt/lustre2/d55g.sanityn/b1/b2' to a subdirectory of itself, '/mnt/lustre2/d55g.sanityn/a1/a2/a3/a4/b2' We survived race with trylock test fail_loc=0x0 PASS 55g (14s) == sanityn test 56a: test llverdev with single large stripe ========================================================== 01:43:49 (1780811029) run_llverdev /mnt/lustre/f56a.sanityn -p -s 16000000 -c 4k llverdev: /mnt/lustre/f56a.sanityn is 16777216000000 bytes (15625 GB) in size timestamp: 1780811030 chunksize: 4096 size: 16777216000000 write offset: 16384000000kB 2.7745 MB/s read offset: 16384000000kB inf MB/s llverdev: data verified successfully PASS 56a (25s) == sanityn test 56b: test llverdev and partial verify of wide stripe file ========================================================== 01:44:14 (1780811054) run_llverdev /mnt/lustre/f56b.sanityn -p -s 32000000 -c 4k llverdev: /mnt/lustre/f56b.sanityn is 33554432000000 bytes (31250 GB) in size timestamp: 1780811055 chunksize: 4096 size: 33554432000000 write offset: 32768000000kB 3.69922 MB/s read offset: 32768000000kB inf MB/s llverdev: data verified successfully PASS 56b (37s) == sanityn test 60: Verify data_version behaviour ======== 01:44:51 (1780811091) striped dir -i0 -c2 -H crush /mnt/lustre/d60.sanityn PASS 60 (2s) == sanityn test 70a: cd directory && rm directory ======== 01:44:53 (1780811093) PASS 70a (3s) == sanityn test 70b: remove files after calling rm_entry ========================================================== 01:44:56 (1780811096) PASS 70b (2s) == sanityn test 71a: correct file map just after write operation is finished ========================================================== 01:44:58 (1780811098) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.00250209 s, 26.2 MB/s SKIP: sanityn test_71a ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71a (1s) == sanityn test 71b: check fiemap support for stripecount > 1 ========================================================== 01:45:00 (1780811100) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.00362248 s, 18.1 MB/s SKIP: sanityn test_71b ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71b (1s) == sanityn test 71c: check FIEMAP_EXTENT_LAST flag with different extents number ========================================================== 01:45:01 (1780811101) SKIP: sanityn test_71c support only ldiskfs ost SKIP 71c (1s) == sanityn test 71d: fiemap corruption test with fm_extent_count=0 ========================================================== 01:45:02 (1780811102) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.00403119 s, 16.3 MB/s SKIP: sanityn test_71d ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71d (1s) == sanityn test 72: getxattr/setxattr cache should be consistent between nodes ========================================================== 01:45:03 (1780811103) llite.lustre-ffff8f15073b7800.xattr_cache=1 llite.lustre-ffff8f150925a000.xattr_cache=1 getfattr: Removing leading '/' from absolute path names user.attr1="value1" getfattr: Removing leading '/' from absolute path names user.attr1="value2" getfattr: Removing leading '/' from absolute path names getfattr: Removing leading '/' from absolute path names b362740ad371eae580e7f85452e8830b - 870e072da1ebfc20ea8c0cd3d6bdf3b6 - PASS 72 (2s) == sanityn test 73: getxattr should not cause xattr lock cancellation ========================================================== 01:45:05 (1780811105) llite.lustre-ffff8f15073b7800.xattr_cache=1 llite.lustre-ffff8f150925a000.xattr_cache=1 getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" PASS 73 (2s) == sanityn test 74: flock deadlock: different mounts ======================================================================== 01:45:07 (1780811107) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00264575 s, 387 kB/s 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00278356 s, 368 kB/s 249054: get lock2 249056: get lock1 249054: done 249054: get lock1 249056: done 249056: get lock2 fcntl cmd 7 failed: Resource deadlock avoided 249054: cannot get lock1: Resource deadlock avoided fcntl cmd 7 failed: Resource deadlock avoided 249056: cannot get lock2: Resource deadlock avoided 249056: exit rc=0 249054: done 249054: exit rc=0 PASS 74 (6s) == sanityn test 75: osc: upcall after unuse lock============================================================================= 01:45:13 (1780811113) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0209764 s, 100 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0103868 s, 101 MB/s fail_loc=0x31d /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 4264: 249647 Killed stat -c %s $DIR1/$tfile fail_loc=0 PASS 75 (7s) == sanityn test 76: Verify MDT open_files listing ======== 01:45:20 (1780811120) striped dir -i0 -c2 -H crush /mnt/lustre/d76.sanityn opening files: ................ closing files: ................ PASS 76 (42s) == sanityn test 77a: check FIFO NRS policy =============== 01:46:02 (1780811162) ost.OSS.ost.nrs_policies=fifo ost.OSS.ost_create.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_out.nrs_policies=fifo ost.OSS.ost_seq.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.113425 s, 148 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0346663 s, 30.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0214989 s, 48.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0637837 s, 16.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104899 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100604 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0990982 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0933925 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104356 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0930898 s, 11.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121092 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0950594 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105335 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1056 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12633 s, 8.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8f150925a000.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff8f150925a000.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.099186 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138449 s, 7.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152879 s, 68.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.018319 s, 57.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0237212 s, 44.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0296785 s, 35.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0318805 s, 32.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0291771 s, 35.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0205068 s, 51.1 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0281322 s, 37.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0233631 s, 44.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0225927 s, 46.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.028173 s, 37.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00856876 s, 122 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0241739 s, 43.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0341983 s, 30.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0270134 s, 38.8 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8f150925a000.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0115749 s, 90.6 MB/s removed '/mnt/lustre/d77a.sanityn/nrs_w_oleg350-client.virtnet' removed '/mnt/lustre/d77a.sanityn/nrs_r_oleg350-client.virtnet' removed directory '/mnt/lustre/d77a.sanityn' PASS 77a (3s) == sanityn test 77b: check CRR-N NRS policy ============== 01:46:05 (1780811165) ost.OSS.ost.nrs_policies=crrn ost.OSS.ost_create.nrs_policies=crrn ost.OSS.ost_io.nrs_policies=crrn ost.OSS.ost_out.nrs_policies=crrn ost.OSS.ost_seq.nrs_policies=crrn ost.OSS.ost.nrs_crrn_quantum=1 ost.OSS.ost_create.nrs_crrn_quantum=1 ost.OSS.ost_io.nrs_crrn_quantum=1 ost.OSS.ost_out.nrs_crrn_quantum=1 ost.OSS.ost_seq.nrs_crrn_quantum=1 policy: crr-n, crrn_quantum 1 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.11787 s, 142 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0331947 s, 31.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0325872 s, 32.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.033488 s, 31.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0623839 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0524848 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0574981 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0685026 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0511541 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0531238 s, 19.7 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8f150925a000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0943166 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105683 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116335 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123276 s, 8.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129203 s, 8.1 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122516 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123104 s, 8.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0224255 s, 46.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0210097 s, 49.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0289197 s, 36.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141566 s, 74.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0191745 s, 54.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132916 s, 78.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0200816 s, 52.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00723176 s, 145 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0223633 s, 46.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151375 s, 69.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00833921 s, 126 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155381 s, 67.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0308274 s, 34.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0106695 s, 98.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00428916 s, 244 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8f150925a000.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffff8f150925a000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00955117 s, 110 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg350-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg350-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost.nrs_crrn_quantum=64 ost.OSS.ost_create.nrs_crrn_quantum=64 ost.OSS.ost_io.nrs_crrn_quantum=64 ost.OSS.ost_out.nrs_crrn_quantum=64 ost.OSS.ost_seq.nrs_crrn_quantum=64 policy: crr-n, crrn_quantum 64 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.130707 s, 128 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0347381 s, 30.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0535751 s, 19.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0643087 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0661073 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0633733 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0709968 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0653415 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0720872 s, 14.5 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8f150925a000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.090901 s, 11.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0762224 s, 13.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0884343 s, 11.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0844464 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101965 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0679253 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0939998 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110184 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0058435 s, 179 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0124711 s, 84.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00340675 s, 308 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0174784 s, 60.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0196857 s, 53.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00827652 s, 127 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0236701 s, 44.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00958206 s, 109 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00740367 s, 142 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149805 s, 70.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137292 s, 76.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0103752 s, 101 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00919959 s, 114 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0120361 s, 87.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161066 s, 65.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0324162 s, 32.3 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8f150925a000.lock_unused_count=2 removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg350-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg350-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77b (4s) == sanityn test 77c: check ORR NRS policy ================ 01:46:09 (1780811169) ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_orr_quantum=1 ost.OSS.ost_io.nrs_orr_offset_type=physical ost.OSS.ost_io.nrs_orr_supported=reads policy: orr, orr_quantum 1, orr_offset_type physical, orr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.10377 s, 162 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0278937 s, 37.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0303687 s, 34.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8f150925a000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.095214 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102251 s, 10.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118311 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120561 s, 8.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108932 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106742 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121353 s, 8.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102878 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119595 s, 8.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100661 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124571 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128854 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129118 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125212 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016078 s, 65.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0119264 s, 87.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0105475 s, 99.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0244951 s, 42.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00495096 s, 212 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00571998 s, 183 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00348776 s, 301 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0188763 s, 55.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00401662 s, 261 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00425846 s, 246 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00359555 s, 292 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8f150925a000.lock_unused_count=5 ldlm.namespaces.lustre-OST0001-osc-ffff8f150925a000.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00950092 s, 110 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0245012 s, 42.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0120087 s, 87.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0040528 s, 259 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00472672 s, 222 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg350-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg350-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=writes ost.OSS.ost_io.nrs_orr_quantum=64 policy: orr, orr_quantum 64, orr_offset_type physical, orr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.103811 s, 162 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0217762 s, 48.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0303158 s, 34.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0331708 s, 31.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0725074 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0492756 s, 21.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0519572 s, 20.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0852771 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0949149 s, 11.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8f150925a000.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff8f150925a000.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.093176 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118129 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106271 s, 9.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118153 s, 8.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103083 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116005 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107642 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11712 s, 9.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8f150925a000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff8f150925a000.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133389 s, 78.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156892 s, 66.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137706 s, 76.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138655 s, 75.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137938 s, 76.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156765 s, 66.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0199071 s, 52.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00397701 s, 264 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134229 s, 78.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00980107 s, 107 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00632575 s, 166 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00408281 s, 257 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0249265 s, 42.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00950681 s, 110 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00909654 s, 115 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00996579 s, 105 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg350-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg350-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=reads_and_writes ost.OSS.ost_io.nrs_orr_offset_type=logical policy: orr, orr_quantum 64, orr_offset_type logical, orr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.145245 s, 116 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0210591 s, 49.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0524885 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0635752 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0937839 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0862925 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116385 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10604 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100609 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111951 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10848 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116008 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107161 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113465 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131839 s, 8.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0110254 s, 95.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00725747 s, 144 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00245564 s, 427 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.02568 s, 40.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0248921 s, 42.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13726 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0218424 s, 48.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.03322 s, 31.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169689 s, 61.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00367677 s, 285 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00384222 s, 273 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149387 s, 70.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122159 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.021102 s, 49.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00365586 s, 287 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00674369 s, 155 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0219787 s, 47.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8f150925a000.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff8f150925a000.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00253198 s, 414 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg350-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg350-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77c (4s) == sanityn test 77d: check TRR nrs policy ================ 01:46:13 (1780811173) ost.OSS.ost_io.nrs_policies=trr ost.OSS.ost_io.nrs_trr_quantum=1 ost.OSS.ost_io.nrs_trr_offset_type=physical ost.OSS.ost_io.nrs_trr_supported=reads policy: trr, trr_quantum 1, trr_offset_type physical, trr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.105715 s, 159 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0292371 s, 35.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0316721 s, 33.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0388152 s, 27.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0646657 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0572951 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0696202 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.066895 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0866921 s, 12.1 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8f150925a000.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0910988 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.091597 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103507 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0989753 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0884292 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107501 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.03523 s, 29.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0499601 s, 21.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8f150925a000.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0179263 s, 58.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144574 s, 72.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0232249 s, 45.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0110385 s, 95.0 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0213222 s, 49.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0202765 s, 51.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0258093 s, 40.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.021644 s, 48.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00927998 s, 113 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0214882 s, 48.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0086685 s, 121 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0245347 s, 42.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0120874 s, 86.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0223379 s, 46.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00832236 s, 126 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0221958 s, 47.2 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg350-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg350-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=writes ost.OSS.ost_io.nrs_trr_quantum=64 policy: trr, trr_quantum 64, trr_offset_type physical, trr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.118506 s, 142 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0424168 s, 24.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0599523 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0608989 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0669683 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.079828 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0745821 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0813432 s, 12.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0675031 s, 15.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0740847 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0799957 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1059 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0985213 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103021 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.09084 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0905795 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0939858 s, 11.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0201166 s, 52.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00949156 s, 110 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147576 s, 71.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155691 s, 67.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0194499 s, 53.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0230598 s, 45.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0218538 s, 48.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00631519 s, 166 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0196004 s, 53.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8f150925a000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff8f150925a000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0198894 s, 52.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014777 s, 71.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143593 s, 73.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00867355 s, 121 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0269053 s, 39.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131495 s, 79.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00156278 s, 671 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg350-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg350-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=reads_and_writes ost.OSS.ost_io.nrs_trr_offset_type=logical policy: trr, trr_quantum 64, trr_offset_type logical, trr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.116614 s, 144 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0191993 s, 54.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0552727 s, 19.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0560556 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.046528 s, 22.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0831849 s, 12.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0758065 s, 13.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.097101 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0833623 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0821392 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0713397 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10695 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0867524 s, 12.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110483 s, 9.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109343 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118992 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122153 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127514 s, 82.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145277 s, 72.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0192954 s, 54.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144985 s, 72.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0080901 s, 130 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00754552 s, 139 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.02561 s, 40.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0233838 s, 44.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.025442 s, 41.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0174562 s, 60.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00634394 s, 165 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0224288 s, 46.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0111423 s, 94.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0063246 s, 166 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157092 s, 66.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0344457 s, 30.4 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8f150925a000.lock_unused_count=1 removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg350-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg350-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77d (5s) == sanityn test 77e: check TBF NID nrs policy ============ 01:46:18 (1780811178) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_tbf_rule=start localhost nid={0@lo} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start clients nid={192.168.203.50@tcp} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start others nid={*.*.*.*@tcp} rate=50 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.186153 s, 90.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0222888 s, 47.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0296796 s, 35.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0672428 s, 15.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0683551 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0609884 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.054537 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0672947 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0635065 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0703163 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.052505 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0919193 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0615515 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0944995 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0881601 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105807 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11162 s, 9.4 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8f150925a000.lock_unused_count=4 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0426794 s, 24.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0526029 s, 19.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0413416 s, 25.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0504913 s, 20.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0397516 s, 26.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0523917 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.049169 s, 21.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.061406 s, 17.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.052002 s, 20.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0471842 s, 22.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0494365 s, 21.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0505053 s, 20.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0656534 s, 16.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0539224 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0467185 s, 22.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0519967 s, 20.2 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg350-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg350-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change localhost rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change clients rate=101 ost.OSS.ost_io.nrs_tbf_rule=change others rate=51 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.129247 s, 130 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0269078 s, 39.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0435775 s, 24.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0752094 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0486537 s, 21.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0727959 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0803718 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0743115 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0700796 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0797764 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0991081 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0994908 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0949586 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0903017 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0788651 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0996251 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0911952 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137076 s, 76.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0197784 s, 53.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133472 s, 78.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135152 s, 77.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0486127 s, 21.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.023705 s, 44.2 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0233863 s, 44.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0217339 s, 48.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.029199 s, 35.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0518112 s, 20.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0538987 s, 19.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.020939 s, 50.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.049727 s, 21.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0318839 s, 32.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.027092 s, 38.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0287348 s, 36.5 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg350-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg350-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop localhost ost.OSS.ost_io.nrs_tbf_rule=stop clients ost.OSS.ost_io.nrs_tbf_rule=stop others 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.114174 s, 147 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0300625 s, 34.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.035726 s, 29.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0827291 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107166 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10111 s, 10.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121886 s, 8.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111841 s, 9.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108232 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116114 s, 9.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8f150925a000.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff8f150925a000.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128435 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124175 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12841 s, 8.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104433 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116935 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10692 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106333 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150879 s, 69.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149346 s, 70.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137523 s, 76.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155567 s, 67.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0231279 s, 45.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183784 s, 57.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00342321 s, 306 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016131 s, 65.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0231985 s, 45.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013909 s, 75.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016486 s, 63.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00529835 s, 198 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0121276 s, 86.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0072338 s, 145 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00324072 s, 324 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00687485 s, 153 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg350-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg350-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.120183 s, 140 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0208245 s, 50.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0317538 s, 33.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0887123 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0909404 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0827831 s, 12.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112516 s, 9.3 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0895891 s, 11.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0960228 s, 10.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108325 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108473 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104169 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0992329 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12436 s, 8.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8f150925a000.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff8f150925a000.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0973253 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112526 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119829 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140985 s, 74.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0105421 s, 99.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146066 s, 71.8 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8f150925a000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169244 s, 62.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0051242 s, 205 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139003 s, 75.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00377589 s, 278 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00451438 s, 232 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0035178 s, 298 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0113235 s, 92.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00417572 s, 251 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00430064 s, 244 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0116872 s, 89.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00392854 s, 267 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00369464 s, 284 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00948188 s, 111 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg350-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg350-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' PASS 77e (7s) == sanityn test 77f: check TBF JobID nrs policy ========== 01:46:25 (1780811185) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start runas jobid={iozone.500 dd.500 tiotest.500} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start iozone_runas jobid={iozone.500} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg350-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.159636 s, 105 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.04494 s, 23.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0676309 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0669524 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0864635 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0754016 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0759404 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0925651 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0874192 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101691 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0992723 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10725 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105975 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0956635 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109825 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103566 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123397 s, 8.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016803 s, 62.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134225 s, 78.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0203013 s, 51.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0222967 s, 47.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0218655 s, 48.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0104839 s, 100 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00946218 s, 111 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0184114 s, 57.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0176581 s, 59.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0237788 s, 44.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0260393 s, 40.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00357617 s, 293 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00897186 s, 117 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013581 s, 77.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0256129 s, 40.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00572491 s, 183 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg350-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg350-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change runas rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change iozone_runas rate=101 ost.OSS.ost_io.nrs_tbf_rule=change dd_runas rate=51 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg350-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.102905 s, 163 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0186598 s, 56.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0802524 s, 13.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0828059 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0821156 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0867154 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0870839 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0856305 s, 12.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0797043 s, 13.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0882915 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0753248 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.088384 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0855204 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0953853 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0941358 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0763234 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0902258 s, 11.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00898034 s, 117 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015389 s, 68.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0171251 s, 61.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135473 s, 77.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0113638 s, 92.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00827998 s, 127 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170196 s, 61.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00387688 s, 270 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0223557 s, 46.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0207508 s, 50.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00293593 s, 357 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0211929 s, 49.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0228201 s, 45.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00359981 s, 291 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00541884 s, 194 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0017481 s, 600 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg350-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg350-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop runas ost.OSS.ost_io.nrs_tbf_rule=stop iozone_runas ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg350-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.127329 s, 132 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0258387 s, 40.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0278323 s, 37.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0858825 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.093522 s, 11.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0741524 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0975991 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104866 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103189 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0996004 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103052 s, 10.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8f150925a000.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffff8f150925a000.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101706 s, 10.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105929 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103895 s, 10.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124266 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127048 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127379 s, 8.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied ldlm.namespaces.lustre-OST0001-osc-ffff8f150925a000.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0100633 s, 104 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151347 s, 69.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016308 s, 64.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165188 s, 63.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0268752 s, 39.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00709639 s, 148 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0217265 s, 48.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163065 s, 64.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0225249 s, 46.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.017441 s, 60.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0112713 s, 93.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00786593 s, 133 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00774595 s, 135 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.02384 s, 44.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0312082 s, 33.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00828664 s, 127 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg350-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg350-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_policies=fifo running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg350-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.119929 s, 140 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0296905 s, 35.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0483937 s, 21.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0242364 s, 43.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0632336 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0582497 s, 18.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8f150925a000.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0874276 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0783051 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0941889 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0908491 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112119 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0968422 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107368 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123355 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11043 s, 9.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied ldlm.namespaces.lustre-OST0001-osc-ffff8f150925a000.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123014 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121588 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00959405 s, 109 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153432 s, 68.3 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0226846 s, 46.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0209698 s, 50.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0244036 s, 43.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012916 s, 81.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0271308 s, 38.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0200201 s, 52.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0326333 s, 32.1 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0300031 s, 34.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158 s, 66.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0203707 s, 51.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0234366 s, 44.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0086791 s, 121 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0109004 s, 96.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0114317 s, 91.7 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg350-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg350-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' PASS 77f (7s) == sanityn test 77g: Change TBF type directly ============ 01:46:32 (1780811192) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 ost.OSS.ost_io.nrs_policies=fifo PASS 77g (3s) == sanityn test 77h: Wrong policy name should report error, not LBUG ========================================================== 01:46:35 (1780811195) oleg350-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=abc: No such file or directory oleg350-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='abc': No such file or directory pdsh@oleg350-client: oleg350-server: ssh exited with exit code 2 oleg350-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc: Invalid argument oleg350-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc': Invalid argument pdsh@oleg350-client: oleg350-server: ssh exited with exit code 22 oleg350-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf reg abc: Invalid argument oleg350-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf reg abc': Invalid argument pdsh@oleg350-client: oleg350-server: ssh exited with exit code 22 oleg350-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc efg: Invalid argument oleg350-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc efg': Invalid argument pdsh@oleg350-client: oleg350-server: ssh exited with exit code 22 PASS 77h (4s) == sanityn test 77i: Change rank of TBF rule ============= 01:46:39 (1780811199) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start before jobid={jobid} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start after jobid={jobid} rate=1000 rank=default ost.OSS.ost_io.nrs_tbf_rule=start target jobid={jobid} rate=1000 rank=after Move before itself ost.OSS.ost_io.nrs_tbf_rule=change target rank=target Move to higher rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=before Move to lower rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=after Move before default ost.OSS.ost_io.nrs_tbf_rule=change target rank=default ost.OSS.ost_io.nrs_policies=fifo PASS 77i (6s) == sanityn test 77j: check TBF-OPCode NRS policy ========= 01:46:45 (1780811205) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf opcode ost.OSS.ost_io.nrs_tbf_rule=start ost_r opcode={ost_read} rate=5 ost.OSS.ost_io.nrs_tbf_rule=start ost_w opcode={ost_write} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.121257 s, 138 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0400009 s, 26.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0511048 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0507171 s, 20.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0537122 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0554725 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0706833 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0580627 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0577751 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0582444 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0587045 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0503016 s, 20.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0778348 s, 13.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104877 s, 10.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0910892 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0988133 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101408 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.018132 s, 57.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0115751 s, 90.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0346719 s, 30.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0191771 s, 54.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0033753 s, 311 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00393653 s, 266 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0264959 s, 39.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0350444 s, 29.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0213565 s, 49.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0228847 s, 45.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0207171 s, 50.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0229431 s, 45.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0273011 s, 38.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00578511 s, 181 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0426146 s, 24.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00388367 s, 270 MB/s removed '/mnt/lustre/d77j.sanityn/nrs_w_oleg350-client.virtnet' removed '/mnt/lustre/d77j.sanityn/nrs_r_oleg350-client.virtnet' removed directory '/mnt/lustre/d77j.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 5 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.92732 s, 21.3 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.4274 s, 5.4 MB/s Read runtime is 20 s, speed is 5.000000 IOPS cleanup_tbf /mnt/lustre/d77j.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_r ost.OSS.ost_io.nrs_tbf_rule=stop ost_w ost.OSS.ost_io.nrs_policies=fifo PASS 77j (46s) == sanityn test 77ja: check TBF-UID/GID NRS policy ======= 01:47:31 (1780811251) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg350-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg350-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg350-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg350-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg350-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.445694 s, 37.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0599871 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0648806 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0839272 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0822923 s, 12.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0948644 s, 11.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.076521 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0896996 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0663701 s, 15.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121504 s, 8.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112873 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147385 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13619 s, 7.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143261 s, 7.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132418 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137185 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136563 s, 7.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0224142 s, 46.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0225006 s, 46.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168423 s, 62.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.024481 s, 42.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0224787 s, 46.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0188629 s, 55.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0184775 s, 56.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168034 s, 62.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0251227 s, 41.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0216899 s, 48.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00704739 s, 149 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0049255 s, 213 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00380617 s, 275 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0194068 s, 54.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0274896 s, 38.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00474858 s, 221 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg350-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg350-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8247 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9541 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg350-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg350-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg350-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg350-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg350-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.452877 s, 37.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0696986 s, 15.0 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.074748 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102848 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0758139 s, 13.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100641 s, 10.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0983476 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.094104 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0953482 s, 11.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108039 s, 9.7 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119108 s, 8.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113167 s, 9.3 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11802 s, 8.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114947 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123241 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138104 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122074 s, 8.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00488933 s, 214 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154567 s, 67.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0115084 s, 91.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016632 s, 63.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0194813 s, 53.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0184623 s, 56.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0179547 s, 58.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00661687 s, 158 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161092 s, 65.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0230787 s, 45.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0209637 s, 50.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00531812 s, 197 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0225837 s, 46.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00354517 s, 296 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00485981 s, 216 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00149616 s, 701 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg350-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg350-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8273 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.959 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77ja (127s) == sanityn test 77jb: check TBF-UID/GID NRS policy on files that don't belong to us ========================================================== 01:49:38 (1780811378) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg350-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg350-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg350-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg350-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg350-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.2221 s, 13.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0776844 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.074793 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0706483 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0791214 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0733977 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0810708 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0823695 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0798939 s, 13.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16252 s, 6.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162554 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141118 s, 7.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162905 s, 6.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178811 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174855 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171998 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166759 s, 6.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147174 s, 71.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0178326 s, 58.8 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0207216 s, 50.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0220211 s, 47.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0311686 s, 33.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0256302 s, 40.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00758638 s, 138 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0336001 s, 31.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0234135 s, 44.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147838 s, 70.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00938317 s, 112 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0252096 s, 41.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0111462 s, 94.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0334032 s, 31.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0248201 s, 42.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00179312 s, 585 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg350-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg350-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8278 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9598 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg350-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg350-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg350-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg350-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg350-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.444469 s, 37.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0415557 s, 25.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0514684 s, 20.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0636598 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0619393 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0747246 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0669684 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0893148 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0752558 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123156 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105359 s, 10.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126756 s, 8.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128826 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12506 s, 8.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145059 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15136 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142109 s, 7.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0226743 s, 46.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0230729 s, 45.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.020698 s, 50.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0175941 s, 59.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00407347 s, 257 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.031121 s, 33.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0273361 s, 38.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00419738 s, 250 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00904116 s, 116 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0103083 s, 102 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00882514 s, 119 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150735 s, 69.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139997 s, 74.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0108787 s, 96.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00520495 s, 201 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.025226 s, 41.6 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg350-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg350-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.6587 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9528 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77jb (133s) == sanityn test 77k: check TBF policy with UID/GID/JobID/OPCode expression ========================================================== 01:51:51 (1780811511) ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg350-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.242377 s, 69.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0335846 s, 31.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0534428 s, 19.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0698918 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.056341 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0684412 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0884602 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0969739 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0863016 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.098021 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0804076 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106193 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0779768 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0848145 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0916133 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0299788 s, 35.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0997234 s, 10.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0104309 s, 101 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01124 s, 93.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168943 s, 62.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141658 s, 74.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158297 s, 66.2 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0190997 s, 54.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127173 s, 82.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157839 s, 66.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00929128 s, 113 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0237435 s, 44.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0213568 s, 49.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0107838 s, 97.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00843616 s, 124 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0244829 s, 42.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00529703 s, 198 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00457982 s, 229 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg350-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg350-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.91753 s, 21.3 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.72582 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.203.50@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.203.50@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.12267 s, 137 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0277131 s, 37.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0251158 s, 41.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0718806 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0595021 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0825516 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.082395 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0727397 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104334 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0817022 s, 12.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109062 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0985381 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0949267 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184886 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174013 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222474 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215011 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0117117 s, 89.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126548 s, 82.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125744 s, 83.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0185496 s, 56.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014694 s, 71.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0111096 s, 94.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00389815 s, 269 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183796 s, 57.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158149 s, 66.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0204487 s, 51.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0212789 s, 49.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00380561 s, 276 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00442711 s, 237 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00386825 s, 271 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0252067 s, 41.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00392261 s, 267 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg350-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg350-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.91772 s, 21.3 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.72704 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.203.50@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg350-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.122757 s, 137 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0252352 s, 41.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0485728 s, 21.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0563543 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0459553 s, 22.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0542675 s, 19.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0485939 s, 21.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0558758 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0838875 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0776981 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106277 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102732 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103289 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106468 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100011 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103047 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111799 s, 9.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128636 s, 81.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169804 s, 61.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157272 s, 66.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0186148 s, 56.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140241 s, 74.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00478704 s, 219 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0235646 s, 44.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00417576 s, 251 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0262064 s, 40.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0217497 s, 48.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0105316 s, 99.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0228515 s, 45.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00343048 s, 306 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0258877 s, 40.5 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0265908 s, 39.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00857822 s, 122 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg350-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg350-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.91975 s, 21.3 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.87512 s, 21.5 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg350-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.12946 s, 130 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0241397 s, 43.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.027557 s, 38.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0768259 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0727578 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101878 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0885106 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0975278 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10921 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0912352 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102032 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0922508 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.096301 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173904 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160279 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208965 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.214297 s, 4.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133611 s, 78.5 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134377 s, 78.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0190078 s, 55.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151371 s, 69.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0182989 s, 57.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0198896 s, 52.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01137 s, 92.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0210577 s, 49.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00498629 s, 210 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0206021 s, 50.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0186701 s, 56.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00373087 s, 281 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.021208 s, 49.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0248541 s, 42.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00210991 s, 497 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00190942 s, 549 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg350-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg350-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.76724 s, 10.7 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.72653 s, 10.8 MB/s Read runtime is 10 s, speed is 10.000000 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.89756 s, 21.4 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.72366 s, 10.8 MB/s Read runtime is 10 s, speed is 10.000000 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg350-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.838456 s, 20.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0213296 s, 49.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0384145 s, 27.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0473838 s, 22.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0566761 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0650169 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0664089 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.067365 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0558096 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165766 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191974 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17521 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188192 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193503 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20606 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190116 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190553 s, 5.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138431 s, 75.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154901 s, 67.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144021 s, 72.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129976 s, 80.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153653 s, 68.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0182591 s, 57.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00411939 s, 255 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170415 s, 61.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161306 s, 65.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0192038 s, 54.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164803 s, 63.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00452575 s, 232 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0230564 s, 45.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00736848 s, 142 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129204 s, 81.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0224178 s, 46.8 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg350-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg350-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8233 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.4257 s, 5.4 MB/s Read runtime is 20 s, speed is 5.000000 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_ug ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg350-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.119099 s, 141 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0503796 s, 20.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0750267 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112902 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127746 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11481 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109307 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128449 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133657 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144877 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15287 s, 6.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153771 s, 6.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14232 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156136 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173038 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149419 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180267 s, 5.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147487 s, 71.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151513 s, 69.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0184851 s, 56.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0214125 s, 49.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00901289 s, 116 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0230699 s, 45.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0304982 s, 34.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0200102 s, 52.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0289528 s, 36.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146543 s, 71.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153789 s, 68.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.031557 s, 33.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.011996 s, 87.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0178903 s, 58.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00438319 s, 239 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00949725 s, 110 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg350-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg350-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.93046 s, 21.3 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.7286 s, 10.8 MB/s Read runtime is 10 s, speed is 10.000000 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg350-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.113002 s, 148 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0200878 s, 52.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0503351 s, 20.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0542034 s, 19.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0711982 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0517026 s, 20.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0670437 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0735005 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0623231 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0670166 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0848872 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0934245 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0943838 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168669 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172111 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.239323 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.235038 s, 4.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137064 s, 76.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151622 s, 69.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133961 s, 78.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00343091 s, 306 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141359 s, 74.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0043791 s, 239 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0117435 s, 89.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0243967 s, 43.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00392342 s, 267 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00393984 s, 266 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.019151 s, 54.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00825198 s, 127 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0120372 s, 87.1 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0043324 s, 242 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00977698 s, 107 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00402216 s, 261 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg350-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg350-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.77393 s, 10.7 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.72645 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.76206 s, 10.7 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.72759 s, 10.8 MB/s Read runtime is 10 s, speed is 10.000000 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_policies=fifo PASS 77k (361s) == sanityn test 77kb: Check different granular TBF type combination: jobid+opcode ========================================================== 01:57:52 (1780811872) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/nrs_r_oleg350-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.244495 s, 68.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0698901 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0776985 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0929495 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105282 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124361 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132099 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133841 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13319 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13077 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13031 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135716 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120174 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132721 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130653 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129331 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137424 s, 7.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128427 s, 81.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129307 s, 81.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0171679 s, 61.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0213968 s, 49.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0178814 s, 58.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0285609 s, 36.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0272798 s, 38.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00828955 s, 126 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167671 s, 62.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0241321 s, 43.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145741 s, 71.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.018611 s, 56.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166075 s, 63.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0120017 s, 87.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0212346 s, 49.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150524 s, 69.7 MB/s removed '/mnt/lustre/d77kb.sanityn/nrs_w_oleg350-client.virtnet' removed '/mnt/lustre/d77kb.sanityn/nrs_r_oleg350-client.virtnet' removed directory '/mnt/lustre/d77kb.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.91532 s, 21.3 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.72544 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kb (35s) == sanityn test 77kc: Check different granular TBF type combination: nid+opcode ========================================================== 01:58:27 (1780811907) ost.OSS.ost_io.nrs_policies=tbf nid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.203.50@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.203.50@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.117276 s, 143 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0629934 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0639577 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.080339 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0739459 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0801053 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.083528 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0912683 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0776936 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0913812 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126422 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130295 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152555 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176071 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174988 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227329 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.217308 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0174911 s, 59.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0101145 s, 104 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167649 s, 62.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0216112 s, 48.5 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0260306 s, 40.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0231219 s, 45.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00403652 s, 260 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0353954 s, 29.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.02745 s, 38.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0354119 s, 29.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0205829 s, 50.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159316 s, 65.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0243359 s, 43.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142183 s, 73.7 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8f150925a000.lock_unused_count=1 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138497 s, 75.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0219641 s, 47.7 MB/s removed '/mnt/lustre/d77kc.sanityn/nrs_w_oleg350-client.virtnet' removed '/mnt/lustre/d77kc.sanityn/nrs_r_oleg350-client.virtnet' removed directory '/mnt/lustre/d77kc.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.9162 s, 21.3 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.72739 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kc.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kc (38s) == sanityn test 77kd: Check different granular TBF type combination: nid+jobid ========================================================== 01:59:05 (1780811945) ost.OSS.ost_io.nrs_policies=tbf nid+jobid ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.203.50@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/nrs_r_oleg350-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.246302 s, 68.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.022346 s, 46.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0363214 s, 28.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0660361 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0811848 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0721005 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0862687 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0992077 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0829272 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0875096 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0885919 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0934085 s, 11.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0875727 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0848531 s, 12.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108581 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103647 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0957327 s, 11.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00848808 s, 124 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0179962 s, 58.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167168 s, 62.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0171153 s, 61.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0182052 s, 57.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0201596 s, 52.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.020007 s, 52.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126103 s, 83.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0236365 s, 44.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0205222 s, 51.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00326812 s, 321 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013854 s, 75.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0244521 s, 42.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0111066 s, 94.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0091702 s, 114 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0024196 s, 433 MB/s removed '/mnt/lustre/d77kd.sanityn/nrs_w_oleg350-client.virtnet' removed '/mnt/lustre/d77kd.sanityn/nrs_r_oleg350-client.virtnet' removed directory '/mnt/lustre/d77kd.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.9196 s, 21.3 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kd.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.96525 s, 21.1 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kd.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kd (34s) == sanityn test 77ke: Check different granular TBF type combination: jobid+opcode ========================================================== 01:59:39 (1780811979) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/nrs_r_oleg350-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.461845 s, 36.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0367087 s, 28.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0318361 s, 32.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0641621 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0689507 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0555096 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0625106 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.058472 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108957 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107278 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107946 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114151 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12239 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.301372 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.290589 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.355013 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.355026 s, 3.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0117599 s, 89.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0182463 s, 57.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0119574 s, 87.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0100473 s, 104 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0187103 s, 56.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0191218 s, 54.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0249063 s, 42.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0100725 s, 104 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173487 s, 60.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0189223 s, 55.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0067174 s, 156 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0241909 s, 43.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0226725 s, 46.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00708659 s, 148 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00237852 s, 441 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00178437 s, 588 MB/s removed '/mnt/lustre/d77ke.sanityn/nrs_r_oleg350-client.virtnet' removed '/mnt/lustre/d77ke.sanityn/nrs_w_oleg350-client.virtnet' removed directory '/mnt/lustre/d77ke.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.76531 s, 10.7 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ke.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.72662 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.90216 s, 21.4 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.7258 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77ke (73s) == sanityn test 77kf: Check different granular TBF type combination: uid+gid ========================================================== 02:00:52 (1780812052) ost.OSS.ost_io.nrs_policies=tbf uid+gid ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/nrs_r_oleg350-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.113197 s, 148 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.024408 s, 43.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0460889 s, 22.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0359366 s, 29.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0591073 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0438395 s, 23.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0426363 s, 24.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.058147 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.057765 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0854354 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119175 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116822 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0917036 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118749 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130181 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124348 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139254 s, 7.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00924106 s, 113 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0060546 s, 173 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0117814 s, 89.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173433 s, 60.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130026 s, 80.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0110701 s, 94.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0069533 s, 151 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00430253 s, 244 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0201824 s, 52.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0221006 s, 47.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00626686 s, 167 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0231604 s, 45.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.02334 s, 44.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0122522 s, 85.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00814294 s, 129 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00485629 s, 216 MB/s removed '/mnt/lustre/d77kf.sanityn/nrs_r_oleg350-client.virtnet' removed '/mnt/lustre/d77kf.sanityn/nrs_w_oleg350-client.virtnet' removed directory '/mnt/lustre/d77kf.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.6585 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kf.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9594 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77kf.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kf (67s) == sanityn test 77kg: check different granular TBF type combination: uid+opcode ========================================================== 02:01:59 (1780812119) ost.OSS.ost_io.nrs_policies=tbf uid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg350-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.116849 s, 144 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0269808 s, 38.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0525658 s, 19.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0331842 s, 31.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0596032 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0532131 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0447513 s, 23.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0568834 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0898346 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0798968 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10002 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111792 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0965682 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0980397 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111666 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109796 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109751 s, 9.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0174029 s, 60.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016445 s, 63.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0297128 s, 35.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0312037 s, 33.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.024432 s, 42.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183112 s, 57.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0243668 s, 43.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00804637 s, 130 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00583025 s, 180 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00999453 s, 105 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0231816 s, 45.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00413524 s, 254 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00694451 s, 151 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00434829 s, 241 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0289233 s, 36.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00597716 s, 175 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg350-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg350-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.95788 s, 21.1 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.74021 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg350-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.447712 s, 37.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0220149 s, 47.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.036292 s, 28.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0406085 s, 25.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.064163 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0912638 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0932161 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109215 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102851 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0983433 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100243 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145118 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157886 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212509 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215028 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.251388 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.270859 s, 3.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161524 s, 64.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014682 s, 71.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149923 s, 69.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166697 s, 62.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0215086 s, 48.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0207376 s, 50.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0192509 s, 54.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0203816 s, 51.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00959944 s, 109 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00508898 s, 206 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00784476 s, 134 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00844176 s, 124 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0279985 s, 37.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00581869 s, 180 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159972 s, 65.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00398815 s, 263 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg350-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg350-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.7724 s, 10.7 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.72559 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.75296 s, 10.8 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.72635 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kg (122s) == sanityn test 77kh: Verify that Project ID support for NRS TBF rule ========================================================== 02:04:01 (1780812241) using SAVE_PROJECT_SUPPORTED=0 Stopping clients: oleg350-client.virtnet /mnt/lustre (opts:) Stopping client oleg350-client.virtnet /mnt/lustre opts: Stopping clients: oleg350-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg350-client.virtnet /mnt/lustre2 opts: Stopping /mnt/lustre-mds1 (opts:-f) on oleg350-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg350-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg350-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg350-server sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime) proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) devtmpfs on /dev type devtmpfs (rw,nosuid,mode=755) securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime) tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev) devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000) tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755) tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755) cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd) pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime) bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700) cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event) cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset) cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma) cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids) cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio) cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio) cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer) cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct) cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb) cgroup on /sys/fs/cgroup/misc type cgroup (rw,nosuid,nodev,noexec,relatime,misc) cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory) cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices) none on /sys/kernel/tracing type tracefs (rw,relatime) configfs on /sys/kernel/config type configfs (rw,relatime) /dev/nbd0 on / type ext4 (ro,relatime,stripe=32) rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) debugfs on /sys/kernel/debug type debugfs (rw,relatime) mqueue on /dev/mqueue type mqueue (rw,relatime) hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M) none on /mnt type ramfs (rw,relatime) none on /var/lib/stateless/writable type tmpfs (rw,relatime) /dev/vda on /home/green/git/lustre-release type squashfs (ro,relatime) none on /var/cache/man type tmpfs (rw,relatime) none on /var/log type tmpfs (rw,relatime) none on /var/lib/dbus type tmpfs (rw,relatime) none on /tmp type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) none on /var/tmp type tmpfs (rw,relatime) none on /var/lib/NetworkManager type tmpfs (rw,relatime) none on /var/lib/systemd/random-seed type tmpfs (rw,relatime) none on /var/spool type tmpfs (rw,relatime) none on /var/lib/nfs type tmpfs (rw,relatime) none on /var/lib/gssproxy type tmpfs (rw,relatime) none on /var/lib/logrotate type tmpfs (rw,relatime) none on /etc type tmpfs (rw,relatime) none on /var/lib/rsyslog type tmpfs (rw,relatime) none on /root/.ssh type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) 192.168.200.253:/exports/state/oleg350-client.virtnet on /var/lib/stateless/state type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.203.50,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg350-client.virtnet/boot on /boot type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.203.50,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg350-client.virtnet/etc/kdump.conf on /etc/kdump.conf type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.203.50,local_lock=none,addr=192.168.200.253) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=378180k,mode=700) 192.168.200.253:/exports/testreports/64756/testresults/sanityn-zfs-DNE-rocky8.10_x86_64-rocky8.10_x86_64 on /tmp/testlogs type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.203.50,local_lock=none,addr=192.168.200.253) /dev/vda on /usr/sbin/mount.lustre type squashfs (ro,relatime) Checking servers environments Checking clients oleg350-client.virtnet environments /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing load_modules_local oleg350-server: Loading modules from /home/green/git/lustre-release/lustre oleg350-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg350-server: detected 4 online CPUs by sysfs oleg350-server: MODOPTS_LIBCFS= oleg350-server: Force libcfs to create 2 CPU partitions Setup mgs, mdt, osts Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg350-server: oleg350-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0000 Start mds2: mount -t lustre -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg350-server: oleg350-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0001 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg350-server: oleg350-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-OST0000 Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg350-server: oleg350-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-OST0001 Starting client: oleg350-client.virtnet: -o user_xattr,flock 192.168.203.150@tcp:/lustre /mnt/lustre Starting client oleg350-client.virtnet: -o user_xattr,flock 192.168.203.150@tcp:/lustre /mnt/lustre Started clients oleg350-client.virtnet: 192.168.203.150@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) Starting client: oleg350-client.virtnet: -o user_xattr,flock 192.168.203.150@tcp:/lustre /mnt/lustre2 Starting client oleg350-client.virtnet: -o user_xattr,flock 192.168.203.150@tcp:/lustre /mnt/lustre2 Started clients oleg350-client.virtnet: 192.168.203.150@tcp:/lustre on /mnt/lustre2 type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8f1511031800.idle_timeout=debug osc.lustre-OST0000-osc-ffff8f152bf36800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8f1511031800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8f152bf36800.idle_timeout=debug disable quota as required ost.OSS.ost_io.nrs_policies=tbf projid ost.OSS.ost_io.nrs_tbf_rule=start ext_proj_rw projid={100} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.249045 s, 67.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0270226 s, 38.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0646153 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0642887 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0929183 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10009 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0727862 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0882014 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126204 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123128 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136494 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130401 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157047 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.235244 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.231848 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.293922 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.279953 s, 3.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0233086 s, 45.0 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0216964 s, 48.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0202077 s, 51.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0192237 s, 54.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0121812 s, 86.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0303418 s, 34.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0291751 s, 35.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00429555 s, 244 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.024467 s, 42.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0231249 s, 45.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0180839 s, 58.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125529 s, 83.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142351 s, 73.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00864644 s, 121 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00399624 s, 262 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00544128 s, 193 MB/s removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg350-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg350-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.94539 s, 21.2 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.97442 s, 21.1 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_proj_rw ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf projid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w projid={100}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r projid={100}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.25532 s, 65.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0254779 s, 41.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0418879 s, 25.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0458016 s, 22.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0686056 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0710825 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.068464 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0858737 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0920588 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0850957 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106599 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0916902 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.095458 s, 11.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.252606 s, 4.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.245736 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.30763 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.289761 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131072 s, 80.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0185209 s, 56.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170246 s, 61.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0187431 s, 55.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016409 s, 63.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0198346 s, 52.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0244727 s, 42.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00974315 s, 108 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0191697 s, 54.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00705035 s, 149 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138244 s, 75.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0258277 s, 40.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00839719 s, 125 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0263903 s, 39.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0269964 s, 38.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00475716 s, 220 MB/s removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg350-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg350-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.9274 s, 21.3 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.72507 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77kh (135s) == sanityn test 77ki: Add rule with unsupported TBF type should fail for generic TBF ========================================================== 02:06:16 (1780812376) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 oleg350-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_fail projid={100}&opcode={ost_write} rate=20: Invalid argument oleg350-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_fail projid={100}&opcode={ost_write} rate=20': Invalid argument pdsh@oleg350-client: oleg350-server: ssh exited with exit code 22 ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77ki (7s) == sanityn test 77l: check the output of NRS policies for generic TBF ========================================================== 02:06:23 (1780812383) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_policies=fifo PASS 77l (3s) == sanityn test 77m: check NRS Delay slows write RPC processing ========================================================== 02:06:26 (1780812386) ost.OSS.ost_io.nrs_policies=delay ost.OSS.ost_io.nrs_delay_min=4 ost.OSS.ost_io.nrs_delay_max=4 ost.OSS.ost_io.nrs_delay_pct=100 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg350-client.virtnet] [bs=1M] [count=1] [oflag=direct] [conv=fdatasync] 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 3.22147 s, 325 kB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg350-client.virtnet] [bs=1M] [count=10] [oflag=direct] [conv=fdatasync] 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 40.0139 s, 262 kB/s ost.OSS.ost_io.nrs_policies=fifo PASS 77m (50s) == sanityn test 77n: check wildcard support for TBF JobID NRS policy ========================================================== 02:07:16 (1780812436) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={*.500} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.110062 s, 152 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0541077 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0657902 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0578373 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.064639 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107144 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119614 s, 8.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105356 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123831 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114207 s, 9.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135941 s, 7.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126046 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129724 s, 8.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135584 s, 7.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146718 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115026 s, 9.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8f1511031800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff8f1511031800.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144984 s, 7.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.021483 s, 48.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.024432 s, 42.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0257905 s, 40.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0211207 s, 49.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0286601 s, 36.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0118436 s, 88.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0222698 s, 47.1 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0319634 s, 32.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0260668 s, 40.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0289619 s, 36.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0290007 s, 36.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0325852 s, 32.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00868311 s, 121 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0454817 s, 23.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0367172 s, 28.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0115952 s, 90.4 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg350-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg350-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77n.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.92198 s, 21.3 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77n.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.96776 s, 21.1 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.*} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.239774 s, 70.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0309614 s, 33.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0576768 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115026 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126631 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15329 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144646 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15365 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173627 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150986 s, 6.9 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152783 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158627 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172318 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.254757 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.231257 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.288265 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.315165 s, 3.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.031234 s, 33.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0213683 s, 49.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0380044 s, 27.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0271773 s, 38.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0395396 s, 26.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.038166 s, 27.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00967178 s, 108 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0300746 s, 34.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0366355 s, 28.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0334121 s, 31.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0270776 s, 38.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0252166 s, 41.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.041251 s, 25.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0244762 s, 42.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00719635 s, 146 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00505513 s, 207 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg350-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg350-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.91314 s, 21.3 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.96376 s, 21.1 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_policies=fifo PASS 77n (67s) == sanityn test 77o: Changing rank should not panic ====== 02:08:23 (1780812503) mds.MDS.mdt.nrs_policies=tbf nid mds.MDS.mdt.nrs_tbf_rule=start name nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=start name1 nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=change name1 rank=name mds.MDS.mdt.nrs_tbf_rule=stop name mds.MDS.mdt.nrs_policies=fifo PASS 77o (4s) == sanityn test 77q: Parallel TBF rule definitions should not panic ========================================================== 02:08:27 (1780812507) mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_policies=fifo PASS 77q (34s) == sanityn test 77p: Check validity of rule names for TBF policies ========================================================== 02:09:01 (1780812541) mds.MDS.mdt.nrs_policies=tbf oleg350-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_overflo uid={500} rate=500: File name too long oleg350-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_overflo uid={500} rate=500': File name too long pdsh@oleg350-client: oleg350-server: ssh exited with exit code 36 oleg350-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start uid={500} rate=500: Invalid argument oleg350-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start uid={500} rate=500': Invalid argument pdsh@oleg350-client: oleg350-server: ssh exited with exit code 22 oleg350-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_empty: Invalid argument oleg350-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_empty': Invalid argument pdsh@oleg350-client: oleg350-server: ssh exited with exit code 22 oleg350-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p@spec uid={500} rate=500: Invalid argument oleg350-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p@spec uid={500} rate=500': Invalid argument pdsh@oleg350-client: oleg350-server: ssh exited with exit code 22 oleg350-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p.spec uid={500} rate=500: Invalid argument oleg350-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p.spec uid={500} rate=500': Invalid argument pdsh@oleg350-client: oleg350-server: ssh exited with exit code 22 oleg350-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p~spec uid={500} rate=500: Invalid argument oleg350-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p~spec uid={500} rate=500': Invalid argument pdsh@oleg350-client: oleg350-server: ssh exited with exit code 22 oleg350-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p#spec uid={500} rate=500: Invalid argument oleg350-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p#spec uid={500} rate=500': Invalid argument pdsh@oleg350-client: oleg350-server: ssh exited with exit code 22 oleg350-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p/spec uid={500} rate=500: Invalid argument oleg350-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p/spec uid={500} rate=500': Invalid argument pdsh@oleg350-client: oleg350-server: ssh exited with exit code 22 oleg350-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p^spec uid={500} rate=500: Invalid argument oleg350-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p^spec uid={500} rate=500': Invalid argument pdsh@oleg350-client: oleg350-server: ssh exited with exit code 22 oleg350-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p%spec uid={500} rate=500: Invalid argument oleg350-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p%spec uid={500} rate=500': Invalid argument pdsh@oleg350-client: oleg350-server: ssh exited with exit code 22 oleg350-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p*spec uid={500} rate=500: Invalid argument oleg350-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p*spec uid={500} rate=500': Invalid argument pdsh@oleg350-client: oleg350-server: ssh exited with exit code 22 oleg350-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p;spec uid={500} rate=500: Invalid argument oleg350-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p;spec uid={500} rate=500': Invalid argument pdsh@oleg350-client: oleg350-server: ssh exited with exit code 22 oleg350-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p,spec uid={500} rate=500: Invalid argument oleg350-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p,spec uid={500} rate=500': Invalid argument pdsh@oleg350-client: oleg350-server: ssh exited with exit code 22 oleg350-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p?spec uid={500} rate=500: Invalid argument oleg350-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p?spec uid={500} rate=500': Invalid argument pdsh@oleg350-client: oleg350-server: ssh exited with exit code 22 oleg350-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77pspec uid={500} rate=500: Invalid argument oleg350-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p>spec uid={500} rate=500': Invalid argument pdsh@oleg350-client: oleg350-server: ssh exited with exit code 22 oleg350-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p:spec uid={500} rate=500: Invalid argument oleg350-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p:spec uid={500} rate=500': Invalid argument pdsh@oleg350-client: oleg350-server: ssh exited with exit code 22 oleg350-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p+spec uid={500} rate=500: Invalid argument oleg350-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p+spec uid={500} rate=500': Invalid argument pdsh@oleg350-client: oleg350-server: ssh exited with exit code 22 oleg350-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p=spec uid={500} rate=500: Invalid argument oleg350-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p=spec uid={500} rate=500': Invalid argument pdsh@oleg350-client: oleg350-server: ssh exited with exit code 22 oleg350-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p)spec uid={500} rate=500: Invalid argument oleg350-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p)spec uid={500} rate=500': Invalid argument pdsh@oleg350-client: oleg350-server: ssh exited with exit code 22 oleg350-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p(spec uid={500} rate=500: Invalid argument oleg350-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p(spec uid={500} rate=500': Invalid argument pdsh@oleg350-client: oleg350-server: ssh exited with exit code 22 oleg350-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p{spec uid={500} rate=500: Invalid argument oleg350-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p{spec uid={500} rate=500': Invalid argument pdsh@oleg350-client: oleg350-server: ssh exited with exit code 22 oleg350-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p}spec uid={500} rate=500: Invalid argument oleg350-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p}spec uid={500} rate=500': Invalid argument pdsh@oleg350-client: oleg350-server: ssh exited with exit code 22 oleg350-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p|spec uid={500} rate=500: Invalid argument oleg350-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p|spec uid={500} rate=500': Invalid argument pdsh@oleg350-client: oleg350-server: ssh exited with exit code 22 oleg350-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p[spec uid={500} rate=500: Invalid argument oleg350-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p[spec uid={500} rate=500': Invalid argument pdsh@oleg350-client: oleg350-server: ssh exited with exit code 22 oleg350-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p]spec uid={500} rate=500: Invalid argument oleg350-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p]spec uid={500} rate=500': Invalid argument pdsh@oleg350-client: oleg350-server: ssh exited with exit code 22 oleg350-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p!spec uid={500} rate=500: Invalid argument oleg350-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p!spec uid={500} rate=500': Invalid argument pdsh@oleg350-client: oleg350-server: ssh exited with exit code 22 oleg350-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p&spec uid={500} rate=500: Invalid argument oleg350-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p&spec uid={500} rate=500': Invalid argument pdsh@oleg350-client: oleg350-server: ssh exited with exit code 22 oleg350-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p$spec uid={500} rate=500: Invalid argument oleg350-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p$spec uid={500} rate=500': Invalid argument pdsh@oleg350-client: oleg350-server: ssh exited with exit code 22 oleg350-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p`spec uid={500} rate=500: Invalid argument oleg350-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p`spec uid={500} rate=500': Invalid argument pdsh@oleg350-client: oleg350-server: ssh exited with exit code 22 oleg350-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p\spec uid={500} rate=500: Invalid argument oleg350-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p\spec uid={500} rate=500': Invalid argument pdsh@oleg350-client: oleg350-server: ssh exited with exit code 22 mds.MDS.mdt.nrs_policies=fifo PASS 77p (10s) == sanityn test 77r: Change type of tbf policy at run time ========================================================== 02:09:11 (1780812551) striped dir -i0 -c1 -H fnv_1a_64 /mnt/lustre/d77r.sanityn mds.MDS.mdt.nrs_policies=tbf uid mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf gid mds.MDS.mdt.nrs_tbf_rule=start md_rule gid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start md_rule jobid={*.500} rate=1 mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=fifo check the number of requests in queue: fifo: 0 crrn: 0 tbf: 0 delay: 0 fifo: 0 crrn: 0 tbf: 0 delay: 0 cleanup 77r 293483 kill md_thread xargs PASS 77r (41s) == sanityn test 77s: Check TBF LRU shrinker ============== 02:09:53 (1780812593) jobid_name=%e.%p jobid_var=TEST_77s_JOBVAR mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start touch jobid={touch.*} rate=1000 mds.MDS.mdt.nrs_tbf_rule=start unlink jobid={unlink.*} rate=1000 touch jobid={touch.*} 1000, ref 10 touch jobid={touch.*} 1000, ref 0 unlink jobid={unlink.*} 1000, ref 10 unlink jobid={unlink.*} 1000, ref 0 mds.MDS.mdt.nrs_policies=fifo jobid_var=procname_uid jobid_name=%e.%u PASS 77s (16s) == sanityn test 78: Enable policy and specify tunings right away ========================================================== 02:10:08 (1780812608) ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_orr_quantum=1 ost.OSS.ost_io.nrs_policies=fifo PASS 78 (5s) == sanityn test 79: xattr: intent error ================== 02:10:13 (1780812613) striped dir -i1 -c2 -H all_char /mnt/lustre/d79.sanityn File: /mnt/lustre/d79.sanityn Size: 8192 Blocks: 16 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 162129637319639051 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-06-07 02:10:13.000000000 -0400 Modify: 2026-06-07 02:10:13.000000000 -0400 Change: 2026-06-07 02:10:13.000000000 -0400 Birth: 2026-06-07 02:10:13.000000000 -0400 fail_val=0 fail_loc=0x80000160 fail_val=0 fail_loc=0x80000131 # file: mnt/lustre/d79.sanityn trusted.name1="value1" PASS 79 (13s) == sanityn test 80a: migrate directory when some children is being opened ========================================================== 02:10:26 (1780812626) total: 10 open/close in 0.06 seconds: 168.94 ops/second multiop /mnt/lustre2/d80a.sanityn/f80a.sanityn vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.8387 Finish migration, then checking.. PASS 80a (5s) == sanityn test 80b: Accessing directory during migration ========================================================== 02:10:31 (1780812631) start migration thread 304511 accessing the migrating directory for 1 minute... sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete concurrent access 1279 failures, 1673 successes PASS 80b (79s) == sanityn test 81a: rename and stat under striped directory ========================================================== 02:11:50 (1780812710) stat: cannot statx '/mnt/lustre/d81a.sanityn/d0/0': No such file or directory File: /mnt/lustre/d81a.sanityn/d0/0 Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115272381369227 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-06-07 02:11:51.000000000 -0400 Modify: 2026-06-07 02:11:51.000000000 -0400 Change: 2026-06-07 02:11:51.000000000 -0400 Birth: 2026-06-07 02:11:50.000000000 -0400 PASS 81a (2s) == sanityn test 81b: rename under striped directory doesn't deadlock ========================================================== 02:11:52 (1780812712) total: 2500 open/close in 5.31 seconds: 471.05 ops/second /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5002: 309629 Killed ( while true; do touch $DIR1/$tdir; done ) (wd: ~) PASS 81b (98s) == sanityn test 81c: rename revoke LOOKUP lock for remote object ========================================================== 02:13:30 (1780812810) SKIP: sanityn test_81c needs >= 4 MDTs SKIP 81c (1s) == sanityn test 81d: parallel rename file cross-dir on same MDT ========================================================== 02:13:32 (1780812812) total: 2500 open/close in 5.57 seconds: 449.07 ops/second - open/close 2302 (time 1780812831.03 total 10.00 last 230.12) total: 2500 open/close in 10.71 seconds: 233.44 ops/second snapshot_time 1780812863.808236091 secs.nsecs start_time 1780812812.411014975 secs.nsecs elapsed_time 51.397221116 secs.nsecs open 2505 samples [usecs] 72 12090 753783 421320983 close 2504 samples [usecs] 67 664 229976 23090630 mknod 2501 samples [usecs] 181 12014 655237 361360603 unlink 2500 samples [usecs] 368 93886 3041186 28851392920 mkdir 2 samples [usecs] 537 1988 2525 4240513 rmdir 2 samples [usecs] 4372 5599 9971 50463185 rename 2500 samples [usecs] 257 14173 2880143 5329144209 getattr 3766 samples [usecs] 0 7406 94215 92074679 statfs 10 samples [usecs] 12 35 236 6124 parallel_rename_file 2500 samples [usecs] 257 14173 2880143 5329144209 crossdir_rename 2500 samples [usecs] 257 14173 2880143 5329144209 snapshot_time 1780812863.808285848 secs.nsecs start_time 1780812812.411059690 secs.nsecs elapsed_time 51.397226158 secs.nsecs open 2504 samples [usecs] 84 7425 1420859 935736205 close 2504 samples [usecs] 74 5014 449289 137602971 mknod 2500 samples [usecs] 190 7347 1234400 719272704 unlink 2500 samples [usecs] 368 15701 2554676 4710976406 mkdir 2 samples [usecs] 833 11575 12408 134674514 rmdir 2 samples [usecs] 1701 8252 9953 70988905 rename 2500 samples [usecs] 256 11624 3429149 7632378535 getattr 3757 samples [usecs] 7 5252 112815 69270533 statfs 10 samples [usecs] 11 26 192 3946 parallel_rename_file 2500 samples [usecs] 256 11624 3429149 7632378535 crossdir_rename 2500 samples [usecs] 256 11624 3429149 7632378535 PASS 81d (54s) == sanityn test 82: fsetxattr and fgetxattr on orphan files ========================================================== 02:14:26 (1780812866) multiop /mnt/lustre/f82.sanityn vO_ac TMPPIPE=/tmp/multiop_open_wait_pipe.8387 multiop /mnt/lustre2/f82.sanityn vo_Ac TMPPIPE=/tmp/multiop_open_wait_pipe.8387 PASS 82 (3s) == sanityn test 83: access striped directory while it is being created/unlinked ========================================================== 02:14:29 (1780812869) start pid 323824 to create/unlink striped directory start pid 323826 to stat striped directory /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 5987: 323824 Terminated ( cd $DIR1; while true; do $LFS mkdir -i1 -c2 $tdir; rmdir $tdir; done ) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 5987: 323826 Terminated ( cd $DIR2; while true; do stat $tdir > /dev/null 2>&1; done ) PASS 83 (124s) == sanityn test 84: 0-nlink race in lu_object_find() ===== 02:16:33 (1780812993) fail_loc=0x8000060b PASS 84 (8s) == sanityn test 85: Lustre API root cache race =========== 02:16:41 (1780813001) Starting: llapi_root_test /mnt/lustre/f85.sanityn /mnt/lustre2/f85.sanityn Starting test test1 at 1780813002 Finishing test test1 at 1780813002 PASS 85 (4s) == sanityn test 90: open/create and unlink striped directory ========================================================== 02:16:45 (1780813005) start pid 330581 to open/create under striped directory start pid 330582 to unlink striped directory /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 6054: 330581 Terminated ( cd $DIR1; while true; do $LFS mkdir -c$MDSCOUNT $tdir > /dev/null 2>&1; touch $tdir/f{0..3} > /dev/null 2>&1; done ) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 6054: 330582 Terminated ( cd $DIR2; while true; do rm -rf $tdir > /dev/null 2>&1; done ) PASS 90 (183s) == sanityn test 91: chmod and unlink striped directory === 02:19:48 (1780813188) start pid 339307 to chmod striped directory start pid 339309 to unlink striped directory /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 6091: 339307 Terminated ( cd $DIR1; while true; do $LFS mkdir -c$MDSCOUNT $tdir > /dev/null 2>&1; chmod go+w $tdir > /dev/null 2>&1; done ) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 6091: 339309 Terminated ( cd $DIR2; while true; do rm -rf $tdir > /dev/null 2>&1; done ) PASS 91 (183s) == sanityn test 92: create remote directory under orphan directory ========================================================== 02:22:51 (1780813371) fail_loc=0x1408 mkdir: cannot create directory '/mnt/lustre2/d92.sanityn/dir': No such file or directory lfs setdirstripe: cannot resolve path '/mnt/lustre2/d92.sanityn/remote_dir': No such file or directory (2) lfs setdirstripe: '/mnt/lustre2/d92.sanityn/remote_dir' is not on a Lustre filesystem: No such file or directory (2) lfs setdirstripe: cannot create dir '/mnt/lustre2/d92.sanityn/remote_dir': No such file or directory fail_loc=0 PASS 92 (3s) == sanityn test 93: alloc_rr should not allocate on same ost ========================================================== 02:22:54 (1780813374) fail_loc=0x00000163 fail_loc=0x0 /mnt/lustre/f93.sanityn-1/file1 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 6572 0x19ac 0x280000401 1 6568 0x19a8 0x2c0000401 /mnt/lustre2/f93.sanityn-2/file2 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 6573 0x19ad 0x280000401 1 6569 0x19a9 0x2c0000401 rc1=0 and rc2=0 PASS 93 (11s) == sanityn test 94: signal vs CP callback race =========== 02:23:05 (1780813385) osc.lustre-OST0000-osc-ffff8f1511031800.idle_timeout=0 osc.lustre-OST0000-osc-ffff8f152bf36800.idle_timeout=0 osc.lustre-OST0001-osc-ffff8f1511031800.idle_timeout=0 osc.lustre-OST0001-osc-ffff8f152bf36800.idle_timeout=0 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0578184 s, 142 kB/s fail_val=5 fail_loc=0x80000312 fail_val=6 fail_loc=0x80000329 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 3.10071 s, 1.3 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 6178: 348312 Alarm clock dd if=/dev/zero of=$DIR/$tfile conv=notrunc oflag=append bs=4k count=1 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0228922 s, 179 kB/s osc.lustre-OST0000-osc-ffff8f1511031800.idle_timeout=debug osc.lustre-OST0000-osc-ffff8f152bf36800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8f1511031800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8f152bf36800.idle_timeout=debug PASS 94 (11s) == sanityn test 95a: Check readpage() on a page that was removed from page cache ========================================================== 02:23:16 (1780813396) llite.lustre-ffff8f1511031800.fast_read=0 llite.lustre-ffff8f152bf36800.fast_read=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0164123 s, 128 MB/s fail_loc=0x80001421 fail_val=7 fail_loc=0x80001422 fail_val=10 llite.lustre-ffff8f1511031800.fast_read=1 llite.lustre-ffff8f152bf36800.fast_read=1 PASS 95a (14s) == sanityn test 95b: Check readpage() on a page that is no longer uptodate ========================================================== 02:23:30 (1780813410) llite.lustre-ffff8f1511031800.fast_read=0 llite.lustre-ffff8f152bf36800.fast_read=0 1+0 records in 1+0 records out 12288 bytes (12 kB, 12 KiB) copied, 0.00154946 s, 7.9 MB/s fail_loc=0x80001424 fail_val=5 fail_loc=0 llite.lustre-ffff8f1511031800.fast_read=1 llite.lustre-ffff8f152bf36800.fast_read=1 PASS 95b (8s) == sanityn test 100a: DoM: glimpse RPCs for stat without IO lock (DoM only file) ========================================================== 02:23:38 (1780813418) SKIP: sanityn test_100a Reserved for glimpse-ahead SKIP 100a (1s) == sanityn test 100b: DoM: no glimpse RPC for stat with IO lock (DoM only file) ========================================================== 02:23:39 (1780813419) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00198139 s, 2.1 MB/s /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK PASS 100b (4s) == sanityn test 100c: DoM: write vs stat without IO lock (combined file) ========================================================== 02:23:43 (1780813423) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0607022 s, 34.5 MB/s /mnt/lustre/d100c.sanityn/dom has type file OK /mnt/lustre/d100c.sanityn/dom has size 2097152 OK PASS 100c (3s) == sanityn test 100d: DoM: write+truncate vs stat without IO lock (combined file) ========================================================== 02:23:46 (1780813426) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0450983 s, 46.5 MB/s /mnt/lustre/d100d.sanityn/dom has type file OK /mnt/lustre/d100d.sanityn/dom has size 4096 OK PASS 100d (3s) == sanityn test 100e: DoM: read on open and file size ==== 02:23:49 (1780813429) 1+0 records in 1+0 records out 12000 bytes (12 kB, 12 KiB) copied, 0.00265822 s, 4.5 MB/s PASS 100e (3s) == sanityn test 101a: Discard DoM data on unlink ========= 02:23:52 (1780813432) /mnt/lustre/f101a.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00250772 s, 1.6 MB/s PASS 101a (3s) == sanityn test 101b: Discard DoM data on rename ========= 02:23:55 (1780813435) /mnt/lustre/f101b.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00254358 s, 1.6 MB/s PASS 101b (2s) == sanityn test 101c: Discard DoM data on close-unlink === 02:23:57 (1780813437) /mnt/lustre/f101c.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0025051 s, 1.6 MB/s PASS 101c (4s) SKIP: sanityn test_102 skipping ALWAYS excluded test 102 == sanityn test 103: Test size correctness with lockahead ========================================================== 02:24:01 (1780813441) striped dir -i1 -c2 -H crush /mnt/lustre/d103.sanityn fail_loc=0x415 fail_loc=0x415 fail_val=2 Incorrect size expected (no glimpse fix): Starting test test23 at 1780813442 lockahead_test: lockahead_test.c:1171: test23: assertion 'sb.st_size == sb2.st_size' failed: size on /mnt/lustre/d103.sanityn/f103.sanityn and /mnt/lustre2/d103.sanityn/f103.sanityn differs: 0 vs 1048576 fail_loc=0 fail_loc=0x214 fail_val=2 Starting test test23 at 1780813445 Finishing test test23 at 1780813445 PASS 103 (9s) == sanityn test 104: Verify that MDS stores atime/mtime/ctime during close ========================================================== 02:24:10 (1780813450) SKIP: sanityn test_104 ldiskfs only test SKIP 104 (1s) == sanityn test 105: Glimpse and lock cancel race ======== 02:24:11 (1780813451) striped dir -i1 -c2 -H crush /mnt/lustre/d105.sanityn fail_loc=0x416 PASS 105 (17s) == sanityn test 106a: Verify the btime via statx() ======= 02:24:28 (1780813468) SKIP: sanityn test_106a Test only for ldiskfs and statx() supported SKIP 106a (1s) == sanityn test 106b: Glimpse RPCs test for statx ======== 02:24:29 (1780813469) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00287158 s, 357 kB/s File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115272381375684 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-06-07 02:24:30.000000000 -0400 Modify: 2026-06-07 02:24:30.000000000 -0400 Change: 2026-06-07 02:24:30.000000000 -0400 Birth: 2026-06-07 02:24:29.000000000 -0400 /mnt/lustre/f106b.sanityn 144115272381375684 -rw-r--r-- 1 0 0 1780813469 1780813470 1780813470 File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115272381375684 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-06-07 02:24:30.000000000 -0400 Modify: 2026-06-07 02:24:30.000000000 -0400 Change: 2026-06-07 02:24:30.000000000 -0400 Birth: 2026-06-07 02:24:29.000000000 -0400 1780813470 1024 1 PASS 106b (2s) == sanityn test 106c: Verify statx attributes mask ======= 02:24:31 (1780813471) PASS 106c (2s) == sanityn test 107a: Basic grouplock conflict =========== 02:24:33 (1780813473) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.0792026 s, 132 MB/s /mnt/lustre/f107a.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 6579 0x19b3 0x280000401 multiop /mnt/lustre/f107a.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.8387 multiop /mnt/lustre2/f107a.sanityn vO_G16022000r10g16022000c TMPPIPE=/tmp/multiop_open_wait_pipe.8387 First grouplock blocks second one PASS 107a (4s) == sanityn test 107b: Grouplock is added to the head of waiting list ========================================================== 02:24:37 (1780813477) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.080121 s, 131 MB/s /mnt/lustre/f107b.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 6575 0x19af 0x2c0000401 multiop /mnt/lustre/f107b.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.8387 Grouplock blocks IO First grouplock blocks second one Second grouplock blocks IO PASS 107b (8s) == sanityn test 108a: lseek: parallel updates ============ 02:24:45 (1780813485) 1+0 records in 1+0 records out 10000 bytes (10 kB, 9.8 KiB) copied, 0.00161339 s, 6.2 MB/s llite.lustre-ffff8f1511031800.hybrid_io=0 llite.lustre-ffff8f152bf36800.hybrid_io=0 fail_val=4 fail_loc=0x80000414 1+0 records in 1+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 4.1865 s, 2.0 MB/s PASS 108a (6s) == sanityn test 109: Race with several mount instances on 1 node ========================================================== 02:24:51 (1780813491) 192.168.203.150@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg350-client.virtnet /mnt/lustre (opts:) 192.168.203.150@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg350-client.virtnet /mnt/lustre2 (opts:) Starting race between client mounts (duration 30) Iteration 0 Starting client: oleg350-client.virtnet: -o user_xattr,flock 192.168.203.150@tcp:/lustre /mnt/lustre Starting client: oleg350-client.virtnet: -o user_xattr,flock 192.168.203.150@tcp:/lustre /mnt/lustre2 192.168.203.150@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.150@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg350-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg350-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing load_modules_local oleg350-server: Loading modules from /home/green/git/lustre-release/lustre oleg350-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg350-server: detected 4 online CPUs by sysfs oleg350-server: MODOPTS_LIBCFS= oleg350-server: Force libcfs to create 2 CPU partitions Iteration 1 Starting client: oleg350-client.virtnet: -o user_xattr,flock 192.168.203.150@tcp:/lustre /mnt/lustre2 Starting client: oleg350-client.virtnet: -o user_xattr,flock 192.168.203.150@tcp:/lustre /mnt/lustre 192.168.203.150@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.150@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg350-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg350-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing load_modules_local oleg350-server: Loading modules from /home/green/git/lustre-release/lustre oleg350-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg350-server: detected 4 online CPUs by sysfs oleg350-server: MODOPTS_LIBCFS= oleg350-server: Force libcfs to create 2 CPU partitions Iteration 2 Starting client: oleg350-client.virtnet: -o user_xattr,flock 192.168.203.150@tcp:/lustre /mnt/lustre Starting client: oleg350-client.virtnet: -o user_xattr,flock 192.168.203.150@tcp:/lustre /mnt/lustre2 192.168.203.150@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.150@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg350-client.virtnet /mnt/lustre (opts:) Stopping client oleg350-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing load_modules_local oleg350-server: Loading modules from /home/green/git/lustre-release/lustre oleg350-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg350-server: detected 4 online CPUs by sysfs oleg350-server: MODOPTS_LIBCFS= oleg350-server: Force libcfs to create 2 CPU partitions Starting client: oleg350-client.virtnet: -o user_xattr,flock 192.168.203.150@tcp:/lustre /mnt/lustre Starting client: oleg350-client.virtnet: -o user_xattr,flock 192.168.203.150@tcp:/lustre /mnt/lustre2 PASS 109 (36s) == sanityn test 110: do not grant another lock on resend ========================================================== 02:25:27 (1780813527) fail_loc=0x525 fail_val=3 fail_loc=0x534 File: /mnt/lustre/f110.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115272431697923 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-06-07 02:25:28.000000000 -0400 Modify: 2026-06-07 02:25:28.000000000 -0400 Change: 2026-06-07 02:25:28.000000000 -0400 Birth: 2026-06-07 02:25:27.000000000 -0400 fail_loc=0 fail_val=0 Connected clients: oleg350-client.virtnet oleg350-client.virtnet PASS 110 (66s) == sanityn test 111: A racy rename/link an open file should not cause fs corruption ========================================================== 02:26:33 (1780813593) fail_loc=0x8000018a /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 6977: 364463 Terminated $MULTIOP $DIR2/$tdir/mdt0dir/foodir/file2 Ow4096_c PASS 111 (5s) == sanityn test 112: update max-inherit in default LMV === 02:26:39 (1780813599) lmv.lustre-clilmv-ffff8f15063b9800.qos_threshold_rr=90 lmv.lustre-clilmv-ffff8f1509259800.qos_threshold_rr=90 PASS 112 (5s) == sanityn test 113: check servers of specified fs ======= 02:26:44 (1780813604) lustre-OST0000-osc-ffff8f15063b9800 active. lustre-OST0001-osc-ffff8f15063b9800 active. lustre-OST0000-osc-ffff8f15063b9800 active. lustre-OST0001-osc-ffff8f15063b9800 active. MGC192.168.203.150@tcp active. PASS 113 (3s) == sanityn test 114: implicit default LMV inherit ======== 02:26:47 (1780813607) /mnt/lustre/d114.sanityn/l1/l2/l3 lmv_stripe_count: 1 lmv_stripe_offset: -1 lmv_hash_type: none lmv_max_inherit: 1 lmv_max_inherit_rr: 0 /mnt/lustre2/d114.sanityn/l1/l2/l3 lmv_stripe_count: 1 lmv_stripe_offset: -1 lmv_hash_type: none lmv_max_inherit: 1 lmv_max_inherit_rr: 0 /mnt/lustre/d114.sanityn/l1/l2/l3 lmv_stripe_count: 2 lmv_stripe_offset: 1 lmv_hash_type: none lmv_max_inherit: 3 lmv_max_inherit_rr: 0 /mnt/lustre2/d114.sanityn/l1/l2/l3 lmv_stripe_count: 2 lmv_stripe_offset: 1 lmv_hash_type: none lmv_max_inherit: 3 lmv_max_inherit_rr: 0 /mnt/lustre/d114.sanityn/l1 lmv_stripe_count: 2 lmv_stripe_offset: 2 lmv_hash_type: none lmv_max_inherit: 2 lmv_max_inherit_rr: 0 /mnt/lustre2/d114.sanityn/l1 lmv_stripe_count: 2 lmv_stripe_offset: 2 lmv_hash_type: none lmv_max_inherit: 2 lmv_max_inherit_rr: 0 /mnt/lustre/d114.sanityn/l1/l2/l3 lmv_stripe_count: 2 lmv_stripe_offset: 1 lmv_hash_type: none lmv_max_inherit: 3 lmv_max_inherit_rr: 0 /mnt/lustre2/d114.sanityn/l1/l2/l3 lmv_stripe_count: 2 lmv_stripe_offset: 1 lmv_hash_type: none lmv_max_inherit: 3 lmv_max_inherit_rr: 0 /mnt/lustre/d114.sanityn/l1/l2/l3 lmv_stripe_count: 1 lmv_stripe_offset: -1 lmv_hash_type: none lmv_max_inherit: 1 lmv_max_inherit_rr: 0 /mnt/lustre/d114.sanityn/l1/l2/l3 lmv_stripe_count: 1 lmv_stripe_offset: -1 lmv_hash_type: none lmv_max_inherit: 1 lmv_max_inherit_rr: 0 /mnt/lustre2/d114.sanityn/l1/l2/l3 lmv_stripe_count: 1 lmv_stripe_offset: -1 lmv_hash_type: none lmv_max_inherit: 1 lmv_max_inherit_rr: 0 /mnt/lustre2/d114.sanityn/l1/l2/l3 lmv_stripe_count: 1 lmv_stripe_offset: -1 lmv_hash_type: none lmv_max_inherit: 1 lmv_max_inherit_rr: 0 PASS 114 (11s) == sanityn test 115: ldiskfs doesn't check direntry for uniqueness ========================================================== 02:26:58 (1780813618) SKIP: sanityn test_115 ldiskfs only test SKIP 115 (1s) == sanityn test 116: DNE: Set default LMV layout from a remote client ========================================================== 02:26:59 (1780813619) llite.lustre-ffff8f15063b9800.intent_mkdir=1 llite.lustre-ffff8f1509259800.intent_mkdir=1 MD layout /mnt/lustre/d116.sanityn: lmv_stripe_count: 2 lmv_stripe_offset: 0 lmv_hash_type: crush mdtidx FID[seq:oid:ver] 0 [0x2000013a0:0x3b2:0x0] 1 [0x240000bd2:0x35c:0x0] mkdir /mnt/lustre/d116.sanityn/tdir0 setdirstripe -D -i1 /mnt/lustre2/d116.sanityn/tdir0 mkdir /mnt/lustre/d116.sanityn/tdir0/tdir11 lmv_stripe_count: 0 lmv_stripe_offset: 0 lmv_hash_type: none lmv_stripe_count: 0 lmv_stripe_offset: 1 lmv_hash_type: none PASS 116 (3s) == sanityn test 121: trunc append race =================== 02:27:03 (1780813623) 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00670887 s, 3.1 MB/s fail_loc=0x80001436 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0293365 s, 698 kB/s 0000000 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0005000 0000 0000 0000 0000 0000 0000 0000 0000 * 000c800 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0011800 PASS 121 (5s) == sanityn test 200: service remains healthy while able to process request ========================================================== 02:27:08 (1780813628) at_max=10 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=9 striped dir -i0 -c2 -H crush2 /mnt/lustre/d200.sanityn fail_loc=0x1e1 fail_val=20 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0848 s, 0.2 kB/s 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0812 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0926 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.098 s, 0.2 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0853 s, 0.2 kB/s 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0873 s, 0.2 kB/s4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0831 s, 0.2 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1448 s, 0.1 kB/s 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1466 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1576 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1424 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1496 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1476 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1449 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2323 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.238 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2313 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2369 s, 0.1 kB/s at_max=600 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=116 oleg350-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8f15063b9800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8f15063b9800.ost_server_uuid in FULL state after 0 sec PASS 200 (67s) cleanup: ====================================================== == sanityn test complete, duration 5808 sec ============== 02:28:15 (1780813695) === sanityn: start cleanup 02:28:16 (1780813696) === Stopping clients: oleg350-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg350-client.virtnet /mnt/lustre2 opts: === sanityn: finish cleanup 02:29:41 (1780813781) ===